n-dimensional structures via FFI

[Roman Glebenkov]

Hello.

I am starting to get into FFI and have been successful to a degree.
However, I can't figure out how to make a 2-dimensional structure that can
contain multiple sub-structures

So, for example, right now I have a table called 'world', which contains
several sub-tables, namely 'blocks', 'backgrounds', 'trees', 'stillWater',
'movingWater' and and integer 'treeCount', in code it looks like this:

world = {}
world.blocks = {}
world.trees = {}
world.backgrounds = {}
world.stillWater = {}
world.movingWater = {}

My question is, how would I turn all of this in a C structure, with the
structures being accessable via [x][y] indexes. For reference, here's what
I managed to make use of FFI as of now:

ffi.cdef[[
typedef struct {uint8_t n, h; bool p;} block_t;
]]

world = {}
world.blocks = {}
world.blocks[x] = {}
world.blocks[x][y] = ffi.new('world_t", 1)

^ and so on for trees, backgrounds, stillWater, movingWater, etc.

Ideally, I want to turn the 'world' table into a pure C structure, but have
no idea how. Any help would be greatly appreciated.

Thanks in advance.

[Wolfgang Pupp]

-- Here is a simple example.

local ffi = require'ffi'
local cdef, typeof = ffi.cdef, ffi.typeof

-- fixed-size elements are very straightforward:
cdef[[
typedef struct {
uint8_t n, h;
bool p;
} block_t;
]]

-- variable-size elements are slightly harder, and need to
-- be allocated individually ("leafs" is just a pointer):
cdef[[
typedef struct {
int leaf_cnt;
uint8_t *leafs;
} tree_t;
]]

-- this is a "constructor" for a leaf-array.
-- Desired number of elements needs to be passed as argument.
-- Returns a VLA (variable-length-array)-- the elements are *not*
-- guaranteed to be zero-initialized.
local leaf_array = typeof'uint8_t[?]'

local function new_world(size_x, size_y)
local w = typeof([[
struct {
int xs, ys;
block_t blocks[$][$];
tree_t trees[$][$];
/* etc. */
}]],
-- those are type-template parameters
-- see http://luajit.org/ext_ffi_semantics.html#param
size_x, size_y,
size_x, size_y
)(
-- initial values go here (we provide only the first two)
size_x, size_y
)

-- "blocks" are NOT a VLA and thus zero-initialized.

-- careful: zero-based arrays
for x=0,w.xs-1 do
for y=0,w.ys-1 do
local k = math.random(1, 100)
w.trees[x][y].leaf_cnt = k
-- we allocate our leafs here
local leafs = leaf_array(k)

-- inside the world struct, we set a pointer to the leafs-VLA's
w.trees[x][y].leafs = leafs
-- Note how we lost size information (same thing happens in C)
print("sizeof(VLA)", ffi.sizeof(leafs))
print("sizeof(VLA-ptr)", ffi.sizeof(w.trees[x][y].leafs))

for i=0,k-1 do
w.trees[x][y].leafs[i] = i
end
end
end

return w
end

local w = new_world(1, 5)

--Wolfgang

[Roman Glebenkov]

I just tried doing a small test:

ffi.cdef[[
typedef struct {uint8_t n, h; bool p;} block_t;
typedef struct {uint8_t p, fp, f;} light_t;
typedef struct {uint8_t n, h; bool p;} back_t;
typedef struct {uint8_t n, h, a, i; bool p;} tree_t;
typedef struct {uint8_t ticks; float fullness;} water_t;
typedef struct {long ticks, count; float fullness;} water_idle_check_t;
]]

local test = ffi.typeof([[struct {
block_t blocks[$][$];
tree_t trees[$][$];
back_t backgrounds[$][$];
water_t stillWater[$][$];
water_t movingWater[$][$];
}]], 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)

print(test.blocks)

and it doesn't work, it's saying that the structure has no member named
'blocks'.

Post by Wolfgang Pupp
-- Here is a simple example.
local ffi = require'ffi'
local cdef, typeof = ffi.cdef, ffi.typeof
cdef[[
typedef struct {
uint8_t n, h;
bool p;
} block_t;
]]
-- variable-size elements are slightly harder, and need to
cdef[[
typedef struct {
int leaf_cnt;
uint8_t *leafs;
} tree_t;
]]
-- this is a "constructor" for a leaf-array.
-- Desired number of elements needs to be passed as argument.
-- Returns a VLA (variable-length-array)-- the elements are *not*
-- guaranteed to be zero-initialized.
local leaf_array = typeof'uint8_t[?]'
local function new_world(size_x, size_y)
local w = typeof([[
struct {
int xs, ys;
block_t blocks[$][$];
tree_t trees[$][$];
/* etc. */
}]],
-- those are type-template parameters
-- see http://luajit.org/ext_ffi_semantics.html#param
size_x, size_y,
size_x, size_y
)(
-- initial values go here (we provide only the first two)
size_x, size_y
)
-- "blocks" are NOT a VLA and thus zero-initialized.
-- careful: zero-based arrays
for x=0,w.xs-1 do
for y=0,w.ys-1 do
local k = math.random(1, 100)
w.trees[x][y].leaf_cnt = k
-- we allocate our leafs here
local leafs = leaf_array(k)
-- inside the world struct, we set a pointer to the leafs-VLA's
w.trees[x][y].leafs = leafs
-- Note how we lost size information (same thing happens in C)
print("sizeof(VLA)", ffi.sizeof(leafs))
print("sizeof(VLA-ptr)", ffi.sizeof(w.trees[x][y].leafs))
for i=0,k-1 do
w.trees[x][y].leafs[i] = i
end
end
end
return w
end
local w = new_world(1, 5)
--Wolfgang

[Mike Pall]

Post by Roman Glebenkov
local test = ffi.typeof([[struct {
block_t blocks[$][$];
tree_t trees[$][$];
back_t backgrounds[$][$];
water_t stillWater[$][$];
water_t movingWater[$][$];
}]], 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
print(test.blocks)
and it doesn't work, it's saying that the structure has no member named
'blocks'.

ffi.typeof returns a type. You need to instantiate it (call it) to
get an actual struct:

...
local test_t = ffi.typeof( ... ... )
local test = test_t()
print(test.blocks)

--Mike

[Roman Glebenkov]

That works, thank you so much.

Post by Mike Pall

Post by Roman Glebenkov
local test = ffi.typeof([[struct {
block_t blocks[$][$];
tree_t trees[$][$];
back_t backgrounds[$][$];
water_t stillWater[$][$];
water_t movingWater[$][$];
}]], 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
print(test.blocks)
and it doesn't work, it's saying that the structure has no member named
'blocks'.

ffi.typeof returns a type. You need to instantiate it (call it) to
...
local test_t = ffi.typeof( ... ... )
local test = test_t()
print(test.blocks)
--Mike

[Michal Kottman]

Post by Wolfgang Pupp
-- careful: zero-based arrays
for x=0,w.xs-1 do
for y=0,w.ys-1 do
local k = math.random(1, 100)
w.trees[x][y].leaf_cnt = k
-- we allocate our leafs here
local leafs = leaf_array(k)
-- inside the world struct, we set a pointer to the leafs-VLA's
w.trees[x][y].leafs = leafs

I may be wrong, but you should 'anchor' leafs in some other way than just
using it in the struct, because the GC does not traverse cdata, and will
sooner or later decide that the 'local leafs' is no longer needed and will
free it from under you.

One way to solve it is use the standard C library malloc/free and do memory
management yourself, or append the 'leafs' to some Lua table (which is
traversed by GC and will keep the object alive).

[Roman Glebenkov]

I came across an interesting problem, not sure whether it's LuaJIT, plain
Lua or something else causing it.
If I call write string.char(26) into a file, it will simply not write it,
but if I do string.byte(string.char(26)), then it will work just fine.
Any ideas?

Post by Michal Kottman

Post by Wolfgang Pupp
-- careful: zero-based arrays
for x=0,w.xs-1 do
for y=0,w.ys-1 do
local k = math.random(1, 100)
w.trees[x][y].leaf_cnt = k
-- we allocate our leafs here
local leafs = leaf_array(k)
-- inside the world struct, we set a pointer to the leafs-VLA's
w.trees[x][y].leafs = leafs

I may be wrong, but you should 'anchor' leafs in some other way than just
using it in the struct, because the GC does not traverse cdata, and will
sooner or later decide that the 'local leafs' is no longer needed and will
free it from under you.
One way to solve it is use the standard C library malloc/free and do
memory management yourself, or append the 'leafs' to some Lua table (which
is traversed by GC and will keep the object alive).

[Michal Kottman]

Post by Roman Glebenkov
I came across an interesting problem, not sure whether it's LuaJIT, plain

Lua or something else causing it.

Post by Roman Glebenkov
If I call write string.char(26) into a file, it will simply not write it,

but if I do string.byte(string.char(26)), then it will work just fine.

Post by Roman Glebenkov
Any ideas?

string.char(26) is Ctrl+Z, which on Windows means end-of-file [1]. My guess
is that you are using Windows and the 'text' mode when opening the file
(the default, i.e. using io.open(file, 'w')). This causes some processing
when writing to the file, such as expanding \n to \r\n and I guess also
stripping the Ctrl+Z.

Try opening the file in 'binary' mode using io.open(file, 'wb').

[1] http://en.m.wikipedia.org/wiki/Control-Z

[Roman Glebenkov]

Yes, that's exactly what I'm using, I am writing the data via wb+ and the
issue is still there.

Post by Roman Glebenkov
I came across an interesting problem, not sure whether it's LuaJIT,

plain Lua or something else causing it.

Post by Roman Glebenkov
If I call write string.char(26) into a file, it will simply not write

it, but if I do string.byte(string.char(26)), then it will work just fine.

Post by Roman Glebenkov
Any ideas?

string.char(26) is Ctrl+Z, which on Windows means end-of-file [1]. My
guess is that you are using Windows and the 'text' mode when opening the
file (the default, i.e. using io.open(file, 'w')). This causes some
processing when writing to the file, such as expanding \n to \r\n and I
guess also stripping the Ctrl+Z.
Try opening the file in 'binary' mode using io.open(file, 'wb').
[1] http://en.m.wikipedia.org/wiki/Control-Z