metatype __gc inconsistency

[Tomash Brechko]

Hello!

Consider the following code:

ffi = require("ffi")
ffi.cdef[[ struct S {}; struct S *malloc(size_t size); ]]
ffi.metatype("struct S", {
__len = function() return 5 end,
__gc = function() print("GC") end
})
s1 = ffi.new("struct S")
s2 = ffi.C.malloc(1)
print(#s1) -- Outputs "5" - good
print(#s2) -- Outputs "5" - good

Now we do the following:

collectgarbage()
s1 = nil
collectgarbage() -- Outputs "GC" - good
s2 = nil
collectgarbage() -- Output nothing - why?

Not claming that it's a bug, my guess this is intentional and one has to do
ffi.gc(s2, ...), but why? What is the rationale for not calling metatype
__gc when object is allocated outside of LuaJIT?

[Mike Pall]

Post by Tomash Brechko
ffi = require("ffi")
ffi.cdef[[ struct S {}; struct S *malloc(size_t size); ]]
ffi.metatype("struct S", {
__len = function() return 5 end,
__gc = function() print("GC") end
})
s1 = ffi.new("struct S")

This returns a struct.

Post by Tomash Brechko
s2 = ffi.C.malloc(1)

This returns a pointer to a struct.

Post by Tomash Brechko
collectgarbage()
s1 = nil
collectgarbage() -- Outputs "GC" - good
s2 = nil
collectgarbage() -- Output nothing - why?

Well, why should it do anything? The pointer cdata object is
collected, not the struct it points to.

Post by Tomash Brechko
Not claming that it's a bug, my guess this is intentional and one has to do
ffi.gc(s2, ...), but why? What is the rationale for not calling metatype
__gc when object is allocated outside of LuaJIT?

Because memory allocated outside of LuaJIT is (obviously) not
under control of the GC.

The pointer cdata object is under control of the GC, but it can't
just go out of it's way and call __gc for the corresponding struct
if any pointer object is collected. You could have many pointers
to the same struct. The GC doesn't track external memory, unless
you explicitly tell it to do so via an anchor pointer object.

--Mike

[Tomash Brechko]

Post by Mike Pall

Post by Tomash Brechko
s1 = ffi.new("struct S")

This returns a struct.

Post by Tomash Brechko
s2 = ffi.C.malloc(1)

This returns a pointer to a struct.

Got it! Now let me change the example:

ffi = require("ffi")
ffi.cdef[[ struct S { void *p; }; struct S malloc(size_t size); ]] --
return struct S by value
ffi.metatype("struct S", {
__len = function() return 5 end,
__gc = function() print("GC") end
})
s1 = ffi.new("struct S")
s2 = ffi.C.malloc(1)
print(s1) -- cdata<struct S>: ... - a struct
print(s2) -- cdata<struct S>: ... - a struct too
s1 = nil
collectgarbage() -- Outputs "GC"
s2 = nil
collectgarbage() -- Outputs nothing

Could you please elaborate what am I missing now?

Thanks in advance!

[Wolfgang Pupp]

If malloc actually returned a struct, the __gc metamethod of the
corresponding type would not be called because you never "created" the
instance (malloc did), and the __gc function is only used for an "implicit
ffi.gc call during creation of an instance".

This makes sense, because objects created outside of ffi.new usually need
custom GC behavior anyway (malloc'd pointers need freeing, objects obtained
from some C-API usually need to be passed to a special free-function from
said API, etc.). Blindly invoking __gc functions for structs obtained from
C functions would probably be disastrous more often than not.

Note that passing/returning structs by value currently prevents JIT
compilation of surrounding code and is seldomly seen in C-APIs anyway.

If you want to use malloc instead of ffi.new, it is IMHO best to keep
your object as pointer, because you *need* the pointer when free'ing the
object, and there is no "ffi.addressof" (by design).
Dereferencing mallocs return value immediately is just not worth it IMO.

----------------------------------------------------------------------------
-- Here is a simple malloc example that shows how LuaJIT deals with
-- pointers-to-structs vs structs, and what the (few) differences in
-- behavior are:
----------------------------------------------------------------------------
local ffi = require'ffi'

ffi.cdef[[
void * malloc(size_t bytes);
void free(void *ptr);
]]

local typeof, sizeof = ffi.typeof, ffi.sizeof

local malloc
do
local libc, fill, cast, gc = ffi.C, ffi.fill, ffi.cast, ffi.gc
malloc = function(ctype)
local ptr = cast(typeof('$ *', ctype), libc.malloc(sizeof(ctype)))
fill(ptr, sizeof(ctype)) --zero-fill
gc(ptr, libc.free)
return ptr
end
end

local PointF = typeof'struct {double x; double y;}'
do
local sqrt = math.sqrt
local mt = {
__index = {
distance = function(a, b) return sqrt((a.x-b.x)^2 + (a.y-b.y)^2) end,
},
}
ffi.metatype(PointF, mt)
end

local p = PointF(1, 1)
local p2 = malloc(PointF)

print(p:distance{x=2, y=2})

-- LuaJIT dereferences the pointer for us.
-- C/C++ would force us to write "p2->distance" here.
print(p2:distance{x=2, y=2})

print(sizeof(p), sizeof(p2)) --> 16, <pointer size>

-- In C/C++, one would need to write "p2->x"
print(p.x, p2.x) --> 1, 0

-- dereferencing before accessing the member only works with the pointer
print(p2[0].x) --> 0

[Mike Pall]

Post by Wolfgang Pupp
local typeof, sizeof = ffi.typeof, ffi.sizeof
local malloc
do
local libc, fill, cast, gc = ffi.C, ffi.fill, ffi.cast, ffi.gc
malloc = function(ctype)
local ptr = cast(typeof('$ *', ctype), libc.malloc(sizeof(ctype)))
fill(ptr, sizeof(ctype)) --zero-fill
gc(ptr, libc.free)
return ptr
end
end

Umm ...

1. Don't use ffi.typeof() in a fast path.

2. Be careful when creating parameterized types repeatedly. Ok, so
pointer types happen to be interned, but others aren't. If not
interned, it'll create a distinguished ctype every time. You don't
want that, since the JIT-compiler always specializes to a ctype.

3. Separating declaration and assignment of a local function kills
the immutable upvalue optimization (e.g. when it's called from
within another function after that). Don't do that.


The way to design such an API is to use a factory function:

local C, cast, fill, gc = ffi.C, ffi.cast, ffi.fill, ffi.gc

local function malloc_type(ct)
local ctp, cts = ffi.typeof("$ *", ct), ffi.sizeof(ct)
return function()
local p = cast(ctp, C.malloc(cts))
fill(p, cts)
return gc(p, C.free)
end
end

local point_new = malloc_type(PointF)

local function whatever()
local p1 = point_new()
local p2 = point_new()
...
end

But note you always pay for the allocation of the GC-managed
pointer object, anyway. Such a use of malloc() only makes sense if
the allocated types are very large. Otherwise it's just an arduous
ffi.new().

--Mike

[Tomash Brechko]

Post by Wolfgang Pupp
If malloc actually returned a struct, the __gc metamethod of the
corresponding type would not be called because you never "created" the
instance (malloc did), and the __gc function is only used for an "implicit
ffi.gc call during creation of an instance".

Well, I have a feeling that other people have much broader "obvious" domain
than I have :). Because I see it differently: malloc() in my example is an
arbitrary function that returns struct by value, and then LuaJIT creates a
box around a copy, and __gc is a property of that boxed copy. So original
Mike's explanation about value vs. pointer doesn't apply here. Some time
earlier when I complained that ffi.metatype() creates an impression that it
blesses with metatable only its argument I was corrected by Mike that
ffi.metatype() operates on the C type globally, not on a particular type
instance that ffi.metatype() returns (there's no such thing as "type
instance"). But now the story goes that there actually are "type
flavours": it makes a difference how you produced a value of a type (via
ffi.new() or a native call).

Imagine I first implement some func() in LuaJIT that internally allocates
some data and wraps it in a struct created with ffi.new() and blessed with
metatable having __gc:

ffi.cdef[[ struct S { MyType *p; /* maybe some other fields */ } ]]
ffi.metatype("struct S", { __gc = function(v) freeMyType(v.p) end })

function func()
local data = allocateMyType();
-- some heavy lifting on data here
return ffi.new("struct S", { data })
end

(I understand that I can implement it differently but let's say I have some
reasons to use struct with metatable and not ffi.gc()). Now if Lua
performance is not satisfactory I may decide to rewrite func() in C,
leaving the rest intact. I.e. C implementation of func() will return a
struct by value, and I'd like to expect __gc to be called as before, which
won't happen for some "obvious" reason that's not really so obvious ;).

Post by Wolfgang Pupp
Blindly invoking __gc functions for structs obtained from
C functions would probably be disastrous more often than not.

It's not a mere blind invocation, but something explicitly requested by the
user. Otherwise you are requiring the programmer to keep track of how each
object has been allocated (via ffi.new() or native call) and act
accordingly (and even ffi.istype() and tostring() can't tell the
difference, so it's completely up to the programmer to control full
lifecycle, which is not always possible).

I still view it as an inconsistency, OTOH I'm not pursuing the goal to
convince anyone that it must be fixed.

[Coda Highland]

On Thu, Sep 18, 2014 at 4:29 PM, Tomash Brechko

"type flavours"

There are pointers to values, and there are values. malloc returns the
former. LuaJIT just allows you to dereference through a pointer to
access the fields of the value being pointed to.

/s/ Adam

[Tomash Brechko]

Post by Coda Highland
There are pointers to values, and there are values.

Yup, but that part of discussion is agreed upon and closed. Look carefully
into the second example - no pointers there (don't let "malloc()" mislead
you, in its place you may use

ffi.cdef[[ struct S { double d; }; struct S sin(double a); ]]
...
s2 = ffi.C.sin(0) -- not a pointer either

just as well - any function returning a struct will do).

[lex pops]

On Thu, Sep 18, 2014 at 4:29 PM, Tomash Brechko

Post by Tomash Brechko
Imagine I first implement some func() in LuaJIT that internally allocates
some data and wraps it in a struct created with ffi.new() and blessed with
ffi.cdef[[ struct S { MyType *p; /* maybe some other fields */ } ]]
ffi.metatype("struct S", { __gc = function(v) freeMyType(v.p) end })
function func()
local data = allocateMyType();
-- some heavy lifting on data here
return ffi.new("struct S", { data })
end
(I understand that I can implement it differently but let's say I have some
reasons to use struct with metatable and not ffi.gc()). Now if Lua
performance is not satisfactory I may decide to rewrite func() in C, leaving
the rest intact. I.e. C implementation of func() will return a struct by
value, and I'd like to expect __gc to be called as before, which won't
happen for some "obvious" reason that's not really so obvious ;).

Your example seems particularly contrived. Why not still use ffi.new()
and have your C function 'fill up' the struct?

[Tomash Brechko]

Post by lex pops
Your example seems particularly contrived. Why not still use ffi.new()
and have your C function 'fill up' the struct?

The example _is_ contrived. However the existence of a workaround doesn't
justify nor explain the behaviour itself, does it? The question raised is
why it has to work this way and should it stay so. You don't seem to know
the answer either, so welcome, let's wait until someone shed the light or
Mike confirms that it just happened to be so, without any particular
reason. Or until the thread just fade out :)

[Cosmin Apreutesei]

ffi.cdef[[ struct S { void *p; }; struct S malloc(size_t size); ]] -- return struct S by value

It's not ok to lie to the ffi that malloc returns a struct-by-value.
The ffi might look into the wrong register or stack slot for it
depending on your ABI. Is it possible that you're hitting undefined
behavior and just escaped a crash in this particular example?

[Tomash Brechko]

Post by Cosmin Apreutesei
Is it possible that you're hitting undefined
behavior and just escaped a crash in this particular example?

No, on Linux x86-64 the ABI for a scalar and a struct wrapping scalar is
the same (likely so on any ABI as there are little reasons to do
otherwise). I faked malloc/sin just for the sake of example, I don't know
any universally available function that returns a struct by value. Also
note that accessing struct value (in example with sin()) and other
metatable methods (__len) work as expected, so ABI part is safe (and just
for the case I actually checked it right now with the real function
returning a struct :D - works the same way).

As has been cited earlier, the docs say "The __gc metamethod only applies
to struct/union types and performs an implicit ffi.gc()
<http://luajit.org/ext_ffi_api.html#ffi_gc> call during creation of an
instance." So my guess that at some moment this particular case has been
overlooked and no internal ffi.gc() call has been added for structs
returned by value, though they are likely boxed with the code sequence
similar to ffi.new() call. Other metamethods work because they also have
to work for pointer to struct and hence take a different code path.