[BUG] Wrong ctype metatable association

[Tomash Brechko]

Hello,

According to documentation,

ctype = ffi.metatype(ct, metatable)

Creates a ctype object for the given ct and associates it with a
metatable.

As I read it it _first_ creates a ctype, and _then_ associates metatable
with the _newly created_ ctype. This is logical and expected. However:
-- code
ffi = require("ffi")
ffi.cdef("struct S { int i; }")
t1 = ffi.metatype("struct S", {
__newindex = function() print("META") end
})
v1 = ffi.new(t1)
v1.x = 0 -- prints "META" as expected

v2 = ffi.new("struct S")
v2.x = 0 -- ERR: prints "META" with plain "struct
S"

t3 = ffi.metatype("struct S", {}) -- ERR: throws 'cannot change a protected
metatable'
-- end

Looks like the metatable is associated with the ct passed to the function,
not the one returned. Tested with 2.0.3 and git v2.1.

[Mike Pall]

Post by Tomash Brechko
According to documentation,
ctype = ffi.metatype(ct, metatable)
Creates a ctype object for the given ct and associates it with a
metatable.
As I read it it _first_ creates a ctype, and _then_ associates metatable
with the _newly created_ ctype.

Declaration of C types is not the same as creating a ctype object.
Metatypes are always associated with the underlying C type. The
ctype object is just an identifier with special functionality.

Post by Tomash Brechko
ffi.cdef("struct S { int i; }")

This declares the C type "struct S".

Post by Tomash Brechko
t1 = ffi.metatype("struct S", {
__newindex = function() print("META") end
})

This uses the existing C type declaration, creates a ctype object
for that type and associates the underlying C type with a metatable.

It couldn't work in any other way since C works that way. Two
instances of "struct S" must be compatible, no matter how they
were created.

Post by Tomash Brechko
v1 = ffi.new(t1)
v1.x = 0 -- prints "META" as expected
v2 = ffi.new("struct S")
v2.x = 0 -- ERR: prints "META" with plain "struct S"

Nope, that's correct since "struct S" is the same C type.

Post by Tomash Brechko
t3 = ffi.metatype("struct S", {}) -- ERR: throws 'cannot change a protected metatable'

Sure, this is the same C type, again.

--Mike

[Tomash Brechko]

Post by Mike Pall
This uses the existing C type declaration, creates a ctype object
for that type and associates the underlying C type with a metatable.

OK, I got the point. API and docs are misleading however, ffi.metatable()
should have not return anything, otherwise it behaves the same as
ffi.typeof() except for the first time when it also has a side effect on
first argument.

I'm not totally buying the argument "because C works this way", metatables
are kinda above C and belong to Lua level. For instance
structure-to-structure copy bypasses metatable now, so potentially you
could have same structures with different metatables still
assign-compatible (what would have to be changed is == for ctype objects
themselves, and tostring() to include metatable address).

But I guess things won't change, so I think it's at least worth clarifying
what *it* refers to in "Creates a ctype object for the given ct and
associates *it* with a metatable."

Sorry for false alarm ;)

[Mike Pall]

Post by Tomash Brechko
API and docs are misleading however, ffi.metatable()
should have not return anything, otherwise it behaves the same as
ffi.typeof() except for the first time when it also has a side effect on
first argument.

The point *is* to behave similar to ffi.typeof(), because the
returned ctype object is typically used to create instances.

Post by Tomash Brechko
I'm not totally buying the argument "because C works this way", metatables
are kinda above C and belong to Lua level.

They match 1:1 with C declarations. There's no other way to do it,
or you'd get inconsistencies all over the place.

Post by Tomash Brechko
For instance
structure-to-structure copy bypasses metatable now, so potentially you
could have same structures with different metatables still
assign-compatible (what would have to be changed is == for ctype objects
themselves, and tostring() to include metatable address).

If you enjoy the complexity of C++, you know where to find it.

--Mike

[Tomash Brechko]

Post by Mike Pall
The point *is* to behave similar to ffi.typeof(), because the
returned ctype object is typically used to create instances.

My point is that one could as well write

ffi.cdef("struct S { int i; }")
ffi.metatype("struct S", { ... })
v = ffi.new("struct S")

I don't get why it's important to return a ctype that is not any way
special from the original "struct S", and main purpose of ffi.metatable()
is to affect its argument. The docs should have said "Associates given
metatable with ct and returns ctype as ffi.typeof(ct) would." (and then
it's clear that it's a "two-in-one" design flaw).

Post by Mike Pall

Post by Tomash Brechko
I'm not totally buying the argument "because C works this way",

metatables

Post by Tomash Brechko
are kinda above C and belong to Lua level.

They match 1:1 with C declarations. There's no other way to do it,
or you'd get inconsistencies all over the place.

You can't befriend metatables with structure-to-structure copy, and you
know it ;). So metatables aren't really connected to C. In Lua there's a
standard trick with shadow table and metatable to implement virtually
everything, from read-only tables to classes, and it works because
metatable is associated with the instance, not the "type". I thought in
LuaJIT they are associated with the "type instance" (I thought that what
"ctypes" are). But the design is apparently different.

Post by Mike Pall
If you enjoy the complexity of C++, you know where to find it.

I enjoy LuaJIT, but I could enjoy it a bit more :)

[Tomash Brechko]

Post by Tomash Brechko
I don't get why it's important to return a ctype that is not any way
special from the original "struct S"

OK, got this one: because of anonymous structures.

[Tomash Brechko]

Post by Tomash Brechko
OK, got this one: because of anonymous structures.

OTOH one could always write

T = ffi.typeof("struct {}")
ffi.metatype(T, { ... })

without two-in-one stuff. OK, nevermind, thanks for you answer, I'll try
to figure out the other way around for my task.

[Coda Highland]

Post by Tomash Brechko

Post by Tomash Brechko
OK, got this one: because of anonymous structures.

OTOH one could always write
T = ffi.typeof("struct {}")
ffi.metatype(T, { ... })
without two-in-one stuff. OK, nevermind, thanks for you answer, I'll try to
figure out the other way around for my task.

This is exactly the possibility I was going for with my question -- to
see if you can use anonymous structs for each different kind of
metatable you need to attach.

/s/ Adam

[Tomash Brechko]

Post by Coda Highland
This is exactly the possibility I was going for with my question -- to
see if you can use anonymous structs for each different kind of
metatable you need to attach.

Thinking a bit more now I realize that what I was after is metatables on
cdata (i.e. on data instances as they are in pure Lua). I had an
impression that metatable on ctype will affect only instances created with
given ctype and not other ctypes rooted in the same C type (thought ctypes
are similar to file descriptors that refer to the same file but each can be
decorated with different file position and flags). Now I realize that even
if that were the case types "T" and "T with metatable" should still be
considered different types so that there won't be an ambiguity on how to
compile the code (and thus pointers to "T" and "T with {}" won't be
compatible either). The only possible advantage would be
structure-to-structure copy between "T" and "T with metatable" (since such
copy bypasses __index and __newindex anyway), but that doesn't worth it.
So now I'm ready to agree with Mike that current design is the most sound
one. Sorry for being slow :).


My final question on the matter: if ctype/C type has been used for a while,
and then ffi.metatype() is called on it, are all compiled traces that are
using this ctype invalidated at this point, or are they continue to work as
before (without metatable)?


Thanks!

[Coda Highland]

Post by Mike Pall
Declaration of C types is not the same as creating a ctype object.
Metatypes are always associated with the underlying C type. The
ctype object is just an identifier with special functionality.

Post by Tomash Brechko
ffi.cdef("struct S { int i; }")

This declares the C type "struct S".

Is it the case that anonymous structs are distinct? Is "struct { int
i; }" different from "struct { int i; }" if invoked independently?

/s/ Adam

[Tomash Brechko]

Post by Coda Highland
Is it the case that anonymous structs are distinct? Is "struct { int
i; }" different from "struct { int i; }" if invoked independently?

Anonymous structures are not assign-compatible. Mike's decision was to
link metatables with underlying C type, from the docs I expected them to be
linked with ctype object, and I expected such objects to be
assign-compatible (BTW I wonder how you convey the difference between "C
type" vs. "ctype" orally :)). In my first example was trying to implement
a callback on structure field update. For this I used shadow structure to
store the value and metatable with __index and __newindex functions with
upvalue referencing shadow structure. But turns out I can't have two
ctypes that are assign-compatible but have different metatables, thus can't
have different upvalues, thus can't have different shadow values for
different instances, unless they have totally unrelated types. But such
instances were supposed to be assign-compatible, so no luck :/

[Mike Pall]

Post by Coda Highland
Is it the case that anonymous structs are distinct? Is "struct { int
i; }" different from "struct { int i; }" if invoked independently?

Yes, they are distinct. Every "struct { ... }" creates a new type.
C has nominal typing and anonymous structs get different (internal)
names.

[The opposite would be a structural type system, that's e.g. how
Go's interfaces work.]

--Mike