C data reflection in LuaJIT

[Peter Colberg]

Hi Mike,

The FFI reflect module [1] by Peter Cawley is very useful for
serializing C data objects, e.g., for reading and writing HDF5
datasets of multi-dimensional arrays with struct element type.

[1] https://github.com/corsix/ffi-reflect

However, the module depends on LuaJIT-internal data structures that
could change with new major releases. Would you consider including
the functionality of the reflect module in the ffi.* API?

Regards,
Peter

[Konstantin Olkhovskiy]

I would "plus" this. That was the first thing I though about when looking
at the code.

[Mike Pall]

Post by Peter Colberg
However, the module depends on LuaJIT-internal data structures that
could change with new major releases. Would you consider including
the functionality of the reflect module in the ffi.* API?

Sorry, no. That would mean I'd have to maintain it as part of the
core and update it for any changes to the internals.

--Mike

[Peter Colberg]

Post by Mike Pall
Sorry, no. That would mean I'd have to maintain it as part of the
core and update it for any changes to the internals.

I understand that this was a too big request to make.

Could you take a quick look at the attached patch and test? It adds
three minimal functions ffi.elemtypeof(), ffi.fieldtypeof(), and
ffi.fieldsof().

ffi.fieldsof() returns an iterator to retrieve the field names of a
composite type, which may then be passed to ffi.offsetof() and
ffi.fieldtypeof().

Please let me know if these minimal functions would be acceptable
maintainance-wise; then I will complete the contribution with
documentation.

Thanks,
Peter

[Mike Pall]

Post by Peter Colberg
Could you take a quick look at the attached patch and test? It adds
three minimal functions ffi.elemtypeof(), ffi.fieldtypeof(), and
ffi.fieldsof().

I think that API is not minimal enough and incomplete, too. I'd go
for a single low-level function that only does a ctype_get() with
bounds-checked CTypeID, returning a CType* cdata object or nil.
Adding the definition of CType* to the predefined types is
probably the trickiest part of that.

--Mike

[Peter Colberg]

Post by Mike Pall
I think that API is not minimal enough and incomplete, too. I'd go
for a single low-level function that only does a ctype_get() with
bounds-checked CTypeID, returning a CType* cdata object or nil.
Adding the definition of CType* to the predefined types is
probably the trickiest part of that.

Would you accept a CType userdata that exposes ctype_ macros on
ct->info as methods and ct->name as a string? That would avoid
replicating parts of the internal API of lj_obj.h and lj_ctype.h.

Peter

[Mike Pall]

Post by Peter Colberg
Would you accept a CType userdata that exposes ctype_ macros on
ct->info as methods and ct->name as a string? That would avoid
replicating parts of the internal API of lj_obj.h and lj_ctype.h.

Ah, I forgot about the GCref. Ok, so it might make more sense to
offer an API similar to jit.util.traceir(). I.e. split the struct
up into multiple return values.

--Mike

[Peter Colberg]

Post by Mike Pall
Ah, I forgot about the GCref. Ok, so it might make more sense to
offer an API similar to jit.util.traceir(). I.e. split the struct
up into multiple return values.

Please see the patch for a bounds-checked low-level API.

It would be tremendous if LuaJIT could include a high-level API in
form of a tiny Lua module that provides a minimal set of functions
to decode the "info" field.

Peter

[Mike Pall]

Post by Peter Colberg
Please see the patch for a bounds-checked low-level API.

Applied to the v2.1 branch.

Post by Peter Colberg
It would be tremendous if LuaJIT could include a high-level API in
form of a tiny Lua module that provides a minimal set of functions
to decode the "info" field.

I'm not sure I'd want to add such an API before it's fleshed out
sufficiently by practical use.

--Mike

[Peter Cawley]

For reference, I've updated https://github.com/corsix/ffi-reflect to
use ffi.typeinfo when available.

Post by Mike Pall

Post by Peter Colberg
Please see the patch for a bounds-checked low-level API.

Applied to the v2.1 branch.

Post by Peter Colberg
It would be tremendous if LuaJIT could include a high-level API in
form of a tiny Lua module that provides a minimal set of functions
to decode the "info" field.

I'm not sure I'd want to add such an API before it's fleshed out
sufficiently by practical use.
--Mike

[Peter Colberg]

Post by Mike Pall
I'm not sure I'd want to add such an API before it's fleshed out
sufficiently by practical use.

What is your opinion on the ffi-reflect API? Specifically, what would
you change about the mapping of CType to attributes and iterators (see
quick index at bottom of page)?

http://corsix.github.io/ffi-reflect/

The API has been in practical use for a bit longer than two years;
though it is difficult to assess whether it’s fleshed out without
user feedback.

Peter