Luajit interpreter question

[Dibyendu Majumdar]

Hi,

I have a question regarding the Luajit interpreter. Is it possible to write
the Luajit bytecode interpreter in C - or does it rely upon features only
available in assembly language?

Regards
Dibyendu

[Daniel Kolesa]

Post by Dibyendu Majumdar
Hi,
I have a question regarding the Luajit interpreter. Is it possible to
write the Luajit bytecode interpreter in C - or does it rely upon features
only available in assembly language?

I don't see why it wouldn't be possible to write a VM that interprets
LuaJIT bytecode. But you're gonna have to deal with the FFI (which will
need either platform specific code or use something like libffi) if you
want it truly compatible.

Post by Dibyendu Majumdar
Regards
Dibyendu

[Mike Pall]

Post by Daniel Kolesa

Post by Dibyendu Majumdar
I have a question regarding the Luajit interpreter. Is it possible to
write the Luajit bytecode interpreter in C - or does it rely upon features
only available in assembly language?

I don't see why it wouldn't be possible to write a VM that interprets
LuaJIT bytecode. But you're gonna have to deal with the FFI (which will
need either platform specific code or use something like libffi) if you
want it truly compatible.

Sure ... and the transitions to and from JIT-compiled code,
including register and stack snapshot handling. And error handling
plus frame unwinding. And bytecode recording. And metamethod
continuations. And the builtins written in assembler. Oh, and
performance won't be stellar either ...

--Mike

[Karel Tuma]

Post by Mike Pall

Post by Dibyendu Majumdar
I have a question regarding the Luajit interpreter. Is it possible to
write the Luajit bytecode interpreter in C - or does it rely upon features
only available in assembly language?

Sure ... and the transitions to and from JIT-compiled code,
including register and stack snapshot handling. And error handling
plus frame unwinding. And bytecode recording. And metamethod
continuations. And the builtins written in assembler. Oh, and
performance won't be stellar either ...

On a more technical note, it is possible to skew the current "most of hot paths
in asm" to "bare minimum glue asm". Some low level machinery of MM/C/ASM/JIT mcode
must be implemented in assembly as Mike mentions, however most of the actual
opcodes can be written in C. Easiest way is to adapt vm_record for your
"C stub" opcode implementations and use second level dispatch in C
for your C opcode handlers (in similiar fashion how trace recorder does it,
minus the restart). The C stub would return optional MM fallback for asm
glue to continue on.

Abusing fastfunc or lj_dispatch.c for this purpose is not advised as it is
rather invasive and would be difficult merge-wise with LuaJIT.

Also, meta dispatch is already mixed asm/c code (grep lj_meta_call),
the asm part is mostly the bare minimum to call C fallback.

All in all, it's not worth it IMO. You need to have fair understanding of
the target ISA already for LuaJIT, so might as well port the whole
interpreter and keep it fast at the same time.

With fairly cosmetic changes to LuaJIT and Lua5.2+luaffi, it is possible to use both
interchangeably, ie the desired result with the least effort.

[Daniel Kolesa]

Post by Dibyendu Majumdar

Post by Mike Pall

Post by Dibyendu Majumdar
I have a question regarding the Luajit interpreter. Is it possible to
write the Luajit bytecode interpreter in C - or does it rely upon

features

Post by Mike Pall

Post by Dibyendu Majumdar
only available in assembly language?

Sure ... and the transitions to and from JIT-compiled code,
including register and stack snapshot handling. And error handling
plus frame unwinding. And bytecode recording. And metamethod
continuations. And the builtins written in assembler. Oh, and
performance won't be stellar either ...

On a more technical note, it is possible to skew the current "most of hot paths
in asm" to "bare minimum glue asm". Some low level machinery of MM/C/ASM/JIT mcode
must be implemented in assembly as Mike mentions, however most of the actual
opcodes can be written in C. Easiest way is to adapt vm_record for your
"C stub" opcode implementations and use second level dispatch in C
for your C opcode handlers (in similiar fashion how trace recorder does it,
minus the restart). The C stub would return optional MM fallback for asm
glue to continue on.
Abusing fastfunc or lj_dispatch.c for this purpose is not advised as it is
rather invasive and would be difficult merge-wise with LuaJIT.
Also, meta dispatch is already mixed asm/c code (grep lj_meta_call),
the asm part is mostly the bare minimum to call C fallback.
All in all, it's not worth it IMO. You need to have fair understanding of
the target ISA already for LuaJIT, so might as well port the whole
interpreter and keep it fast at the same time.
With fairly cosmetic changes to LuaJIT and Lua5.2+luaffi, it is possible to use both
interchangeably, ie the desired result with the least effort.

LuaFFI is far from being actually useful. When I tried it was incredibly
buggy, only supported 2 or so architectures, didn't map to LuaJIT FFI
semantics closely and I managed to get it to fuck up on first non-trivial
example. So portability-wise and functionality-wise you're better off
actually using LuaJIT if you intend to use the FFI.

[Karel Tuma]

Post by Daniel Kolesa
LuaFFI is far from being actually useful. When I tried it was incredibly
buggy, only supported 2 or so architectures, didn't map to LuaJIT FFI
semantics closely and I managed to get it to fuck up on first non-trivial
example. So portability-wise and functionality-wise you're better off
actually using LuaJIT if you intend to use the FFI.

Indeed it needs some dusting off. The good part is that the Lua side of
API is already implemented. As for cosmetic changes, those amount mostly to
fixing Lua interop and ABI archdep:

* Rip out all the archdep stuff, its mostly broken anyway. Dynasm
does not support archs beyond what LuaJIT does already either.
* Replace with libffi which is robust and well maintained.
* Patch Lua to make it aware of ctype compare semantics.
* Teach libffi new arch if it is missing your target (has plenty, but not all).

It's not as hard as it sounds, though there is no code in the open
(both libffi and luaffi) as there are interesting bussiness cases
for this in the embedded/mobile space (aarch64, thumb2 .. yay for MIT).

The idea here is to get good long term portability with not much effort,
something I've been unable to achieve with LuaJIT (though eventually will
need to port LJ to thumb2, as speed of current solution is rather abysmal).

[Daniel Kolesa]

Post by Karel Tuma

Post by Daniel Kolesa
LuaFFI is far from being actually useful. When I tried it was incredibly
buggy, only supported 2 or so architectures, didn't map to LuaJIT FFI
semantics closely and I managed to get it to fuck up on first non-trivial
example. So portability-wise and functionality-wise you're better off
actually using LuaJIT if you intend to use the FFI.

Indeed it needs some dusting off. The good part is that the Lua side of
API is already implemented. As for cosmetic changes, those amount mostly to
* Rip out all the archdep stuff, its mostly broken anyway. Dynasm
does not support archs beyond what LuaJIT does already either.
* Replace with libffi which is robust and well maintained.
* Patch Lua to make it aware of ctype compare semantics.
* Teach libffi new arch if it is missing your target (has plenty, but not all).
It's not as hard as it sounds, though there is no code in the open
(both libffi and luaffi) as there are interesting bussiness cases
for this in the embedded/mobile space (aarch64, thumb2 .. yay for MIT).
The idea here is to get good long term portability with not much effort,
something I've been unable to achieve with LuaJIT (though eventually will
need to port LJ to thumb2, as speed of current solution is rather abysmal).

Have you seen the Python cffi module? It uses libffi and is closely modeled
after the LuaJIT FFI.

[Karel Tuma]

Post by Daniel Kolesa
Have you seen the Python cffi module? It uses libffi and is closely modeled
after the LuaJIT FFI.

Yes, it's a good example on how to go by about using libffi in LJ-esque
setting. Unfortunately it is mostly written in python, not really usable
as a foundation where Lua is used because of constrained resources in the
first place... If someone would be willing to do Python->Lua source port, it
might be interesting OTOH.

As for libffi, almost everything with FFI uses it :)

[lex pops]

Post by Mike Pall

Post by Daniel Kolesa

Post by Dibyendu Majumdar
I have a question regarding the Luajit interpreter. Is it possible to
write the Luajit bytecode interpreter in C - or does it rely upon features
only available in assembly language?

I don't see why it wouldn't be possible to write a VM that interprets
LuaJIT bytecode. But you're gonna have to deal with the FFI (which will
need either platform specific code or use something like libffi) if you
want it truly compatible.

Sure ... and the transitions to and from JIT-compiled code,
including register and stack snapshot handling. And error handling
plus frame unwinding. And bytecode recording. And metamethod
continuations.

What are metamethod continuations?

[lex pops]

Post by lex pops

Post by Mike Pall

Post by Daniel Kolesa

Post by Dibyendu Majumdar
I have a question regarding the Luajit interpreter. Is it possible to
write the Luajit bytecode interpreter in C - or does it rely upon features
only available in assembly language?

I don't see why it wouldn't be possible to write a VM that interprets
LuaJIT bytecode. But you're gonna have to deal with the FFI (which will
need either platform specific code or use something like libffi) if you
want it truly compatible.

Sure ... and the transitions to and from JIT-compiled code,
including register and stack snapshot handling. And error handling
plus frame unwinding. And bytecode recording. And metamethod
continuations.

What are metamethod continuations?

nm, some grepping of the source revealed lj_vm.h:121

/* Continuations for metamethods. */
LJ_ASMF void lj_cont_cat(void); /* Continue with concatenation. */
...

Though I'm not quite sure what these do, it's a starting point.

[Mike Pall]

Post by lex pops
What are metamethod continuations?

It's an implementation-choice for metamethods which allows a fully
resumable VM so you can yield across a metamethod call. This means
the complete state is contained in the Lua stack, there's no
implicit state in the C stack.

The simpler implementation choice (e.g. what Lua 5.1 does) is to
recursively invoke the interpreter on the metamethod. But then you
can't yield from a metamethod, since the C stack is 'in the way'.

Metamethods in LuaJIT 2.x implicitly do a function call. Apart
from constructing the new frame for the call it also leaves a
continuation there to differentiate it from a regular call.

When the call returns, the continuation is invoked to perform the
second half of the metmethod behavior. E.g. storing the result at
a location which depends on the invoking bytecode. Or branching,
depending on the truthiness of the result.

--Mike