possible memory leak in 2.1 (not in 2.0.3)

[Valerio Schiavoni]

Hello,
I think there's a memory leak somewhere in 2.1, which does not exhibit in 2.0.3.

To reproduce the bug:

git clone https://github.com/vschiavoni/all-distance.git
cd all-distance
luajit all-distance.lua

This is a 22-lines single-file Lua program: it imports a big table
from 3 other lua files (also in the repository) and it iterates over
the entries.
On my machine (Mac OSX 10.9), LuaJIT 2.1 (freshly cloned) crashes more
or less around the the time the program prints '651' (thought slight
variations might occur).
LuaJIT 2.0.3 works just fine.

Is this a known problem ? Am I missing something ?

Thanks,
Valerio

[Mike Pall]

Post by Valerio Schiavoni
On my machine (Mac OSX 10.9), LuaJIT 2.1 (freshly cloned) crashes more
or less around the the time the program prints '651' (thought slight
variations might occur).
LuaJIT 2.0.3 works just fine.

I get the out-of-memory condition in various versions and 32/64 bit
variants, just at different places. It may happen for plain Lua, too.

Post by Valerio Schiavoni
Is this a known problem ? Am I missing something ?

The Lua/LuaJIT garbage collector is only driven forward on regular
allocations, but not when growing a table.

There are only two allocations here: the 'local i_dist={}' and the
implicit string creation in 'print(i)'. LuaJIT doesn't even need an
allocation for the latter, i.e. it only performs half the GC steps.

But there are lots of table growths inbetween. So, even if the GC
starts a cycle at some point in time, it may not catch up and
perform enough incremental steps to complete the cycle before
memory runs out.

There have been various reports on the Lua mailing list about this
issue, esp. with big userdata allocations. IMHO it's quite
difficult to modify the current GC to handle that better. Tuning
GC heuristics to perform well under all kinds of mutator loads is
really, really hard.

--Mike

[Valerio Schiavoni]

Post by Mike Pall

Post by Valerio Schiavoni
On my machine (Mac OSX 10.9), LuaJIT 2.1 (freshly cloned) crashes more
or less around the the time the program prints '651' (thought slight
variations might occur).
LuaJIT 2.0.3 works just fine.

I get the out-of-memory condition in various versions and 32/64 bit
variants, just at different places. It may happen for plain Lua, too.

I could not reproduce it with plain Lua 5.2.3, at least not on mac osx
10.9 or ubuntu 14 lts.

Post by Mike Pall
The Lua/LuaJIT garbage collector is only driven forward on regular
allocations, but not when growing a table.
There are only two allocations here: the 'local i_dist={}' and the
implicit string creation in 'print(i)'. LuaJIT doesn't even need an
allocation for the latter, i.e. it only performs half the GC steps.
But there are lots of table growths inbetween. So, even if the GC
starts a cycle at some point in time, it may not catch up and
perform enough incremental steps to complete the cycle before
memory runs out.
There have been various reports on the Lua mailing list about this
issue, esp. with big userdata allocations. IMHO it's quite
difficult to modify the current GC to handle that better. Tuning
GC heuristics to perform well under all kinds of mutator loads is
really, really hard.

Would you suggest, as possible workaround, a pre-allocation of the
i_dist table with nil values, thus avoiding the table growths during
the iteration ?
Would that somehow help the luajit vm ?

Thanks,
Valerio

[Mike Pall]

Post by Valerio Schiavoni
Would you suggest, as possible workaround, a pre-allocation of the
i_dist table with nil values, thus avoiding the table growths during
the iteration ?

The problem looks like it only needs fixed arrays of fixed types,
albeit rather large ones. So I'd suggest to use FFI arrays/structs
to store all data.

Also, don't convert the input to Lua source and then load that. If
the input file is mostly static, better convert it to a binary
stream of doubles (once). Then simply map that file into memory
for each run.

--Mike

[Nicolas Hillegeer]

Post by Valerio Schiavoni
I could not reproduce it with plain Lua 5.2.3, at least not on mac osx
10.9 or ubuntu 14 lts.

On debian 7:

➜ all-distance git:(master) lua -v
Lua 5.1.5 Copyright (C) 1994-2012 Lua.org, PUC-Rio
➜ all-distance git:(master) lua alldistances.lua
...
830
831
832
lua: not enough memory

Next trying lua 5.2 (after some apt-get juggling):

➜ all-distance git:(master) lua -v
Lua 5.2.1 Copyright (C) 1994-2012 Lua.org, PUC-Rio
➜ all-distance git:(master) lua alldistances.lua
...
10923
...
and counting (no crash yet)

So it seems that either lua 5.2 received some changes to the garbage
collector, or it's a fluke. Since you couldn't reproduce it either I would
go with gc changes.