Getting the most out of the GC

[Geoff Leyland]

Hi,

I often find myself with code that will push LuaJIT to an out of memory panic, and I suspect I’m tickling the GC incorrectly. It’s probably best to explain by example.

The following (very contrived) example runs smoothly:

local tree = {}
local branch = tree

for i = 1, 500000 do
local sum = 0
branch.next = {}
branch = branch.next
for i = 1, 10 do branch[i] = { sum } end

if i % 10000 == 0 then
io.stderr:write(("%10d: %.3fMB\n"):format(i, collectgarbage("count") / 1024))
end
end


while this panics after about 3M iterations, even though all the extra memory allocated is collectible:

local tree = {}
local branch = tree

for i = 1, 500000 do
local sum = 0
local temp = {}
for i = 1, 10000 do temp[i] = i end
for i = 1, 10000 do sum = sum + temp[i] end
branch.next = {}
branch = branch.next
for i = 1, 10 do branch[i] = { sum } end

if i % 10000 == 0 then
io.stderr:write(("%10d: %.3fMB\n"):format(i, collectgarbage("count") / 1024))
end
end


My remedy has been to collect garbage manually (as well as re-use temporary tables as much as possible):

local tree = {}
local branch = tree

for i = 1, 500000 do
local sum = 0
local temp = {}
for i = 1, 10000 do temp[i] = i end
for i = 1, 10000 do sum = sum + temp[i] end
branch.next = {}
branch = branch.next
for i = 1, 10 do branch[i] = { sum } end

if i % 10000 == 0 then
collectgarbage()
io.stderr:write(("%10d: %.3fMB\n"):format(i, collectgarbage("count") / 1024))
end
end


I get the feeling there must be something smarter that can be done. Anyone have any bright ideas? Should I just collectgarbage(“setpause”, 120)?


Thanks in advance,
Geoff