C stack overflow in coroutine.wrap, possible other FF

[Karel Tuma]

With LuaJIT, it is possible to blow up C stack:

$ lua
Lua 5.2.3 Copyright (C) 1994-2013 Lua.org, PUC-Rio

function x() coroutine.wrap(x)() end x()

stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1:
stdin:1:
stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: stdin:1: C stack overflow
stack traceback:
[C]: in ?
stdin:1: in function 'x'
stdin:1: in main chunk
[C]: in ?

^D

$ git describe
v2.0.3-257-ge107525
$ ./luajit
LuaJIT 2.1.0-alpha -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/
JIT: ON SSE2 SSE3 fold cse dce fwd dse narrow loop abc sink fuse

function x() coroutine.wrap(x)() end x()

Segmentation fault (core dumped)

This happens with anything recursively calling through C. C frame
usage should be probably capped somehow, like Lua does. Lazy sketch of fix
is:

https://github.com/katlogic/ljx/commit/6647d950136bb32fee73b7bbf7e15545ee0d4089

This is rather wrong, as the check should go into all lj_vm_*.dasc,
whenever L->cframe is saved or somesuch, or maybe implement Lua nCcalls
(I like size usage more though, as it allows to place hard caps on small
pthread stacks).