0

I've been working on a utility that makes judicious use of tables and I'm faced with a problem:

The dreaded ~2GB mem limit for Luajit. I'm using luajit 2.0.5. I'm working with very large flat files. I've tried forcing garbage collection from lua; it works some but continues to grow and eventually breaks with "not enough memory"

I researched and found one way to get around this is use the luajit ffi interface to allocate memory outside of lua control. Great! How exactly?

I've done some research and found the following link:

https://www.develcuy.com/en/playing-luajit-ffi

This actually shows how to work with the C library; not exactly what I wanted as I was hoping to allocate memory from FFI C interface so that the script did not break.

I looked a little more and found:

Why is LuaJIT's memory limited to 1-2 GB on 64-bit platforms?

Which looks like RaptorJIT and OpenResty Luajit2 have patches to avoid this entirely. . . . and on top of that. . . looks like maybe the luaJIT beta has it as well.

So the question is what is the smartest way to get past the 2GB limit without a re-write? I finally have the code working the way I want and it's pretty fast and seems bug free--expect for that pesky 2GB limit which I didn't find till I tested with a 20GB input file. :-(

Argh Tastic
  • 33
  • 5
  • 1
    Just use the latest LuaJIT 2.1, it doesn't have 2Gb limit problem. – Egor Skriptunoff Feb 08 '20 at 07:55
  • Thanks for confirming; now if I can just get it to compile on Windows. . . 2.0 compiled fine. 2.1 beta3 not so much. – Argh Tastic Feb 08 '20 at 22:22
  • I used the git bash shell to do the compiling; MSVC would not work for me and neither would MinGW64. The luajit build page list MSVC--which I am not at all familiar with, but the included batch file for beta3 doesn't work with the community edition of MSVC. cloning the repo and running make -j8 worked like a charm though. – Argh Tastic Feb 09 '20 at 03:41

0 Answers0