2

I am tuning a data import script, and occasionally I find an approach puts too much into memory in one call (usually this is because I am writing inefficient code). The "failed to allocate" message is only sort of useful in that it tells you how much memory was needed (without an informative traceback) and only if the allocation fails. Regular profiling requires that enough memory be available for allocation (and contiguously placed), which changes depending on the circumstances under which the code is run, and is very slow.

Is there a function that simulates a call to see how much memory will be used, or otherwise efficiently profiles how much memory a line of R will need whether it succeeds or fails? Something that could wrap an existing line of code in a script like System.time would be ideal.

Edit: lsos() does not work for this because it only describes what is stored after a command is run. (see: Reserved memory of R is twice the size of an allocated array)

Community
  • 1
  • 1
bright-star
  • 6,016
  • 6
  • 42
  • 81
  • How much memory a function (or line of code) needs, does not only depend on the function, but also on its input. Calculating how much memory will be needed should be possible for a specific function with specific input data. You can also profile memory demand for different sizes of input data and extrapolate. However, e.g., for reading text files memory demand not only depends on the size of the file, but also on its content (characters, numerics, factors, ..., number of columns vs number of rows, ...). – Roland Jun 16 '14 at 07:21
  • I see. I'll edit the question to be more precise. – bright-star Jun 16 '14 at 07:24
  • 2
    http://adv-r.had.co.nz/memory.html#memory-profiling – Roland Jun 16 '14 at 07:29
  • That library's functions are predicated on the call succeeding, though, right? – bright-star Jun 16 '14 at 07:52
  • If I do `x <- mean(outer(seq_len(1e6), seq_len(1e6))); traceback()` I get an error telling me how much memory was needed (and not available) and in which function the error occured. Your additional needs are unclear to me. – Roland Jun 16 '14 at 08:07
  • You're correct in that there are three or more *different* things I can do to accomplish this task depending on whether or not the line of code fails or succeeds, or whether I want to spend the time to run it or try to simulate based on object sizes and function calls. I was curious if there was an existing general solution that didn't involve putting them all together myself in a wrapper with error catching etc. – bright-star Jun 16 '14 at 08:47

0 Answers0