-3

Now I am using R to integrate different stock information(like RCurl for Web info and quantmod for financial report and trade info)

After I get data I put them in MySQL database.

The program works find BUT after loop some time the R session consumes too much memory(some 1000 stocks takes 4GB memory)

Is there anyway in R I can check which objects takes the most memory? Or is there any tools I can check the possible memory leak problem? Thanks!

Colin.Fu
  • 15
  • 2
  • 5
    See the [Previous SO Post](http://stackoverflow.com/questions/1358003/tricks-to-manage-the-available-memory-in-an-r-session) – G5W Dec 19 '16 at 16:28
  • Are you growing an object in a loop? – Roland Dec 19 '16 at 16:38
  • yes, the basic logic in a loop is: download html code, parse it and extract value i want, load it into an object then write to it to mysql. – Colin.Fu Dec 21 '16 at 15:21

1 Answers1

1

Great thought. I would encourage taking a look at Hadley Wickham's article on R memory use at http://adv-r.had.co.nz/memory.html. A Couple of notes:

  • You can find an object size by using the function object_size(x) from the pryr package
  • You can find the total size of all objects in memory with mem_used() from the pryr package
  • There is an awesome package called lineprof (now deprecated in favour of profvis) that allows you to see memory use at every line of your code.

Hope it helps.

Ben Bolker
  • 211,554
  • 25
  • 370
  • 453
jean
  • 11
  • 1
  • thanks jean and others, I have read all references you offered before, it helps. Since I use the XML package to parse html code and it seems it is the most suspicious one! As recommends, I use xml2 package to replace XML, it works *almost* fine which it dramatically low the consumption of memory. Maybe the lower growth rate of memory seizing is caused by other reasons. – Colin.Fu Dec 25 '16 at 05:57
  • I am using xml2::read_html to replace RCurl::getURL; using xml_find_all to replace XML::getNodeSet(which seems the most suspicious function cause memory leaks). then use xml_remove(x, free = TRUE) to free memory explicitly. hope this helps. – Colin.Fu Dec 25 '16 at 06:00