0

Using the packages RMySQL and dbConnect, I have a program that pulls tables from my Company's MySQL database. I'm an intern here leaving in a few weeks, and need to make my code robust for the guy that will be running it weekly once I'm gone (I don't want it to break on him / he will have issues debugging without me) - one of the things I'm worried about is the size of the queries, and whether or not R will always be able to handle / fit the size of the data pulled in the query. Currently, the code involves running a 10 - 30 minute query that results in a dataframe with X rows and Y columns.

Is there anyway to check (1) how much space in memory this dataframe is taking up, (2) how big this dataframe could become before R has an issue with it (other than by brute force testing the query with a very large pull until it's too big)?

Thanks,

Canovice
  • 9,012
  • 22
  • 93
  • 211
  • R just stores things in RAM, so the limiting factor to consider is the amount of RAM on the machine running the code compared to the max size the table could be. – joran Aug 16 '16 at 14:24
  • got it, okay that's what i thought, thanks – Canovice Aug 16 '16 at 14:29
  • ...but you can do some rough estimation using something like `object.size` on maybe a portion of one of the tables. – joran Aug 16 '16 at 14:31
  • Also take into account that you'll use roughly three times the dataset size in RAM because of the overhead of working with the data, see also http://stackoverflow.com/questions/21335132/r-ram-amounts-and-specific-limitations-to-avoid-memory-errors/21335288#21335288. – Paul Hiemstra Aug 16 '16 at 14:37
  • Possible duplicate of [Assessing how large a file (How much RAM it will take) will be in R before loading it](http://stackoverflow.com/questions/24299125/assessing-how-large-a-file-how-much-ram-it-will-take-will-be-in-r-before-loadi) – polka Aug 16 '16 at 19:09

0 Answers0