16

I wonder if there exists way to work with large files in Mathematica ? Currently I have a file about 500Mb with table data.

Import["data.txt","Table"];

What is alternate way?

Igor Konoplyanko
  • 9,176
  • 6
  • 57
  • 100
  • it takes too much time to load large file. and I think too much memory too. – Igor Konoplyanko Mar 03 '10 at 13:08
  • 3
    Similar question was asked again here: http://stackoverflow.com/questions/7525782/import-big-files-arrays-with-mathematica . I gave two solutions for dense and sparse tables, which those coming to this page may find useful. They are still rather slow, but much more memory-efficient than `Import`. – Leonid Shifrin Sep 25 '11 at 19:02

3 Answers3

13

Use OpenRead["file"] which gives you an InputStream object on which you can use Read[stream]. Depending on the formatting of the data file you may need to set custom option values in Read[] for RecordSeparators.

Example:

In[1]:= str = OpenRead["ExampleData/USConstitution.txt"]    
Out[1]= InputStream["ExampleData/USConstitution.txt", 24]   

In[2]:= Read[str, Word]    
Out[2]= "We"    
In[3]:= Read[str, Word]
Out[3]= "the"    
In[4]:= Read[str, Record]
Out[4]= "People of the United States, in Order to form a more perfect Union,"
Timo
  • 4,246
  • 6
  • 29
  • 42
  • 8
    You need to remember to close the stream when you are done. Also, if you abort the operation, the stream will remain open. This can result in nasty surprises. So, I'd recommend wrapping your read code in a `CheckAbort` to catch the abort, and then close the stream regardless of whether the operation has been aborted or not. – rcollyer Mar 05 '10 at 15:43
5

You could also load your data into a database (for example MySQL) and access it from Mathematica using DatabaseLink

gdelfino
  • 11,053
  • 6
  • 44
  • 48
2

The function DumpSave can also be helpful for saving large datasets. It saves data in Mathematica's internal format, so it's more efficient in both time and file size.

Andrew Barber
  • 39,603
  • 20
  • 94
  • 123
Robert
  • 21
  • 1