1

I am looking for a way to rewrite a Mathematica function which reads a text file of numerical values (several hundred Mb in size) and creates a Mathematica array out of it. I use ReadList to read the file and this goes as quickly as one could expect, but my problem is how do I get this array back out to a variable outside this function without it being copied or anything silly like that?

Currently my function is written like

LoadData[vars_...]:=Module[{localvars_...},...Return[giant array];];

and try to assign the giant array to a variable like this:

table = LoadData[vars_...]

However the time spent taking the array from "giant array" and putting it in "table" is longer than the file read itself. Clearly the whole thing is being copied or something silly; it will probably overflow the memory if I go to much larger data.

How can I prevent this happening? How can I get Mathematica to just rename the array rather than copy all the data? In case it is relevant, the function is defined in a separate package to the one where I am using it.

SOLVED:

Ahh, thanks belisarius, simple question on passing data between functions indeed gives me the hint I need, i.e. setting

Attributes[LoadData]={HoldFirst}

in the package seems to make the function behave sensibly.

Community
  • 1
  • 1
Ben Farmer
  • 2,387
  • 1
  • 25
  • 44
  • 1
    How did you determine that the copy operation was the problem? – rcollyer Sep 19 '11 at 05:12
  • 3
    Related http://stackoverflow.com/questions/7376032/simple-question-on-passing-data-between-functions – Dr. belisarius Sep 19 '11 at 05:20
  • I put a print statement just before the Return call in the function and one just after the assignment to "table" in the parent notebook, and it took a very long time to get from one to the other. – Ben Farmer Sep 19 '11 at 05:21
  • Note that for the HoldFirst to work as you intend, the huge array should be the first argument. Otherwise, a HoldAll might be used. Furthermore, a `Return` is probably not necessary as you modify the array in-place. – Sjoerd C. de Vries Sep 19 '11 at 11:41
  • 1
    @belisarius Given that you more or less gave the answer in the comment I think it would be good to post it as an answer too. The answer count is still zero and hence it looks unanswered. – Sjoerd C. de Vries Sep 19 '11 at 11:43
  • @Sjoerd If that answer solved the problem, I think it is a dup. Voting to close instead of posting an answer! ...and BTW, without having read the other Q. with care yet, I must confess I don't fully understand why HoldFirst solves the problem :) – Dr. belisarius Sep 19 '11 at 12:03
  • @belisarius. Now that I come to think of it: the questions are different (why slow? vs how to pass by reference?), but the answer is the same. Is it a dup then? – Sjoerd C. de Vries Sep 19 '11 at 12:19
  • @Sjoerd I thought like you until http://meta.stackexchange.com/questions/104227/is-it-acceptable-to-add-a-duplicate-answer-to-several-questions – Dr. belisarius Sep 19 '11 at 13:06

1 Answers1

0

You can also make the variable Dynamic and update it from within the function - load data straight into it. But Unevaluated works just as well for this cause.

Gregory Klopper
  • 2,285
  • 1
  • 14
  • 14