3

Here is the result of a profiled simulation run of my MATLAB program. I need to run this simulation several hundred thousand times (~100,000 times).

The abnormally slow xlsread

Thus I need a faster way to read the Excel file.

Specifications: The Excel file is of 10000x2 cells and each simulation run is reading one such sheet each from 5 separate Excel files.

UPDATE: I put the xlsread in basic mode and also reduced the number of calls by combining my input into a single file. Next target is xlswrite now. Ah, that sinking feeling. :|

Updated Profile Summary

NOTE: Although writing to a CSV file using dlmread is very fast (around 20 times), I need to use the comfort of separate sheets that an .xls file provides.

Amro
  • 123,847
  • 25
  • 243
  • 454
OrangeRind
  • 4,798
  • 13
  • 45
  • 57
  • 1
    Have you tried to alter the function xlsread itself? Find the relevant script file, copy it and then turn off error checking that happens at the beginning (eg. is string empty? etc.) since you know that the input to the program is always going to be an excel file of 10000x2 cells. – Sriram May 30 '11 at 08:06
  • I'd turn the Excel into a .mat file as a separate batch process, then at least you only need to do that once. – Alex May 30 '11 at 13:25
  • 4
    A lot of time is consumed by activeX which slows down the data loading. I am not 100% sure but you should speed up it using option 'basic'. http://www.mathworks.com/help/techdoc/ref/xlsread.html#inputarg_'basic' – Jirka cigler May 30 '11 at 18:07
  • Jirka cigler is right. The 'basic' option makes it blindingly fast. – Rich C May 30 '11 at 22:46
  • Bloody hell. Its speeding up by an order of magnitude! You can put that as an answer surely! – OrangeRind May 31 '11 at 01:45
  • I just updated the new profile with the suggested changes. looks green, but want greener. :) – OrangeRind May 31 '11 at 02:18
  • @OrangeRind: Check my answer to a similar question: http://stackoverflow.com/a/6784516/97160 (especially the second part) – Amro Jun 09 '12 at 15:51

1 Answers1

1

I don't think you would be able to wring much out of xlswrite if you need Excel sheets as the output.

How about parallelizing?

Do you have access to the parallel computing toolbox? Or maybe you can run two instances of MATLAB if your box supports it. If so, you could consider two approaches:

  1. Have the first process do the xlsread part, the simulation part and then write to mat files/plain binary/CSV, whatever is the fastest while preserving your data integrity. Have another process convert the matfiles/intermediate data files into Excel using xlswrite.

  2. Have N MATLAB instances/workers (N depends on your physical machine capacity). Parallelize the whole read-process-write part across N workers. Note, I am not sure how Excel would scale when called by N workers! (xlswrite uses activeX/MS Excel to write the data).

As any parallel approach, your mileage will vary on the complexity of the simulation vs. required file I/O and its performance.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Ashish Uthama
  • 1,331
  • 1
  • 9
  • 14
  • Haven't done this before. Thus jumping on it rightaway. Will hit again in case of any problems. :) – OrangeRind Jun 01 '11 at 01:07
  • Thanks. much improvement. However, a newer problem has come about which is entirely on a different plane. So I'll accept this as an answer, and post the new problem separately. – OrangeRind Jun 01 '11 at 03:57
  • Connecting question idhar hai - http://stackoverflow.com/questions/6196653/non-linear-performance-of-java-function-in-parellel-matlab need to make it faster. :) – OrangeRind Jun 01 '11 at 05:26