0

I'm trying to determine the reason why my code runs roughly four times slower than I thought it would.

In my code I'm iterating over a loop 100000 times. Once every 500 loops I print a line into a file (opening it in append mode, printing the line and closing it). Is it possible that on a decent computer (HPC cluster) this turns a 12h run into 48h run?

Hope my question is clear.

Yotam
  • 10,295
  • 30
  • 88
  • 128
  • This question needs a lot more info. How are you measuring, both overall time and the file i/o? Can you post some relevant code? Have you tried using some kind of profiling tool to see where your program is spending the most time? – Joe Nov 16 '11 at 18:27
  • 1
    Do the math. 100000/500 = 200 writes. 36 additional hours * 60 minutes per hour / 200 writes = 10.8 minutes per write. I would say that it is extremely unlikely that the file write is the whole cause. – AShelly Nov 16 '11 at 19:12
  • 2
    @AShelly 10.8 minutes makes a lot of sense if the OP is writing log messages... [to tape](http://www.quantum.com/Products/Autoloaders/SuperLoader3/Index.aspx) – ObscureRobot Nov 16 '11 at 22:17
  • I'd like to hear a direct answer to the title of this question. – User Feb 23 '12 at 23:00

2 Answers2

2

Try running an A/B test:

In test A, you do the logging as you describe - open and close the file for each log message.

In test B, try opening the file at the very beginning, keeping it open throughout the run, and only closing it when your program exits.

The time it takes to open and close files will depend on your operating system, C IO library implementation and your storage subsystem. I suspect that test B will be faster, but you need to run some experiments on your own software and hardware to be sure.

ObscureRobot
  • 7,306
  • 2
  • 27
  • 36
1

Just pause it a few times, and look at the stack. Whatever is making it take 4 times as long will appear on roughly 3 out of 4 stack samples. You can't miss it.

It's possible the problem is opening/closing files, but I doubt it if you're only doing it 200 times. Anyway, there's no need to hypothesize. Just pause it a few times and find out for certain what the problem is.

Community
  • 1
  • 1
Mike Dunlavey
  • 40,059
  • 14
  • 91
  • 135