Write to file many times in loop or create huge string then write to file?

Question

I was just wondering which is considered to be more efficient in PHP.

I need to loop through MySQL results and output it to a file using a specific format, I am interested to know whether it's more efficient to create a huge string (I'm going to be going through over 100,000 rows) and write the string to a file, or write (append) what I want to add to the file using fwrite, as I'm going through the results?

From the outset, I believe using a very long string will be more efficient than keep writing to a file but I am wondering whether PHP is going to have any issues with processing a very long string?

Thanks.

So, basically your question is not what you have mentioned but whether PHP has any problem in handling (very)long strings or not. Right? — Bhavik Shah, Jan 25 '13 at 11:52
@BhavikShah No, my question is what I stated. Which is more efficient? I believe that building up a huge string will be more efficient, but I'm not entirely sure and my worry concerning it is whether PHP has any string limitations. — EM-Creations, Jan 25 '13 at 11:54

score 3 · Accepted Answer · answered Jan 25 '13 at 11:54

PHP wont have an issue with creating a very long string, it will however use a lot of memory. As long as you have enough memory available then this won't be an issue. Reading and writing from memory is much faster than writing to disk so going down the memory route could be an option although with 100,000 rows, you could split it up and get the best of both worlds.

What I mean by this is every 10,000 rows append your string into the file and then empty the string, this way, you'll only be writing to file 10 times and storing a smaller amount of data in memory.

Thank you very much for your response. I think I'll go ahead with your suggestion. — EM-Creations, Jan 25 '13 at 11:56

score 1 · Answer 2 · answered Jan 25 '13 at 11:53

1

If you create a large string, try to collect all values into an array and join it together before writing. Strings are immutable, if you just append values to a String, this will lead to bad performance.

Best solution would be to collect for example 1000 results, write this part to a file, collet another 1000 results and so on.

answered Jan 25 '13 at 11:53

Gregor

4,306
1
22
37

2

Arrays will use much more memory – nice ass Jan 25 '13 at 11:55
Thank you very much for your response. It's very similar to Prisoner's suggestion, so I think this is what I'll go ahead with. – EM-Creations Jan 25 '13 at 11:56
1

You should read this topic on building large strings in memory: http://stackoverflow.com/questions/124067/php-string-concatenation-performance – Gregor Jan 25 '13 at 11:56

Write to file many times in loop or create huge string then write to file?

2 Answers2