-3

I need to create some random files (say 100,000) each of 1GB or higher. I tried the normal way of open file and writing into it using print/syswrite, but it's taking more than 30mins. Could somebody tell me any other efficient quick way I can achieve it in perl.

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
jais
  • 21
  • 2
  • 1
    May be a good idea to explain what your goal is here as there are probably a lot better ways than what you are currently trying. Are you trying to test something? – Geezer68 Jan 17 '14 at 02:31
  • You don't care about what is in the file? Change its size in the file allocation table. Il will be filled with leftover bytes stored on disk. Windows has an API for it (`SetFilePointer` and `SetEndOfFIle`). You can use system calls in *Nix. See [this answer](http://stackoverflow.com/a/7970410/591064). – ixe013 Jan 17 '14 at 03:45
  • 30 minutes for 1 10G file? that seems excessive; what size of blocks are you writing? have you tried just writing a file with a single unchanging block to see how much of the time is generating the random data? – ysth Jan 17 '14 at 04:14
  • It's unclear whether the OP is saying it takes 30+ minutes to create a single 10G file (in which case something is very wrong) or to create 10,000 (or 100,000) 10G files (which is to be expected, given that most hard drives will require several days, if not weeks, to write that much data). – Dave Sherohman Jan 17 '14 at 08:44

3 Answers3

7

What do you want to use these files for?

A standard hard disk drive will typically write 100MB per second.

That means 100 seconds to write a 10GB file.

10,000 such files will take 1 million seconds, or about 11.5 days, and will occupy 100TB, or twenty-five standard 4TB 3.5in drives.

You can persuade the file system to just allocate the space to the file without writing anything, but there isn't a way to do that that I know of using a scripting language. And you would end up with files containing random data.

Borodin
  • 126,100
  • 9
  • 70
  • 144
5

On posixy systems, just:

dd bs=1M count=10240 if=/dev/urandom of=foo.dat

which should take a few minutes at most.

ysth
  • 96,171
  • 6
  • 121
  • 214
3

Your choice of language doesn't matter here. Perl is as good as anything else. You certainly don't have that much RAM, so the time this takes depends on the speed of your disk. And writing 100TB takes a long time no matter what.

hobbs
  • 223,387
  • 19
  • 210
  • 288