3

My situation is I have a legacy app which I don't have the code for which writes out data to disk every second or so. I have a C# program I wrote which every second reads what was written to disk and uses the data. The data is written to a few text files which I know the file name before its created.

The issue is I have lots of virtual machines running this legacy app and my program. They are not limited by ram or cpu but I can't add more than 10 VMs per machine due to file io bottleneck.

Is there an easy way I can make a file on disk that exists in ram or something else? I heard something about named pipes being an option?

Thanks!

caf
  • 233,326
  • 40
  • 323
  • 462
Luke Belbina
  • 5,708
  • 12
  • 52
  • 75
  • You could always use a RAM disk. I see you've tagged the question that way. Is that because you want to know how to create one? Can you specify the path to the file that the legacy app writes? – Cody Gray - on strike Jan 23 '11 at 04:40
  • 1
    The VMs cause the problem, it prevents data being shared through the file system cache (memory, not disk). Just use *one*. – Hans Passant Jan 23 '11 at 09:57
  • I can specify the path the legacy app writes to so I guess I could just set up a ram disk. Was hoping for something a cooler tho :-) And regarding just using one VM, I can't because the legacy app has hard coded limits on how much it will do one one machine. – Luke Belbina Jan 24 '11 at 15:46

2 Answers2

0

You can search for some kind of memory/temporary file systems.

I'm not sure if you can use pipes here since you legacy app is writing directly to the HDD.

Community
  • 1
  • 1
Elalfer
  • 5,312
  • 20
  • 25
0

Are you sure of the actual IO involved?

Long ago I implemented a very ugly connection sending data from a dos program to a Windows program by means of a file. This was a lot faster than once a second, though--the dos program would send a 4k block anytime anything on it changed, 50 times a second (if it was caught up) the Windows program would read the frame number and then read the 4K block if the frame number differed.

This did NOT cause disk IO! You could sit there causing the dos program to update the frame many times a second for as long as you wanted and the hard drive light would stay off. Windows saw the file was open and being frequently written, the buffers were NOT flushed to disk until the updates stopped.

While I spent a lot of time optimizing the Windows side of the link it was all in what was done with the data, not in the connection--that simply wasn't a bottleneck despite it's apparent ugliness.

It's possible Windows would handle it differently if the file was closed each time. Sticking it on a ramdisk would keep it from doing the disk IO even then.

Loren Pechtel
  • 8,945
  • 3
  • 33
  • 45
  • yeah, just one machine can eat the IO even tho its ugly. however when I have 15 computers all on the same HDD running simultaneously its a bit different. I guess I'll have to use a ram disk. Any suggestions? – Luke Belbina Jan 24 '11 at 15:48
  • @nextgenneo: What you're missing is that Windows buffered the IO, it never went to the disk in the first place. – Loren Pechtel Jan 24 '11 at 15:55