0

I have a long running process that creates a small image (~1-2 kb) and loads it into memory. Because its created using an APK I have to save it to a file before loading it into memory. At that point I no longer need the file. The process can potentially creates many millions of these files.

I am using Visual C++.NET and this is a windows based application. I'd like to know the best way to go about this.

I could delete them as I go, by calling DeleteFile (http://msdn.microsoft.com/en-us/library/aa363915(v=vs.85).aspx) but I wonder if that operation has overhead and if I ought to do it every 10k or 100k files using a wildcard and a naming convention.

Is that worth the effort? Whats the trade offs here?

Frank Conry
  • 2,694
  • 3
  • 29
  • 35
  • Maybe this question can help you. [Link](http://stackoverflow.com/questions/10673468/c-net-how-to-dispose-a-bitmap) – David Merinos Mar 31 '14 at 03:37
  • Well, I never need the file again after I read it it's base64 encoded and sent to another process, I just need some advice on the best way to delete millions of these. – Frank Conry Mar 31 '14 at 03:51
  • MAYBE puting them all into a directory and then just delete the directory in 1 interation. – David Merinos Mar 31 '14 at 03:53
  • 2
    Um, there is no wildcard version of DeleteFile, so your alleged alternative doesn't exist. – Raymond Chen Mar 31 '14 at 03:54
  • May be you can lunch a new thread just to DeleteFile to all your files in that particular directory. – Mantosh Kumar Mar 31 '14 at 04:03
  • Did you try to delete it after loading to memory and then profile the code? Personally I use only boost::filesystem for files/dirs management, but if you need to know what are the costs then measure. – MojaveWastelander Mar 31 '14 at 06:27
  • How big a performance hit does the call to `DeleteFile` incur? What has your profiling told you? – David Heffernan Mar 31 '14 at 08:26

3 Answers3

5

Use FILE_ATTRIBUTE_TEMPORARY and FILE_FLAG_DELETE_ON_CLOSE. Together they tell Windows exactly how you intend to use this file.

The combination means that Windows will keep the file in file cache until you close its handle, at which point it's removed from memory. You don't even need DeleteFile! Since all the semantics are still that of an ordinary file, you don't need to change anything else.

MSalters
  • 173,980
  • 10
  • 155
  • 350
  • 1
    Your solution requires that the OP'er also creates the APK files using the `FILE_ATTRIBUTE_TEMPORARY` flag. If the OP has control over the process that creates the image files then this is the best solution. However if he does not then this wont work. –  Mar 31 '14 at 11:19
4

It's not worth the effort. Just place the files into the user's temp folder, delete the files as you go and don't care about those left behind in case of emergency (i.e. in case of application crash). They are going to be cleaned up by system's cleanup process anyway.

Reason: you are already doing a lot of IO:

  1. Create the file (open for write)
  2. Write the file
  3. Read the file

"Delete" call would be just fourth in the chain - and just as light as creating the file first.

DarkWanderer
  • 8,739
  • 1
  • 25
  • 56
1

Since you have'nt mentioned any data integrity requirements and your temporary files are very small (~1-2 kb) plus there is a lot of disk I/O I would reccommend mapping a RAM Drive on the server and use that drive to store and retrieve the images.

Using a RAM drive works just like a secondary storage (Harddisk) does but everything is actually stored in the primary storage (RAM). This would speed up all (CRUD) I/O like creating a file, reading from a file, writing to a file, and deleting files as RAM has a lot higher data throughput than a harddisk.

Community
  • 1
  • 1
  • Ram drives are a pain to set up, and totally miss the point here. Windows has a file cache, which serves exactly the same point (use RAM as a substitute for disk). – MSalters Mar 31 '14 at 11:10
  • @MSalters I'm just trying to think OOTB since the OP's problem was I/O efficiency. Since he also never mentions data integrity then some kind of primary storage (RAM) solution is the way to go IMO - RAM disks or file caches are all good in that context. Your solution to the problem at hand seem to be preferred over mine if the OP also has control over the process that creates the file also, if he does not then there is a overhead of creating millions of files and he still has to delete them. –  Mar 31 '14 at 11:14