1

Hey I am trying to create a storage system that can replace my current storage system.

I need to be able to save 10's of files per second fairly fast with little to no garbage creation in a way that is compatable with WP7, XBOX and PC using the .Net compact framework.

Originally in the game I am making I just saved the files to the disk as there are 100's of files for different parts of the world and things in it (I have already collapsed files together as best I can), on a background thread these files get batched up and then written out in batches while also streaming new files in.

While this was fine for pc, on the phone people would frequently exit the game or turn of their phone and as such only X of Z files would actually be saved resulting in a unknown save file state. And to make things worse it takes up to 1 second to save a single file on the Xbox and commit it to disk, and also generated heaps of garbage (>30 mb a minute) which causes a 1-3 second pause on the Xbox every 1-3 seconds due to its "very bad" garbage collector running every 1mb of allocated memory. Everything else in the game generates no garbage and is pooled and cached so this was also a problem.

So I then migrated to using a single file to store everything using a zip system. This works but fairly rarely the zip file just corrupts as I am using it in a way that it was never designed for... Basicly I load up the file at game start, then save to and from it while it sits in a memory stream, then every 3-7 minutes based on player progress I dump the single file to the disk.

So basicly zip files (I have tried 2 implimentations) are not doing what I need. Does anyone know if anything else exists that could do what I need or have any ideas on how I should build my own?

tldr;

So in short I need a system that is

  • Single file
  • Can live in memory and just be commited to local disk periodically
  • Generates little to no garbage
  • .Net Compact framework compliant (or with source so I can make it so)
  • File/structure editing is done in-place
  • Reliable first, performant second
  • No need for compression or encryption
  • Doesn't need to be able to just load a small part of the file
  • Could be a type of database / doesn't have to mimic a file stucture

Similar to this question Here

Anyone have any ideas?

Community
  • 1
  • 1
Daniel Armstrong
  • 829
  • 9
  • 22
  • Interesting problem, but I'm sure that there is probably an easier way. Can you elaborate as to why you need to save a couple of "files" to disk every few seconds? That sounds like it's begging for corruption – Earlz Jul 23 '12 at 05:51
  • My game is a blockworld game kinda like minecraft or infinminer but with magic and skyrim like combat etc... It features an infinite world and needs to generate and save the world as well as world edits. I do a whole heap of compression of my own on all the data (my saves are 2800% smaller than minecrafts for the same size data) so its not file size that is the issue but amount of files. The corruption with the zip librarys is the main problem... – Daniel Armstrong Jul 23 '12 at 06:05
  • I find this question kind of annoying, because you've encountered a problem, selected a solution, and are asking for an implementation of that solution. I'm not sure you've selected the right solution here. – Andrew Russell Jul 23 '12 at 12:22
  • I already have an implimentation of the original solution that I came up with. The issue is that it doesn't work, this is why I am asking for other ideas or implimentations of my current solution that can improve on its weeknesses. – Daniel Armstrong Jul 23 '12 at 13:54

0 Answers0