1

For my in-game AI project I use computer vision. So, for supervised learning I capture the screen and pressed keys.
I have a problem to store this huge amount of data (lot's of images of size 320x240) since my hard drive is limited in space. So far I found that saving frames in jpeg performs the best (1000 frames ~20MB).
I also tried to save array of images using numpy (.npy, 1000 frames ~220 MB) and h5py (.h5, 1000 frames ~220MB). In this ways the file sizes were too big to store sufficient amount of data for AI training (even when using gzip compression).

However, saving in jpeg gives very slow read/write speed. So, is there any way to store an array of images in a single file to have high read/write speed and at the same time being compact?
I found interesting research about it (https://stackoverflow.com/a/41425878), but seems in the case of images it is not helpful.

Rustam
  • 21
  • 7
  • maybe use binary file... but the question is do you really need to store all images in 320x240 ? can't you "pre-process" your images do reduce size when it's possible (without loss of information) and then reshape when you'll use it ? the question is also why you should store a so huge amount of data ... ? do you need to keep everything ? – Dadep Mar 23 '18 at 07:45
  • very likely a [x-y problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). you also fail to mention what you're going to do with that data. compression might render your data useless for further processing. I agree with Dadep. Try to extracting features and storing them is usually the better way – Piglet Mar 23 '18 at 08:59
  • @Dadep, it's already "pre-processed", since the original size of images is 800x600. And yes, I have to keep all the data first, because my computer is not too powerful to play and train AI simultaneously. – Rustam Mar 23 '18 at 09:33
  • @Piglet, I'm doing a self-driving car. Therefore, I want to train AI to keep a car staying in one lane (so far, later I'll add functionalities). What features are good to extract in my case? I think jpeg format even with loss will do it's work - AI will able to detect lines and stay in one lane. – Rustam Mar 23 '18 at 09:36
  • @Rustam, I agree with Piglet, you should extract features, and also may consider change the general architecture of your project. If you have computational limitation you should thinks in alternative way. Can you edit your post, to show a general diagram of you system ? – Dadep Mar 23 '18 at 09:44
  • so with 1000 frames (that's 16 seconds at 60fps) consuming 220MB which gives us roughly 1GB per minute you can record 15 hours of gaming footage per TB. so an average HDD these days should be capable of storing several days of training data. of course you don't have to use 60fps. much less should be perfectly fine so what exactly is your problem here? – Piglet Mar 23 '18 at 10:00
  • I am trying to reproduce the way how Nvidia trained it's cars (https://images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf). According to the paper, it needs really big data set, however I didn't find yet how big it should be. I think I will go with a big external HDD for storing data. Thank you guys. – Rustam Mar 23 '18 at 10:58

1 Answers1

0

Well, if you already have the images as (e.g. NumPy) arrays in memory, saving them using numpy.save or h5py is pretty much optimal, as both store the data in binary form (as compared to e.g. numpy.savetxt). To get even smaller file sizes, you can make use of one of the compression filters of HDF5/h5py.

The reason that you can get even lower file sizes by saving as jpeg is because this is a lossy compression format, meaning that you actually loose data. To make an objective comparison between "raw data" and "real image" formats, try saving to png instead.

jmd_dk
  • 12,125
  • 9
  • 63
  • 94