2

My application/device is running on an ARM Cortex M3 (STM32), without OS but with a FatFs) and needs to access many resources files (audio, image, etc..)

  • The code runs from internal flash (ROM, 256Kb).
  • The resources files are stored on external flash (SD card, 4Gb).
  • There is not much RAM (32Kb), so malloc a complete file from package is not an option.

As the user has access to the resources folder for atomic update, I would like to package all theses resources files in a single (.dat, .rom, .whatever) So the user doesn't mishandle theses data.

Can someone point me to a nice solution to do so?

I don't mind remapping fopen, fread, fseek and fclose in my application, but I would not like starting from scratch (coding the serializer, table of content, parser, etc...). My system is quite limited (no malloc, no framework, just stdlib and FatFs)

Thanks for any input you can give me.

note: I'm not looking for a solution where the resources are embedded IN the code (ROM) as obviously they are way too big for that.

Tweepy
  • 75
  • 9
  • 3
    How about using a archive? Zip is pretty popular for such purposes, and you don't need to have the contents compressed, just use the archiving functionality. – Some programmer dude Feb 08 '16 at 14:40
  • Why not keep individual files but mark them with the system and hidden attributes? This should cause Windows to emit a warning when you attempt to remove them. – fuz Feb 08 '16 at 14:51
  • Another possible - create a folder for each version. You could name the folder with a representation of the version/date/time. – Martin James Feb 08 '16 at 14:55
  • Making up a simple compound file format shouldn't be too difficult. That doesn't need more than 100 lines of code. – fuz Feb 08 '16 at 14:56
  • I would rather use TAR instead of ZIP, as it's Makefile/GNU compatible as Brian McFarland pointed. Individual file is not an option, as the user can be Win/Mac/Nux, there is no way of nicely hiding it. – Tweepy Feb 09 '16 at 00:06

3 Answers3

1

To expand on what Joachim said above:

Popular choices of uncompressed (sometimes) archive formats are cpio, tar, and zip. Any of the 3 would work just fine.

Here are a few more in-depth comments on using TAR or CPIO.

TAR

I've used tar before for the exact purpose, on an stm32 with FatFS, so can tell you it works. I chose it over cpio or zip because of its familiarity (most developers have seen it), ease of use, and rich command line tools.

GNU Tar gives you fine-grained control over order in which the files are placed in the archive and regexes to manipulate file names (--xform) or --exclude paths. You can pretty much guarantee you can get exactly the archive you're after with nothing more than GNU Tar and a makefile. I'm not sure the same can be said for cpio or zip.

This means it worked well for my build environment, but your requirements may vary.

CPIO

The cpio has a much worse/harder to use set of command line tools than tar in my opinion. Which is why I steer clear of it when I can. However, its file format is a little lighter-weight and might be even simpler to parse (not that tar is hard).

The Linux kernel project uses cpio for initramfs images, so that's probably the best / most mature example on the internet that you'll find on using it for this sort of purpose.

If you grab any kernel source tree, the tool usr/gen_init_cpio.c can used to generate a cpio from a cpio listing file format described in that source file.

The extraction code is in init/initramfs.c.

ZIP

I've never used the zip format for this sort of purpose. So no real comment there.

Brian McFarland
  • 9,052
  • 6
  • 38
  • 56
  • The system has to be able to read these formats. I wonder if his device has the extra space in ROM for the code to read these formats? – Chimera Feb 09 '16 at 00:50
  • TAR and CPIO are great suggestion but they don't have a table of content (index), meaning I'll have to parse the complete archive (of 1Gb) to find a file. DAR looks like TAR, but with a file index. – Tweepy Feb 09 '16 at 11:13
  • @Tweepy - You don't actually have to read 1Gb from disk to scan a TAR/cpio. The header includes a length that you can use to `f_lseek` (in fatfs terms), which *should* be much faster than reading the whole file. Although, theoretically, having an index could make seeking faster. If DAR does this, I think you answered your own question :-). – Brian McFarland Feb 09 '16 at 19:11
  • @Chimera - You can write a TAR (not tar.gz) parser in 50-150 lines of C depending on how many header fields you can afford to ignore and whether or not you already have `memcmp`, `memset`, `memcpy`, and `strtol` already linked into the image. I would guess no more than 1kB ROM and not much more than 512 bytes RAM (a single raw header/block). The format dates back to the 1970s when high priced mainframes were about as powerful as this guy's MCU is today. TAR="tape archive". – Brian McFarland Feb 09 '16 at 19:25
1

It should be possible to use fatfs recursively.

Drive 0 would be your real device, and drive 1 would be a file on drive 0. You can implement the disk_* functions like this

#define BLOCKSIZE 512
FIL imagefile;
DSTATUS disk_initialize(BYTE drv) {
  UINT r;
  if(drv == 0)
    return SD_initialize();
  else if(drv == 1) {
    r = f_open(&image, "0:/RESOURCE.DAT", FA_READ);
    if(r == FR_OK)
      return 0;
  }
  return STA_NOINIT;
}
DRESULT disk_read(BYTE drv, BYTE *buff, DWORD sector, DWORD count) {
  UINT br, r;
  if(drv == 0)
    return SD_read_blocks(buff, sector, count);
  else if(drv == 1) {
    r = f_seek(&imagefile, sector*BLOCKSIZE);
    if(r != FR_OK)
      return RES_ERROR;
    r = f_read(&imagefile, buff, count*BLOCKSIZE, &br);
    if((r == FR_OK) && (br == count*BLOCKSIZE))
      return RES_OK;
  }
  return RES_ERROR;
}

To create the filesystem image on Linux or other similar systems you'd need mkfs.msdos and the mtools package. See this SO post on how to do it. Might work on Windows with Cygwin, too.

Community
  • 1
  • 1
followed Monica to Codidact
  • 7,779
  • 1
  • 23
  • 39
  • 1
    Clever. I think this is actually the path to least amount of code change on the target platform. Not sure I'd call it "trivial" to make the image though--I think using "standard" Linux tools, you have to make/mount a loopback device to put anything in the image, no? If so, that which points to needing root privileges to build. I don't think there's an a widely available equivalent to `genisoimage` or `mksquashfs` for FAT16/32 – Brian McFarland Feb 09 '16 at 19:09
  • OK, maybe not exactly trivial, but doesn't need root, see the edited answer. Anyway, it needed some explanation, thanks for pointing it out. – followed Monica to Codidact Feb 10 '16 at 04:21
1

Berendi found a very clever solution: use the existing fat library to access it recursively!

The implementation is quite simple, and after extensive testing, I'd like to post the code to use FatFs recursively and the commands used for single file fat generation.

  1. First, lets generate a 100Mb FAT32 file:

dd if=/dev/zero of=fat.fs bs=1024 count=102400

mkfs.vfat -F 32 -r 112 -S 512 -v fatfile.fs

  1. Create/push content into it:

echo HelloWorld on Virtual FAT >> helloworld.txt

mcopy -i fatfile.fs helloworld.txt ::/

  1. Change the diskio.c file, to add Berendi's code but also:
DSTATUS disk_status ()
{
  DSTATUS status = STA_NOINIT;
  switch (pdrv)
  {
  case FATFS_DRIVE_VIRTUAL:
      printf("disk_status: FATFS_DRIVE_VIRTUAL\r\n" );
  case FATFS_DRIVE_ATA:    /* SD CARD */
      status = FATFS_SD_SDIO_disk_status();
  }
}
  1. Dont forget to add the enum for the drive name, and the number of volumes:

#define _VOLUMES 2

  1. Then mount the virtual FAT, and access it:
f_mount(&VirtualFAT, (TCHAR const*)"1:/", 1);
f_open(&file, "1:/test.txt", FA_READ);

Thanks a lot for your help.

Tweepy
  • 75
  • 9