0

I stumbled upon a neat trick that I've started using to write binary files into (flash) memory on arduino/esp8266 using a library someone posted to one of the esp8266 forums. I've been trying a number of ways to expand upon it. Most recently I've been minifying and compressing my web content files and compiling them in with sketches on my ESP. The script he posted first uses the output of the unix command xxd -i to write the binary file into an array of hex. The second part uses a struct to combine the file details with a pointer to the array that you can reference from the code whenever the server gets a uri request that matches an entry in the array.

What I would like to do is create a second array of these things with 'default' tools already pre-compressed so I don't have to go through it every time and/or modify my script that builds the header file any time I create a new server sketch. Basically compress and xxd stuff like jquery.js, bootstrap.css and bootstrap.js (or more often their smaller counterparts like backbone or barekit)

Currently once a file is dumped to hex, for example:

FLASH_ARRAY(uint8_t, __js__simple_js,
  0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x4b, 0x2b,
  0xcd, 0x4b, 0x2e, 0xc9, 0xcc, 0xcf, 0x53, 0xc8, 0xad, 0xf4, 0xcf, 0xf3,
  0xc9, 0x4f, 0x4c, 0xd1, 0xd0, 0xac, 0x4e, 0xcc, 0x49, 0x2d, 0x2a, 0xd1,
  0x50, 0x0a, 0xc9, 0xc8, 0x2c, 0x56, 0x00, 0xa2, 0xc4, 0x3c, 0x85, 0xfc,
  0xbc, 0x1c, 0xa0, 0x94, 0x42, 0x6e, 0x6a, 0x71, 0x71, 0x62, 0x7a, 0xaa,
  0x92, 0xa6, 0x75, 0x51, 0x6a, 0x49, 0x69, 0x51, 0x9e, 0x42, 0x49, 0x51,
  0x69, 0x6a, 0x2d, 0x00, 0x16, 0xa6, 0x25, 0xe5, 0x43, 0x00, 0x00, 0x00);

The existing code added them all at once along with the struct definition:

struct t_websitefiles {
  const char* path;
  const char* mime;
  const unsigned int len;
  const char* enc;
  const _FLASH_ARRAY<uint8_t>* content;
} files[] = {
{
    .path = "/js/simple.js",
    .mime = "application/javascript",
    .len = 84,
    .enc = "gzip",
    .content = &__js__simple_js,
  },
  {
 /* details for file2 ...*/
  },
  {
 /* details for file3 ...*/
  }
};

Building an array of the structs representing the various files.

My questions amount to noob questions regarding the language syntax. Can I assume that I can use an identical populated struct in the place of what is inside the curly brackets? For example, if I had a second header file with my regularly used libraries, and jquery was compressed in an array called 'default_files' at position 3, could I use something like &default_files[3] in the place of { /* definitions stuffs */ }. Such as:

struct t_websitefiles {
  const char* path;
  const char* mime;
  const unsigned int len;
  const char* enc;
  const _FLASH_ARRAY<uint8_t>* content;
} files[] = {
{
    .path = "/js/simple.js",
    .mime = "application/javascript",
    .len = 84,
    .enc = "gzip",
    .content = &__js__simple_js,
  },
  &default_files[1],
  &default_files[3],
{
    .path = "/text/readme.txt",
    .mime = "text/text",
    .len = 112,
    .enc = "",
    .content = &__text__readme_txt,
  }
};

(I'm guessing based on what I've learned thus far it needs the & in front of it?)

I also assume rather than re-writing the struct definition twice,I could do it as a typedef and then just do:

t_websitefiles files[] = { {/*definitions*/},{ /*stuffs*/ } };

Is that correct? Any help is appreciated. It's hard sometimes to find details on the syntax for specific use cases in documentation covering basics. (I would just try it, but I'm not conveniently in front of a compiler at the moment nor do I have direct access to my codebase but want to work on it later when I might not have direct access to the net)

NathanOliver
  • 171,901
  • 28
  • 288
  • 402
Scott
  • 7,983
  • 2
  • 26
  • 41
  • Initialization of structures with denoting components explicitly is supported since C99 but not in C++. Your C++ compiler might support this but it's not standard. (If you don't understand what I mean, please, see [C++ Structure Initialization](https://stackoverflow.com/q/11516657/7478597).) – Scheff's Cat Jul 28 '17 at 16:08

1 Answers1

2

From what I understand, you want create an array of structs such contains both compound literals and items from another array, all defined in header information. I don't think this is possible - or at least not in the exact way you suggest. I'll try and provide an alternative though.

Can I assume that I can use an identical populated struct in the place of what is inside the curly brackets?

No - you're mixing your types. 'files' is defined as an array of 'struct t_website'. The code

struct t_websitefiles files[] = {
    ...
    &default_files[1],
    ...
}

won't compile as you are mixing your types. files is defined as an array of struct t_websitefile, but &default_files[1] is a pointer. C makes a distinction between pointers and non-pointers. They are seperate types.

The obvious option that I can see to do what you want is to use pointers. This will allow you to define everything in header information.

struct t_websitefiles default_files[] = {
   ....
}
struct t_websitefiles files[] = {
   ....
}

// An array of pointers
struct t_websitefiles *files_combined[] = {
    &files[0],
    &files[1],
    &default_files[0],
    // Or whatever values you want here
    ...
}

// Example main, just iterates through the combined list
//  of files
int main(int argc, char* argv[]) {
    int i;
    int files_combined_len = sizeof(files_combined)/sizeof(struct t_websitefiles);

    for (i=0; i<files_combined_len; i++) {
        printf("File %s\r\n", files_combined[i]->path);
    }

    return 0;
}

Hope this helps.

  • cool! thanks! That's why I asked. I had a feeling there was some subtlety I was missing. Yeah, my only concern is to be able to pull from the semi-static 'defaults' that I'm not changing, and the script generated list of my own files that I am updating regularly. It would be nice to be able to access both in my main code with just a list of references to each source of file content and info. – Scott Jul 28 '17 at 18:46
  • This is the best way I've found to date to serve web content without needing to attach an SD card when using those ESP modules. It's so damned easy to have the array of items with a property that represents a uri. Then you just scan the array when a request is made to see if the resource is available, or throw a 404. Easy cheesy. And compressing it helps the ESP serve it up better without going to the trouble of using the (more complex) asynchronous server code. – Scott Jul 28 '17 at 18:49
  • by the way - how does the syntax for the property references work when using that style of referencing? I see the line you have with files_combined[i]->path. Is that all there is to it? (no other fancy symbols other than the ->?) – Scott Aug 01 '17 at 14:29
  • as an aside, I think the best solution for what I'm currently working on is in the auto-generation script. So I can add or remove files to keep the final .h file small, I'm going to modify my compressor script to first check my web tree and compress any uncompressed files into my gz tree. I was storing things by directory (e.g. gz/js, gz/htm, gz/css). But generating the file list as I went. Instead I'll compress then generate the list by again scanning the gz/... structure for the compressed files. That way I can just put the more 'permanent' stuff in there and leave it, already compressed. – Scott Aug 01 '17 at 14:32
  • The _->_ operator works exactly the same as a _._, except it is for use on a pointer (as opposed to an object). _files_combined[i]_ is a pointer to some memory containing a _struct t_websitefiles_. At a base level, this is just some number, so _files_combined[i].path_ doesn't make any sense. _._ doesn't make sense. However _files_combined[i]->path_ tells the compiler to look at at whatever memory address that the _files_combined[i]_ points to. I remember finding pointers confusing at first but they are super useful once you get your head round them. – thewaspsknees Aug 01 '17 at 16:57
  • I'm a bit confused by your last comment but I think you're on the right path. As an aside I'd recomment taking a look at [this thread](https://stackoverflow.com/questions/4864866/c-c-with-gcc-statically-add-resource-files-to-executable-library), in particular using objcopy or similar to add data (e.g. files) without putting it in the source code itself. This has the advantage of detaching resources from the raw source code resulting in smaller code and easier to modify data. – thewaspsknees Aug 01 '17 at 17:17
  • oh just saying that since I'm using a combination of 'find' and xxd -i to create the hex array, it's easier to just incorporate handling both types of files in the script I'm using to detect types, minimize and compress when necessary. Currently I'm doing it all at once, but I can split it into two steps and leave unmodified 'library' type files in minimized-compressed form to be detected after the non-static files are auto-processed. – Scott Aug 02 '17 at 18:03
  • (the idea behind this setup is that I can modify the web content in real time along with the program code, then run one script prior to compiling to create the header file with the hex-array form of the files to store in the FLASH memory of the device in question. When I use an SD card, I just edit them right on the card. When I store them in internal FLASH, I need to embed them in the program code that is uploaded, which means any modifications to web content have to be made prior to compiling and uploading the sketch so they can be incorporated with the upload) – Scott Aug 02 '17 at 18:05