Create vector from CSV at compile time in C++

Question

I'm trying to create a lookup table for my Xilinx Zynq SoC (the ARM Cortex).

I have a CSV file with 1330 entries which I can not read or parse during runtime. The latest I can do that is at compile time. I have read it is possible to embed a file into an executable so it can be used during runtime.

In other words, I want to read and parse the CSV file at runtime, without the original file actually being on any filesystem since it's an embedded device. So I would need to somehow embed the CSV file into the executable. How would I achieve something like this?

The CSV file looks like this (full file is here);

0,0,48,112,160,208,272,320,368,....,65440,65488

@teivaz I have added a link to the the csv. I need to figure out a way to get all those values in an array and have them be available at runtime. — Nick, Jun 09 '16 at 20:59
Is there a reason you cannot just compile it into the code as an array? — Soren, Jun 09 '16 at 21:16

score 2 · Accepted Answer · edited May 23 '17 at 12:24

You asked for a vector but I'm not sure why you'd necessarily want that. The data will unavoidably occupy space in the application's read-only (".text" or ".rodata" or something like that) section, and while you can convert it into a vector if necessary (which will consume heap space and require runtime construction and initialization from the data in the read-only .text/.rodata section), you might as well just use it as a POD array since I doubt you'll be changing the data at runtime. So to create a const POD array of the data you could just do something like this....

const int myArray[] =
{
    #include "myCsvFile.csv"
};

If the number of elements is not fixed, your program can determine the number with sizeof(myArray)/sizeof(myArray[0]). Even if it is a fixed size, this technique is probably best anyway. And of course if all of your entries are unsigned and can fit within 16 bits (a cursory examination suggested so), instead of defining it as an array of int, you can define it as an array of unsigned short or uint16_t to save space.

I should also mention that the const keyword is important here: Without that, your array will occupy twice as much memory: first, it will occupy space in the read-only section (.text or .rodata or whatever), and during application initialization, the runtime will make a read/writable copy of the read-only data in the read/write data section (.data probably), where myArray is allocated. To avoid that, define it as const and then the address of myArray will be in the read-only section, and it won't be copied to the read/write data section.

score 1 · Answer 2 · answered Jun 09 '16 at 21:17

As your data is a plain array of unsigned integers you can use precompiler

Assuming you csv file is in data.csv file.

Then simply in .cpp file you can following:

const unsigned int k_Data[] = {
#include "data.csv" // << Preprocessor will replace this line with the contents of data.csv
};

#include <iostream>

int main()
{
    std::cout << k_Data[3];
}

112

Soren · Answer 3 · 2016-06-09T21:27:50.350

For the specific type of CSV data you have, e.g.

0,0,48,112,160,208,272,320,368,432,480,512,576,640,704,752,800,848,896,......

which is basically just a bunch of numbers, you should be able to include them using an actual #include statement, so;

const unsigned short myCSV[] ={
#include "./theCSV.data"
};

Im using unsigned short since the largest number looks like 64k and it would save you some space -- but you may want to use int instead, if you beleive the number can be larger than 64k

Create vector from CSV at compile time in C++

3 Answers3