How to upload binary data (html + favicon) to ESP32 using ESP-IDF?

Question

I am writing a file-serving http server for ESP32 using ESP-IDF and PlatformIO, but I just can't make data upload to SPIFFS work. I am trying to send html and favicon to flash, so it can be served on http.

Code of the server is taken from an example https://github.com/espressif/esp-idf/tree/master/examples/protocols/http_server/file_serving. Noticeable difference is that example project uses only ESP-IDF tools (without platformio) and that in an example data files are in the same directory as source files, where in my projects I have separated directories for /src and /data. SPIFFS is configured using custom partition table.

I was following instructions from both PlatformIO documents (https://docs.platformio.org/en/latest/platforms/espressif32.html?utm_source=platformio&utm_medium=piohome) as well as from ESP (https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html#embedding-binary-data).

I have custom partisions.csv file (identical to the example) + changed menuconfig to use it.

In platformio.ini I added:

board_build.partitions = partitions.csv
board_build.embed_txtfiles =
    data/favicon.ico
    data/upload_script.html

I also changed project CMakeLists file to embed data like this:

cmake_minimum_required(VERSION 3.16.0)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(HTTP-server)
target_add_binary_data(HTTP-server.elf "data/favicon.ico" TEXT)
target_add_binary_data(HTTP-server.elf "data/upload_script.html" TEXT)

/src/CMakeLists stayed unchanged:

FILE(GLOB_RECURSE app_sources ${CMAKE_SOURCE_DIR}/src/*.*)
idf_component_register(SRCS ${app_sources})

But even with all this configuration when I try to use this data in file_server.c like this:

extern const unsigned char favicon_ico_start[] asm("_binary_favicon_ico_start");
extern const unsigned char favicon_ico_end[] asm("_binary_favicon_ico_end");
extern const unsigned char upload_script_start[] asm("_binary_upload_script_html_start");
extern const unsigned char upload_script_end[] asm("_binary_upload_script_html_end"); 

I get compilation errors

Linking .pio/build/nodemcu-32s/firmware.elf
/home/artur/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/nodemcu-32s/src/file_server.o:(.literal.http_resp_dir_html+0x14): undefined reference to `_binary_upload_script_html_end'
/home/artur/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/nodemcu-32s/src/file_server.o:(.literal.http_resp_dir_html+0x18): undefined reference to `_binary_upload_script_html_start'
/home/artur/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/nodemcu-32s/src/file_server.o:(.literal.favicon_get_handler+0x0): undefined reference to `_binary_favicon_ico_end'
/home/artur/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/nodemcu-32s/src/file_server.o:(.literal.favicon_get_handler+0x4): undefined reference to `_binary_favicon_ico_start'
collect2: error: ld returned 1 exit status
*** [.pio/build/nodemcu-32s/firmware.elf] Error 1
================================================================================= [FAILED] Took 65.20 seconds =================================================================================

I tried changing extern definitions to:

extern const unsigned char favicon_ico_start[] asm("_binary_data_favicon_ico_start");

But it didn't change anything.

Additionally when running "Build Filesystem Image" task I get this error:

*** [.pio/build/nodemcu-32s/spiffs.bin] Implicit dependency `data/favicon' not found, needed by target `.pio/build/nodemcu-32s/spiffs.bin'.
================================================================================= [FAILED] Took 5.70 seconds =================================================================================
The terminal process "platformio 'run', '--target', 'buildfs', '--environment', 'nodemcu-32s'" terminated with exit code: 1.

Any help would be very much appreciated, as I feel that I did everything that the documentation stated.

Answer 1

I suspect you need to declare them as BINARY instead of TEXT. TEXT creates a null-terminated string which probably won't generate the _binary_..._end aliases.

target_add_binary_data(HTTP-server.elf "data/favicon.ico" BINARY)
target_add_binary_data(HTTP-server.elf "data/upload_script.html" BINARY)

Also something's off with your SPIFFS image generation. You know that the CMake macros target_add_binary_data() and idf_component_register(... EMBED_TXTFILES...) embed the stuff only into the application binary, right? You cannot use them add stuff to a pre-generated SPIFFS partition. For that you need to use spiffsgen-py script.

Answer 2

The problem was that I was a little bit confused with the difference between embedding files into an app and sending stuff to SPIFFS partition.

Solution was to move both .html and .ico files from /data to /src. I think the reason behind this is that this code: asm("_binary_favicon_ico_start") can't reference files in other directories, but I am not sure. I also reversed project CMakeFile to default and added this line to /src/CMakeFile: idf_component_register(SRCS ${app_sources} EMBED_FILES "favicon.ico" "upload_script.html")

I also didn't need to create any filesystem images as SPIFFS partition was only needed for webserver itself.