platformio clion – automation of gzip encoded files as bin2hex encode header files.

This works for me on OSX Catalina 10.15.4

Summary: If you wanted to automated include files, you can use the script shown to help automatically gzip, and convert bin to hex string representation for inclusion into C/C++ headers. Automating the process.

This post explains the process to take html/css/icon (what have you) compresses them to gzip, and converts them to c/c++ include files. This is extremely useful for ESP8266/ESP32 webservers.

FOLDER	Description
<project root>	your project root
+/data/	temporary hold spot (making it compatible with spiffs/littlefs location, in case you want to use them later.
+/workdata/	where you actually edit/changes your working files.
+/src/data/	location of include files.
+/src/html.h	generated file, that updates with the script.

The bash script does the following (<project_root>/makeBinaryIncludeFiles.sh):

#!/usr/bin/env bash

rm -rf data
mkdir data
rsync -a workdata/* data/
gzip -r data/*

rm src/html.h
touch src/html.h

mkdir -p src/data

find data -type f -print0 | while IFS= read -r -d '' filename; do
   xxd -i $filename > src/data/${filename##*/}.h
   sed -i .bak 's/unsigned char/const char/g;
   s/\] \=/] PROGMEM =/g' src/data/${filename##*/}.h
   rm src/data/${filename##*/}.h.bak
   echo "#include \"data/${filename##*/}.h\"" >> src/html.h
done

Platformio doesn’t run bash scripts (or at least I noticed it didn’t like to work), but doesn’t have a problem with python. So I wrote a wrapper to run the bash script:

#makeBinarywrapper.py
import subprocess
rc = subprocess.Popen(['bash', './makeBinaryIncludeFiles.sh'])

Add the following line to your platform.ini

#platformio.ini
extra_scripts = pre:makeBinaryWrapper.py

With that peices together… you have have HTML/CSS/JS files, being submitted with the following command:

// ...
#include <html.h>
// ...


void handleHtmlGZ(const char *encodingString, const char *gzString, const int size) {
    digitalWrite(led, 0);

    server.sendHeader("content-encoding", "gzip");
    server.sendHeader("Expires", "Mon, 1 Jan 2222 10:10:10 GMT");
    server.send_P(200, encodingString, gzString, size);

    digitalWrite(led, 1);
}


void setup(){
.....

server.on("/", []{
        handleHtmlGZ("text/html", data_index_html_gz, data_index_html_gz_len);
    });
    server.on("/css/bootstrap.min.css", []{
        handleHtmlGZ("text/css", data_css_bootstrap_min_css_gz, data_css_bootstrap_min_css_gz_len);
    });

    server.on("/favicon.ico", []{
        handleHtmlGZ("image/x-icon", data_favicon_ico_gz, data_favicon_ico_gz_len);
    });

....
}

Making your c/c++ code much cleaner, and separated.