Flash footprint optimization

[1technophile]

I'm thinking that we could reduce the decoder footprint by optimizing the size of the keys used in the devices decoder.
Without changing the API it seems that there is some significant opportunities, example:
Replacing "servicedata" by "svd" and "manufacturerdata" by "mfd"

Example:

const char* _SBBT_json = "{\"brand\":\"SwitchBot\",\"model\":\"Blind Tilt\",\"model_id\":\"W270160X\",\"tag\":\"0d22\",\"condition\":[\"uuid\",\"index\",0,\"0d00\",\"|\",\"uuid\",\"index\",0,\"fd3d\",\"&\",\"svd\",\"=\",6,\"index\",0,\"78\",\"&\",\"mfd\",\">=\",24,\"index\",0,\"6909\"],\"properties\":{\"open\":{\"decoder\":[\"value_from_hex_data\",\"mfd\",20,2,false,false],\"post_proc\":[\"&\",127,\"-\",50,\"*\",2,\"±\",100,\"abs\"]},\"direction\":{\"decoder\":[\"value_from_hex_data\",\"mfd\",20,2,false,false],\"post_proc\":[\"&\",127,\"-\",50,\"*\",2,\"SBBT-dir\"]},\"motion\":{\"decoder\":[\"bit_static_value\",\"mfd\",20,3,false,true]},\"calibrated\":{\"decoder\":[\"bit_static_value\",\"mfd\",19,0,false,true]},\"lightlevel\":{\"decoder\":[\"value_from_hex_data\",\"mfd\",18,1,false,false]},\"batt\":{\"decoder\":[\"value_from_hex_data\",\"svd\",4,2,false,false],\"post_proc\":[\"&\",127]},\"mac\":{\"decoder\":[\"mac_from_hex_data\",\"mfd\",4]}}}";
/*R""""(
{
   "brand":"SwitchBot",
   "model":"Blind Tilt",
   "model_id":"W270160X",
   "tag":"0d22",
   "condition":["uuid", "index", 0, "0d00", "|", "uuid", "index", 0, "fd3d", "&", "svd", "=", 6, "index", 0, "78", "&", "mfd", ">=", 24, "index", 0, "6909"],
   "properties":{
      "open":{
         "decoder":["value_from_hex_data", "mfd", 20, 2, false, false],
         "post_proc":["&", 127, "-", 50, "*", 2, "±", 100, "abs"]
      },
      "direction":{
         "decoder":["value_from_hex_data", "mfd", 20, 2, false, false],
         "post_proc":["&", 127, "-", 50, "*", 2, "SBBT-dir"]
      },
      "motion":{
         "decoder":["bit_static_value", "mfd", 20, 3, false, true]
      },
      "calibrated":{
         "decoder":["bit_static_value", "mfd", 19, 0, false, true]
      },
      "lightlevel":{
         "decoder":["value_from_hex_data", "mfd", 18, 1, false, false]
      },
      "batt":{
         "decoder":["value_from_hex_data", "svd", 4, 2, false, false],
         "post_proc":["&", 127]
      },
      "mac":{
         "decoder":["mac_from_hex_data", "mfd", 4]
      }
   }
})"""";*/

Save around 11500 bytes, I haven't tested the change yet but wanted to bounce the idea.

[1technophile]

If we push the idea further by also replacing:

• "post_proc" by "pp"
• "condition" by "cnd"
• "value_from_hex_data" by "vfhd"
• "string_from_hex_data" by "sfhd"
• "bit_static_value" by "bsv"
  We are saving around 23Kb

[DigiH]

I'll look at this when I'm back home next week. There should also still be branch from @h2zero from a while back.

[h2zero]

I'd suggest making a python script to do the text replacement at build time to avoid changing the documentation/api. Better yet is to precompile into a array of JSON objects, this would be more work though.

[1technophile]

I pushed this branch which improve the size by >20Kb while keeping the same API for information
https://github.com/theengs/decoder/tree/flash-size

[1technophile]

Another possibility is to replace the json char array with a struct based approach, savings would be huge

[1technophile]

Here is an example generated with Claude:

#ifndef XOSSX2_STRUCT_H
#define XOSSX2_STRUCT_H

#include "../decoder_structs.h"

#ifdef __cplusplus
extern "C" {
#endif

// XOSSX2 Heart Rate Sensor device configuration
// Original JSON: {"brand":"XOSS","model":"X2 Heart Rate Sensor","model_id":"XOSSX2","tag":"0b01",
//                "cond":["mfd","=",12,"ind",0,"04ff"],"properties":{"bpm":{"decoder":["vfhd","mfd",10,2,false,false]},
//                "batt":{"decoder":["vfhd","mfd",6,2,false,false],"pprc":["&",127]}}}

// Post-processing for battery (& 127)
static const post_process_t xossx2_batt_pprc[] = {
    {PROC_AND, 127}
};

// Match conditions: mfd = 12, ind 0 "04ff"
static const condition_t xossx2_conditions[] = {
    {COND_EQUAL, DATA_MFD, 0, NULL, 12, 0},  // mfd length = 12
    {COND_IND, DATA_MFD, 0, "04ff", 0, 4}    // ind 0 "04ff" (4 hex chars = 2 bytes)
};

// Device properties
static const property_def_t xossx2_properties[] = {
    {
        .key = "bpm",
        .decoder = {DECODER_VFHD, DATA_MFD, 10, 2, false, false},
        .pprc = NULL,
        .pprc_count = 0,
        .conditions = NULL,
        .cond_count = 0,
        .lookup = NULL,
        .lookup_count = 0,
        .unit = "bpm",
        .name = "heart rate"
    },
    {
        .key = "batt",
        .decoder = {DECODER_VFHD, DATA_MFD, 6, 2, false, false},
        .pprc = (post_process_t*)xossx2_batt_pprc,
        .pprc_count = 1,
        .conditions = NULL,
        .cond_count = 0,
        .lookup = NULL,
        .lookup_count = 0,
        .unit = "%",
        .name = "battery"
    }
};

// Device configuration
static const device_config_t XOSSX2_config = {
    .brand = "XOSS",
    .model = "X2 Heart Rate Sensor",
    .model_id = "XOSSX2",
    .tag = "0b01",
    .match = {
        .conditions = (condition_t*)xossx2_conditions,
        .cond_count = 2
    },
    .properties = (property_def_t*)xossx2_properties,
    .prop_count = 2
};

#ifdef __cplusplus
}
#endif

#endif // XOSSX2_STRUCT_H

[1technophile]

@DigiH , thoughts about this approach ?

[DigiH]

While this might give the most savings, I personally find this solution more awkwardly human readable whenever decoder debugging and/or amendments are necessary.

However, as I'll be taking a sebactical from active development involvement soon, any solution should suit whomever is taking over most of the Theengs Decoder development.

[1technophile]

Thanks for the comment, let me check if we can have something more human readable.