FFat seems to be slower than SPIFFS

[Vincent-Stragier]

Hi Felix,

I try to run the FFat version of the code, but it looks like it is slower to run... also I have modified your code.

I have heavily modified your code to reduce de complexity (of the firmware and of the Python OTA script). Note that I removed all the code stored in the loop. Nearly all is managed in the onWrite callback and a new task is created to perform the update in the end. On the other side I changed the update sequence, I first send the file size, then MTU/Part size and finally I initiate the update by getting the mode.
It is not perfect but by removing most of the global variable, the code gets simpler:

/*
   MIT License

   Copyright (c) 2021 Felix Biego

   Permission is hereby granted, free of charge, to any person obtaining a copy
   of this software and associated documentation files (the "Software"), to deal
   in the Software without restriction, including without limitation the rights
   to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
   copies of the Software, and to permit persons to whom the Software is
   furnished to do so, subject to the following conditions:

   The above copyright notice and this permission notice shall be included in
   all copies or substantial portions of the Software.

   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
   OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
   SOFTWARE.
*/
#include "FS.h"
#include <Arduino.h>
#include <BLE2902.h>
#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <Update.h>

// #define DEBUG_BLE_OTA_DFU_TX
// #define DEBUG_BLE_OTA_DFU_RX
const uint8_t BUILTIN_LED = 2;

const bool FORMAT_FLASH_IF_MOUNT_FAILED = true;

#define USE_SPIFFS // comment to use FFat

#ifdef USE_SPIFFS
// SPIFFS write is slower
#include "SPIFFS.h"
#define FLASH SPIFFS
const bool FASTMODE = false;
#else
// FFat is faster
#include "FFat.h"
#define FLASH FFat
const bool FASTMODE = true;
#endif

// enum mode { OTA_IDLE = 0, OTA_UPLOAD, OTA_INSTALL };

const char SERVICE_UUID[] = "fe590001-54ae-4a28-9f74-dfccb248601d";
const char CHARACTERISTIC_UUID_RX[] = "fe590002-54ae-4a28-9f74-dfccb248601d";
const char CHARACTERISTIC_UUID_TX[] = "fe590003-54ae-4a28-9f74-dfccb248601d";

BLECharacteristic *pCharacteristicTX;
BLECharacteristic *pCharacteristicRX;

bool device_connected = false;

void reboot_ESP_with_reason(String reason) {
  ESP_LOGI(TAG, "Rebooting ESP32 with reason: %s", reason.c_str());
  delay(5000);
  ESP.restart();
}

uint16_t write_binary(fs::FS &file_system, const char *path, uint8_t *data,
                      uint16_t length) {
  // Append data to the file
  ESP_LOGI(TAG, "Write binary file %s\r\n", path);
  File file = file_system.open(path, FILE_APPEND);

  if (!file) {
    ESP_LOGE(TAG, "Failed to open the file to write");
    return 0;
  }

  file.write(data, length);
  file.close();
  return length;
}

void perform_update(Stream &update_stream, size_t update_size) {
  String result = (String)(char)0x0F;
  // Init update
  if (Update.begin(update_size)) {
    // Perform the update
    size_t written = Update.writeStream(update_stream);
    if (written == update_size) {
      ESP_LOGI(TAG, "Written: %d successfully", written);
    } else {
      ESP_LOGI(TAG, "Written: %d/%d. Retry?", written, update_size);
    }
    result += "Written : " + String(written) + "/" + String(update_size) +
              " [" + String((written / update_size) * 100) + " %] \n";

    // Check update
    if (Update.end()) {
      ESP_LOGI(TAG, "OTA done!");
      result += "OTA Done: ";
      if (Update.isFinished()) {
        ESP_LOGI(TAG, "Update successfully completed. Rebooting...");
        result += "Success!\n";
      } else {
        ESP_LOGE(TAG, "Update not finished? Something went wrong!");
        result += "Failed!\n";
      }
    } else {
      Serial.println("Error Occurred. Error #: " + String(Update.getError()));
      result += "Error #: " + String(Update.getError());
    }
  } else {
    ESP_LOGE(TAG, "Not enough space to begin BLE OTA DFU");
    result += "Not enough space to begin BLE OTA DFU";
  }

  if (device_connected) {
    // Return the result to the client (tells the client if the update was a
    // success or not)
    pCharacteristicTX->setValue(result.c_str());
    pCharacteristicTX->notify();
    delay(10);
    delay(5000);
  }
}

void update_from_FS(fs::FS &file_system) {
  // Update the board from the flash.

  // Open update.bin file.
  File update_binary = file_system.open("/update.bin");

  // If the file can be loaded
  if (update_binary) {
    // Verify that the file is not a directory
    if (update_binary.isDirectory()) {
      ESP_LOGE(TAG, "Error, update.bin is not a file");
      update_binary.close();
      return;
    }

    // Get binary file size
    size_t update_size = update_binary.size();

    // Proceed to the update if the file is not empty
    if (update_size > 0) {
      ESP_LOGI(TAG, "Trying to start update");
      perform_update(update_binary, update_size);
    } else {
      ESP_LOGE(TAG, "Error, update file is empty");
    }

    update_binary.close();

    // When finished remove the binary from spiffs
    // to indicate the end of the process
    ESP_LOGI(TAG, "Removing update file");
    file_system.remove("/update.bin");

    reboot_ESP_with_reason("complete OTA update");
  } else {
    ESP_LOGE(TAG, "Could not load update.bin from spiffs root");
  }
}

void task_install_update(void *parameters) {
  delay(5000);
  update_from_FS(FLASH);
  ESP_LOGI(TAG, "Installation is complete");
}

class MyServerCallbacks : public BLEServerCallbacks {
  // Somehow generic
  void onConnect(BLEServer *pServer) { device_connected = true; }
  void onDisconnect(BLEServer *pServer) { device_connected = false; }
};

class BLEOverTheAirDeviceFirmwareUpdate : public BLECharacteristicCallbacks {
private:
  bool selected_updater = true;
  uint8_t updater[2][16384];
  uint16_t write_len[2] = {0, 0};
  uint16_t parts = 0, MTU = 0;
  uint16_t current_progression = 0;
  uint32_t received_file_size, expected_file_size;
  //    void onStatus(BLECharacteristic* pCharacteristic, Status s, uint32_t
  //    code) {
  //      Serial.print("Status ");
  //      Serial.print(s);
  //      Serial.print(" on characteristic ");
  //      Serial.print(pCharacteristic->getUUID().toString().c_str());
  //      Serial.print(" with code ");
  //      Serial.println(code);
  //    }
public:
  void onNotify(BLECharacteristic *pCharacteristic) {
#ifdef DEBUG_BLE_OTA_DFU_TX
    uint8_t *pData;
    std::string value = pCharacteristic->getValue();
    uint16_t len = value.length();
    pData = pCharacteristic->getData();
    if (pData != NULL) {
      ESP_LOGD(TAG, "Notify callback for characteristic %s  of data length %d",
               pCharacteristic->getUUID().toString().c_str(), len);

      // Print transferred packets
      Serial.print("TX  ");
      for (uint16_t i = 0; i < len; i++) {
        Serial.printf("%02X ", pData[i]);
      }
      Serial.println();
    }
#endif
  }

  void onWrite(BLECharacteristic *pCharacteristic) {
    uint8_t *pData;
    std::string value = pCharacteristic->getValue();
    uint16_t len = value.length();
    pData = pCharacteristic->getData();

    if (pData != NULL) { // Check that data have been received
#ifdef DEBUG_BLE_OTA_DFU_RX
      ESP_LOGD(TAG, "Write callback for characteristic %s  of data length %d",
               pCharacteristic->getUUID().toString().c_str(), len);
      Serial.print("RX  ");
      for (int i = 0; i < len; i++) {
        Serial.printf("%02X ", pData[i]);
      }
      Serial.println();
#endif
      switch (pData[0]) {
      case 0xEF: { // Format the flash and send total and used sizes
        FLASH.format();

        // Send flash size
        uint16_t total_size = FLASH.totalBytes();
        uint16_t used_size = FLASH.usedBytes();
        uint8_t flash_size[] = {0xEF,
                                (uint8_t)(total_size >> 16),
                                (uint8_t)(total_size >> 8),
                                (uint8_t)total_size,
                                (uint8_t)(used_size >> 16),
                                (uint8_t)(used_size >> 8),
                                (uint8_t)used_size};
        pCharacteristicTX->setValue(flash_size, 7);
        pCharacteristicTX->notify();
        delay(10);
      } break;

      case 0xFB: // Write parts to RAM
        // pData[1] is the position of the next part
        for (uint16_t index = 0; index < len - 2; index++) {
          updater[!selected_updater][(pData[1] * MTU) + index] =
              pData[index + 2];
        }
        break;

      case 0xFC: { // Write updater content to the flash
        selected_updater = !selected_updater;
        write_len[selected_updater] = (pData[1] * 256) + pData[2];
        current_progression = (pData[3] * 256) + pData[4];

        received_file_size +=
            write_binary(FLASH, "/update.bin", updater[selected_updater],
                         write_len[selected_updater]);

        if ((current_progression < parts - 1) && !FASTMODE) {
          uint8_t progression[] = {0xF1,
                                   (uint8_t)((current_progression + 1) / 256),
                                   (uint8_t)((current_progression + 1) % 256)};
          pCharacteristicTX->setValue(progression, 3);
          pCharacteristicTX->notify();
          delay(10);
        }

        ESP_LOGI(TAG, "Upload progress: %d/%d", current_progression + 1, parts);
        if (current_progression + 1 == parts) {
          // If all the file has been received, send the progression
          uint8_t progression[] = {0xF2,
                                   (uint8_t)((current_progression + 1) / 256),
                                   (uint8_t)((current_progression + 1) % 256)};
          pCharacteristicTX->setValue(progression, 3);
          pCharacteristicTX->notify();
          delay(10);

          if (received_file_size != expected_file_size) {
            received_file_size +=
                write_binary(FLASH, "/update.bin", updater[selected_updater],
                             write_len[selected_updater]);

            if (received_file_size > expected_file_size) {
              ESP_LOGW(TAG, "Unexpected size:\n Expected: %d\nReceived: %d",
                       expected_file_size, received_file_size);
            }

          } else {
            ESP_LOGI(TAG, "Installing update");
            xTaskCreate(task_install_update, "task_install_update", 8192, NULL,
                        5, NULL);
          }
        }
      } break;

      case 0xFD: // Remove previous file and send transfer mode
      {
        // Remove previous (failed?) update
        if (FLASH.exists("/update.bin")) {
          ESP_LOGI(TAG, "Removing previous update");
          FLASH.remove("/update.bin");
        }

        // Send mode ("fast" or "slow")
        uint8_t mode[] = {0xAA, FASTMODE};
        pCharacteristicTX->setValue(mode, 2);
        pCharacteristicTX->notify();
        delay(10);
      } break;

      case 0xFE: // Keep track of the received file and of the expected file
                 // sizes
        received_file_size = 0;
        expected_file_size = (pData[1] * 16777216) + (pData[2] * 65536) +
                             (pData[3] * 256) + pData[4];

        ESP_LOGI(TAG, "Available space: %d\nFile Size: %d\n",
                 FLASH.totalBytes() - FLASH.usedBytes(), expected_file_size);
        break;

      case 0xFF: // Switch to update mode
        parts = (pData[1] * 256) + pData[2];
        MTU = (pData[3] * 256) + pData[4];
        break;

      default:
        ESP_LOGW(TAG, "Unknown command: %02X", pData[0]);
        break;
      }
    }
    delay(1);
  }
};

void initBLE() {
  BLEDevice::init("ESP32 OTA");
  BLEServer *pServer = BLEDevice::createServer();
  pServer->setCallbacks(new MyServerCallbacks());

  BLEService *pService = pServer->createService(SERVICE_UUID);
  pCharacteristicTX = pService->createCharacteristic(
      CHARACTERISTIC_UUID_TX, BLECharacteristic::PROPERTY_NOTIFY);
  pCharacteristicRX = pService->createCharacteristic(
      CHARACTERISTIC_UUID_RX,
      BLECharacteristic::PROPERTY_WRITE | BLECharacteristic::PROPERTY_WRITE_NR);
  pCharacteristicRX->setCallbacks(new BLEOverTheAirDeviceFirmwareUpdate());
  pCharacteristicTX->setCallbacks(new BLEOverTheAirDeviceFirmwareUpdate());
  pCharacteristicTX->addDescriptor(new BLE2902());
  pCharacteristicTX->setNotifyProperty(true);
  pService->start();

  // BLEAdvertising *pAdvertising = pServer->getAdvertising();
  // The above is still working for backward compatibility
  BLEAdvertising *pAdvertising = BLEDevice::getAdvertising();
  pAdvertising->addServiceUUID(SERVICE_UUID);
  pAdvertising->setScanResponse(true);
  // functions that help with iPhone connections issue
  pAdvertising->setMinPreferred(0x06);
  pAdvertising->setMinPreferred(0x12);
  BLEDevice::startAdvertising();
  Serial.println("Characteristic defined! Now you can read it in your phone!");
}

void setup() {
  Serial.begin(115200);
  Serial.println("Starting BLE OTA sketch");

  pinMode(BUILTIN_LED, OUTPUT);

#ifdef USE_SPIFFS
  if (!SPIFFS.begin(FORMAT_FLASH_IF_MOUNT_FAILED)) {
    Serial.println("SPIFFS Mount Failed");
    return;
  }
#else
  if (!FFat.begin()) {
    Serial.println("FFat Mount Failed");
    if (FORMAT_FLASH_IF_MOUNT_FAILED)
      FFat.format();
    return;
  }
#endif

  initBLE();
}

void loop() {
  // switch (mode) {
  // case OTA_IDLE:
  //   if (device_connected) {
  //     digitalWrite(BUILTIN_LED, HIGH);
  //     delay(100);
  //     // your loop code here (if a client is needed)
  //   } else {
  //     digitalWrite(BUILTIN_LED, LOW);
  //     delay(500);
  //   }
  //   // or here (if no client is needed)
  //   break;

  // case OTA_UPLOAD:
  //   break;

  // case OTA_INSTALL:
  //   break;
  // }
  delay(100); // Try to increase stability
}

"""
  
MIT License

Copyright (c) 2021 Felix Biego

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""

from __future__ import print_function
import os
import asyncio
import math
import sys
import re

from bleak import BleakClient, BleakScanner
# from bleak.exc import BleakError

header = """#####################################################################
    ------------------------BLE OTA update---------------------
    Arduino code @ https://github.com/fbiego/ESP32_BLE_OTA_Arduino
#####################################################################"""

UART_SERVICE_UUID = "fe590001-54ae-4a28-9f74-dfccb248601d"
UART_RX_CHAR_UUID = "fe590002-54ae-4a28-9f74-dfccb248601d"
UART_TX_CHAR_UUID = "fe590003-54ae-4a28-9f74-dfccb248601d"

PART = 16000
MTU = 500

ble_ota_dfu_end = False
global_client = None
file_bytes = None
total = 0


async def start_ota(ble_address: str, filename: str):
    device = await BleakScanner.find_device_by_address(ble_address, timeout=20.0)
    disconnected_event = asyncio.Event()

    def handle_disconnect(_: BleakClient):
        print("Device disconnected !")
        disconnected_event.set()

    async def handle_rx(_: int, data: bytearray):
        # print(f'\nReceived: {data = }\n')
        match data[0]:
            case 0xAA:
                print("Starting transfer, mode:", data[1])
                print_progress_bar(0, total, prefix='Upload progress:',
                                   suffix='Complete', length=50)

                match data[1]:
                    case 0:  # Slow mode
                        # Send first part
                        await send_part(0, file_bytes, global_client)
                    case 1:  # Fast mode
                        for index in range(file_parts):
                            await send_part(index, file_bytes, global_client)
                            print_progress_bar(index + 1, total,
                                               prefix='Upload progress:',
                                               suffix='Complete', length=50)

            case 0xF1:  # Send next part and update progress bar
                next_part_to_send = int.from_bytes(
                    data[2:3], byteorder='little')
                # print("Next part:", next_part_to_send, "\n")
                await send_part(next_part_to_send, file_bytes, global_client)
                print_progress_bar(next_part_to_send + 1, total,
                                   prefix='Upload progress:',
                                   suffix='Complete', length=50)

            case 0xF2:  # Install firmware
                # ins = 'Installing firmware'
                # print("Installing firmware")
                pass

            case 0x0F:
                print("OTA result: ", str(data[1:], 'utf-8'))
                global ble_ota_dfu_end
                ble_ota_dfu_end = True

    def print_progress_bar(iteration: int, total: int, prefix: str = '', suffix: str = '', decimals: int = 1, length: int = 100, filler: str = '█', print_end: str = "\r"):
        """
        Call in a loop to create terminal progress bar
        @params:
            iteration   - Required  : current iteration (Int)
            total       - Required  : total iterations (Int)
            prefix      - Optional  : prefix string (Str)
            suffix      - Optional  : suffix string (Str)
            decimals    - Optional  : positive number of decimals in percent complete (Int)
            length      - Optional  : character length of bar (Int)
            filler      - Optional  : bar fill character (Str)
            print_end   - Optional  : end character (e.g. "\r", "\r\n") (Str)
        """
        percent = ("{0:." + str(decimals) + "f}").format(100 *
                                                         (iteration / float(total)))
        filled_length = (length * iteration) // total
        bar = filler * filled_length + '-' * (length - filled_length)
        print(f'\r{prefix} |{bar}| {percent} % {suffix}', end=print_end)
        # Print new line upon complete
        if iteration == total:
            print()

    async def send_part(position: int, data: bytearray, client: BleakClient):
        start = position * PART
        end = (position + 1) * PART

        if len(data) < end:
            end = len(data)

        data_length = end - start
        parts = data_length // MTU
        for part_index in range(parts):
            to_be_sent = bytearray([0xFB, part_index])
            for mtu_index in range(MTU):
                to_be_sent.append(
                    data[(position*PART)+(MTU * part_index) + mtu_index])
            await send_data(client, to_be_sent)

        if data_length % MTU:
            remaining = data_length % MTU
            to_be_sent = bytearray([0xFB, parts])
            for index in range(remaining):
                to_be_sent.append(
                    data[(position*PART)+(MTU * parts) + index])
            await send_data(client, to_be_sent)

        await send_data(client, bytearray([0xFC, data_length//256, data_length %
                                           256, position//256, position % 256]), True)

    async def send_data(client: BleakClient, data: bytearray, response: bool = False):
        await client.write_gatt_char(UART_RX_CHAR_UUID, data, response)

    if not device:
        print("-----------Failed--------------")
        print(f"Device with address {ble_address} could not be found.")
        return
        #raise BleakError(f"A device with address {ble_address} could not be found.")

    async with BleakClient(device, disconnected_callback=handle_disconnect) as client:
        # Set the UUID of the service you want to connect to and the callback
        await client.start_notify(UART_TX_CHAR_UUID, handle_rx)
        await asyncio.sleep(1.0)

        # Set global client to be the current client
        global global_client
        global_client = client

        # Send file size
        print("Reading from: ", filename)
        global file_bytes
        file_bytes = open(filename, "rb").read()
        file_parts = math.ceil(len(file_bytes) / PART)
        file_length = len(file_bytes)

        print(f'File size: {len(file_bytes)}')
        # Send file length
        await send_data(client, bytearray([0xFE,
                                           file_length >> 24 & 0xFF,
                                           file_length >> 16 & 0xFF,
                                           file_length >> 8 & 0xFF,
                                           file_length & 0xFF]))

        # Send number of part and MTU value
        global total
        total = file_parts

        await send_data(client, bytearray([0xFF,
                                           file_parts//256,
                                           file_parts % 256,
                                           MTU // 256,
                                           MTU % 256]))

        # Remove previous update and receive transfer mode (start the update)
        await send_data(client, bytearray([0xFD]))

        # Wait til upload is complete
        while not ble_ota_dfu_end:
            await asyncio.sleep(1.0)

        print("Waiting for disconnect... ", end="")

        await disconnected_event.wait()
        print("-----------Complete--------------")


def is_valid_address(value: str = None) -> bool:
    # Regex to check valid MAC address
    regex_0 = (r"^([0-9A-Fa-f]{2}[:-])"
               r"{5}([0-9A-Fa-f]{2})|"
               r"([0-9a-fA-F]{4}\\."
               r"[0-9a-fA-F]{4}\\."
               r"[0-9a-fA-F]{4}){17}$")
    regex_1 = (r"^[{]?[0-9a-fA-F]{8}"
               r"-([0-9a-fA-F]{4}-)"
               r"{3}[0-9a-fA-F]{12}[}]?$")

    # Compile the ReGex
    regex_0 = re.compile(regex_0)
    regex_1 = re.compile(regex_1)

    # If the string is empty return false
    if value is None:
        return False

    # Return if the string matched the ReGex
    if re.search(regex_0, value) and len(value) == 17:
        return True

    return re.search(regex_1, value) and len(value) == 36


if __name__ == "__main__":
    print(header)
    # Check if the user has entered enough arguments
    # sys.argv.append("C8:C9:A3:D2:60:8E")
    # sys.argv.append("firmware.bin")

    if len(sys.argv) < 3:
        print("Specify the device address and firmware file")
        import sys
        import os
        filename = os.path.join(os.path.dirname(
            __file__), 'PIO', 'ESP32_BLE_OTA_DFU', '.pio', 'build', 'esp32dev', 'firmware.bin')
        filename = filename if os.path.exists(filename) else "firmware.bin"
        print(f"$ {sys.executable} {__file__} \"C8:C9:A3:D2:60:8E\" \"{filename}\"")
        exit(1)

    print("Trying to start OTA update")
    ble_address = sys.argv[1]
    filename = sys.argv[2]

    # Check if the address is valid
    if not is_valid_address(ble_address):
        print(f"Invalid Address: {ble_address}")
        exit(2)

    # Check if the file exists
    if not os.path.exists(filename):
        print(f"File not found: {filename}")
        exit(3)

    asyncio.run(start_ota(ble_address, filename))

Best,
Vincent

[fbiego]

Did you try the code before modification?

[SuperTankMan]

Hi Vincent,
I found this code clean up and modification to be quite stable, no disrespect to fbeigo who has done an excellent job of this project.
Vincent I do not know much about FreeRTOS and hence ESP32 tasks etc. But I was wondering if the previous code was increasing speed by swapping between two buffer. i.e. while one buffer is being downloaded the other buffer is being used to write to file. However in the case of " case 0xFC: { // Write updater content to the flash" while the file is being written it does not release callback function until it has finished writing the buffer and hence "case 0xFB: // Write parts to RAM" can not be processed at the same time although I am not sure if write_binary(..) is processed in the background. Anyway if it is blocking then is it possible to write anther task to write_binary so that " case 0xFC: { // Write updater content to the flash" can release and process "case 0xFB: // Write parts to RAM" and process the writing to the file in the background?
I do not have enough experience of FreeRTOS to write a that task and I was wondering if you can do that which would improve the download speed. I'd be grateful if you can write this modification. I look forward to your response.
Raj

[Vincent-Stragier]

Hi,

@fbiego, I will try to use the FFAT mode, but not right now. I'll keep you inform.

@SuperTankMan, I did modify the code further since last time. @fbiego did a great job of implementing the OTA, the only issue is that it is hard to integrate in other projects (due to the numerous global variables, etc.). In the joined zip, there is a PlatformIO project with the OTA functionality enclosed in a library (feel free to test/modify/use it). Note that the Python script do not complete (probably a bug on the ESP32 side)…

ble_ota_dfu_example.zip

When not using the fast mode, only one slot at a time is used during the upload (I don't know what happen when using the fast mode though)…

I tried to implement the write_binary function in another task, but it was slower or less stable or both (I do not remember exactly, but I already tried it). The reason of the added delay/instability is that you need a queue to share variable between task, and it takes some time.

Another optimization (that I hope is correctly implemented) is to keep the file open in write_binary instead of opening/closing it for each chunk (opening and closing files is time-consuming).

Hope it helps,
Vincent

[fbiego]

Hello @Vincent-Stragier
I think this is a great solution and also simplifies the existing code.

Maybe we can add the code to the repo for easy access.
Can you create a PR with your code in a folder named esp32_ble_ota_compact?

I will have to update the android app as well to work with your code

[SuperTankMan]

Hi Vincent,
Do you have any updates on the NON Nimble version? The version at the top of this issue?

[Vincent-Stragier]

Hi @SuperTankMan,

I did not manage to work on that issue, I will have more time to do so after the 8th of June.

Best,
Vincent

[SuperTankMan]

Hi @Vincent-Stragier @fbiego
Any updates on the above code at the top? have you forked this in repository? I can see you created one on this repository as #18 but what about the bare code you sent above? any chances of that one being forked too as its BLE library instead of the NimBLE library?

[NoelCav]

Hey @SuperTankMan. I believe the program is swapping between two buffers because the ESP32 has a max Global variable allocation limit of around 160KB. It does seem weird to have 2 buffers since they are never used at the same time, one buffer could just be reused instead. Also, since the data isn't being saved to FLASH simultaneously, it seems like the transmission would get faster the larger you set the "PART" Constant. Is there any negative to replacing updater1 and updater2 with one updater that is 65536 Bytes and set PART to 64000? That should cut File IO calls by 4.

Also @Vincent-Stragier, I really can't figure out the point/benefit of running the update as an RTOS task rather than a simple function call. I really don't understand RTOS so I would love to know why I'm missing and why you added them to the code.

[Vincent-Stragier]

Hi @NoelCav and @SuperTankMan ,

In #18, I kept most of the algorithm, which include the updater “1” and “2”. I planned to use those in parallel, but I do not remember why I did not pursue in that direction. For PART, it is to be tried. There might be some stability issues.

I see at least two benefits of using RTOS. First, it is easier to integrate to your projects (note that can also be achieved by rewriting the code in a library, like in #19). Second, you are not dependent on the main running task (loop), which means that if you upload a code that freeze the loop you can still flash the board with a new code.

@SuperTankMan, it is not interesting (and complex) to implement #18 for BLE. I tried, but if I remember well, it was instable and maybe too big. The other issue is that BLE is not supported as well as NimBLE (see https://github.com/nkolban/esp32-snippets vs https://github.com/h2zero/NimBLE-Arduino).