ESP32/8266 Serial File Transfer and Sync
This is a single file library and companion tool written in Python to allow easy and efficient syncing of the contents of a native file system with the SPIFFS on an ESP8266 or ESP32.
The library works by inserting itself into the serial stream, if it detects sync data, it will automatically start syncing the file system contents.
There are tools already to do this, but they create a whole binary SPIFFS image and then flash that which is SLOW. Especially for modules with large flashes.
Eg, To update a file that is 1K, at 115200 bps, on a 14MB SPIFFS, uploading a image will take more than 21 minutes. Even at a theoretical maximum data rate of 3Mbps (Max for USB to UART Interfaces) the filesystem sync will take ~49 seconds. Unbearably slow when one is developing and needing to iterate web page designs.
Whereas if the transfer was efficient, the 1K file would take ~0.08 seconds to transfer at 115200.
I explored using off the shelf protocols, the most applicable seemed to be ZMODEM, but all existing code is horrible, probably because the specification is horrible.
So I rolled my own, simple transparent file transfer protocol for serial lines.
The packet is BASED on DDCMP principles, but is not a DDCMP packet. The advantage of a DDCMP type packet is its data transparent, there is no need to escape bytes, like Async HDLC, etc, does.
A Packet has an 8 byte Header, and an optional Data section (Up to 16MB in size) This means there is fixed overhead, which is large for small files, but insignificant for large files.
| Code | Value | Description |
|---|---|---|
| STX | 0x02 | Start Message Sentinel |
| CMN | 0x20-0x3F 0x40-0x5F |
Request Cyclic Message Number Reply Cyclic Message Number |
| FUN | 0x06[ACK],0x15[NAK],0x60-0x7E | Function Code |
| SIZ OPT |
0x000000-0xFFFFFF Option Data |
Data Payload Size Function Option Data |
| CHK | 0x0000-0xFFFF | FLETCHER-16 checksum of all data from STX to the end of SIZ/OPT |
| DATA | 0x00-0xFF x N | Variable Length data body. |
| CHK2 | 0x00000000-0xFFFFFFFF | ADLER-32 for Data Payloads |
ALL Multi-byte fields are sent MOST SIGNIFICANT BYTE FIRST (Network Byte Order).
This is a traditional start of message character for asynchronous protocols, and is defined by ASCII as 0x02.
This message number starts at 0x20 and for each unique message sent increments to 0x3F where it wraps back to 0x20. The reply will echo this CMN but will add 0x20 to it before replying. The CMN is used to detect duplicate transmissions, in a re-transmission scenario.
The Function of the message. Defines how the Size/Option bytes will be interpreted.
| CODE | FUNCTION | Description |
|---|---|---|
| 0x06 | ACK | Optional, sent immediately after a valid header if processing may be slow. [OPT] |
| 0x15 | NAK | Sent IF there is some problem with the data payload only. Header errors are ignored. [OPT] |
| 0x60 | Set Time | Set RTC Time [SIZE] |
| 0x61 | Format | Format the SPIFFS [SIZE] |
| 0x62 | List | Get complete file listing of the SPIFFS [SIZE] |
| 0x63 | Remove | Remove the named file [SIZE] |
| 0x64 | Rename | Rename a file [SIZE] |
| 0x65 | File | Send a File [SIZE] |
| 0x70 | Time Set | Response to the set time command [SIZE] |
| 0x71 | Formated | Response to the reply command [SIZE] |
| 0x72 | Listing | Response to the list command [SIZE] |
| 0x73 | Removed | Response to the Remove command [SIZE] |
| 0x74 | Renamed | Response to the Rename command [SIZE] |
| 0x75 | Received | Response to the File command [SIZE] |
For most messages, this is the size of the data payload following the header. For ACK and NAK there are no data payloads and these three bytes are options associated with the ACK or NAK reply.
A Value of 0x000000 to 0xFFFFFF sent Most Significant Byte first. The size is this value allowing from 0 bytes thru to 16,777,215 bytes (16MB-1) of data to be sent following the header. This limits a single file to a size of 16MB-1, but given the largest ESP32/8266 Flash is only 16MB this is not a practical limitation. If the payload size is 0, no data payload follows the header.
The control messages ACK and NAK have no data body. Instead this is an option value for the function.
In the case of ACK, which is only sent in the case of a long running function, the first 2 bytes of the OPT value specifies the number of milliseconds the sender should wait to receive a reply, before concluding the message failed. The last byte is unused and MUST be 0x5A. This response is optional, short run functions can immediately reply with their response. The ACK can be sent before all Data in the Data payload has been transmitted, provided the header is valid. The timeout is +1 this value giving timeouts of 1ms to ~1 Minute in length. The timeout is sent Most Significant Byte first.
In the case of NAK, the first byte is an error code, the remaining 16 bits may specify additional data related to the error, and IF they are not used are set to 0xA55A.
This is the Fletcher-16 Checksum of all bytes from STX through to the end of SIZ/OPT. This checksum was chosen because it is almost as good as a CRC-16 but uses far less program and data space, and is very fast.
If the message data size is not zero (True for every message other than ACK/NAK) then immediately following the header is the RAW Binary data being transmitted, there is no character escaping. The number of bytes are declared by the header, and the size excludes the checksum that follows the data.
The 32 bit data payload checksum is an ADLER-32 checksum of all bytes in the Data payload ONLY.
The checksum value is appended to the data stream Most significant byte first.
If the checksum fails to evaluate correctly, the Slave will send a NAK indicating a checksum error, and no action is taken on the data. In the case of a master receiving a data payload checksum error, the master can resend the original request, to trigger a re-transmission of the reply, OR can abort the transfer.
The Master sends a message, and the receiver either processes it and replies, OR ignores it if there is an error in the formatting of the header. The only Negative replies are sent in response to a data payload error.
This allows the protocol to transparently co-exist with other serial communications on the same line, IF and only IF a properly formatted header is received will the receiver process the messages.
If a Slave receives a command from the Master that may take some long time to execute, it can send the Master an ACK reply, which signals that the message is being processed, but that it could take "up to" the number of milliseconds specified to complete, so the Master can wait the appropriate time before retrying. It is optional.
As a special case, A Master may send an ACK message to a slave. The slave does nothing, except reply with an ACK message.
A Master never sends NAK, and a Slave ignores it if received.
IF and ONLY IF there is a data payload error, the Slave will send an NAK signaling the error. Error codes are:
| Code | Error | Description |
|---|---|---|
| 0x21 | TIMEOUT | Data payload not received in time |
| 0x22 | CHKSUM | Data payload checksum in error |
| 0x23 | FORMAT | Data payload formatting error detected |
| 0x24 | FSERR | File System Error occurred |
| 0x25 | FNOTF | File Not Found |
| 0x26 | FNAMERR | File Name too long or contains invalid characters |
| 0x27 | FSIZERR | File Too Big |
| 0x28 | FEXISTS | File already exists |
This message allows the Master to set the time of the Slave, so that file operations can be properly recorded with the time.
The Data is the Time:
| Field | Size | Description |
|---|---|---|
| DAY | 1 | Day (1-31) |
| MONTH | 1 | Month (1-12) |
| YEAR | 1 | Year (0-255) Where the Year is encoded as years since 2019. |
| HOUR | 1 | Hour (0-23) |
| MINUTE | 1 | Minute (0-59) |
| SECOND | 1 | Second (0-59) |
| CHK2 | 4 | Checksum of DAY thru SECOND |
IF the data is valid, then the time is set as requested, and an 0x70 Time Set message is returned as a reply.
Otherwise the time is not changed, a NAK is returned, and the Error code is 0x23 FORMAT.
This message is sent in reply to 0x60 and contains no data. Data size is 0.
This message instructs the Slave to format its SPIFFS. Formatting can take a long time, the Slave should reply with an ACK, indicating the maximum duration the Master should wait for the Format to complete, followed by a 0x71, Formatted reply when the format operation is complete.
If the Master does not see the ACK or 0x71 reply, it may re-transmit the format message. If the CMN remains unchanged, the Slave will simply reply with the status of the operation. Otherwise if CMN is changed, a new format is started.
It is valid for the Slave to ignore further messages received during the format. The Master should not send further messages until a 0x71 is received, or a timeout occurs, triggering a retransmit or a failed communications session.
If the format fails due to an error in the SPIFFS code, a NAK with a 0x24 FSERR code will be sent, aborting the session.
Indicates the 0x61 command completed successfully. The reply contains the available SPIFFS size.
The Data is the SPIFFS Size:
| Field | Size | Description |
|---|---|---|
| SIZE | 4 | Size of the SPIFFS |
| USED | 4 | Number of bytes used by the SPIFFS |
| NSIZ | 1 | Maximum Name Length |
| CHK2 | 4 | Checksum of SIZE thru NSIZ |
This instructs the Slave to return a list of ALL files on the SPIFFS. The slave replies with a 0x72, Listing reply. There is a single byte of data, so the data size is 1.
The Data is:
| Field | Size | Description |
|---|---|---|
| OPT | 1 | Bit 0 = 0 -> No Time Requested. Bit 0 = 1 -> File Time Requested Bit 1 = 0 -> Adler-32 Checksum NOT Requested. Bit 1 = 1 -> Adler-32 Checksum of each file Requested. Bit 2-7 Ignored |
| CHK2 | 4 | Checksum of OPT |
This reply contains a full list of all files in the SPIFFS and relevant information about them. The first fields specify global data about the SPIFFS, the following fields is a list of file entries.
The Data is:
| Field | Size | Description |
|---|---|---|
| SIZE | 4 | Size of the SPIFFS |
| FREE | 4 | Free space in the SPIFFS |
| NSIZ | 1 | The MAX SIZE of a file name |
| OPT | 1 | Bit 0 = 0 -> File Time Not Present. Bit 0 = 1 -> File Time Present Bit 1 = 0 -> Adler-32 Checksum NOT Present. Bit 1 = 1 -> Adler-32 Checksum of each file Present. Bit 2-7 Ignored |
| FILE 1 | X | File 1 fields |
| ... | ... | ... |
| FILE N | X | File N fields. |
| CHK2 | 4 | Checksum of all Data |
The File fields are:
| Field | Size | Description |
|---|---|---|
| NAME | X | The Files Name, this is fixed length and always = NSIZ, if the filename is shorter, it is padded with NULL 0x00 bytes |
| FSIZ | 4 | The Size of the File |
| DATE | 6 | Modified Time and Date of the File (ONLY PRESENT IF OPT SPECIFIES IT, Otherwise Field not present) |
| FCHK | 4 | File Adler-32 Checksum (ONLY PRESENT IF OPT SPECIFIES IT, Otherwise Field not present) |
Each File entry is fixed size, and so is the number of files in the file system, which allows the Slave to quickly calculate the size of the reply, for inclusion in the header, without pre-buffering the entire reply. The File Entry fields can be returned progressively, and as the Checksum may take some time to complete, there may be a pause between individual file entries. The Master is to buffer the reply, and it is ONLY valid if the checksum matches a the end.
The DATE field has the exact same format as the fields in the Set Time message.
Causes the file named in the data to be deleted. The Slave will reply with a ACK, if the delete operation will take a significant amount of time, and will indicate how long the Master should wait before retrying or giving up. Once the delete operation is complete, the Slave will reply with 0x73, Removed. The Data is simply the file name to delete. If the file does not exist, the Slave will reply with a NAK, and a FNOTF Error code.
The Data is:
| Field | Size | Description |
|---|---|---|
| NAME | X | The Name of the File to delete, not padded |
| CHK2 | 4 | Checksum of NAME |
Reply to the Remove File command. Data indicates the maximum and remaining space available if the SPIFFS.
The Data is:
| Field | Size | Description |
|---|---|---|
| SIZE | 4 | Size of the SPIFFS |
| FREE | 4 | Free space in the SPIFFS |
| CHK2 | 4 | Checksum of SIZE & FREE |
Causes the file named in the data to be renamed to the second name in the data. The Slave will reply with a ACK, if the rename operation will take a significant amount of time, and will indicate how long the Master should wait before retrying or giving up. Once the rename operation is complete, the Slave will reply with 0x74, Renamed. The Data is simply the file name to rename, and the new name. If the file does not exist, the Slave will reply with a NAK, and a FNOTF Error code. If the file to rename to already exists, it will not be over written and a NAK will reply with FEXISTS Error Code.
The Data is:
| Field | Size | Description |
|---|---|---|
| NLEN | 1 | The length of the File Name |
| NAME | X | The Name of the File to rename, not padded, no zero termination |
| RLEN | 1 | The length of the Renamed File name |
| RNAME | X | The new Name of the File to rename, not padded, no zero termination |
| CHK2 | 4 | Checksum of NLEN thru to end of RNAME |
This message is sent in reply to 0x64 and contains no data. Data size is 0.
This message sends a file to the Slave, the Slave DOES NOT verify if this is the same as a previous file of the same name, and over-writes it if it exists, It is the responsibility of the Master to ensure only necessary files are sent to be stored. The data is the files name, its file time, and its data.
The Data is:
| Field | Size | Description |
|---|---|---|
| NSIZ | 1 | The Size of the File Name 1-255 |
| NAME | X | The Name of the File, NSIZ Bytes long, not Padded |
| DATE | 6 | The Date and Time of the file |
| FDAT | X | File Data, may be zero bytes long |
| CHK2 | 4 | Adler-32 Checksum of all Data |
If the Name Size is 0 or too large for the SPIFFS to store or the NAME field has any other problems, NAK is replied, with a FNAMERR code. If the Date is not properly formatted, NAK is replied with a FORMAT error code. The Date Field has the same format as the Set Time message. If there is not enough space to store the file NAK will be replied with FSIZERR. If any filesystem errors occur, NAK will be replied with FSERR code.
To save buffering in RAM, the Slave will immediately start writing the file to a temporary file name. When the Checksum is received, IF and ONLY IF it is valid, the temporary file is renamed to the destination file name. The Temporary file name is "///TEMP" and the Slave will refuse to receive a file of this name, it will also delete any file of this name on start up.
Reply to the File command. Data indicates the maximum and remaining space available if the SPIFFS.
The Data is:
| Field | Size | Description |
|---|---|---|
| SIZE | 4 | Size of the SPIFFS |
| FREE | 4 | Free space in the SPIFFS |
| CHK2 | 4 | Checksum of SIZE & FREE |