support maccor binary files

[martinjrobins]

Is your feature request related to a problem? Please describe.

Would be great to be able to import maccor binary files as the test is progressing

Describe the solution you'd like

maccor parser currently supports text versions of maccor output files, need to extend to handle binary files

Additional context

blocker: currently can't find any information about the maccor binary file format or programs/libraries for import

[momoson]

I am currently looking into decoding the binary format, as I did not receive any response from Maccor themselves regarding their format.

I am not sure how much the binary format differs between different hardware/software versions of the Maccor suite. Could you provide me some example data files?

[martinjrobins]

@adamL-D @davidhowey, do you have a variety of binary maccor files that can be used for this issue?

[adamL-D]

@martinjrobins Yes, I will send some to you via email as I can't link them here for some reason.

[martinjrobins]

Upload from @adamL-D 's email:
maccor_bin.zip

[momoson]

Thank you very much for the example datasets. Unfortunately, the file format seems to be bit complexer than I thought.

The file starts with ~25kbytes long header, which consists of the battery program (in table and xml form, both ASCII coded), the name of the used device (ASCII), probably some calibration data and some more, unfortunately unknown, information in a binary format. The header is followed by 264 byte-long records, each representing one data point including its timestamp, voltage, current, current step, cycle number, etc.. While the location of e.g. current inside of the 264 byte-long record seems to be the same for different devices, the factor needed to obtain the physical current in Ampere differs between different data sets.

A few more details:

Data file access

viewData.exe accesses the data file by the following read calls. Italic font represents offset or length values which differ between different example datasets.

read offset in bytes	read length in bytes	Content of fragment
0	6 (as 3 x 2)	File signature? maybe file format version? In examples always 00 00 a1 01 00 00
0	416	Unknown
0	34	Unknown
0	416	Unknown
22684	8	Name of the used device, e.g. Maccor #1
904	64	Unknown
968	44	Program name
4302	255	Unknown
750 $+~i~\cdot$ 777	77	Probably info about $i$th AUX channel, only if present, starts with $i = 0$
416	330	Unknown
4164	138	Maybe calibration date?
22005	666	Mostly unknown, ends with description of some network interface
22956	264	First datapoint
4557 $+~i~ \cdot$ 816	816	$i$-th line of battery testing program, starts with $i = 0$
12717	6481	XML representation of battery testing program
22956 $+~i~\cdot$ 264	264	$i$-th datapoint, starts with $i=1$

Datapoint structure

Each datapoint record consists of at least the following elements:

Byte range inside 264 byte-long record	Meaning	Conversion to physical unit
0-7	DPT Time	interpret as float64, days since 1899-12-30 00:00:00
8	Unknown
9	Mode	interpret as uint8, see table below
10	ES, reason for record, e.g. time or step ended	interpret as uint8, see viewData help
11-14	Cycle#	bitmasked by 0x80ffffff, interpret as int32
15	Step#	interpret as uint8, add 1
16-17	Current	interpret as uint16, then /65535*10 (not for all examples)
18-19	Voltage	interpret as uint16, then /65535*10
20	Unknown
21-28	Test time	bitmasked by 0x00ffffffffffffff, interpret as uint64, then /6000 -> minutes (not for all examples)
29-36	Step time	bitmasked by 0x00ffffffffffffff, interpret as uint64, then /6000 -> minutes (not for all examples)
37-44	Capacity	interpret as int64, then *10/65535/360 -> mAh
45-52	Energy	interpret as int64, then *100/65535/65535/360 -> mWh
53-229	Unknown, aux channels are here
230-233	Record#	interpret as int32
234-262	Unknown, bytes here have some influence on the way data is interpret
263	Is tested on being 0x34, unknown

Mode code

The current mode is given by the following table. It seems like the mode is only coded by the 5 LSB. Changing the 0x40 or 0x80 bit also has some influence, however.

Coded mode	mode
0	P
1	C
2	D
4	R
8	EC
9	ED
19	PC
20	PD
21	DIG
23	SC
24	SD
40	EC
41	ED

Example extraction code (does not always work)

Based on above info, the following function extracts the known elements of each data points and converts them to the appropriate physical units, if possible. However, this does only work for some example dataset files, while some others require additional factors of e.g. 8 for the current values.

import numpy as np
import pandas as pd
import struct

def maccor_load_file(filename):
    with open(filename, 'rb') as f:
        fb = f.read()
    num_records = struct.unpack("I",fb[-264+230:-264+230+4])[0]

    data = {}

    data["dpt"] = np.frombuffer(fb[-num_records*264+0:], count=(num_records-1)*264//8+1, dtype=np.float64)[::264//8]
    data["dpt"] = (np.round(data["dpt"]*1e9*3600*24).astype(np.int64)+np.datetime64("1899-12-30","ns").astype(np.int64)).astype("datetime64[ns]", copy=False)
    data["mode"] = np.frombuffer(fb[-num_records*264+9:], count=(num_records-1)*264//1+1, dtype=np.uint8)[::264//1]
    data["es"] = np.frombuffer(fb[-num_records*264+10:], count=(num_records-1)*264//1+1, dtype=np.uint8)[::264//1]
    data["cycle"] = np.frombuffer(fb[-num_records*264+11:], count=(num_records-1)*264//4+1, dtype=np.uint32)[::264//4]
    data["step"] = np.frombuffer(fb[-num_records*264+15:], count=(num_records-1)*264//1+1, dtype=np.uint8)[::264//1] + 1
    data["current (A)"] = np.frombuffer(fb[-num_records*264+16:], count=(num_records-1)*264//2+1, dtype=np.uint16)[::264//2]/65535*10
    data["voltage (V)"] = np.frombuffer(fb[-num_records*264+18:], count=(num_records-1)*264//2+1, dtype=np.uint16)[::264//2]/65535*10
    data["test_time"] = (np.frombuffer(fb[-num_records*264+21:], count=(num_records-1)*264//8+1, dtype=np.uint64)[::264//8]/6000*60*1e9).astype("timedelta64[ns]")
    data["step_time"] = (np.frombuffer(fb[-num_records*264+29:], count=(num_records-1)*264//8+1, dtype=np.uint64)[::264//8]/6000*60*1e9).astype("timedelta64[ns]")
    data["capacity (Ah)"] = np.frombuffer(fb[-num_records*264+37:], count=(num_records-1)*264//8+1, dtype=np.int64)[::264//8]/65535/360/1e3
    data["energy (Wh)"] = np.frombuffer(fb[-num_records*264+45:], count=(num_records-1)*264//8+1, dtype=np.int64)[::264//8]/65535/65535/360/1e1

    record = np.frombuffer(fb[-num_records*264+230:], count=(num_records-1)*264//4+1, dtype=np.uint32)[::264//4]

    pd_data = pd.DataFrame(data=data, index=pd.Index(data=record, name="Record number"))

    return pd_data

[martinjrobins]

thanks @momoson this is great information! can I ask how you normally go about interrogating the binary file to be able to work this out? Its not something I've had previous experience in

[momoson]

I've sent you an email regarding the binary interrogation process

[martinjrobins]

Thanks @momoson, this was very helpful (sorry for the delay in response I've mostly been on leave). Currently we've not had any information from Maccor regarding their file format, @adamL-D I think you contacted them about this, any response?

[chintanp]

Also, wanted to add to the discussion that an NI plugin exists to read the Maccor data files: https://www.ni.com/en-us/support/downloads/dataplugins/download.maccor-dataplugin.html#426304 . And I know anecdotally that Voltaiq platform can read the Maccor binary files. So, in the past, Maccor has been open about how the binary file can be read. They should, ideally, make this public. Their suggested approach is to use the MIMS server.

[martinjrobins]

also looks like you can directly get data from maccor cyclers via MacNet, e.g. using this library: https://github.com/BattGenie/pymacnet