embody-file

Embody File

This is a Python based implementation for parsing binary files from the Aidee EmBody device.

Features

• Converts binary embody files to HDF, CSV, etc
• Integrates with the EmBody Protocol Codec project
• CLI (command line interface)
• Can be used as package in other projects
• Type safe code using mypy for type checking

Requirements

• Python 3.11+

Installation

You can install Embody File via pip:

$ pip install embody-file

Usage

To use the command line, first install this library either globally or using venv:

$ pip install embody-file

When this library has been installed, a new command is available, embody-file which can be used according to the examples below:

Get help

To get an updated overview of all command line options:

embody-file --help

Print version number

embody-file --version

Convert binary embody file to HDF

To convert to a HDF 5 (hierarcical data format) format, run the following:

embody-file testfiles/v5_0_0_test_file.log --output-format HDF

The file will be named the same as the input file, with the .hdf extension at the end of the file name.

Performance Note: HDF exports now use optimized batched writes for improved performance when exporting multiple datasets.

Reading HDF Metadata

The HDF files store sampling frequency as metadata attributes rather than on the DataFrame index. This approach handles real-world sensor data that may have timing variations or gaps.

To access the sampling frequency from HDF files:

import pandas as pd

# Read the data
df = pd.read_hdf('your_file.hdf', key='multidata')

# Access sampling frequency metadata
with pd.HDFStore('your_file.hdf', mode='r') as store:
    attrs = store.get_storer('multidata').attrs
    if hasattr(attrs, 'sample_frequency_hz'):
        print(f"Sampling frequency: {attrs.sample_frequency_hz} Hz")
        print(f"Sample period: {attrs.sample_period_ms} ms")

Available metadata attributes:

• sample_frequency_hz: The sampling frequency in Hertz
• sample_period_ms: The sampling period in milliseconds

Convert binary embody file to CSV

To convert to CSV format, run the following:

embody-file testfiles/v5_0_0_test_file.log --output-format CSV

The file will be named the same as the input file, with the .csv extension at the end of the file name.

Convert to multiple formats at once

It is possible to convert to multiple formats at once. For example, to convert to both HDF and CSV, run the following:

embody-file testfiles/v5_0_0_test_file.log --output-format HDF CSV

Print statistics for binary embody file

To print stats without conversion:

embody-file testfiles/v5_0_0_test_file.log --print-stats

Fail on parse errors

The parser is lenient by default, accepting errors in the input file. If you want to the parsing to fail on any errors, use the --strict flag:

embody-file testfiles/v5_0_0_test_file.log --strict

Troubleshooting

I get an error in the middle of the file - how do I start finding the root cause?

To get the best overview, start by running the parser in strict mode and with debug logging, so it stops at the first error:

embody-file troublesomefile.log --strict --log-level DEBUG

This provides positional information per message so it's easier to continue searching for errors.

If this doesn't give us enough information, look at the protocol documentation and start looking and the problematic areas in the input file.

There are several command line tools you can use. On MAC and Linux, one good example is to use the hexdump tool:

hexdump -C -n 70 -s 0 troublesomefile.log

Here, -n 70 is the amount of bytes to print in hex format, and -s 0 tells hexdump to start at position 0 in the file. Adjust these parameters according to your needs.

Make a note from the parser's error output of what position the first error started from, and based on that:

• Look at the preceding bytes to see whether there were any errors in the previous protocol message
• Look at the bytes from the reported (error) position to see if there are just a few bytes before a new, plausible protocol message starts

Issues

If you encounter any problems, please file an issue along with a detailed description.