Welcome to the Joulescope® File Format project. The goal of this project is to provide performant data storage for huge, simultaneous, one-dimensional signals. This repository contains:
Items marked with 🔜 are under development and coming soon. Items marked with ⏳ are planned for future release.
As of June 2023, the JLS v2 file structure is well-defined and stable. However, the compression storage formats are not yet defined and corrupted file recovery is not yet implemented.
The world is already full of file formats, and we would rather not create another one. However, we could not identify a solution that met these requirements. HDF5 meets the large storage requirements, but not the reliability and rapid load requirements. The Saleae binary export file format v2 is also not suitable since it buffers stores single, contiguous blocks. Sigrok v2 is similar. The Sigrok v3 format (under development as of June 2023) is better in that it stores sequences of "packets" containing data blocks, but it still will does not allow for fast seek or summaries.
Timeseries databases, such as InfluxDB, are powerful tools. However, they are not well-designed for fast sample-rate data.
Media containers are another option, especially the ISO base media file format used by MPEG4 and many others:
However, the standard does not include the ability to store the signal summaries and our specific signal types. While we could add these features, these formats are already complicated, greatly reducing the advantage of repurposing them.
This file format is based upon JLS v1 designed for pyjoulescope and used by the Joulescope test instrument. We leveraged the lessons learned from v1 to make v2 better, faster, and more extensible.
The JLS v1 format has been great for the Joulescope ecosystem and has accomplished the objective of long data captures (days) with fast sampling rates (MHz). However, it now has a long list of issues including:
The JLS v2 file format addressed all of these issues, dramatically improved performance, and added new capabilities, such as signal compression.
At its lowest layer, JLS is an enhanced tag-length-value (TLV) format. TLV files form the foundation of many reliable image and video formats, including MPEG4 and PNG. The enhanced header contains additional fields to speed navigation and improve reliability. The JLS file format calls each TLV a chunk. The enhanced tag-length component the chunk header or simply header. The file also contains a file header, not to be confused with the chunk header. A chunk may have zero payload length, in which case the next header follows immediately. Otherwise, a chunk consists of a header followed by a payload.
The JLS file format defines sources that produce data. The file allows the application to clearly define and label the source. Each source can have any number of associated signals.
Signals are 1-D sequences of values over time consisting of a single, fixed data type. Each signal can have multiple tracks that contain data associated with that signal. The JLS file supports two signal types: fixed sample rate (FSR) and variable sample rate (VSR). FSR signals store their sample data in the FSR track using FSR_DATA and FSR_SUMMARY. FSR time is denoted by samples using timestamp. FSR signals also support:
VSR signals store their sample data in the VSR track. VSR signals specify time in UTC (wall-clock time). VSR signals also support annotations with the ANNOTATION track. The JLS file format supports VSR signals that only use the ANNOTATION track and not the VSR track. Such signals are commonly used to store UART text data where each line contains a UTC timestamp.
Signals support DATA chunks and SUMMARY chunks. The DATA chunks store the actual sample data. The SUMMARY chunks store the reduced statistics, where each statistic entry represents multiple samples. FSR tracks store the mean, min, max, and standard deviation. Although standard deviation requires the writer to compute the square root, standard deviation keeps the same units and bit depth requirements as the other fields. Variance requires twice the bit size for integer types since it is squared.
Before each SUMMARY chunk, the JLS file will contain the INDEX chunk which contains the starting time and offset for each chunk that contributed to the summary. This SUMMARY chunk enables fast O(log n) navigation of the file. For FSR tracks, the starting time is calculated rather than stored for each entry.
The JLS file format design supports SUMMARY of SUMMARY. It supports the DATA and up to 15 layers of SUMMARIES. timestamp is given as a 64-bit integer, which allows each summary to include only 20 samples and still support the full 64-bit integer timestamp space. In practice, the first level summary increases a single value to 4 values, so summary steps are usually 50 or more.
Many applications, including the Joulescope UI, prioritize read performance, especially visualizing the waveform quickly following open, over write performance. Waiting to scan through a 1 TB file is not a valid option. The reader opens the file and scans for sources and signals. The application can then quickly load the highest summary of summaries for every signal of interest. The application can very quickly display this data, and then start to retrieve more detailed information as requested.
sof
header
USER_DATA(0, NULL) // Required, point to first real user_data chunk
SOURCE_DEF(0) // Required, internal, reserved for global annotations
SIGNAL_DEF(0, 0.VSR) // Required, internal, reserved for global annotations
TRACK_DEF(0.VSR)
TRACK_HEAD(0.VSR)
TRACK_DEF(0.ANNO)
TRACK_HEAD(0.ANNO)
SOURCE_DEF(1) // input device 1
SIGNAL_DEF(1, 1, FSR) // our signal, like "current" or "voltage"
TRACK_DEF(1.FSR)
TRACK_HEAD(1.FSR)
TRACK_DEF(1.ANNO)
TRACK_HEAD(1.ANNO)
TRACK_DEF(1.UTC)
TRACK_HEAD(1.UTC)
USER_DATA // just because
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_INDEX(1.FSR, lvl=1)
TRACK_SUMMARY(1.FSR, lvl=2)
USER_DATA // just because
END
eof
Note that TRACK_HEAD(1.FSR) points to the first TRACK_INDEX(1.FSR, lvl=0) and TRACK_INDEX(1.FSR, lvl=1). Each TRACK_DATA(1.FSR) is in a doubly-linked list with its next and previous neighbors. Each TRACK_INDEX(1.FSR, lvl=0) is likewise in a separate doubly-linked list, and the payload of each TRACK_INDEX points to the summarized TRACK_DATA instances. TRACK_INDEX(1.FSR, lvl=1) points to each TRACK_INDEX(1.FSR, lvl=0) instance. As more data is added, the TRACK_INDEX(1.FSR, lvl=1) will also get added to the INDEX chunks at the same level.
This project is Copyright © 2017-2023 Jetperch LLC and licensed under the permissive Apache 2.0 License.
FAQs
Joulescope™ file format
We found that pyjls demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.