@logsn/benchmarks

Log Store Benchmarks

This package provides a benchmarking tool for the Log Store Network. It's built with Vitest and Tinybench, and allows running benchmarks either as normal Vitest tests or through a Command Line Interface (CLI).

Architecture

• Vitest: Used for running the benchmarks as tests.
• Tinybench: Utilized for measuring performance.

Dependencies

Ensure that you have installed all the dependencies specified in the package.json file.

Running Benchmarks

There are two ways to run benchmarks:

• As Normal Vitest Tests: Run the benchmarks as normal Vitest tests by executing the following command:

pnpm benchmark

This command will run all *.benchmark.ts test files.

• (WIP): Using CLI Tool: You can also run the benchmarks through the command line interface.

To run without installing globally:

pnpx @logsn/benchmarks run

Alternatively, install globally and then run from terminal:

pnpm install -g @logsn/benchmarks
logsn-benchmarks run

Output

The benchmarks produce JSON output with statistics. When running as normal Vitest tests, the results are saved in the results directory.

Each benchmark produces two sets of results: Cold and Hot. Cold start tests are executed first with fewer iterations and mainly serve to warm up the process. Hot tests run immediately after. This distinction is necessary for analyzing the performance during the warmup phase.

Note: New results will override old results by test name. This means that it doesn't delete the last file, but instead replaces it with new results if the test name is equal.

Results reference

We are collecting and exporting the standard statistics provided by Tinybench, except for samples propery. The following table describes each of the properties in the results object.

Property	Type	Description
error	unknown?	The last error that was thrown while running the task. Optional.
totalTime	number	The amount of time in milliseconds to run the benchmark task (cycle).
min	number	The minimum value in the samples.
max	number	The maximum value in the samples.
hz	number	The number of operations per second.
period	number	How long each operation takes in milliseconds.
mean	number	Samples mean/average (estimate of the population mean).
variance	number	Samples variance (estimate of the population variance).
sd	number	Samples standard deviation (estimate of the population standard deviation).
sem	number	Standard error of the mean (also known as the standard deviation of the sampling distribution of the sample mean).
df	number	Degrees of freedom.
critical	number	Critical value of the samples.
moe	number	Margin of error.
rme	number	Relative margin of error.
p75	number	75th percentile.
p99	number	99th percentile.
p995	number	99.5th percentile.
p999	number	99.9th percentile.
population	number	Number of iterations / samples.

Note: The ommited samples property references each individual task interation time, so 1000 iterations would mean large files. But if default statistics provided by Tinybench are not enough, we can always access the samples property in the results object and build our own statistics.

Configurations

• Number of Iterations: You can configure the number of iterations via the TEST_TIMEOUT environment variable for test mode, or through the appropriate argument when using the CLI.
• Log Level: You can set the log level by setting the LOG_LEVEL environment variable. Acceptable values include info, debug, error, fatal, silent, warn, trace. E.g., LOG_LEVEL='trace'.
• Network URLs: Network URLs (e.g., RPC) can be configured by setting the STREAMR_DOCKER_DEV_HOST environment variable or the corresponding CLI argument. For more details, refer to the config documentation.

CLI

The benchmarking tool also offers a CLI which allows running benchmarks through the terminal. The CLI can be used to specify various options for the benchmark run.

The command to use the CLI is:

logsn-benchmarks run [options]
# or pnpx @logsn/benchmarks run [options] if not installed

Options

• c, --config <string>: Specifies the config file. Note: This is not working currently.
• o, --outDir <string>: Specifies the output directory for the benchmark results. The default is ./results.
• s, --streamrHost <string>: Specifies the Streamr host. The default is localhost.
• n, --numberOfIterations <number>: Specifies the number of iterations for the benchmark. The default is 5.
• l, --logLevel <string>: Specifies the log level. Acceptable values are debug, info, warn, error, fatal, silent. The default is info.

Example Usage

logsn-benchmarks run -o ./my-results -s my-streamr-host -n 10 -l debug

This example runs the benchmarks and saves the results in a directory named my-results. It uses my-streamr-host as the Streamr host, sets the number of iterations to 10, and the log level to debug.

Development

Building

To build the project, run:

pnpm build

This will trigger the build script defined in build.ts using ESBuild.