io.sirix:sirix-benchmarks
Advanced tools
Stores small-sized, immutable snapshots of your data in an append-only manner. It facilitates querying and reconstructing the entire history as well as easy audits.
Download ZIP | Join us on Discord | Community Forum
Working on your first Pull Request? You can learn how from this free series How to Contribute to an Open Source Project on GitHub and another tutorial: How YOU can contribute to OSS, a beginners guide
"Remember that you're lucky, even if you don't think you are, because there's always something that you can be thankful for." - Esther Grace Earl (http://tswgo.org)
SirixDB uses a huge persistent (in the functional sense) tree of tries, wherein the committed snapshots share unchanged pages and even common records in changed pages. The system only stores page-fragments during a copy-on-write out-of-place operation instead of full pages during a commit to reduce write-amplification. During read operations, the system reads the page-fragments in parallel to reconstruct an in-memory page (thus, a fast, random access storage device as a PCIe SSD is best suited or even byte addressable storage as Intel DC optane memory in the near future -- as we store fine granular not page aligned modifications in a single file (page fragments are currently word-aligned)).
SirixDB currently supports the storage and (time travel) querying of both XML - and JSON-data in our binary encoding, tailored to support versioning. The index-structures and the whole storage engine has been written from scratch to support versioning natively. We might also implement the storage and querying of other data formats as relational data.
Please consider sponsoring our Open Source work if you like the project.
Note: Work on a frontend built with SolidJS, D3.js, and Typescript has just begun.
Discuss it in the Community Forum.
We could write quite a bunch of stuff, why it's often of great value to keep all states of your data in a storage system. Still, recently we stumbled across an excellent blog post, which explains the advantages of keeping historical data very well. In a nutshell, it's all about looking at the evolution of your data, finding trends, doing audits, implementing efficient undo-/redo-operations. The Wikipedia page has a bunch of examples. We recently also added use cases over here.
Our firm belief is that a temporal storage system must address the issues, which arise from keeping past states way better than traditional approaches. Usually, storing time-varying, temporal data in database systems that do not support the storage thereof natively results in many unwanted hurdles. They waste storage space, query performance to retrieve past states of your data is not ideal, and usually, temporal operations are missing altogether.
The DBS must store data in a way that storage space is used as effectively as possible while supporting the reconstruction of each revision, as the database saw it during the commits. All this should be handled in linear time, whether it's the first revision or the most recent revision. Ideally, query time of old/past revisions and the most recent revision should be in the same runtime complexity (logarithmic when querying for specific records).
SirixDB not only supports snapshot-based versioning on a record granular level through a novel versioning algorithm called sliding snapshot, but also time travel queries, efficient diffing between revisions and the storage of semi-structured data to name a few.
Executing the following time-travel query on our binary JSON representation of Twitter sample data gives an initial impression of the possibilities:
let $statuses := jn:open('mycol.jn','mydoc.jn', xs:dateTime('2019-04-13T16:24:27Z'))=>statuses
let $foundStatus := for $status in $statuses
let $dateTimeCreated := xs:dateTime($status=>created_at)
where $dateTimeCreated > xs:dateTime("2018-02-01T00:00:00") and not(exists(jn:previous($status)))
order by $dateTimeCreated
return $status
return {"revision": sdb:revision($foundStatus), $foundStatus{text}}
The query opens a database/resource in a specific revision based on a timestamp (2019–04–13T16:24:27Z) and searches for all statuses, which have a created_at timestamp, which has to be greater than the 1st of February in 2018 and did not exist in the previous revision. => is a dereferencing operator used to dereference keys in JSON objects, array values can be accessed as shown looping over the values or through specifying an index, starting with zero: array[[0]] for instance specifies the first value of the array. Brackit, our query processor also supports Python-like array slices to simplify tasks.
In order to verify changes in a node or its subtree, first select the node in the revision and then
query for changes using our stored merkle hash tree, which builds and updates hashes for each node and it's subtree and check the hashes with sdb:hash($item). The function jn:all-times delivers the node in all revisions in which it exists. jn:previous delivers
the node in the previous revision or an empty sequence if there's none.
let $node := jn:doc('mycol.jn','mydoc.jn')=>fieldName[[1]]
let $result := for $node-in-rev in jn:all-times($node)
return
if ((not(exists(jn:previous($node-in-rev))))
or (sdb:hash($node-in-rev) ne sdb:hash(jn:previous($node-in-rev)))) then
$node-in-rev
else
()
return [
for $jsonItem in $result
return { "node": $jsonItem, "revision": sdb:revision($jsonItem) }
]
Emit all diffs between the revisions in a JSON format:
let $maxRevision := sdb:revision(jn:doc('mycol.jn','mydoc.jn'))
let $result := for $i in (1 to $maxRevision)
return
if ($i > 1) then
jn:diff('mycol.jn','mydoc.jn',$i - 1, $i)
else
()
return [
for $diff at $pos in $result
return {"diffRev" || $pos || "toRev" || $pos + 1: jn:parse($diff)=>diffs}
]
We support easy updates as in
let $array := jn:doc('mycol.jn','mydoc.jn')
return insert json {"bla":true} into $array at position 0
to insert a JSON object into a resource, whereas the root node is an array at the first position (0). The transaction is implicitly committed, thus a new revision is created and the specific revision can be queried using a single third argument, either a simple integer ID or a timestamp. The following query issues a query on the first revision (thus without the changes).
jn:doc('mycol.jn','mydoc.jn',1)
Omitting the third argument simply opens the resource in the most recent revision, but you could in this case also specify revision number 2. You can also use a timestamp as in:
jn:open('mycol.jn','mydoc.jn',xs:dateTime('2022-03-01T00:00:00Z'))
A simple join (whereas joins are optimized in our query processor called Brackit):
(* first: store stores in a stores resource *)
sdb:store('mycol.jn','stores','
[
{ "store number" : 1, "state" : "MA" },
{ "store number" : 2, "state" : "MA" },
{ "store number" : 3, "state" : "CA" },
{ "store number" : 4, "state" : "CA" }
]')
(* second: store sales in a sales resource *)
sdb:store('mycol.jn','sales','
[
{ "product" : "broiler", "store number" : 1, "quantity" : 20 },
{ "product" : "toaster", "store number" : 2, "quantity" : 100 },
{ "product" : "toaster", "store number" : 2, "quantity" : 50 },
{ "product" : "toaster", "store number" : 3, "quantity" : 50 },
{ "product" : "blender", "store number" : 3, "quantity" : 100 },
{ "product" : "blender", "store number" : 3, "quantity" : 150 },
{ "product" : "socks", "store number" : 1, "quantity" : 500 },
{ "product" : "socks", "store number" : 2, "quantity" : 10 },
{ "product" : "shirt", "store number" : 3, "quantity" : 10 }
]')
let $stores := jn:doc('mycol.jn','stores')
let $sales := jn:doc('mycol.jn','sales')
let $join :=
for $store in $stores, $sale in $sales
where $store=>"store number" = $sale=>"store number"
return {
"nb" : $store=>"store number",
"state" : $store=>state,
"sold" : $sale=>product
}
return [$join]
SirixDB through Brackit also supports array slices. Start index is 0, step is 1 and end index is 1 (exclusive) in the next query:
let $array := [{"foo": 0}, "bar", {"baz": true()}]
return $array[[0:1:1]]
The query returns the first object {"foo":0}.
With the function sdb:nodekey you can find out the internal unique node key of a node, which will never change. You for instance might be interested in which revision it has been removed. The following query uses the function sdb:select-item which as the first argument needs a context node and as the second argument the key of the item or node to select. jn:last-existing finds the most recent version and sdb:revision retrieves the revision number.
sdb:revision(jn:last-existing(sdb:select-item(jn:doc('mycol.jn','mydoc.jn',1), 26)))
SirixDB has three types of indexes along with a path summary tree, which is basically a tree of all distinct paths:
We base the indexes on the following serialization of three revisions of a very small SirixDB ressource.
{
"sirix": [
{
"revisionNumber": 1,
"revision": {
"foo": [
"bar",
null,
2.33
],
"bar": {
"hello": "world",
"helloo": true
},
"baz": "hello",
"tada": [
{
"foo": "bar"
},
{
"baz": false
},
"boo",
{},
[]
]
}
},
{
"revisionNumber": 2,
"revision": {
"tadaaa": "todooo",
"foo": [
"bar",
null,
103
],
"bar": {
"hello": "world",
"helloo": true
},
"baz": "hello",
"tada": [
{
"foo": "bar"
},
{
"baz": false
},
"boo",
{},
[]
]
}
},
{
"revisionNumber": 3,
"revision": {
"tadaaa": "todooo",
"foo": [
"bar",
null,
23.76
],
"bar": {
"hello": "world",
"helloo": true
},
"baz": "hello",
"tada": [
{
"foo": "bar"
},
{
"baz": false
},
"boo",
{},
[
{
"foo": "bar"
}
]
]
}
}
]
}
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $stats := jn:create-name-index($doc, ('foo','bar'))
return {"revision": sdb:commit($doc)}
The index is created for "foo" and "bar" object fields. You can query for "foo" fields as for instance:
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $nameIndexNumber := jn:find-name-index($doc, 'foo')
for $node in jn:scan-name-index($doc, $nameIndexNumber, 'foo')
order by sdb:revision($node), sdb:nodekey($node)
return {"nodeKey": sdb:nodekey($node), "path": sdb:path($node), "revision": sdb:revision($node)}
Second, whole paths are indexable.
Thus, the following path index is applicable to both queries: =>sirix[]=>revision=>tada[]=>foo and =>sirix[]=>revision=>tada[][[4]]=>foo. Thus, essentially both foo nodes are indexed and the first child has to be fetched afterwards. For the second query also the array index 4 has to be checked if the indexed node is really on index 4.
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $stats := jn:create-path-index($doc, '/sirix/[]/revision/tada//[]/foo')
return {"revision": sdb:commit($doc)}
The index might be scanned as follows:
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $pathIndexNumber := jn:find-path-index($doc, '/sirix/[]/revision/tada//[]/foo')
for $node in jn:scan-path-index($doc, $pathIndexNumber, '/sirix/[]/revision/tada//[]/foo')
order by sdb:revision($node), sdb:nodekey($node)
return {"nodeKey": sdb:nodekey($node), "path": sdb:path($node)}
CAS indexes index a path plus the value. The value itself must be typed (so in this case we index only decimals on a path).
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $stats := jn:create-cas-index($doc, 'xs:decimal', '/sirix/[]/revision/foo/[]')
return {"revision": sdb:commit($doc)}
We can do an index range-scan as for instance via the next query (2.33 and 100 are the min and max, the next two arguments are two booleans which denote if the min and max should be retrieved or if it's >min and <max). The last argument is usually a path if we index more paths in the same index (in this case we only index /sirix/[]/revision/foo/[]).
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $casIndexNumber := jn:find-cas-index($doc, 'xs:decimal', '/sirix/[]/revision/foo/[]')
for $node in jn:scan-cas-index-range($doc, $casIndexNumber, 2.33, 100, false(), true(), ())
order by sdb:revision($node), sdb:nodekey($node)
return {"nodeKey": sdb:nodekey($node), "node": $node}
You can also create a CAS index on all string values on all paths (all object fields: //*; all arrays: //[]):
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $stats := jn:create-cas-index($doc,'xs:string',('//*','//[]'))
return {"revision": sdb:commit($doc)}
To query for a string values with a certain name (bar) on all paths (empty sequence ()):
let $doc := jn:doc('mycol.jn','mydoc.jn')
let $casIndexNumber := jn:find-cas-index($doc, 'xs:string', '//*')
for $node in jn:scan-cas-index($doc, $casIndexNumber, 'bar', 0, ())
order by sdb:revision($node), sdb:nodekey($node)
return {"nodeKey": sdb:nodekey($node), "node": $node, "path": sdb:path(sdb:select-parent($node))}
The argument 0 means check for equality of the string. Other values which might make more sense for integers, decimals... are -2 for <, -1 for <=, 1 for >= and 2 for >.
SirixDB is a log-structured, temporal NoSQL document store, which stores evolutionary data. It never overwrites any data on-disk. Thus, we're able to restore and query the full revision history of a resource in the database.
Some of the most important core principles and design goals are:
Keeping the revision history is one of the main features in SirixDB. You can revert any revision into an earlier version or back up the system automatically without the overhead of copying. SirixDB only ever copies changed database-pages and, depending on the versioning algorithm you chose during the creation of a database/resource, only page-fragments, and ancestor index-pages to create a new revision.
You can reconstruct every revision in O(n), where n denotes the number of nodes in the revision. Binary search is used on an in-memory (linked) map to load the revision, thus finding the revision root page has an asymptotic runtime complexity of O(log n), where n, in this case, is the number of stored revisions.
Currently, SirixDB offers two built-in native data models, namely a binary XML store and a JSON store.
Articles published on Medium:
SirixDB as of now has not been tested in production. It is recommended for experiments, testing, benchmarking, etc., but is not recommended for production usage. Let us know if you'd like to use SirixDB in production and get in touch. We'd like to test real-world datasets and fix issues we encounter along the way.
Please also get in touch if you like our vision and you want to sponsor us or help with man-power or if you want to use SirixDB as a research system. We'd be glad to get input from the database and scientific community.
git clone https://github.com/sirixdb/sirix.git
or use the following dependencies in your Maven or Gradle project.
SirixDB uses Java 18, thus you need an up-to-date Gradle (if you want to work on SirixDB) and an IDE (for instance IntelliJ or Eclipse).
At this stage of development, you should use the latest SNAPSHOT aSrtifacts from the OSS snapshot repository to get the most recent changes.
Just add the following repository section to your POM or build.gradle file:
<repository>
<id>sonatype-nexus-snapshots</id>
<name>Sonatype Nexus Snapshots</name>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
repository {
maven {
url "https://oss.sonatype.org/content/repositories/snapshots/"
mavenContent {
snapshotsOnly()
}
}
}
Note that we changed the groupId from com.github.sirixdb.sirix to io.sirix. Most recent version is 0.9.6-SNAPSHOT.
Maven artifacts are deployed to the central maven repository (however please use the SNAPSHOT-variants as of now). Currently, the following artifacts are available:
Core project:
<dependency>
<groupId>io.sirix</groupId>
<artifactId>sirix-core</artifactId>
<version>0.9.6-SNAPSHOT</version>
</dependency>
compile group:'io.sirix', name:'sirix-core', version:'0.9.6-SNAPSHOT'
Brackit binding:
<dependency>
<groupId>io.sirix</groupId>
<artifactId>sirix-xquery</artifactId>
<version>0.9.6-SNAPSHOT</version>
</dependency>
compile group:'io.sirix', name:'sirix-xquery', version:'0.9.6-SNAPSHOT'
Asynchronous, RESTful API with Vert.x, Kotlin and Keycloak (the latter for authentication via OAuth2/OpenID-Connect):
<dependency>
<groupId>io.sirix</groupId>
<artifactId>sirix-rest-api</artifactId>
<version>0.9.4-SNAPSHOT</version>
</dependency>
compile group: 'io.sirix', name: 'sirix-rest-api', version: '0.9.6-SNAPSHOT'
Other modules are currently not available (namely the GUI, the distributed package as well as an outdated Saxon binding).
The REST-API is asynchronous at its very core. We use Vert.x, which is a toolkit built on top of Netty. It is heavily inspired by Node.js but for the JVM. As such, it uses event loop(s), which is thread(s), which never should by blocked by long-running CPU tasks or disk-bound I/O. We are using Kotlin with coroutines to keep the code simple. SirixDB uses OAuth2 (Password Credentials/Resource Owner Flow) using a Keycloak authorization server instance.
For setting up the SirixDB HTTP-Server and a basic Keycloak-instance with a test realm:
git clone https://github.com/sirixdb/sirix.gitsudo docker-compose up keycloakYou can set up Keycloak as described in this excellent tutorial. Our docker-compose file imports a sirix realm with a default admin user with all available roles assigned. You can skip steps 3 - 7 and 10, 11, and simply recreate a client-secret and change oAuthFlowType to "PASSWORD". If you want to run or modify the integration tests, the client secret must not be changed. Make sure to delete the line "build: ." in the docker-compse.yml file for the server image if you want to use the Docker Hub image.
Clients => accountaccess-type is set to confidentialCredentials tabclient secret into the SirixDB HTTP-Server configuration file. Change the value of "client.secret" to whatever Keycloak set up.direct access grant on the settings tab must be enabled.${databaseName}- prefixed roles.The following command will start the docker container
sudo docker-compose upTo created a fat-JAR. Download our ZIP-file for instance, then
cd bundles/sirix-rest-apigradle build -x testAnd a fat-JAR with all required dependencies should have been created in your target folder.
Furthermore, a key.pem and a cert.pem file are needed. These two files have to be in your user home directory in a directory called "sirix-data", where Sirix stores the databases. For demo purposes they can be copied from our resources directory.
Once also Keycloak is set up we can start the server via:
java -jar -Duser.home=/opt/sirix sirix-rest-api-*-SNAPSHOT-fat.jar -conf sirix-conf.json -cp /opt/sirix/*
If you like to change your user home directory to /opt/sirix for instance.
The fat-JAR in the future will be downloadable from the maven repository.
In order to run the integration tests under bundles/sirix-rest-api/src/test/kotlin make sure that you assign your admin user all the user-roles you have created in the Keycloak setup (last step). Make sure that Keycloak is running first and execute the tests in your favorite IDE for instance.
Note that the following VM-parameters currently are needed: -ea --add-modules=jdk.incubator.foreign --enable-preview
We ship a (very) simple command-line tool for the sirix-xquery bundle:
Get the latest sirix-xquery JAR with dependencies.
We are currently working on the documentation. You may find first drafts and snippets in the documentation and in this README. Furthermore, you are kindly invited to ask any question you might have (and you likely have many questions) in the community forum (preferred) or in the Discord channel. Please also have a look at and play with our sirix-example bundle which is available via maven or our new asynchronous RESTful API (shown next).
If you have any questions or are considering to contribute or use Sirix, please use the Community Forum to ask questions. Any kind of question, may it be an API-question or enhancement proposal, questions regarding use-cases are welcome... Don't hesitate to ask questions or make suggestions for improvements. At the moment also API-related suggestions and critics are of utmost importance.
You may find us on Discord for quick questions.
SirixDB is maintained by
And the Open Source Community.
As the project was forked from a university project called Treetank, my deepest gratitude to Marc Kramis, who came up with the idea of building a versioned, secure and energy-efficient data store, which retains the history of resources of his Ph.D. Furthermore, Sebastian Graf came up with a lot of ideas and greatly improved the implementation for his Ph.D. Besides, a lot of students worked and improved the project considerably.
Thanks goes to these wonderful people, who greatly improved SirixDB lately. SirixDB couldn't exist without the help of the Open Source community:
Contributions of any kind are highly welcome!
This work is released under the BSD 3-clause license.
FAQs
Unknown package
We found that io.sirix:sirix-benchmarks demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
Socket researchers have discovered malicious npm packages targeting crypto developers, stealing credentials and wallet data using spyware delivered through typosquats of popular cryptographic libraries.
Security News
Socket's package search now displays weekly downloads for npm packages, helping developers quickly assess popularity and make more informed decisions.
Security News
A Stanford study reveals 9.5% of engineers contribute almost nothing, costing tech $90B annually, with remote work fueling the rise of "ghost engineers."
