github.com/caltechlibrary/dataset/v2

Dataset Project

The Dataset Project provides tools for working with collections of JSON documents stored on the local file system in a pairtree or in a SQL database supporting JSON columns. Two primary tools are provided by the project -- a command line interface (dataset) and a RESTful web service (datasetd).

dataset, a command line tool

dataset is a command line tool for working with collections of JSON documents. Collections can be stored on the file system in a pairtree directory structure or stored in a SQL database that supports JSON columns (currently SQLite3 or MySQL 8 are supported). Collections using the file system store the JSON documents in a pairtree. The JSON documents are plain UTF-8 source. This means the objects can be accessed with common Unix text processing tools as well as most programming languages.

The dataset command line tool supports common data management operations such as initialization of collections; document creation, reading, updating and deleting; listing keys of JSON objects in the collection; and associating non-JSON documents (attachments) with specific JSON documents in the collection.

enhanced features include

• aggregate objects into data frames
• generate sample sets of keys and objects
• clone a collection
• clone a collection into training and test samples

See Getting started with dataset for a tour and tutorial.

datasetd, dataset as a web service

datasetd is a RESTful web service implementation of the dataset command line program. It features a sub-set of capability found in the command line tool. This allows dataset collections to be integrated safely into web applications or used concurrently by multiple processes. It achieves this by storing the dataset collection in a SQL database using JSON columns.

Design choices

dataset and datasetd are intended to be simple tools for managing collections JSON object documents in a predictable structured way.

dataset is guided by the idea that you should be able to work with JSON documents as easily as you can any plain text document on the Unix command line. dataset is intended to be simple to use with minimal setup (e.g. dataset init mycollection.ds creates a new collection called 'mycollection.ds').

• dataset and datasetd store JSON object documents in collections.
  • Storage of the JSON documents may be either in a pairtree on disk or in a SQL database using JSON columns (e.g. SQLite3 or MySQL 8)
  • dataset collections are made up of a directory containing a collection.json and codemeta.json files.
  • collection.json metadata file describing the collection, e.g. storage type, name, description, if versioning is enabled
  • codemeta.json is a codemeta file describing the nature of the collection, e.g. authors, description, funding
  • collection objects are accessed by their key, a unique identifier made of lower case alpha numeric characters
  • collection names are usually lowered case and usually have a .ds extension for easy identification

datatset collection storage options

• pairtree is the default disk organization of a dataset collection
  • the pairtree path is always lowercase
  • non-JSON attachments can be associated with a JSON document and found in a directories organized by semver (semantic version number)
  • versioned JSON documents are created along side the current JSON document but are named using both their key and semver
• SQL store stores JSON documents in a JSON column
  • SQLite3 and MySQL 8 are the current SQL databases support
  • A "DSN URI" is used to identify and gain access to the SQL database
  • The DSN URI maybe passed through the environment

datasetd is a web service

• is intended as a back end web service run on localhost
  • by default it runs on localhost port 8485
  • supports collections that use the SQL storage engine
• should never be used as a public facing web service
  • there are no user level access mechanisms
  • anyone with access to the web service end point has access to the dataset collection content

The choice of plain UTF-8 is intended to help future proof reading dataset collections. Care has been taken to keep dataset simple enough and light weight enough that it will run on a machine as small as a Raspberry Pi Zero while being equally comfortable on a more resource rich server or desktop environment. dataset can be re-implement in any programming language supporting file input and output, common string operations and along with JSON encoding and decoding functions. The current implementation is in the Go language.

Features

dataset supports

• Initialize a new dataset collection
  • Define metadata about the collection using a codemeta.json file
  • Define a keys file holding a list of allocated keys in the collection
  • Creates a pairtree for object storage
• Codemeta file support for describing the collection contents
• Simple JSON object versioning
• Listing Keys in a collection
• Object level actions
  • create
  • read
  • update
  • delete
  • keys
  • has-key
  • sample
  • clone
  • clone-sample
  • Documents as attachments
    • attachments (list)
    • attach (create/update)
    • retrieve (read)
    • prune (delete)
• The ability to create data frames from while collections or based on keys lists
  • frames are defined using a list of keys and a lost dot paths describing what is to be pulled out of a stored JSON objects and into the frame
  • frame level actions
    • frames, list the frame names in the collection
    • frame, define a frame, does not overwrite an existing frame with the same name
    • frame-def, show the frame definition (in case we need it for some reason)
    • frame-keys, return a list of keys in the frame
    • frame-objects, return a list of objects in the frame
    • refresh, using the current frame definition reload all the objects in the frame given a key list
    • reframe, replace the frame definition then reload the objects in the frame using the existing key list
    • has-frame, check to see if a frame exists
    • delete-frame remove the frame

datasetd supports

• List collections available from the web service
• List a collection's metadata
• List a collection's Keys
• Object level actions
  • create
  • read
  • update
  • delete
  • Documents as attachments
    • attach
    • retrieve
    • prune
• The ability to create data frames from collections or based on keys lists and dot paths to form a new object
  • dot paths describing what is to be pulled out of a stored JSON objects

Both dataset and datasetd maybe useful for general data science applications needing JSON object management or in implementing repository systems in research libraries and archives.

Limitations of dataset and datasetd

dataset has many limitations, some are listed below

• the pairtree implementation it is not a multi-process, multi-user data store
• it is not a general purpose database system
• it stores all keys in lower case in order to deal with file systems that are not case sensitive, compatibility needed by a pairtree
• it stores collection names as lower case to deal with file systems that are not case sensitive
• it does not have a built-in query language, search or sorting
• it should NOT be used for sensitive or secret information

datasetd is a simple web service intended to run on "localhost:8485".

• it does not include support for authentication
• it does not support a query language, search or sorting
• it does not support access control by users or roles
• it does not provide auto key generation
• it limits the size of JSON documents stored to the size supported by with host SQL JSON columns
• it limits the size of attached files to less than 250 MiB
• it does not support partial JSON record updates or retrieval
• it does not provide an interactive Web UI for working with dataset collections
• it does not support HTTPS or "at rest" encryption
• it should NOT be used for sensitive or secret information

Read next ...

• About the dataset command
• About datasetd web service
• Installation
• License
• Contributing
• Code of conduct
• Explore dataset and datasetd
  • Getting Started with Dataset
  • How To guides
  • Reference Documentation.
  • Topics

Authors and history

• R. S. Doiel
• Tommy Morrell

Releases

Compiled versions are provided for Linux (x86), Mac OS X (x86 and M1), Windows 11 (x86) and Raspberry Pi OS (ARM7).

github.com/caltechlibrary/dataset/releases

Related projects

You can use dataset from Python via the py_dataset package. You can use dataset from Deno+TypeScript by running datasetd and access it with ts_dataset.