WebSite | Repository | Wiki | Bug Reports | Live Help (discord) | By Jay Kuri
DTL, short for Data Transformation Language, is a versatile tool ideal for managing and transforming structured data, especially from sources like NoSQL databases and APIs. It seamlessly integrates into your JavaScript projects, whether you're working in a Node.js environment or directly in the browser. Additionally, DTL offers powerful command-line tools that excel in handling various formats, including JSON, YAML, CSV, and plain text. These tools are perfect for processing and exploring existing data sets.
Whether you need something as simple as JSON templating, more complex tasks like format conversion, or intricate data restructuring, DTL stands out as an invaluable resource for developers dealing with any form of structured data manipulation.
If your work involves manipulating structured data, and what development doesn't, DTL offers an efficient and easy to understand toolset to help you.
DTL looks like JSON and has a very familiar syntax:
{
"out": {
"name": "(: &( $first_name ' ' $last_name ) :)",
"order_total": "(: $order_subtotal + $tax_amt :)",
"origin": "Import from legacy database",
"address": {
"street": "(: $address1 :)",
"unit": "(: $address2 :)",
"city": "(: $addr_city :)",
"state": "(: $addr_state :)",
"postal": "(: $addr_zip :)"
}
}
}
We'll explore this example in detail a bit later.
DTL can be installed within your project, or globally for command-line use.
For global access to DTL's command-line tools, install DTL globally using npm or yarn:
Using npm
npm install -g dtl-js
Using yarn
yarn global add dtl-js
To install DTL as part of your project, navigate to your project directory and run:
Using npm
npm install dtl-js
Using yarn
yarn add dtl-js
After installation, you can verify the installation of DTL:
For global installations, check the version to ensure it's installed correctly:
dtl --version
For local installations, you can check your package.json to see if dtl-js is listed under dependencies.
DTL, is built for transforming and manipulating data structures. At its most basic level, DTL can act as a templating tool for JSON or YAML, enabling straightforward data reformatting and restructuring. However, DTL's capabilities extend far beyond simple templating. DTL is ideal for complex tasks such as data extraction, conversion, and preparation for analysis or reporting. Its simple syntax and robust functions facilitate the clear and concise expression of complex data transformations, making DTL an extremely useful tool for a wide range of data processing needs.
In DTL, transforms are generally defined within a JSON object, which
forms the transform library. Each key in this JSON object is a distinct
transform, functioning like a template or function for how input data should
be outputted. The out key is special as it's the default transform used by
DTL.apply(), akin to the main function in other languages.
In most cases, the value of out will be an object that looks like your
desired output structure, guiding the transformation of input data to this
format. This library can have multiple transforms, each capable of
referencing and calling one another. When a transform is executed through
DTL.apply(), the provided data is represented as $. within the
transform.
Every DTL expression must be enclosed in happy tags: (: :). Happy tags
tell DTL to do something. Anything not in Happy tags is passed through
untouched. Happy tags are called that because they resemble happy emoji.
Reference data using dot notation, with the root represented as $.
For nested data access, use dot notation, e.g., $user.address.street
Data can be referenced with or without a leading ., in other words,
$.user and $user are equivalent.
And here's a bit of good news: in DTL, trying to access a property of a
non-existent or non-object value won't send you into the dreaded JavaScript
spiral of Cannot read properties of undefined. In DTL, it just sensibly
returns undefined. No drama, no crash – just the peace of mind knowing DTL
has got your back in these scenarios.
In DTL expressions, commas are treated like whitespace and are not necessary.
Use built-in helper functions for complex operations, e.g.,
math.round($number). To call another transform in the library, use
($input_data -> transform_name). In this example, $input_data would be
sent to transform_name and the result would be the new data.
Operations look like this: $data op $otherdata, e.g., $number1 + $number2.
Use parentheses to control the order that things should be done, e.g.,
($number1 + $number2) * $number3. DTL understands mathematical order of
operations, but there's no PEMDAS for data types, so parenthesis will help
make your intentions clearer.
For conditionals, use ?(: $condition $truevalue $falsevalue). You can use
this to send data to one or another transform depending on some condition:
?(!empty($address) ($. -> process_address) ($. -> no_address))
In this example, ?(!empty($address) ($. -> process_address) ($. -> no_address)),
DTL checks if $address is not empty. If it's not, the data
is processed with the process_address transform; if $address is empty,
the no_address transform is used instead. This showcases DTL's ability to
elegantly handle conditional data routing based on the presence or absence
of data.
A mix of static (literal values) and dynamic (DTL expressions) is common in transforms. Only dynamic expressions use happy tags.
You can loop or iterate over lists of data. In all helpers that iterate,
like map or grep, use $item, $index, and $all for referencing.
$item is the current item. $index is the index of the current item and
$all is the entire list or object.
Transforms only have access to what was given to them (from DTL.apply() or
another transform in the library) and have no way to access information
other than that. This is a good thing, it means your transform will always
do exactly what you want. Within a transform, $. refers to the input data
of that transform.
In DTL, mathematical operations are carried out using Arbitrary Precision Mathematics. This approach is a game-changer, especially when you compare it to JavaScript, where 0.1 + 0.2 equals 0.30000000000000004 (seriously, try it in your web or Node.js console). In DTL, it simply equals 0.3, just as you would logically expect. This means that when you're doing calculations in DTL, you're getting the exactly correct answer every time. DTL's precision makes it ideal for scenarios where mathematical accuracy isn't just a preference, it's a necessity.
In DTL, the concept of empty is key. Empty in DTL means the data has no
meaningful value, e.g. undefined, null, an empty string, array, or object.
The empty() function returns true if a value is 'empty'
If you've ever wondered why [] evaluates to true or been bitten by
if(formfield) evaluating to false if someone entered 0, we feel your pain.
In most cases, you really want to know if a variable has a meaningful
value. empty() is here to help.
Complementing this, the fne() (First Non-Empty) function scans a list of
values and returns the first one that is not considered empty, streamlining
the selection of valid data from multiple sources. It looks like this:
fne($first_place_to_look $second_place_to_look defaultvalue)
Here's an example of DTL:
{
"out": {
"name": "(: &( $first_name ' ' $last_name ) :)",
"order_total": "(: $order_subtotal + $tax_amt :)",
"origin": "Import from legacy database",
"address": {
"street": "(: $address1 :)",
"unit": "(: $address2 :)",
"city": "(: $addr_city :)",
"state": "(: $addr_state :)",
"postal": "(: $addr_zip :)"
}
}
}
In this JSON transformation example:
name field uses the &() helper to concatenate $first_name and
$last_name with a space in between.order_total field does some simple math, adding $order_subtotal
and $tax_amt together.origin is a static string, "Import from legacy database".address field is an object containing multiple expressions. This shows
how your transforms mirror your desired output format.
### There is no step 3.
Using DTL is as straightforward as it gets:
1. **Create Your Transform**: Defining how you want your output data to
look. This step involves crafting the template or transformation logic in
a JSON object, specifying how the input data should be processed and
structured.
2. **Call `DTL.apply()`**: Once your transform is ready, the next step is to
apply it to your data. This is done using the `DTL.apply()` method, which
takes your input data and the transform, performs the transformation as defined,
and returns the result.
And that's it — there is no step 3. DTL's design is centered around this
simplicity, making data transformations easy and accessible.
Below is a practical example demonstrating these two steps in a Node.js script:
### Putting It All Together: Example Node.js Script
Now that we've covered the basics of DTL, let's see it in action with a
simple Node.js script. This example demonstrates how to use DTL to
transform some sample data and output the result.
```javascript
// Import the DTL module
const DTL = require('dtl-js');
// Example input data
const inputData = {
first_name: "John",
last_name: "Doe",
order_subtotal: 100,
tax_amt: 20,
address1: "123 Main St",
address2: "Apt 4",
addr_city: "Springfield",
addr_state: "IL",
addr_zip: "62704"
};
// Example DTL transform
const transformLibrary = {
"out": {
"name": "(: &( $first_name ' ' $last_name ) :)",
"order_total": "(: $order_subtotal + $tax_amt :)",
"origin": "Import from legacy database",
"address": "(: $. -> formatAddress :)"
},
"formatAddress": {
"street": "(: $address1 :)",
"unit": "(: $address2 :)",
"city": "(: $addr_city :)",
"state": "(: $addr_state :)",
"postal": "(: $addr_zip :)"
}
};
// Applying the transform to the input data
const transformedData = DTL.apply(inputData, transformLibrary);
console.log("Transformed Data:", JSON.stringify(transformedData, undefined, 4));
This script demonstrates how to import DTL, define a transform library, apply it to some sample data, and output the transformed result. It's a practical example of how DTL can be utilized in a typical Node.js environment to manipulate and transform data structures.
As you can see, integrating DTL into your project is super simple. With its simple syntax and powerful capabilities, DTL is all set to make your job easier. Now, go use it to streamline your code, make your workflows more efficient or just do some awesome stuff!
Working with DTL is made significantly easier thanks to comprehensive editor support. Full syntax highlighting in the editor bundles not only clarifies DTL syntax but also enhances your coding efficiency. With snippets for all built-in functions, these bundles facilitate a more intuitive and error-free coding experience. We encourage you to download the appropriate bundle for your editor of choice.
Look for DTL in the VSCode extension marketplace, or if you use Vim, Sublime Text or other Textmate compatible editors, you can download the language support files from the Releases page of the DTL Repository Streamline your coding with these powerful features for an improved, error-free DTL coding experience.
DTL is adept at addressing a wide range of frequently encountered data transformation challenges. Here are some common problems where DTL can be particularly beneficial:
Creating Data Templates: Effortlessly generate structured data from templates. DTL excels in scenarios like auto-generating JSON configurations or preparing data payloads for APIs, making these tasks quick and error-free.
Simple Data Transformations: Whether it's renaming keys in JSON objects, restructuring data (such as flattening nested structures), or performing basic aggregations and calculations, DTL simplifies these common tasks.
Format Conversion: DTL streamlines the conversion between various data formats. Transforming data from JSON to CSV, XML to JSON, or YAML to JSON becomes a hassle-free process.
Data Cleaning and Normalization: Clean or normalize data from diverse sources to fit specific standards. Tasks like removing unwanted fields, standardizing date formats, or converting data types are made straightforward with DTL.
Data Validation: Ensure that incoming data conforms to specified criteria or formats. DTL's data validation capabilities are particularly useful for data from external sources or user inputs.
Extracting Specific Data from Complex Structures: Extracting necessary information from complex or nested data structures, such as certain fields from deeply nested JSON objects, is made efficient and simple.
Automating Repetitive Data Tasks: DTL helps automate repetitive data manipulation tasks, saving time and minimizing the risk of errors in routine operations.
Using DTL, you can address these common data transformation challenges more efficiently, allowing you to focus on the more intricate aspects of your projects.
DTL Repl demo
To open the dtlr data exploration environment:
$ dtlr filename.json
To do bulk data manipulation on the command line:
See available command line options:
$ dtl -h
(Movie examples below use sample movie data from Awesome JSON Datasets )
# Convert movies.json to yaml
$ dtl -o movies.yaml movies.json
# How many movies are listed?
$ dtl -e 'length($.)' movies.json`
# Get all the movies where Bill Murray was in the cast
$ dtl -e 'grep($. "(: member($item.cast `Bill Murray`) :)")' movies.json
# Extract the `name` field in the JSON file in uppercase.
$ dtl -e 'uc($name)' data.json
# Apply the transform expression specified in the `transform.dtl` file to the CSV input file.
$ dtl -f transform.dtl input.csv
# Process `input.json`, output CSV data in `output.csv` including all fields found in input.json.
$ dtl -o output.csv input.json`
# Process `input.json`, output CSV data on stdout using the fields `first_name`, `last_name` and `primary_email`
dtl -O csv -C first_name,last_name,primary_email input.json
Unlike regular code, the output of DTL can only include the information provided to the DTL call, so DTL transforms are much safer to use than the code that would be required to produce the same output. They're also a heck of a lot easier to read... AND since they are self-contained and don't refer to your code, they are safe and easy to share.
DTL can be used within javascript code (node.js and browser, and even inside MongoDB) or it can be used on the command line with the DTL cli tools.
DTL is interesting for several reasons:
Clarity - DTL is purpose-built for data transformation and only data transformation. It is not intended to be a general-purpose programming language and is therefore simple to learn and free of unnecessary components.
Portable - DTL transforms are self-contained and transferrable between applications. Since they can be stored as JSON, they can even be kept in your database.
Security - DTL transforms only have access to the data that was provided as input. DTL transforms have no other variable or system access, so they are much safer to use than custom code.
Stateless - DTL transforms have no access to previous state, only to the data provided and therefore avoid bugs related to bleed over or inadvertant modification, one of the most common sources of bugs.
Provable - It is trivial to create a DTL transform to verify the output of another. This obviously allows for simple test-creation. What may not be obvious is that these verification transforms can be used to check data at run-time.
Non-linear - DTL transforms define how to arrive at the desired data. They do not define a sequence of steps. This means that each expression is independent and not subject to bugs due to issues that occurred in other expressions.
Stable - DTL has been in use in production since 2013 and has been its own separate project since 2015. It is being used in many production applications, handling many millions of transformations every day.
DTL is a language with an implementation. The DTL npm module is only one implementation of the DTL language. The DTL module contains hundreds of tests that verify the language is behaving properly. This allows DTL to be implemented and verified in any programming language.
Truth be told, DTL was not originally intended to be it's own thing. DTL began as an expression language inside a meta-programming engine built by Jay Kuri (me) during my work at Ionzero, a company I founded. One of the first applications of this engine was a system built to handle linking other systems together. I created the language out of the need for a way to define how to map data from one system to another without resorting to hard-coded custom code.
I also realized during the course of this work that DTL could be used for far more than I ever had originally envisioned. As a result of this realization, over time, I refined the DTL language and eventually extracted it into a self-contained module that could be used in any system and proceeded to do so.
I decided to release DTL as open source in the hopes that others would find it as useful and as powerful as I have.
As mentioned earlier in this document, if you have installed the DTL
package globally with npm or yarn, you will have two command line tools for
working with DTL. The dtl cli tool works on bulk data
If you want to just take DTL for a spin without coding you can use the
dtlr tool. The dtlr cli tool is an interactive REPL (Read
Execute Print Loop) tool you can use to test out expressions and get
help.
The dtl cli tool works on bulk data and is designed to process CSV, yaml
and JSON, as well as JSONLines data. It can produce CSV, yaml, JSON and
JSONLines data as well, regardless of whether the input data was the
same type. You can learn more about how to use it by using the
dtl -h command. Note that by default it sends its output to stdout.
If you'd rather have the output go into a file, use the -o filename.json
option.
We are always looking for constructive feedback. If you have ideas on how we might improve DTL, please reach out. If you are looking for help on how to use DTL, we also want to hear from you.
For help learning DTL, the dtlr tool has help built in by
using the .help command. You can also You can visit the
docs or look at the
DTL Expression Syntax.
You can also view all the helper function docs here.
If you want to see examples of DTL or try it out, head to the DTL website and click the 'Try DTL' button to explore DTL right in your browser.
If you want to see more examples, you can take a look at the Test Suite where you can find an example of just about anything DTL can do.
If you have ChatGPT Plus, you can talk to the DTL Helper to get help with DTL.
If you want something a bit more real-time and real-person, you can talk with us on the DTL discord.
And, if you encounter a bug, please don't hesitate to file an Issue.
FAQs
Data Transformation Language - JSON templates and data transformation
We found that dtl-js demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious PyPI package semantic-types steals Solana private keys via transitive dependency installs using monkey patching and blockchain exfiltration.
Security News
New CNA status enables OpenJS Foundation to assign CVEs for security vulnerabilities in projects like ESLint, Fastify, Electron, and others, while leaving disclosure responsibility with individual maintainers.
Product
Socket MCP brings real-time security checks to AI-generated code, helping developers catch risky dependencies before they enter the codebase.