Security News
The Unpaid Backbone of Open Source: Solo Maintainers Face Increasing Security Demands
Solo open source maintainers face burnout and security challenges, with 60% unpaid and 60% considering quitting.
data-reduction
Advanced tools
A utility for reducing the size of data sets for visualization. This library provides data reduction functionality using filtering and binned aggregation.
One of the most common challenges in data visualization is handling a large amount of data. There have been many discussions on the D3 mailing list about this topic: "Building d3 charts with millions of data", "200MB data to browser with D3?", "Creating chart using d3 with more than thousand records", "data visualization of 100 millions of record" and "D3JS to visualize BIG DATA".
There are two main approaches to solve the problem of "Big Data Visualization":
Pushing the limits of technology using techniques like progressive rendering or WebGL is advocated and practiced by many people, however for information visualization, I would argue that this approach is not ideal. Think about it like this: if there are more data points than pixels, can you really perceive all of the data by plotting one mark per data points? If you cannot perceive all of the marks, are they worth rendering?
The second approach is to use data reduction techniques to reduce the data before rendering it. The idea behind this is to preserve all of the interesting structures in the data that you would perceive if you did plot all of the records (such as density distribution), while reducing the number of marks that need to be rendered. The paper imMens: Real-time Visual Querying of Big Data contains a great overview of data reduction methods (Section 3), which includes
This library exposes JavaScript implementations for filtering and binned aggregation.
Install via NPM: npm install data-reduction
Require in your code: var dataReduction = require("data-reduction");
Here's an example that shows the filtering functionality:
var data1 = [
{ x: 1, y: 3 },
{ x: 5, y: 9 },
{ x: 9, y: 5 },
{ x: 4, y: 0 }
];
// Filters rows where x > 5
var result = dataReduction(data1, {
filter: [
{ column: "x", min: 5 }
]
});
console.log(result);
The following JSON is printed:
[
{ "x": 5, "y": 9 },
{ "x": 9, "y": 5 }
]
The following filter calls are also valid:
// Filters rows where x < 3
var result = dataReduction(data1, {
filter: [
{ column: "x", max: 3 }
]
});
// Use min and max together
var result = dataReduction(data1, {
filter: [
{ column: "x", min: 2, max: 6 }
]
});
// Use equal to match exact values.
var result = dataReduction(data1, {
filter: [
{ column: "x", equal: 5 }
]
});
Here's an example that does aggregation:
var data2 = [
{ foo: "A", bar: 1 },
{ foo: "A", bar: 8 },
{ foo: "A", bar: 6 }, // A sum = 15, count = 3
{ foo: "B", bar: 4 },
{ foo: "B", bar: 3 }, // B sum = 7, count = 2
{ foo: "C", bar: 6 },
{ foo: "C", bar: 1 },
{ foo: "C", bar: 3 },
{ foo: "C", bar: 6 },
{ foo: "C", bar: 4 } // C sum = 20, count = 5
];
var result = dataReduction(data2, {
aggregate: {
dimensions: [{
column: "foo"
}],
measures: [{
outColumn: "total",
operator: "count"
}]
}
});
console.log(result);
The following JSON is printed:
[
{ "foo": "A", "total": 3 },
{ "foo": "B", "total": 2 },
{ "foo": "C", "total": 5 }
]
Filter and aggregation can be specified together, in which case the filtering is applied first, then the aggregation.
For more examples see the unit tests.
The main goal of the this package is to unifying the aggregation computation behind visualizations including:
Also, this package provides the ability to dynamically filter the data before aggregation, which is a common need for linked interactive visualizations.
A frequency chart (bar chart or histogram) where the user can select which column of the data to represent with bars.
A heat map where the user can select two columns to aggregate by, each of which may be categorical, numeric, or temporal.
A calendar view where the user can select which column to sum to compute the value for each day.
Linked views where clicking on a bar in a bar chart will filter the data used as input to another bar chart.
Linked Choropleth Map and Line Chart
FAQs
A library for reducing the size of data sets for visualization.
We found that data-reduction demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Solo open source maintainers face burnout and security challenges, with 60% unpaid and 60% considering quitting.
Security News
License exceptions modify the terms of open source licenses, impacting how software can be used, modified, and distributed. Developers should be aware of the legal implications of these exceptions.
Security News
A developer is accusing Tencent of violating the GPL by modifying a Python utility and changing its license to BSD, highlighting the importance of copyleft compliance.