Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
Socket
Sign inDemoInstall
Socket
Sign inDemoInstall

@apache-arrow/ts

Package Overview
Dependencies
Maintainers
5
Versions
38
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

@apache-arrow/ts

Apache Arrow columnar in-memory format

  • 0.14.1
  • Source
  • npm
  • Socket score
Version published
Weekly downloads
12K
decreased by-11.72%
Maintainers
5
Weekly downloads
 
Created
Source

Apache Arrow in JS

npm version Build Status Coverage Status

Arrow is a set of technologies that enable big data systems to process and transfer data quickly.

Install apache-arrow from NPM

npm install apache-arrow or yarn add apache-arrow

(read about how we package apache-arrow below)

Powering Columnar In-Memory Analytics

Apache Arrow is a columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. The Arrow spec aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.

Apache Arrow is the emerging standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...). By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.

Get Started

Check out our API documentation to learn more about how to use Apache Arrow's JS implementation. You can also learn by example by checking out some of the following resources:

Cookbook

Get a table from an Arrow file on disk (in IPC format)

import { readFileSync } from 'fs';
import { Table } from 'apache-arrow';

const arrow = readFileSync('simple.arrow');
const table = Table.from([arrow]);

console.log(table.toString());

/*
 foo,  bar,  baz
   1,    1,   aa
null, null, null
   3, null, null
   4,    4,  bbb
   5,    5, cccc
*/

Create a Table when the Arrow file is split across buffers

import { readFileSync } from 'fs';
import { Table } from 'apache-arrow';

const table = Table.from([
    'latlong/schema.arrow',
    'latlong/records.arrow'
].map((file) => readFileSync(file)));

console.log(table.toString());

/*
        origin_lat,         origin_lon
35.393089294433594,  -97.6007308959961
35.393089294433594,  -97.6007308959961
35.393089294433594,  -97.6007308959961
29.533695220947266, -98.46977996826172
29.533695220947266, -98.46977996826172
*/

Create a Table from JavaScript arrays

import {
  Table,
  FloatVector,
  DateVector
} from 'apache-arrow';

const LENGTH = 2000;

const rainAmounts = Float32Array.from(
  { length: LENGTH },
  () => Number((Math.random() * 20).toFixed(1)));

const rainDates = Array.from(
  { length: LENGTH },
  (_, i) => new Date(Date.now() - 1000 * 60 * 60 * 24 * i));

const rainfall = Table.new(
  [FloatVector.from(rainAmounts), DateVector.from(rainDates)],
  ['precipitation', 'date']
);

Load data with fetch

import { Table } from "apache-arrow";

const table = await Table.from(fetch(("/simple.arrow")));
console.log(table.toString());

Columns look like JS Arrays

import { readFileSync } from 'fs';
import { Table } from 'apache-arrow';

const table = Table.from([
    'latlong/schema.arrow',
    'latlong/records.arrow'
].map(readFileSync));

const column = table.getColumn('origin_lat');

// Copy the data into a TypedArray
const typed = column.toArray();
assert(typed instanceof Float32Array);

for (let i = -1, n = column.length; ++i < n;) {
    assert(column.get(i) === typed[i]);
}

Usage with MapD Core

import MapD from 'rxjs-mapd';
import { Table } from 'apache-arrow';

const port = 9091;
const host = `localhost`;
const db = `mapd`;
const user = `mapd`;
const password = `HyperInteractive`;

MapD.open(host, port)
  .connect(db, user, password)
  .flatMap((session) =>
    // queryDF returns Arrow buffers
    session.queryDF(`
      SELECT origin_city
      FROM flights
      WHERE dest_city ILIKE 'dallas'
      LIMIT 5`
    ).disconnect()
  )
  .map(([schema, records]) =>
    // Create Arrow Table from results
    Table.from([schema, records]))
  .map((table) =>
    // Stringify the table to CSV with row numbers
    table.toString({ index: true }))
  .subscribe((csvStr) =>
    console.log(csvStr));
/*
Index,   origin_city
    0, Oklahoma City
    1, Oklahoma City
    2, Oklahoma City
    3,   San Antonio
    4,   San Antonio
*/

Getting involved

See DEVELOP.md

Even if you do not plan to contribute to Apache Arrow itself or Arrow integrations in other projects, we'd be happy to have you involved:

We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the github.com/apache/arrow repository.

If you are looking for some ideas on what to contribute, check out the JIRA issues for the Apache Arrow project. Comment on the issue and/or contact dev@arrow.apache.org with your questions and ideas.

If you’d like to report a bug but don’t have time to fix it, you can still post it on JIRA, or email the mailing list dev@arrow.apache.org

Packaging

apache-arrow is written in TypeScript, but the project is compiled to multiple JS versions and common module formats.

The base apache-arrow package includes all the compilation targets for convenience, but if you're conscientious about your node_modules footprint, we got you.

The targets are also published under the @apache-arrow namespace:

npm install apache-arrow # <-- combined es5/UMD, es2015/CommonJS/ESModules/UMD, and TypeScript package
npm install @apache-arrow/ts # standalone TypeScript package
npm install @apache-arrow/es5-cjs # standalone es5/CommonJS package
npm install @apache-arrow/es5-esm # standalone es5/ESModules package
npm install @apache-arrow/es5-umd # standalone es5/UMD package
npm install @apache-arrow/es2015-cjs # standalone es2015/CommonJS package
npm install @apache-arrow/es2015-esm # standalone es2015/ESModules package
npm install @apache-arrow/es2015-umd # standalone es2015/UMD package
npm install @apache-arrow/esnext-cjs # standalone esNext/CommonJS package
npm install @apache-arrow/esnext-esm # standalone esNext/ESModules package
npm install @apache-arrow/esnext-umd # standalone esNext/UMD package

Why we package like this

The JS community is a diverse group with a varied list of target environments and tool chains. Publishing multiple packages accommodates projects of all stripes.

If you think we missed a compilation target and it's a blocker for adoption, please open an issue.

People

Full list of broader Apache Arrow committers.

  • Brian Hulette, committer
  • Paul Taylor, Graphistry, Inc., committer

Powered By Apache Arrow in JS

Full list of broader Apache Arrow projects & organizations.

Open Source Projects

  • Apache Arrow -- Parent project for Powering Columnar In-Memory Analytics, including affiliated open source projects
  • rxjs-mapd -- A MapD Core node-driver that returns query results as Arrow columns
  • Perspective -- Perspective is a streaming data visualization engine by J.P. Morgan for JavaScript for building real-time & user-configurable analytics entirely in the browser.
  • Falcon is a visualization tool for linked interactions across multiple aggregate visualizations of millions or billions of records.

Companies & Organizations

  • CCRi -- Commonwealth Computer Research Inc, or CCRi, is a Central Virginia based data science and software engineering company
  • GOAI -- GPU Open Analytics Initiative standardizes on Arrow as part of creating common data frameworks that enable developers and statistical researchers to accelerate data science on GPUs
  • Graphistry, Inc. - An end-to-end GPU accelerated visual investigation platform used by teams for security, anti-fraud, and related investigations. Graphistry uses Arrow in its NodeJS GPU backend and client libraries, and is an early contributing member to GOAI and Arrow[JS] working to bring these technologies to the enterprise.

License

Apache 2.0

Keywords

FAQs

Package last updated on 23 Jul 2019

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

NVD Backlog Tops 20,000 CVEs Awaiting Analysis as NIST Prepares System Updates

Security News

NVD Backlog Tops 20,000 CVEs Awaiting Analysis as NIST Prepares System Updates

NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.

By Sarah Gooding  -  Nov 19, 2024
SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

About

Packages

npm

Go

Maven

PyPI

Rubygems

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc