avsc

What is avsc?

The avsc npm package is a library for encoding and decoding data in the Avro serialization format. It provides tools for working with Avro schemas, serializing and deserializing data, and performing schema evolution.

What are avsc's main functionalities?

Schema Definition

This feature allows you to define Avro schemas using JSON. The code sample demonstrates how to define a simple Avro schema for a record with 'name' and 'age' fields.

const avro = require('avsc');
const type = avro.Type.forSchema({
  type: 'record',
  fields: [
    {name: 'name', type: 'string'},
    {name: 'age', type: 'int'}
  ]
});

Serialization

This feature allows you to serialize JavaScript objects into Avro binary format. The code sample shows how to serialize an object with 'name' and 'age' fields into a buffer.

const avro = require('avsc');
const type = avro.Type.forSchema({
  type: 'record',
  fields: [
    {name: 'name', type: 'string'},
    {name: 'age', type: 'int'}
  ]
});
const buf = type.toBuffer({name: 'John Doe', age: 30});

Deserialization

This feature allows you to deserialize Avro binary data back into JavaScript objects. The code sample demonstrates how to deserialize a buffer back into an object.

const avro = require('avsc');
const type = avro.Type.forSchema({
  type: 'record',
  fields: [
    {name: 'name', type: 'string'},
    {name: 'age', type: 'int'}
  ]
});
const buf = type.toBuffer({name: 'John Doe', age: 30});
const obj = type.fromBuffer(buf);

Schema Evolution

This feature supports schema evolution, allowing you to read data written with an older schema using a newer schema. The code sample shows how to evolve a schema by adding a new field with a default value.

const avro = require('avsc');
const oldType = avro.Type.forSchema({
  type: 'record',
  fields: [
    {name: 'name', type: 'string'}
  ]
});
const newType = avro.Type.forSchema({
  type: 'record',
  fields: [
    {name: 'name', type: 'string'},
    {name: 'age', type: 'int', 'default': 0}
  ]
});
const buf = oldType.toBuffer({name: 'John Doe'});
const obj = newType.fromBuffer(buf);

Other packages similar to avsc

avro-js

avro-js is another library for working with Avro data in JavaScript. It provides similar functionality for schema definition, serialization, and deserialization. However, avro-js is generally considered to be less performant compared to avsc.

node-avro-io

node-avro-io is a library for Avro serialization and deserialization in Node.js. It offers similar features to avsc but is less actively maintained and has fewer features related to schema evolution.

README

Avsc

Pure JavaScript implementation of the Avro specification.

Features

• Fast! Typically twice as fast as JSON with much smaller encodings.
• Full Avro support, including recursive schemas, sort order, and evolution.
• Serialization of arbitrary JavaScript objects via logical types.
• Unopinionated 64-bit integer compatibility.
• No dependencies, avsc even runs in the browser.

Performance

Representative throughput rates (higher is better):

Libraries compared:

• node-avsc, this package.
• node-json, built-in JSON serializer.
• node-protobuf, most popular Protocol Buffers implementation.
• node-pson, alternative to JSON.
• node-msgpack, official MessagePack implementation.

These rates are for processing a realistic record schema, modeled after a popular open-source API. You can find the raw numbers and more details on the benchmarks page.

Installation

$ npm install avsc

avsc is compatible with all versions of node.js since 0.11 and major browsers via browserify.

Documentation

• Overview
• API
• Advanced usage

Examples

Inside a node.js module, or using browserify:

var avsc = require('avsc');

• Encode and decode objects:

  // We can declare a schema inline:
  var type = avsc.parse({
    name: 'Pet',
    type: 'record',
    fields: [
      {name: 'kind', type: {name: 'Kind', type: 'enum', symbols: ['CAT', 'DOG']}},
      {name: 'name', type: 'string'}
    ]
  });
  var pet = {kind: 'CAT', name: 'Albert'};
  var buf = type.toBuffer(pet); // Serialized object.
  var obj = type.fromBuffer(buf); // {kind: 'CAT', name: 'Albert'}
  

• Generate random instances of a schema:

  // We can also parse a JSON-stringified schema:
  var type = avsc.parse('{"type": "fixed", "name": "Id", "size": 4}');
  var id = type.random(); // E.g. Buffer([48, 152, 2, 123])
  

• Check whether an object fits a given schema:

  // Or we can specify a path to a schema file (not in the browser):
  var type = avsc.parse('./Person.avsc');
  var person = {name: 'Bob', address: {city: 'Cambridge', zip: '02139'}};
  var status = type.isValid(person); // Boolean status.
  

• Get a readable stream of decoded records from an Avro container file (not in the browser):

  avsc.createFileDecoder('./records.avro')
    .on('metadata', function (type) { /* `type` is the writer's type. */ })
    .on('data', function (record) { /* Do something with the record. */ });