Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

abstract-log

Package Overview
Dependencies
Maintainers
3
Versions
10
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

abstract-log

An abstract interface to build data processing applications on a log-based architecture.

  • 3.0.4
  • latest
  • Source
  • npm
  • Socket score

Version published
Weekly downloads
3
decreased by-72.73%
Maintainers
3
Weekly downloads
 
Created
Source

abstract-log

An abstract interface to build data processing applications on a log-based architecture.

build status

The idea would be to have implementations that would allow reading/writing to files, databases, kafka, redis, etc.

This is basically abstract-blob-store for append-only logs.

To understand all this append only log stuff and why it's useful to build data processing systems, watch this great talk by Martin Kleppman.

Basically Kafka is all good and well, but very heavy until you need that kind of scale. This project enables you to build on this kind of architecture and then scale up to bigger logging back-ends when you need to.

Usage

// There's a reference implementation of the memory based log included. Should be its own
// module
const memLog = require('abstract-log/mem-log');

// If this were a database log then you would pass through a connection string
let log = memLog();

// append to the log (currently the offset can be an object, not just an integer)
let offset = await log.append({ msg: 'hello world' });

// read from the log
let data = await log.get(offset);

// stream from the log
let offset = 0; // use 0 as initial offset as this is a memory log
log.createReadStream(offset)
   .on('data', console.log);

// stream to a log
const arrayToStream = require('array-to-stream');
arrayToStream([0, 1, 2, 3, 4].map((i) => ({ msg: `hello world ${i}` })))
  .pipe(log.createWriteStream());

abstract-log Implementations

  • mem-log - Reference implementation
  • fs-log - A very simple JSON text file log
  • knex-log - Uses the knex database library to log to postgres, sqlite, mysql, etc

TODO

NB: Please feel free to contribute implementations. PRs welcome!

  • levelup - log to leveldb
  • kafka - log to kafka
  • hyperlog - P2P distribute logs
  • S3 - batch writes and log to S3, or even an abstract-blob-store
  • redis - log to redis
  • ipfs - log to IPFS using scuttlebut logs for replication

Libraries helpful for implementations

Some logs don't have efficient push mechanisms for changes. So you have to poll for changes. Check out polling-stream which helps you create a perpetual stream of changes.

Docs

WIP. Currently - See tests.

Keywords

FAQs

Package last updated on 08 Jun 2017

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc