Cher
Quick Start
Prep your environment
$ echo "APP_NAME=appName" >> .env
$ echo "DATABASE_URL=postgres://user:password@hostname:port/database" >> .env
$ echo "REDIS_URL=redis://hostname:port/" >> .env
$ echo "ADMIN_USER=foo" >> .env
$ echo "ADMIN_PASS=pa$$word" >> .env
$ echo "NODE_ENV=local" >> .env
$ echo "CALLBACK_URL=http://localhost:3000" >> .env
$ echo "SECRET=secretkey" >> .env
Ensure your database and user is created
createdb dbname
createuser dbuser
Start it up
foreman start
Meanwhile, in your code
const Cher = require('cher');
const cherTwitter = require('cher-twitter');
const cherInstagram = require('cher-instagram');
const tools = require('cher-tools');
const dirtyFilter = tools.pipeline.profanityFilter;
const save = tools.pipeline.save;
const config = tools.utils.config;
const sourceStreams = [
{
name: 'twitter',
module: cherTwitter
},
{
name: 'instagram',
module: cherInstagram
}
];
config.fetch({
pipeline: [ dirtyFilter, save ],
sourceStreams: sourceStreams
})
.then(Cher.construct)
.spread((streams, cher) => {
const twitterSource = cher.getStream('twitter');
twitterSource
.start()
.spread((dataChannel, twitterSettings) => {
dataChannel.on('data', (data) => {
console.log(data)
});
});
twitterSource.on('start', (dataChannel, twitterSettings) => {
});
});
Pipelines
Cher is a stream processing platform. The best way to process a stream of data is through a data pipeline. Cher provides several pipeline steps out of the box for you to utilize including a save()
which simply saves data to the database. In order to create a step in the pipeline just create a function which returns a through stream. The function which returns the through stream will receive one parameter deps
which is an object that provides several components that you can utilize in your pipeline step.
{
models,
ctx,
settings
}
models
An instance of the cher model manager which gives you access to your underlyning data model. You can access it like so
const DataModel = models.get('DataModel');
ctx
An instance of a key/value store that cher has access to which you can use to store arbitrary data.
ctx.set('foo', {bar: 'beep'})
.then(ctx.get('foo'))
.then((foo) {
console.log(foo);
})
settings
The configured settings for the current sourceStream. If the data item coming through the pipeline is from twitter, this will be the twitter settings. If it is from S3, it will be the S3 settings.
Putting it all together
So, what does a custom pipeline step look like? Simple
const es = require('event-stream');
const tools = require('cher-tools');
module.exports = (deps) => {
return es.through((data) {
let item = utils.parse(data);
log.debug(data);
const serialized = utils.serialize(item);
this.emit('data', serialize);
});
};
Related Repositories
cher db
cher twitter
cher instagram
cher facebook
cher tools
cher source interface
dionne