Richard Wen
rrwen.dev@gmail.com
Module for extracting Twitter data to MongoDB databases
Install
- Install Node.js
- Install twitter2mongodb via
npm
- Recommended: Install dotenv
npm install --save twitter2mongodb
npm install --save dotenv
For the latest developer version, see Developer Install.
Usage
It is recommended to use a .env
file at the root of your project directory with the following contents:
- Twitter keys can be obtained from https://apps.twitter.com/
TWITTER_CONSUMER_KEY
: Consumer key (API Key)TWITTER_CONSUMER_SECRET
: Consumer secret (API secret) TWITTER_ACCESS_TOKEN_KEY
: Access tokenTWITTER_ACCESS_TOKEN_SECRET
: Access token secretMONGODB_CONNECTION
: MongoDB connection stringMONGODB_DATABASE
: MongoDB database nameMONGODB_COLLECTION
: MongoDB collection name
TWITTER_CONSUMER_KEY=***
TWITTER_CONSUMER_SECRET=***
TWITTER_ACCESS_TOKEN_KEY=***
TWITTER_ACCESS_TOKEN_SECRET=***
MONGODB_CONNECTION=mongodb://localhost:27017
MONGODB_DATABASE=test
MONGODB_COLLECTION=twitter_data
The .env
file above can be loaded using dotenv:
require('dotenv').config();
See Documentation for more details.
REST API
- Load
.env
file variables - Load
twitter2mongodb
- Create
options
object - Optionally define Twitter API keys and MongoDB connection
- Search keyword
twitter
from GET search/tweets
- Apply a
jsonata
filter for statuses key only - Execute
twitter2mongodb
with the REST API options
require('dotenv').config();
var twitter2mongodb = require('twitter2mongodb');
var options = {
twitter: {},
mongodb: {}
};
options.twitter.method = 'get';
options.twitter.path = 'search/tweets';
options.twitter.params = {q: 'twitter'};
options.jsonata = 'statuses';
options.mongodb.method = 'insertMany';
twitter2mongodb(options)
.then(data => {
console.log(data);
}).catch(err => {
console.error(err.message);
});
Stream API
- Load
.env
file variables - Load
twitter2mongodb
- Create
options
object - Optionally define Twitter API keys and MongoDB connection
- Track keyword
twitter
from POST statuses/filter
- Log the
tweets
when they are received - Execute
twitter2return
with the Stream API options
require('dotenv').config();
var twitter2mongodb = require('twitter2mongodb');
var options = {
twitter: {},
mongodb: {}
};
options.twitter.method = 'stream';
options.twitter.path = 'statuses/filter';
options.twitter.params = {track: 'twitter'};
options.mongodb.method = 'insertOne';
options.mongodb.check = function(tweets) {return(true)};
var stream = twitter2mongodb(options);
stream.on('error', function(error) {
console.error(error.message);
});
Contributions
- Reports for issues and suggestions can be made using the issue submission interface.
- Code contributions are submitted via pull requests
See CONTRIBUTING.md for more details.
Developer Notes
Developer Install
Install the latest developer version with npm
from github:
npm install git+https://github.com/rrwen/twitter2mongodb
Install from git
cloned source:
- Ensure git is installed
- Clone into current path
- Install via
npm
git clone https://github.com/rrwen/twitter2mongodb
cd twitter2mongodb
npm install
Tests
- Clone into current path
git clone https://github.com/rrwen/twitter2mongodb
- Enter into folder
cd twitter2mongodb
- Ensure devDependencies are installed and available
- Run tests with a
.env
file (see tests/README.md) - Results are saved to tests/log with each file corresponding to a version tested
npm install
npm test
Documentation
Use documentationjs to generate html documentation in the docs
folder:
npm run docs
See JSDoc style for formatting syntax.
Upload to Github
- Ensure git is installed
- Inside the
twitter2mongodb
folder, add all files and commit changes - Push to github
git add .
git commit -a -m "Generic update"
git push
Upload to npm
- Update the version in
package.json
- Run tests and check for OK status (see tests/README.md)
- Generate documentation
- Login to npm
- Publish to npm
npm test
npm run docs
npm login
npm publish
Implementation
The module twitter2mongodb uses the following npm packages for its implementation:
npm | Purpose |
---|
twitter2return | Connections to the Twitter API REST and Streaming Application Programming Interfaces (APIs) using twitter, and Filters with jsonata before inserting into MongoDB |
mongodb | Insert Twitter data to MongoDB collections |
twitter2return <-- Extract Twitter data from API and Filter JSON data
|
mongodb <-- Insert filtered Twitter data into MongoDB collection