Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

twitter2mongodb

Package Overview
Dependencies
Maintainers
1
Versions
3
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

twitter2mongodb

Module for extracting Twitter data to MongoDB databases

  • 1.0.1
  • latest
  • Source
  • npm
  • Socket score

Version published
Weekly downloads
3
decreased by-25%
Maintainers
1
Weekly downloads
 
Created
Source

twitter2mongodb

Richard Wen
rrwen.dev@gmail.com

Module for extracting Twitter data to MongoDB databases

npm version Build Status Coverage Status npm GitHub license Donarbox Donate PayPal Donate Twitter

Install

  1. Install Node.js
  2. Install twitter2mongodb via npm
  3. Recommended: Install dotenv
npm install --save twitter2mongodb
npm install --save dotenv

For the latest developer version, see Developer Install.

Usage

It is recommended to use a .env file at the root of your project directory with the following contents:

  • Twitter keys can be obtained from https://apps.twitter.com/
  • TWITTER_CONSUMER_KEY: Consumer key (API Key)
  • TWITTER_CONSUMER_SECRET: Consumer secret (API secret)
  • TWITTER_ACCESS_TOKEN_KEY: Access token
  • TWITTER_ACCESS_TOKEN_SECRET: Access token secret
  • MONGODB_CONNECTION: MongoDB connection string
  • MONGODB_DATABASE: MongoDB database name
  • MONGODB_COLLECTION: MongoDB collection name
TWITTER_CONSUMER_KEY=***
TWITTER_CONSUMER_SECRET=***
TWITTER_ACCESS_TOKEN_KEY=***
TWITTER_ACCESS_TOKEN_SECRET=***
MONGODB_CONNECTION=mongodb://localhost:27017
MONGODB_DATABASE=test
MONGODB_COLLECTION=twitter_data

The .env file above can be loaded using dotenv:

require('dotenv').config();

See Documentation for more details.

REST API

  1. Load .env file variables
  2. Load twitter2mongodb
  3. Create options object
  4. Optionally define Twitter API keys and MongoDB connection
  5. Search keyword twitter from GET search/tweets
  6. Apply a jsonata filter for statuses key only
  7. Execute twitter2mongodb with the REST API options
require('dotenv').config();

var twitter2mongodb = require('twitter2mongodb');

// (options) Initialize options object
var options = {
	twitter: {},
	mongodb: {}
};

// (options_twitter_rest) Search for keyword 'twitter' in path 'GET search/tweets'
options.twitter.method = 'get'; // get, post, or stream
options.twitter.path = 'search/tweets'; // api path
options.twitter.params = {q: 'twitter'}; // query tweets

// (options_jsonata) Filter for statuses array using jsonata
options.jsonata = 'statuses';

// (options_mongodb) MongoDB options
options.mongodb.method = 'insertMany'; // insert many objects due to array

// (twitter2mongodb_rest) Query tweets using REST API into MongoDB collection
twitter2mongodb(options)
	.then(data => {
		console.log(data);
	}).catch(err => {
		console.error(err.message);
	});

Stream API

  1. Load .env file variables
  2. Load twitter2mongodb
  3. Create options object
  4. Optionally define Twitter API keys and MongoDB connection
  5. Track keyword twitter from POST statuses/filter
  6. Log the tweets when they are received
  7. Execute twitter2return with the Stream API options
require('dotenv').config();

var twitter2mongodb = require('twitter2mongodb');

// (options) Initialize options object
var options = {
	twitter: {},
	mongodb: {}
};

// (options_twitter_connection) Track keyword 'twitter' in path 'POST statuses/filter'
options.twitter.method = 'stream'; // get, post, or stream
options.twitter.path = 'statuses/filter'; // api path
options.twitter.params = {track: 'twitter'}; // query tweets

// (options_mongodb) MongoDB options
options.mongodb.method = 'insertOne';
options.mongodb.check = function(tweets) {return(true)}; // only insert if returns true

// (twitter2mongodb_stream) Stream tweets into MongoDB collection
var stream = twitter2mongodb(options);
stream.on('error', function(error) {
	console.error(error.message);
});

Contributions

  1. Reports for issues and suggestions can be made using the issue submission interface.
  2. Code contributions are submitted via pull requests

See CONTRIBUTING.md for more details.

Developer Notes

Developer Install

Install the latest developer version with npm from github:

npm install git+https://github.com/rrwen/twitter2mongodb

Install from git cloned source:

  1. Ensure git is installed
  2. Clone into current path
  3. Install via npm
git clone https://github.com/rrwen/twitter2mongodb
cd twitter2mongodb
npm install

Tests

  1. Clone into current path git clone https://github.com/rrwen/twitter2mongodb
  2. Enter into folder cd twitter2mongodb
  3. Ensure devDependencies are installed and available
  4. Run tests with a .env file (see tests/README.md)
  5. Results are saved to tests/log with each file corresponding to a version tested
npm install
npm test

Documentation

Use documentationjs to generate html documentation in the docs folder:

npm run docs

See JSDoc style for formatting syntax.

Upload to Github

  1. Ensure git is installed
  2. Inside the twitter2mongodb folder, add all files and commit changes
  3. Push to github
git add .
git commit -a -m "Generic update"
git push

Upload to npm

  1. Update the version in package.json
  2. Run tests and check for OK status (see tests/README.md)
  3. Generate documentation
  4. Login to npm
  5. Publish to npm
npm test
npm run docs
npm login
npm publish

Implementation

The module twitter2mongodb uses the following npm packages for its implementation:

npmPurpose
twitter2returnConnections to the Twitter API REST and Streaming Application Programming Interfaces (APIs) using twitter, and Filters with jsonata before inserting into MongoDB
mongodbInsert Twitter data to MongoDB collections
twitter2return   <-- Extract Twitter data from API and Filter JSON data
      |
   mongodb       <-- Insert filtered Twitter data into MongoDB collection

Keywords

FAQs

Package last updated on 22 Jan 2018

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc