Product
Introducing SSO
Streamline your login process and enhance security by enabling Single Sign-On (SSO) on the Socket platform, now available for all customers on the Enterprise plan, supporting 20+ identity providers.
twitter2pg-cli
Advanced tools
Readme
Richard Wen
rrwen.dev@gmail.com
Command line tool for extracting Twitter data to PostgreSQL databases
npm
npm install -g twitter2pg-cli
For the latest developer version, see Developer Install.
Get help:
twitter2pg --help
Open documentation in web browser:
twitter2pg doc twitter2pg
twitter2pg doc twitter
twitter2pg doc pg
See twitter2pg for programmatic usage.
Create a template .env
file for Twitter and PostgreSQL details:
twitter2pg file path/to/.env
Set default for the .env
file:
twitter2pg
command will now use the designated .env
filetwitter2pg set file path/to/.env
Send a query to a PostgreSQL database after defining and setting the default Environment File.
The usage examples require a table named twitter_data
which can be created with the command below:
twitter2pg query "CREATE TABLE twitter_data(tweets jsonb);"
row | tweets |
---|---|
1 | {...} |
2 | {...} |
3 | {...} |
... | ... |
Setup default twitter options:
get
, post
, delete
or stream
)twitter2pg set twitter.method get
twitter2pg set twitter.path search/tweets
twitter2pg set twitter.params "{\"q\":\"twitter\"}"
Setup default PostgreSQL options:
twitter2pg set pg.table twitter_data
twitter2pg set pg.column tweets
twitter2pg set pg.query "INSERT INTO $options.pg.table($options.pg.column) SELECT * FROM json_array_elements($1);"
twitter2pg set jsonata statuses
Extract Twitter data into PostgreSQL table given setup options:
twitter2pg > log.csv
Setup default twitter options:
twitter2pg set twitter.method stream
twitter2pg set twitter.path statuses/filter
twitter2pg set twitter.params "{\"track\":\"twitter\"}"
Setup default PostgreSQL options:
twitter2pg set pg.table twitter_data
twitter2pg set pg.column tweets
twitter2pg set pg.query "INSERT INTO $options.pg.table($options.pg.column) VALUES($1);"
twitter2pg set jsonata statuses
Stream Twitter data into PostgreSQL table given setup options:
twitter2pg > log.csv
Stream Twitter data into a PostgreSQL table as a service:
npm install pm2 -g
)pm2
to run the saved script as a servicetwitter2pg save path/to/script.js
pm2 start path/to/script.js
pm2 save
The logs are in the following Comma-Separated Values (CSV) format:
time_iso8601
: Time and date in ISO 8601 formatstatus
: Status of the logmessage
: Relevant messagesjson
: JSON object containing relevant debugging informationtime_iso8601 | status | message | json |
---|---|---|---|
... | ... | ... | ... |
Reports for issues and suggestions can be made using the issue submission interface.
When possible, ensure that your submission is:
Code contributions are submitted via pull requests:
A template of the code contribution explanation is provided below:
## Purpose
The purpose can mention goals that include fixes to bugs, addition of features, and other improvements, etc.
## Description
The description is a short summary of the changes made such as improved speeds or features, and implementation details.
## Changes
The changes are a list of general edits made to the files and their respective components.
* `file_path1`:
* `function_module_etc`: changed loop to map
* `function_module_etc`: changed variable value
* `file_path2`:
* `function_module_etc`: changed loop to map
* `function_module_etc`: changed variable value
## Notes
The notes provide any additional text that do not fit into the above sections.
For more information, see Developer Install and Implementation.
Install the latest developer version with npm
from github:
npm install -g git+https://github.com/rrwen/twitter2pg-cli
Install from git
cloned source:
npm
git clone https://github.com/rrwen/twitter2pg-cli
cd twitter2pg-cli
npm -g install
git clone https://github.com/rrwen/twitter2pg-cli
cd twitter2pg-cli
.env
file (see tests/README.md)npm install
npm test
twitter2pg-cli
folder, add all files and commit changesgit add .
git commit -a -m "Generic update"
git push
package.json
npm test
npm login
npm publish
The module twitter2pg uses the following npm packages for its implementation:
npm | Purpose |
---|---|
yargs | Command line builder and parser |
twitter2pg | Extracts Twitter data to PostgreSQL |
dotenv | Load environmental variables from a file |
opn | Open online browser documentation |
pg | Send queries to PostgreSQL database |
yargs
|--- twitter2pg <-- default command
|--- dotenv <-- file
|--- opn <-- doc
|--- pg <-- query
FAQs
Command line tool for extracting Twitter data to PostgreSQL databases
The npm package twitter2pg-cli receives a total of 10 weekly downloads. As such, twitter2pg-cli popularity was classified as not popular.
We found that twitter2pg-cli demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Streamline your login process and enhance security by enabling Single Sign-On (SSO) on the Socket platform, now available for all customers on the Enterprise plan, supporting 20+ identity providers.
Security News
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
Security News
As cyber threats become more autonomous, AI-powered defenses are crucial for businesses to stay ahead of attackers who can exploit software vulnerabilities at scale.