pipelinewise-tap-twilio
Singer tap that extracts data from a Twilio API and produces JSON-formatted data following the Singer spec.
This is a PipelineWise compatible tap connector.
This tap:
-
Extracts the following resources:
-
Extracts TaskRouter resources:
-
Extracts Programmable Chat resources (members
and chat_messages
are FULL_TABLE
synced, so take care syncing them, they result in a lot of request/data):
-
Outputs the schema for each resource
-
Incrementally pulls data based on the input state
Streams
Standard Endpoints:
accounts
addresses
dependent_phone_numbers
applications
available_phone_number_countries
available_phone_numbers_local
available_phone_numbers_mobile
available_phone_numbers_toll_free
incoming_phone_numbers
keys
calls
conferences
conference_participants
outgoing_caller_ids
recordings
transcriptions
queues
message_media
usage_records
usage_triggers
alerts
TaskRouter Endpoints:
workspaces
activities
events
tasks
task_channels
task_queues
cumulative_statistics
workers
channels
workflows
Programmable Chat Endpoints:
services
roles
chat_channels
members
chat_messages
users
Authentication
This tap authenticates to the Twilio API using Basic Auth.
To set up authentication simply include your Twilio account_sid
and auth_token
in the tap config.
Quick Start
-
Install
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
-
Create your tap's config.json
file. The api_key
is available in the twilio Console UI (see Authentication above). The date_window_days
is the integer number of days (between the from and to dates) for date-windowing through the date-filtered endpoints (default = 30). The start_date
is the absolute beginning date from which incremental loading on the initial load will start.
{
"account_sid": "YOUR_ACCOUNT_SID",
"auth_token": "YOUR_AUTH_TOKEN",
"start_date": "2019-01-01T00:00:00Z",
"user_agent": "tap-twilio <api_user_email@your_company.com>",
}
Optionally, also create a state.json
file. currently_syncing
is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.
{
"currently_syncing": "registers",
"bookmarks": {
"acounts": "2020-03-23T10:31:14.000000Z",
"...": "2020-03-23T00:00:00.000000Z"
}
}
-
Run the Tap in Discovery Mode
This creates a catalog.json for selecting objects/fields to integrate:
tap-twilio --config config.json --discover > catalog.json
See the Singer docs on discovery mode
here.
-
Run the Tap in Sync Mode (with catalog)
For Sync mode:
> tap-twilio --config tap_config.json --catalog catalog.json
Messages are written to standard output following the Singer specification.
The resultant stream of JSON data can be consumed by a Singer target.
To run tests
- Install python test dependencies in a virtual env and run nose unit and integration tests
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install -e .[test]
- To run unit tests:
pytest tests/unit