ga-extractor

Google Analytics Extractor

A CLI tool for extracting Google Analytics data using Google Reporting API. Can be also used to transform data to various formats suitable for migration to other analytics platforms.

Setup

You will need Google Cloud API access for run the CLI:

• Navigate to Cloud Resource Manager and click Create Project

  • alternatively create project with gcloud projects create $PROJECT_ID

• Navigate to Reporting API and click Enable

• Create credentials:

  • Go to credentials page

  • Click Create credentials, select Service account

  • Give it a name and make note of service account email. Click Create and Continue

  • Open Service account page

  • Select previously created service account, Open Keys tab

  • Click Add Key and Create New Key. Choose JSON format and download it. (store this securely)

• Give SA permissions to GA - guide

  • email: SA email from earlier
  • role: Viewer

Alternatively see https://martinheinz.dev/blog/62.

To install and run:

pip install ga-extractor
ga-extractor --help

Running

ga-extractor --help
# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...

# Create config file:
ga-extractor setup \
  --sa-key-path="analytics-api-24102021-4edf0b7270c0.json" \
  --table-id="123456789" \
  --metrics="ga:sessions" \
  --dimensions="ga:browser" \
  --start-date="2022-03-15" \
  --end-date="2022-03-19"
  
cat ~/.config/ga-extractor/config.yaml  # Optionally, check config

ga-extractor auth  # Test authentication
# Successfully authenticated with user: ...

ga-extractor setup --help  # For options and flags

• Value for --table-id can be found in GA web console - Click on Admin section, View Settings and see View ID field
• All configurations and generated extracts/reports are stored in ~/.config/ga-extrator/...
• You can also use metrics and dimensions presets using --preset with FULL or BASIC, if you're not sure which data to extract

Extract

ga-extractor extract
# Report written to /home/some-user/.config/ga-extractor/report.json

extract perform raw extraction of dimensions and metrics using the provided configs

Migrate

You can directly extract and transform data to various formats. Available options are:

• JSON (Default option; Default API output)
• CSV
• SQL (compatible with Umami Analytics PostgreSQL backend)

ga-extractor migrate --format=CSV
# Report written to /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv

head /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv
# path,browser,os,device,screen,language,country,referral_path,count,date
# /,Chrome,Android,mobile,1370x1370,zh-cn,China,(direct),1,2022-03-18
# /,Chrome,Android,mobile,340x620,en-gb,United Kingdom,t.co/,1,2022-03-18

ga-extractor migrate --format=UMAMI
# Report written to /home/user/.config/ga-extractor/cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql

# IMPORTANT: Verify the data and check test database before inserting into production instance 
# To insert into DB (This should be run against clean database):
cat cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql | psql -Upostgres -a some-db

You can verify the data is correct in Umami web console and GA web console:

• Umami extract
• GA Pageviews

Note: Some data in GA and Umami web console might be little off, because GA displays many metrics based on sessions (e.g. Sessions by device), but data is extracted/migrated based on page views. You can however confirm that percentage breakdown of browser or OS usage does match.

Development

Setup

Requirements:

• Poetry (+ virtual environment)

poetry install
python -m ga_extractor --help

Testing

pytest

Building Package

poetry install
ga-extractor --help

# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...