bq-loader

bq_loader - Interactive Command Line Interface for Google BigQuery

Install

pip install bq_loader

Authentication

This package is built on top of Google libraries. To use this package you need to authenticate with your google account. For more information see: https://cloud.google.com/storage/docs/authentication.

You can use the package gcloud to authenticate yourself.

gcloud auth

Usage

Command Line Interface

bqloader

API

Create a table from a local file or directory

from bq_loader import create_table_from_local

 create_table_from_local(table_id='snapshot',
                         project_id='subugoe-collaborative',
                         dataset_id='resources',
                         file_path='test_data/*',
                         schema_file_path='test_schema/schema_crossref.json',
                         source_format='jsonl',
                         write_disposition='WRITE_APPEND',
                         table_description='Test Table generated by bq_loader',
                         ignore_unknown_values=True)

Create a table from a Google Bucket

from bq_loader import create_table_from_bucket

create_table_from_bucket(uri='gs://bigschol/tests/*',
                         table_id='bq_loader_test',
                         project_id='subugoe-collaborative',
                         dataset_id='resources',
                         schema_file_path='test_schema/schema_crossref.json',
                         source_format='jsonl',
                         write_disposition='WRITE_EMPTY',
                         table_description='Test Table generated by bq_loader',
                         ignore_unknown_values=True)

Upload local files to a Google Bucket

from bq_loader import upload_files_to_bucket

upload_files_to_bucket(bucket_name='bigschol',
                       file_path='test_data/*',
                       gcb_dir='tests')