π GraphQL CSV Ingest
β¨ Features
π₯ Professional Data Pipeline in Your Terminal
- π Beautiful CLI: Fun ASCII art and colorful interface
- π Smart CSV Ingestion: Automatic schema detection and type inference
- π PostgreSQL Integration: Seamless database operations
- π GraphQL API: Modern, flexible data querying
- π FastAPI Server: High-performance async web server
- π― Multiple Entry Points: Multiple CLI commands for convenience
- π οΈ Developer Friendly: Type hints, comprehensive error handling
- π¦ Easy Installation: Available on PyPI with all dependencies
π Quick Start
Installation
pip install csv-graphql-cli
python3 -m venv venv
source venv/bin/activate
pip install csv-graphql-cli
git clone https://github.com/cjanowski/graphql-ingest.git
cd graphql-ingest
pip install -e ".[dev]"
Quick Start - Method 1: Using Installed Commands (Recommended)
After running pip install csv-graphql-cli
:
csvgql
csvgql init-db
csvgql ingest -f data/sample_data.csv -t employees
csvgql preview -t employees
csvgql serve
Quick Start - Method 2: Using Wrapper Script
Use the convenient wrapper script (csvgql.py
) if you prefer not to install:
cd graphql-ingest
python csvgql.py
python csvgql.py init-db
python csvgql.py ingest -f data/sample_data.csv -t employees
python csvgql.py preview -t employees
python csvgql.py serve
Quick Start - Method 3: Direct Module Usage
For development, you can also run directly from source:
cd graphql-ingest
python -m src.cli
python -m src.cli init-db
python -m src.cli ingest -f data/sample_data.csv -t employees
python -m src.cli serve
π¬ Demo
Demo video coming soon - showing the CLI interface in action!
π Example Workflow
csvgql
csvgql init-db
csvgql ingest -f data.csv -t users
csvgql preview -t users
csvgql serve
python csvgql.py
python csvgql.py init-db
python csvgql.py ingest -f data.csv -t users
python csvgql.py preview -t users
python csvgql.py serve
π§ CLI Commands
Method 1: Using Installed Package (Recommended)
After running pip install csv-graphql-cli
:
csvgql
csvgql init-db
csvgql ingest -f data.csv -t table
csvgql serve
csvgql preview -t table
csvgql tables
csvgql config-info
Method 2: Using the Wrapper Script
If you prefer not to install the package:
cd graphql-ingest
python csvgql.py
python csvgql.py init-db
python csvgql.py ingest -f data.csv -t table
python csvgql.py serve
python csvgql.py preview -t table
python csvgql.py tables
python csvgql.py config-info
Method 3: Direct Module Execution (Development)
python -m src.cli | Main CLI interface |
python -m src.cli init-db | Initialize database |
python -m src.cli ingest | Ingest CSV files |
python -m src.cli serve | Start GraphQL server |
python -m src.cli preview | Preview table data |
python -m src.cli tables | List available tables |
python -m src.cli config-info | Show current configuration |
All Available Commands Summary
PyPI Package | csvgql | Main CLI interface (shows all commands) |
PyPI Package | csvgql init-db | Initialize database |
PyPI Package | csvgql ingest | Ingest CSV files |
PyPI Package | csvgql serve | Start GraphQL server |
PyPI Package | csvgql preview | Preview table data |
PyPI Package | csvgql tables | List available tables |
PyPI Package | csvgql config-info | Show current configuration |
Development | python csvgql.py | Main CLI interface |
Development | python csvgql.py init-db | Initialize database |
Development | python -m src.cli | Main CLI interface |
Development | python -m src.cli init-db | Initialize database |
π GraphQL Queries
List Tables
{
tables {
name
columns {
name
type
}
}
}
Query Data
{
tableData(tableName: "employees", limit: 10) {
data
total
}
}
Ingest CSV via API
mutation {
ingestCsv(file: "new_data.csv", tableName: "products") {
success
message
rowsInserted
}
}
π οΈ Configuration
Create .env
file in the project root:
DB_HOST=localhost
DB_PORT=5432
DB_NAME=your_database
DB_USER=your_username
DB_PASSWORD=your_password
SERVER_HOST=0.0.0.0
SERVER_PORT=8000
DEBUG=false
You can also copy the example configuration:
cp config/env.example .env
π Project Structure
graphql-ingest/
βββ π¦ src/ # Main application code
β βββ cli.py # Command-line interface
β βββ config.py # Configuration management
β βββ database.py # Database operations
β βββ graphql_schema.py # GraphQL schema definition
β βββ server.py # FastAPI server
βββ π§ͺ tests/ # Test suite
β βββ test_basic.py # Basic functionality tests
β βββ integration_test_examples.py
βββ π examples/ # Usage examples
βββ π docs/ # Documentation
βββ π³ docker/ # Docker configuration
βββ π data/ # Sample data files
β βββ sample_data.csv # Test data
β βββ test_data_large.csv # Larger test dataset
βββ βοΈ config/ # Configuration files
β βββ env.example # Environment template
β βββ setup.cfg # Setup configuration
β βββ MANIFEST.in # Package manifest
βββ π οΈ tools/ # Build and development tools
β βββ setup.py # Package setup script
β βββ coverage.xml # Coverage reports
βββ π .github/ # GitHub workflows
βββ π requirements/ # Dependency management
βββ π CONTRIBUTE.md # Contribution guidelines
βββ π CHANGELOG.md # Change log
βββ π QUICKSTART.md # Quick start guide
βββ π README.md # Project overview
π§ͺ Testing
Setup Testing Environment
python3 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"
Run Tests
pytest
pytest --cov=src
pytest -v
pytest tests/test_basic.py
π³ Docker Support
docker-compose up
docker build -t csv-graphql-cli .
docker run -p 8000:8000 csv-graphql-cli
π Installation Options
Option 1: Install from PyPI (Recommended) π
pip install csv-graphql-cli
Option 2: Install in Virtual Environment
python3 -m venv venv
source venv/bin/activate
pip install csv-graphql-cli
Option 3: Development Setup
git clone https://github.com/cjanowski/graphql-ingest.git
cd graphql-ingest
pip install -e ".[dev]"
π Requirements
- Python 3.9 or higher
- PostgreSQL database
- Modern terminal with Unicode support
π€ Contributing
We welcome contributions! Please see our Contributing Guide for details.
π License
This project is licensed under the MIT License - see the LICENSE file for details.
π Acknowledgments
Made with β€οΈ and lots of πππ!