evolvishub-outlook-ingestor
Advanced tools
Enterprise-ready email ingestion system with unified database connector architecture supporting 8 database types with equal implementation quality
Enterprise-ready email ingestion library with unified database connector architecture.
A streamlined Python library specifically designed for ingesting emails from Microsoft Outlook using Microsoft Graph API. Built as a pure data ingestion library that can be easily integrated into other applications and microservices. Now featuring standardized database connectors with enterprise-grade consistency across all 8 supported database types.
import asyncio
from evolvishub_outlook_ingestor import EmailIngestor, ingest_emails_simple
# Simple usage - minimal configuration
async def simple_example():
result = await ingest_emails_simple(
client_id="your-client-id",
client_secret="your-client-secret",
tenant_id="your-tenant-id",
output_format="json"
)
print(f"Processed {result['processed_emails']} emails")
# Advanced usage - full control
async def advanced_example():
from evolvishub_outlook_ingestor import Settings, IngestionConfig
from evolvishub_outlook_ingestor.adapters.microsoft_graph import MicrosoftGraphAdapter
# Setup
settings = Settings()
settings.graph_api.client_id = "your-client-id"
settings.graph_api.client_secret = "your-client-secret"
settings.graph_api.tenant_id = "your-tenant-id"
adapter = MicrosoftGraphAdapter(settings)
await adapter.initialize()
# Configure ingestion
config = IngestionConfig(
batch_size=100,
include_attachments=True,
progress_callback=lambda p, t: print(f"Progress: {p}/{t}")
)
# Ingest emails
ingestor = EmailIngestor(settings=settings, graph_adapter=adapter)
await ingestor.initialize(config)
result = await ingestor.ingest_emails(
folder_ids=["inbox", "sent"],
output_format="database"
)
print(f"Ingestion completed: {result.processed_emails} emails")
# Run examples
asyncio.run(simple_example())
This library is now focused exclusively on email ingestion. We've removed all non-email functionality (calendar, contacts, etc.) to create a streamlined, reliable tool that does one thing exceptionally well.
All 8 supported database types now have enterprise-grade consistency! We've eliminated architectural bias by standardizing all database connectors to use the unified DatabaseConnector interface, providing equal implementation quality and features across all database types.
# Basic installation (email ingestion only)
pip install evolvishub-outlook-ingestor
# With all database connectors (8 databases supported)
pip install 'evolvishub-outlook-ingestor[database]'
# Individual database connectors
pip install 'evolvishub-outlook-ingestor[postgresql]' # PostgreSQL
pip install 'evolvishub-outlook-ingestor[mongodb]' # MongoDB
pip install 'evolvishub-outlook-ingestor[sqlite]' # SQLite
pip install 'evolvishub-outlook-ingestor[cockroachdb]' # CockroachDB
pip install 'evolvishub-outlook-ingestor[mariadb]' # MariaDB
pip install 'evolvishub-outlook-ingestor[mssql]' # MS SQL Server
pip install 'evolvishub-outlook-ingestor[oracle]' # Oracle Database
pip install 'evolvishub-outlook-ingestor[clickhouse]' # ClickHouse
# Development installation
pip install 'evolvishub-outlook-ingestor[dev]'
All 8 supported database types now implement the same standardized DatabaseConnector interface, eliminating architectural bias and providing enterprise-grade consistency:
from evolvishub_outlook_ingestor.connectors.database_connector import create_database_connector, DatabaseConfig
# Same interface for all 8 database types!
config = DatabaseConfig(
database_type="postgresql", # or "mongodb", "sqlite", "cockroachdb",
# "mariadb", "mssql", "oracle", "clickhouse"
host="localhost",
database="emails",
username="user",
password="password"
)
# Factory function creates the appropriate connector
connector = create_database_connector(config)
# All connectors support the same methods
await connector.connect()
await connector.create_schema()
await connector.store_email_batch(emails)
count = await connector.get_total_email_count()
await connector.disconnect()
| Feature | All 8 Databases |
|---|---|
| Async Operations | ✅ Full async/await support |
| Connection Pooling | ✅ High-performance connection pools |
| Batch Processing | ✅ Optimized batch operations |
| Error Handling | ✅ Comprehensive exception management |
| Security | ✅ Credential encryption, secure connections |
| Monitoring | ✅ Structured logging and metrics |
| Database-Specific Optimizations | ✅ MERGE, UPSERT, columnar operations |
The following table provides a comprehensive overview of all supported components, connectors, and features:
| Component | Type | Status | Key Features |
|---|---|---|---|
| PostgreSQL | Database | ✅ Standardized | DatabaseConnector interface, async operations, connection pooling, ACID compliance |
| MongoDB | Database | ✅ Standardized | DatabaseConnector interface, Motor async driver, GridFS support, replica sets |
| SQLite | Database | ✅ Standardized | DatabaseConnector interface, zero-config setup, file-based storage, ACID properties |
| CockroachDB | Database | ✅ Standardized | DatabaseConnector interface, distributed SQL, UPSERT operations, multi-region support |
| MariaDB | Database | ✅ Standardized | DatabaseConnector interface, MySQL compatibility, ON DUPLICATE KEY UPDATE, clustering |
| Microsoft SQL Server | Database | ✅ Standardized | DatabaseConnector interface, MERGE statements, enterprise security, Always Encrypted |
| Oracle Database | Database | ✅ Standardized | DatabaseConnector interface, enterprise MERGE, JSON support, high availability |
| ClickHouse | Database | ✅ Standardized | DatabaseConnector interface, columnar storage, analytics optimizations, horizontal scaling |
| AWS S3 | Storage | Production Ready | Unlimited scalability, multiple storage classes, server-side encryption, AWS ecosystem |
| Azure Blob Storage | Storage | Production Ready | Multi-tier storage, Azure AD integration, geo-redundancy, threat protection |
| Google Cloud Storage | Storage | Production Ready | Multi-regional options, lifecycle management, GCP AI integration, strong consistency |
| MinIO | Storage | Production Ready | S3-compatible, high-performance, Kubernetes-native, multi-cloud gateway |
| Delta Lake | Storage | Production Ready | ACID transactions, schema evolution, time travel, Spark integration |
| Apache Iceberg | Storage | Production Ready | Schema evolution, hidden partitioning, time travel, multi-engine compatibility |
| Real-time Email Streaming | Streaming | Production Ready | Redis pub/sub, low-latency delivery, pattern subscriptions, auto-failover |
| Kafka Integration | Streaming | Production Ready | High-throughput messaging, exactly-once semantics, stream processing, multi-datacenter |
| Change Data Capture (CDC) | Streaming | Production Ready | Real-time change detection, event sourcing, conflict resolution, lineage tracking |
| Event-driven Architecture | Streaming | Production Ready | Event sourcing patterns, CQRS, saga pattern, event replay |
| Analytics Engine | Processing | Production Ready | Communication analysis, network mapping, trend detection, BI dashboards |
| ML Service | Processing | Production Ready | Email classification (95%+ accuracy), spam detection, priority prediction, sentiment analysis |
| Data Quality Validator | Processing | Production Ready | Anomaly detection, completeness checks, duplicate detection, quality scoring |
| NLP Processor | Processing | Production Ready | Multi-language analysis, NER, sentiment detection, topic modeling, text summarization |
| Intelligent Caching | Processing | Production Ready | Multi-level caching (LRU/LFU/TTL), predictive warming, distributed sync |
| Data Governance | Governance | Production Ready | GDPR/CCPA compliance, lineage tracking, automated validation, privacy assessments |
| Multi-tenant Management | Governance | Production Ready | Tenant isolation, resource quotas, RBAC, audit logging |
| Advanced Monitoring | Monitoring | Production Ready | Prometheus metrics, Grafana dashboards, distributed tracing, APM |
| Security & Compliance | Security | Production Ready | End-to-end encryption, OAuth 2.0/OIDC, certificate auth, audit trails |
All components are designed for:
from evolvishub_outlook_ingestor import Settings
from evolvishub_outlook_ingestor.connectors.database_connector import DatabaseConfig
settings = Settings()
# Unified database configuration (works with all 8 database types!)
database_config = DatabaseConfig(
database_type="postgresql", # or any of the 8 supported types
host="localhost",
port=5432,
database="outlook_emails",
username="user",
password="password",
table_name="emails",
batch_size=100,
max_connections=10
)
# Microsoft Graph API
settings.protocols.graph.client_id = "your-client-id"
settings.protocols.graph.client_secret = "your-client-secret"
settings.protocols.graph.tenant_id = "your-tenant-id"
# PostgreSQL
postgresql_config = DatabaseConfig(
database_type="postgresql",
host="localhost",
port=5432,
database="emails"
)
# MongoDB
mongodb_config = DatabaseConfig(
database_type="mongodb",
host="localhost",
port=27017,
database="emails"
)
# CockroachDB
cockroachdb_config = DatabaseConfig(
database_type="cockroachdb",
host="localhost",
port=26257,
database="emails",
sslmode="require"
)
# ClickHouse
clickhouse_config = DatabaseConfig(
database_type="clickhouse",
host="localhost",
port=8123,
database="emails",
secure=True,
compression=True
)
# MS SQL Server
mssql_config = DatabaseConfig(
database_type="mssql",
host="localhost",
port=1433,
database="emails",
encrypt=True,
trust_server_certificate=False
)
# Enable advanced features
settings.enable_analytics = True
settings.enable_ml = True
settings.enable_governance = True
settings.enable_monitoring = True
# Streaming configuration
settings.streaming.backend = "redis"
settings.streaming.redis_url = "redis://localhost:6379"
# ML configuration
settings.ml.enable_spam_detection = True
settings.ml.enable_classification = True
settings.ml.enable_priority_prediction = True
# Governance configuration
settings.governance.enable_compliance_monitoring = True
settings.governance.enable_retention_policies = True
settings.governance.enable_lineage_tracking = True
import asyncio
from evolvishub_outlook_ingestor import (
OutlookIngestor,
AdvancedMonitoringService,
IntelligentCacheManager,
MLService,
DataQualityValidator,
AnalyticsEngine,
GovernanceService,
Settings
)
async def advanced_pipeline():
settings = Settings()
# Initialize core ingestor
ingestor = OutlookIngestor(settings)
# Initialize advanced services
monitoring = AdvancedMonitoringService({'enable_tracing': True})
cache = IntelligentCacheManager({'backend': 'memory'})
ml_service = MLService({'enable_spam_detection': True})
quality_validator = DataQualityValidator({'enable_duplicate_detection': True})
analytics = AnalyticsEngine({'enable_communication_analysis': True})
governance = GovernanceService({'enable_compliance_monitoring': True})
# Initialize all services
await monitoring.initialize()
await cache.initialize()
await ml_service.initialize()
await quality_validator.initialize()
await analytics.initialize()
await governance.initialize()
print("All services initialized successfully!")
print("Advanced email processing pipeline ready")
# Cleanup
await monitoring.shutdown()
await cache.shutdown()
await ml_service.shutdown()
await quality_validator.shutdown()
await analytics.shutdown()
await governance.shutdown()
asyncio.run(advanced_pipeline())
| Configuration | Emails/Minute | Memory Usage | Notes |
|---|---|---|---|
| Basic Processing | 500-1000 | 128MB | Core ingestion with optimizations |
| With Database Storage | 800-1500 | 256MB | PostgreSQL/MongoDB with connection pooling |
| With Redis Caching | 1200-2000 | 384MB | Intelligent caching enabled |
| Full ML Pipeline | 600-1200 | 512MB | Complete ML classification and analysis |
| Enterprise Setup | 1500-3000 | 1GB | All features with monitoring and governance |
| Feature | Status | Performance | Notes |
|---|---|---|---|
| Real-time Streaming | Production Ready | 2000+ emails/min | Redis + Kafka support |
| ML Classification | Production Ready | 1000+ emails/min | Full sklearn/spacy pipeline |
| Analytics Engine | Production Ready | Real-time insights | Complete communication analysis |
| Intelligent Caching | Production Ready | 95%+ hit rate | Multi-level LRU/LFU/TTL strategies |
| Data Governance | Production Ready | Full compliance | GDPR/CCPA monitoring and reporting |
This project is licensed under the Evolvis AI License - see the LICENSE file for details.
For support, please contact Montgomery Miralles m.miralles@evolvis.ai or visit our documentation.
FAQs
Enterprise-ready email ingestion system with unified database connector architecture supporting 8 database types with equal implementation quality
We found that evolvishub-outlook-ingestor demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
/Security News
A malicious Chrome extension posing as an Ethereum wallet steals seed phrases by encoding them into Sui transactions, enabling full wallet takeover.
Security News
Socket is heading to London! Stop by our booth or schedule a meeting to see what we've been working on.
Security News
OWASP’s 2025 Top 10 introduces Software Supply Chain Failures as a new category, reflecting rising concern over dependency and build system risks.