insyt-secure

Insyt Secure

A proprietary service for securely interacting with databases, APIs, and other services within a secure environment. Visit https://insyt.co for more information.

Quick Start

Installation

# Basic installation (core functionality only)
pip install insyt-secure

Installing on a Fresh Ubuntu Instance

Follow these steps to set up insyt-secure on a fresh Ubuntu installation:

# Step 1: Install required packages
sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv python3-full

# Step 2: Create a directory for your project (optional)
mkdir myproject
cd myproject

# Step 3: Create the virtual environment
python3 -m venv venv

# Step 4: Activate the virtual environment
source venv/bin/activate
# Your command prompt should change to show the virtual environment name
# Example: (venv) ubuntu@ip-100-1-11-111:~/myproject$

# Step 5: Install insyt-secure with desired extensions
pip install insyt-secure[mysql,postgres]
pip install python-dotenv

Notes:

• To run the service, use the following command:

insyt-secure --projects "your-project-id-123:your-api-key-xyz"

or run the service in the background:

nohup insyt-secure --projects "your-project-id-123:your-api-key-xyz" &

• Remember to activate the virtual environment each time you want to use this package in a new terminal session.

source venv/bin/activate

• To deactivate the virtual environment when you're done:

deactivate

• To stop the service, use the following command:

pkill -f insyt-secure

• To check the status of the service, use the following command:

ps aux | grep insyt-secure

Advanced Installation Options

Insyt Secure provides flexible installation options to include only the dependencies you need:

# Install with PostgreSQL support
pip install "insyt-secure[postgres]"

# Install with MongoDB support
pip install "insyt-secure[mongodb]"

# Install with multiple database support
pip install "insyt-secure[postgres,mongodb,redis]"

# Install with vector database support
pip install "insyt-secure[pinecone]" # or any other vector DB

# Install with cloud provider support
pip install "insyt-secure[aws]" # or azure

# Install with messaging system support
pip install "insyt-secure[kafka]" # or rabbitmq, pulsar

# Complete installation with all dependencies
pip install "insyt-secure[all]"

Available extension categories:

• SQL databases: postgres, mysql, mssql, oracle, clickhouse, snowflake, duckdb
• NoSQL databases: mongodb, redis, cassandra, neo4j, elasticsearch, couchdb
• Vector databases: pinecone, qdrant, milvus, weaviate, chroma, faiss
• Cloud services: aws, azure
• Messaging systems: kafka, pulsar, rabbitmq

Broader categories are also available: rdbms, nosql, vector, cloud, messaging

Basic Usage

# Run with a single project
insyt-secure --projects "your-project-id-123:your-api-key-xyz"

Multi-Project Support

Insyt Secure supports managing one or more projects simultaneously:

# Run with multiple projects
insyt-secure --projects "project-id-1:api-key-1,project-id-2:api-key-2"

Each project connection is managed independently, allowing for simultaneous interactions with multiple Insyt projects, each with its own topic subscriptions and credential management.

Getting Help

# View all available options and examples
insyt-secure --help

The help command provides detailed information about all parameters, their defaults, and usage examples directly in your terminal.

Advanced Options

# Run with all options (for a single project)
insyt-secure \
  --projects "your-project-id-123:your-api-key-xyz" \
  --max-workers 10 \
  --execution-timeout 60 \
  --allowed-ips "192.168.1.1,10.0.0.1:3456"

Logging Options

By default, logs are user-friendly and redact sensitive information. You can customize logging behavior:

# Enable more detailed debug logs
insyt-secure --projects "your-project-id:your-api-key" --debug

# Show more verbose logs including from third-party libraries
insyt-secure --projects "your-project-id:your-api-key" --verbose

# Output logs in JSON format (useful for log processing systems)
insyt-secure --projects "your-project-id:your-api-key" --json-logs

# Disable redaction of sensitive information (not recommended for production)
insyt-secure --projects "your-project-id:your-api-key" --show-sensitive

You can also control the log level via environment variables:

# Set log level using environment variable
INSYT_LOG_LEVEL=DEBUG insyt-secure --projects "your-project-id:your-api-key"

# Configure projects using environment variable (optional alternative to --projects)
INSYT_PROJECTS="project-id-1:api-key-1,project-id-2:api-key-2" insyt-secure

Cross-Platform Compatibility

Insyt Secure is designed to work seamlessly on all major platforms:

• Windows: Fully supported natively, no additional configuration needed
• macOS: Fully supported
• Linux: Fully supported

The service uses paho-mqtt with a platform-agnostic implementation to ensure consistent behavior across all operating systems.

Command Line Parameters

Parameter	Description	Default
--projects	Comma-separated list of project-id:api-key pairs for managing one or more projects	-
--max-workers	Maximum number of concurrent code executions per project	5
--execution-timeout	Default maximum execution time in seconds per code snippet	30
--allowed-ips	Comma-separated list of allowed IPs/hostnames	All allowed
--verbose	Enable more verbose logs	False
--debug	Enable debug level logging	False
--json-logs	Output logs in JSON format	False
--show-sensitive	Show sensitive information in logs	False

Credentials Management

This service automatically retrieves and manages connection credentials:

• When the service starts, it gets credentials from the Insyt API for each configured project
• If the connection drops or credentials expire, it automatically requests new credentials
• Each project's credentials are managed independently

Project Management

Insyt Secure supports managing one or more projects simultaneously:

Configuration Options

You can configure projects in two ways:

• Command line parameter:

  insyt-secure --projects "project-id-1:api-key-1,project-id-2:api-key-2"
  

• Environment variable:

  export INSYT_PROJECTS="project-id-1:api-key-1,project-id-2:api-key-2"
  insyt-secure
  

Benefits of Multi-Project Mode

• Resource Efficiency: All projects share a single process but maintain independent connections
• Simplified Management: Manage multiple projects from a single service instance
• Independent Reconnection: Each project can reconnect independently if its connection fails
• Shared DNS Cache: All projects benefit from the DNS cache for improved resilience

API Usage for Multiple Projects

You can also use the project management programmatically:

from insyt_secure.project_manager import ProjectManager
import asyncio

async def run_projects():
    manager = ProjectManager()
    
    # Add one or more projects
    await manager.add_project("project-id-1", "api-key-1", "https://api.account.insyt.co/")
    await manager.add_project("project-id-2", "api-key-2", "https://api.account.insyt.co/")
    
    # Start the manager
    await manager.start()
    
    # Keep the service running
    while True:
        await asyncio.sleep(60)

if __name__ == "__main__":
    asyncio.run(run_projects())

Use Cases

Data Processing Microservice

Perfect for running data transformation code that connects to various data sources:

insyt-secure --projects "your-project-id:your-api-key" --max-workers 15

Secure Environment for Code Testing

Create a sandboxed environment with restricted network access:

insyt-secure --projects "your-project-id:your-api-key" \
  --allowed-ips "10.0.0.1,192.168.1.100" --execution-timeout 20

Multi-Project Service

Run a single service that executes code for multiple projects:

insyt-secure --projects "PRJ-A235466:api-key-1,PRJ-A235477:api-key-2,ai-models:api-key-3" \
  --max-workers 10 --execution-timeout 60

Containerized Deployment

docker run -d --name insyt-secure \
  insyt-secure insyt-secure \
  --projects "your-project-id:your-api-key"

System Requirements and Dependencies

Insyt Secure is designed with a modular dependency structure to minimize installation size and resource usage. Below is a breakdown of what's included in each installation option:

Core Dependencies (included in base install)

The base installation includes:

• HTTP client capabilities via httpx
• MQTT connectivity via paho-mqtt
• Basic data science libraries: NumPy, Pandas, SciPy
• JSON and date handling utilities

Optional Dependencies

SQL Database Connectors

• postgres: High-performance PostgreSQL client (asyncpg)
• mysql: MySQL client libraries
• mssql: Microsoft SQL Server connectivity
• oracle: Oracle database connectivity
• clickhouse: ClickHouse analytics database client
• snowflake: Snowflake data warehouse client
• duckdb: Embedded analytical database

NoSQL Database Connectors

• mongodb: MongoDB client with async support
• redis: Redis client
• cassandra: Apache Cassandra and ScyllaDB clients
• neo4j: Neo4j graph database client
• elasticsearch: Elasticsearch search engine client
• couchdb: CouchDB document database client

Vector Database Connectors

• pinecone: Pinecone vector database client
• qdrant: Qdrant vector search engine client
• milvus: Milvus vector database client
• weaviate: Weaviate vector search engine
• chroma: ChromaDB for AI embeddings
• faiss: Facebook AI Similarity Search

Cloud Services

• aws: AWS SDK (boto3) with S3, Dynamo, etc.
• azure: Azure clients for Cosmos DB, Blob Storage, etc.

Messaging Systems

• kafka: Apache Kafka client
• pulsar: Apache Pulsar client
• rabbitmq: RabbitMQ client

Performance Considerations

The base installation already includes the core ML libraries (numpy, pandas, etc.). If you're installing on a resource-constrained environment, consider using only the specific connector extensions you need rather than the broader categories.

For production deployments, we recommend specifying exact dependencies rather than using broader categories:

# Good (minimal dependencies)
pip install "insyt-secure[postgres,redis]"

# Less efficient (pulls in many unused dependencies)
pip install "insyt-secure[rdbms,nosql]"

Platform-Specific Installation

Insyt Secure is designed to work on all major platforms without modification:

Windows

# Install on Windows
pip install insyt-secure

# Run (in PowerShell or Command Prompt)
insyt-secure --projects "your-project-id-123:your-api-key-xyz"

macOS/Linux

# Install on macOS/Linux
pip install insyt-secure

# Run
insyt-secure --projects "your-project-id-123:your-api-key-xyz"

Docker

# Create a simple Dockerfile
echo 'FROM python:3.10-slim
RUN pip install insyt-secure
ENTRYPOINT ["insyt-secure"]' > Dockerfile

# Build the Docker image
docker build -t insyt-secure .

# Run in Docker
docker run insyt-secure --projects "your-project-id-123:your-api-key-xyz"

Platform-Specific Considerations

• Windows: Works natively without WSL, using a cross-platform MQTT implementation
• MacOS/Linux: All features fully supported
• Docker: Ideal for deployment in containerized environments