flow-frame-core

Flow-Frame

Flow-Frame is an advanced automation platform designed to orchestrate complex workflows involving AI, media processing, browser interactions, and email management.

It acts as a central nervous system for automation, connecting LLMs (OpenAI) with real-world interfaces (Gmail, Chrome, File System) and heavy computation tasks (Image/Audio processing).

🚀 Quick Start

• Install Dependencies:

  npm install
  

• Setup Environment: Create a .env file with your API keys.
• Start Server:

  npm start
  

🧪 Testing

This project uses Jest for unit and integration testing.

Running Tests

To run the automated test suite:

npm test

Writing Tests

• Place unit tests alongside source files (e.g., src/utils.test.ts).
• Tests should use import syntax (ESM).
• Existing manual testing scripts (e.g. *.manual-test.ts) are excluded from the automated runner.

📚 Documentation

Detailed documentation is available in the docs/ folder:

• Setup & Installation: Configuration including src/config.json and GCP.
• Architecture: System design, services, and resiliency.
• API Reference: Endpoints for workflows, config files, media.
• Browser Automation: robotjs integration and extension details.
• Email Automation: Gmail polling and filtering.
• Workflows: How to build and run workflows.

🔧 Tool Reference (for Agent Integration)

flow-frame-core is designed to be used as a tool library by AI agents. The complete tool glossary is in TOOLS.md, covering every function and REST endpoint organized by capability:

Category	Tools	Integration
LLM / AI	runPrompt, runPromptGrok, runImagePromptGrok	Import or POST /api/prompts/chat
Prompt Chains	executeChain — serial/parallel steps with auto-repair	Import
Workflow Engine	executeFlow — 25+ node types in a DAG	Import or REST
JSON Utilities	extractJSON, cleanJSONString, containsJSON	Import
Schema & Code Gen	inferSchema, generateJsTransformFromPrompt	Import
Prompt Optimization	runAutoPromptSearch — iterative prompt refinement	Import
Image Analysis	classifyImageQuery, findImageInImage	Import or REST
Audio / TTS	processAudio — ElevenLabs TTS	Import or POST /generateAudio
PDF Processing	extractPdf — text, images, page renders	Import or GET /pdf/extract
Email	extractGmailBodyText, polling endpoints	Import or REST
Web Scraping	crawl — full-site depth-first crawl	Import or POST /scrape
Storage	getItem, setItem, profile management	Import or REST
Queue	queueManager — serial job queue with events	Import or REST
File System	30+ file utilities (read, write, download, zip)	Import or REST
Templates	generateTextFromTemplate — LLM variable generation	Import or REST
Browser / Desktop	Chrome control, mouse, keyboard, app management	Import or REST
Screenshots	captureScreenshotBase64, captureFullScreenshot	Import or REST
Self-Learning Vision	Ingest, recognize, discover screens, transition graphs	Import or REST
UI Planning	UiPlanner, StepByStepAiPlanner — NL → automation steps	Import
Workflows CRUD	Create, load, save, validate .workflow files	REST
Extensions	Plugin discovery and execution via api.json	Import or REST
Media	Video/audio dimensions, YOLO training data	Import or REST
Config	File-based config CRUD with backup/validation	REST

See TOOLS.md for full function signatures, parameters, return types, and REST endpoint details.

🌟 Key Features

• Agentic UI Planning: Deeply integrated AI that maps web pages and uses robotjs to simulate human clicks and typing.
• Dynamic Workflows: Can materialize file structures from JSON and execute them (build-workflow.js).
• Production Resilience: Global error handlers prevent server crashes (server.js).
• Media Pipeline: Native support for processing Audio, Images, and PDFs.

📂 Project Structure

• src/server.js: Main entry point with error handling.
• src/operations.js: Low-level OS automation (mouse/keyboard).
• src/configEndpoints.js: API for managing file-based configs.
• src/services/: Business logic and AI services.
• src/extension/: Chrome extension for browser integration.
• src/gmail-poller/: Email automation tools.

License

Private / Proprietary.