@ashutosh0x/jarvis

JARVIS - Local-First Desktop Assistant

Install

npm install -g @ashutosh0x/jarvis
jarvis

Or without installing anything permanently:

npx @ashutosh0x/jarvis

No API key is required to start. Web search works out of the box. Every key you add unlocks a feature, and the app degrades honestly without one rather than erroring — run jarvis doctor to see exactly what is and is not available on your machine:

  Jarvis 0.1.0

  Runtime
    ✓ Node                   v22.14.0
    ✓ Platform               win32 x64
    ✓ Electron               installed
    ✓ Interface built
    ✓ jarvis command         %LOCALAPPDATA%\Jarvis\bin\jarvis.cmd

  Optional services
    · GEMINI_API_KEY         unset — conversational answers disabled
    ✓ Ollama                 http://127.0.0.1:11434
    · SearXNG                unset — using public search providers

Everything marked · is optional.

The jarvis command

Whichever way you install it, jarvis becomes a command you can type in any terminal. Installing an app normally gives you a Start-menu entry, not a command, so setup closes that gap itself: it writes a small launcher to a per-user directory and puts that directory on your PATH.

Windows	%LOCALAPPDATA%\Jarvis\bin\jarvis.cmd, registered in HKCU\Environment
macOS / Linux	~/.local/bin/jarvis, with a guarded block in your shell rc if that directory is not already on PATH

jarvis link      # do it now, if setup could not
jarvis unlink    # undo it completely
jarvis doctor    # shows where the command resolves from

What it will not do:

• No setx. It truncates PATH at 1024 characters and rewrites REG_EXPAND_SZ as REG_SZ, so %VAR% entries stop expanding. The registry is written directly, preserving the original value kind. (The PATH on the machine this was built on is 2,152 characters — setx would have destroyed it.)
• No elevation, no machine-wide change. HKCU only.
• No second jarvis. If one is already on PATH — npm i -g puts one there — it is left alone and reported rather than shadowed.
• No claimed success. The previous PATH is saved to disk first, the new value is read back after writing, and the change is verified against the stored PATH rather than this process's environment.

A shell inherits its environment when it starts, so the terminal you ran it from cannot see the change — open a new one. Every message says so rather than implying otherwise.

The package bundles Electron, so the first install downloads a platform binary (~100 MB). That is the price of npm i -g producing a working app instead of a list of instructions. Prefer a native installer? See releases for signed .exe, .dmg, .AppImage, .deb and .rpm builds, each with a SHA-256 checksum.

Use the search engine as a library

The search engine has no Electron dependency and no DOM — it is plain Node, independently tested, and installable on its own terms:

import { search } from '@ashutosh0x/jarvis';

const { results, answer, providers } = await search('rust async runtime comparison');
console.log(answer);      // extracted answer, or null
console.log(providers);   // ['crates', 'github', 'hn', …]

Individual exports

import {
  rrfFuse,          // Reciprocal Rank Fusion, k=60
  bm25Search,       // BM25, k1=1.2 b=0.75
  editDistance,     // Damerau-Levenshtein
  isTimeSensitive,  // does this answer change by the hour?
  gatherAll,        // parallel provider fan-out, returns on quorum
  SearchCache,
  hedgedRace,       // race N RPC endpoints, take the first good one
} from '@ashutosh0x/jarvis';

import { stats, rollup } from '@ashutosh0x/jarvis/metrics';

Search — search, buildProviders, detectIntents, isTimeSensitive, gatherAll, rrfFuse, bm25Search, rankResults, dedupeResults, extractAnswer, verifyAnswer, providerWeights, editDistance, shouldApplyCorrection, htmlToText, SearchCache

Metrics — stats, windowed, rollup, rollupByDay, pruneRaw, makeSample

Networking — hedgedRace, createStickyOrder, backoffDelay, createDedup, createBlockTracker, prioritizeAlerts

Market analytics — dailyReturns, correlation, beta, peerIndex, realizedVol, trailingReturn, drawdown, PEER_GROUPS

The full architecture, feature reference and configuration guide follow below.

JARVIS is a desktop assistant whose intelligence runs entirely on your own machine. Speech recognition, language understanding, retrieval, and vision all execute locally. No model API keys, no network calls to a model provider, and no conversation data leaving the device.

Some features ask the outside world for facts it alone has — a share price, a headline, the state of a blockchain. Those calls send a ticker or an address and nothing else: no transcript, no memory, no conversation. Everything else works with the network unplugged.

It presents as a frameless, transparent 3D visualizer that floats above your desktop, listens continuously, and answers by voice. A companion Android app extends the same interface and control surface to a paired phone over Wi-Fi, and a spoken "mirror my phone" puts that phone's live screen on the desktop with touch and keyboard control.

Contents

• Install
• What makes this different
• Architecture
• Feature reference
  • Listening to a video
  • Connection queries
• Calendar and meetings
• On-chain intelligence
• Web search
• Retrieval engine
• Evaluation
• Android companion
• Screen mirror
• Globe
• Mineral intelligence
• Routing
• Music
• Installation
• Running
• Configuration
• Network ports
• Troubleshooting
• Known limits

What makes this different

Most assistants send your microphone to a datacenter. This one does not.

Capability	Typical assistant	JARVIS
Speech to text	Cloud ASR	Whisper via transformers.js, local
Language model	Hosted API	Gemma 3 via Ollama, local
Embeddings	Hosted API	nomic-embed-text, local
Vision / screen reading	Cloud vision	Gemma 3 multimodal, local
Conversation storage	Provider servers	Local disk only
Works without internet	No	Yes, except live data lookups
Per-query cost	Metered	Zero

Outbound traffic is limited to fact lookups that cannot be answered locally: keyless web search when a query is search-shaped, quote and news endpoints, and blockchain RPC. Each sends only the subject of the question — a ticker, a search string, an address. Inference never leaves the machine.

Architecture

System overview

Diagram: System overview →

Colour	Layer
Purple	Renderer. Visualizer, voice loop, retrieval, intent routing
Cyan	Electron main. Service supervision, IPC, LAN listeners
Green	Local inference. Everything bound to loopback
Light green	Android companion, reached over Wi-Fi
Red	External network. The single outbound path
Amber	Local persistence

The Electron main process (electron.js) supervises every local service and restarts them on failure. The renderer owns the visualizer, voice loop, and retrieval.

Voice pipeline

Diagram: Voice pipeline →

Two feedback loops are load-bearing. The dashed edges back into the VAD and echo guard are what stop JARVIS transcribing its own voice, and both are required: the ttsActive gate leaks because synthesised audio bypasses Chromium echo cancellation, so the text-level guard catches what the gate misses.

Speech to text is JavaScript, not Python. server/stt-server.mjs runs Whisper (onnx-community/whisper-base.en, q8) through transformers.js on ONNX Runtime, spawned on Electron's own Node with ELECTRON_RUN_AS_NODE=1. The previous implementation shelled out to uv run --python 3.12 --with faster-whisper, which meant every machine needed uv, a Python toolchain and a package download before voice input worked at all — and silently had no voice input if any of that was missing. The wire protocol is unchanged, so anything already speaking to :8770 needs no edit:

Client sends	Server does
binary frames	raw 16 kHz mono PCM16, appended to a buffer
{"type":"end"}	transcribe what has accumulated, reply, clear
{"type":"reset"}	discard without transcribing
	replies {"type":"final","text":"…","ms":123}

Three gates sit in front of the model, because a language model asked to describe silence will invent words rather than return nothing:

• Under 250 ms of audio — dropped without transcribing. Note that this path sends no reply at all, matching the old Python server's behaviour exactly. A client that waits for one final per end will wait forever on a blip, so drive it on arrival rather than on a strict request/response pairing.
• RMS below 0.0015 — the window is effectively silence. Returns "" without invoking the model.
• The artefact list — a whole-utterance match against "you", "thank you", "thanks for watching", "bye", "." and similar is blanked after the fact. Matching is on the entire utterance, so a real sentence that merely contains "thank you" is untouched.

Requests are serialised. ONNX Runtime is not reentrant for a single session, so two overlapping transcriptions would interleave into each other's output; they queue instead, which preserves the ordering everything downstream depends on.

Measured on this machine, 20 Aug 2026, whisper-base.en at q8, after the model is cached:

Input	Transcribe	Result
3.38s utterance	844 ms	exact
5.00s window	1060 ms	exact
5.00s window	1407 ms	exact
3s digital silence	—	"", not transcribed
3s low room noise	—	"", not transcribed

The first run downloads the weights into the transformers.js cache. Every run after that is fully offline.

Process supervision

Diagram: Process supervision →

Two invariants are encoded above. Only kill what you spawned: an Ollama the user started is reused and left running on quit. Preload is not optional: Ollama's default keep_alive is 5 minutes, so without the 60-minute preload the first question after any idle period pays a multi-second cold load.

On launch, electron.js starts and monitors:

Service	Behaviour on failure
Ollama	Reuses an existing instance if one is running; otherwise spawns ollama serve, waits for readiness, preloads the model with keep_alive: 60m, and auto-respawns after 15s
Whisper STT	Runs on Electron's own Node via ELECTRON_RUN_AS_NODE, no Python. Auto-respawns after 15s; port conflicts exit harmlessly and the 30s watchdog reclaims the port if its holder later dies
Phone bridge	Token-authenticated HTTP listener
Companion bridge	WebSocket server plus mDNS advertisement
Downloads watcher	chokidar; new documents are OCR'd and ingested
Clipboard monitor	Scans for leaked secrets, reports masked hints only
Active window tracker	10s cadence
Finance service	60s quote cadence

Services that JARVIS spawns are terminated on quit. Services it merely reused, such as an Ollama you started yourself, are left running.

Feature reference

Voice

Open conversation mode. Every transcript is routed and answered; no wake word is required. Leading "Jarvis" and common mis-hearings are stripped.

• Always-on microphone with adaptive noise-floor VAD
• Deliberate microphone selection, excluding loopback devices such as Stereo Mix which would otherwise capture JARVIS listening to itself
• Streaming speech: each completed sentence is spoken during token generation, cutting time-to-first-word from roughly 5-10s to 1-2s
• Echo guard using word-overlap against recently spoken text, because the synthesis-active flag alone is known to leak
• Self-healing microphone recovery with a 5s watchdog for eventless device death

Listening to a video

Paste or say a YouTube link and JARVIS opens it, listens to it, and says what is being said.

https://www.youtube.com/watch?v=...     start listening
what did he say                         replay the last ~60 words, from the buffer
stop listening                          stop, and report how many passages were spoken

The link is checked before intent detection. A bare URL matches no intent, so it used to fall through to the model and come back as "I encountered an error processing your request".

Two properties the design exists to protect:

The words are the speaker's. No model rewrites the transcript on its way to the speakers. transcriptEngine.js is string comparison with fixed rules, so what JARVIS says is what Whisper heard, minus the repetition the window overlap creates. Summarising is a separate thing you have to ask for.

The audio comes from the player, not the room. Capture is scoped to the player frame through getDisplayMedia rather than system loopback. That is what lets JARVIS speak while still listening — system capture would hear its own voice and transcribe it back into the queue, which compounds into nonsense within a few turns.

Constant	Value	Why
Window	5000 ms	The floor on lag: Whisper cannot start until it has one. Measured at 1.06–1.41s to transcribe, landing end-to-end around two to three seconds behind the source
Overlap	700 ms	A word spanning the cut survives whole in one of the two windows. The engine removes the duplication this creates
Sample rate	16 kHz mono	What Whisper wants; the AudioWorklet in youtube-pcm-processor.js does the conversion

Windows are transcribed concurrently but spoken strictly in order — transcriptEngine.js holds a window until every lower sequence number has arrived, because two overlapping utterances are unintelligible and a re-ordered transcript is worse than a late one.

Connection queries

correlationEngine.js answers how things in the ontology relate. This is the surface that lets a sentence reach it.

how is apple connected to foxconn        paths between two entities
what do apple and foxconn have in common
what is connected to apple               everything adjacent to one entity
show me everything related to apple
what happened around the same time       events that co-occur
what are the bridges                     articulation points — what holds the graph together

Three outcomes are deliberately kept distinct all the way to the spoken sentence, because collapsing them is the tempting bug — "no connection found" and "I have never heard of either of these" sound identical to a listener and mean opposite things:

Case	Answer
The graph has never heard of the entity	Cede — the web may know
The graph holds both, and no path exists	A finding, said plainly
The graph holds a path	Report it, with sources

"Connect" is the most overloaded verb in this assistant: it already means pair my phone, link my calendar, and go online. Every one of those matchers sits further down detectIntent than this one, so an over-broad pattern here would not merely mis-answer a graph question — it would silently break phone pairing. Two guards prevent that. A connection question must name two things, so a bare pronoun or a possessive ("my", "our", "their") disqualifies it; and any utterance containing phone, calendar, wifi, bluetooth, wallet, spotify and the rest of that list is refused outright before parsing starts.

Visualizer

• Icosahedron with Perlin-noise vertex displacement driven by live FFT
• Bass, mid, and treble bands weighted separately
• Frequency-mapped colour, with time-based hue cycling when idle
• Transparent frameless window that floats over other applications
• F2 toggles between orb-only and full HUD

Feedback

Full write-up: docs/FEEDBACK.md

Confirmation on the channels the machine actually has. Nothing vibrates on a desktop — navigator.vibrate is callable in Electron and moves nothing, because there is no motor — so a press is carried by a short animation and a synthesized click, and the vibration channel reports itself unavailable rather than pretending.

• Every heard utterance gets an instant 8 ms acknowledgement, before the answer exists. Speech has no click of its own, so the gap before the first word back was otherwise indistinguishable from not being heard
• Every unprompted event — a phone notification, a download, a whale transfer, a price alert — carries the same rising marker
• Destructive actions get the one effect with a gap in it: pulse, pause, pulse, falling and low. Every other effect is a single gesture, so it cannot be mistaken for a confirmation. A warning falls; a success rises
• Audio is synthesized rather than sampled: no binary assets, nothing to fetch, and the sound is a table of numbers that can be reviewed and tested
• prefers-reduced-motion gates the animation only. It is a statement about motion, not about feedback, and silencing the audio would strip the non-visual confirmation from the user who just asked for less movement
• The paired phone has a real motor, and gets the same vocabulary mapped onto Android's own haptic API — probed by primitive, not by API level

System control

• Application launch through an allowlist
• Volume, brightness, media keys, power state
• Wi-Fi scan, connect to saved profiles, disconnect, and measured link diagnostics reporting real latency and packet loss
• File operations, clipboard read and write
• Windows Settings deep links
• Live CPU, RAM, uptime, and active window telemetry in the HUD

Files and folders

Create folders and files by voice, anywhere inside your own user folders.

"create a folder called notes on the desktop"
"make a file called todo.txt in documents"
"make a file called shopping list saying milk and eggs"

• Parsing is rule-based, never model-driven. These commands write to disk, and a model deciding what "create a file called that thing" means is a model deciding what to name a file on your Desktop
• Confined to Desktop, Documents, Downloads, Pictures, Videos and Music. Containment is checked at a path-separator boundary, so ~/Desktop-evil is not treated as ~/Desktop
• An existing file is never silently overwritten
• Executable types (.exe, .bat, .vbs) are refused. Source files are not — writing .js is the point, and a .js is only dangerous when something runs it, which JARVIS never does
• A name that does not survive sanitising is reported, not replaced with a fallback. A file called untitled appearing because a name was misheard is worse than being told the name was not understood

Writing code

"open vscode and write a binary search in java"
"write a quicksort in python on the desktop"

Gemma writes the file contents; the filename, directory and language are fixed by rule before the model is asked anything. Java and C# get PascalCase names because those languages resolve the type by filename. The file opens in VS Code when an editor was named, and VS Code is resolved to the real Code.exe rather than shelling out to the code shim.

If the model returns nothing, no file is created — an empty file reported as success would be a lie.

Alarms and timers

"set a timer for 40 minutes"
"set an alarm for tomorrow at 2:30 pm"
"set an alarm for an hour and a half"
"set a timer for twenty minutes to check the oven"
"cancel the timer"   ·   "what timers do I have"   ·   "stop"

• Scheduled to the exact instant rather than polled, so a 30-second timer works
• Persists across restarts, and a missed alarm is announced on return rather than silently dropped
• Fires with speech, a synthesised tone and a notification; each is independent, so a suspended AudioContext does not suppress the other two
• Repeats every five seconds until dismissed, stopping after two minutes
• An unresolvable time is refused. "Set a timer for the pasta" asks how long instead of choosing a duration

Waking up

"wake me up tomorrow morning"        ·   "wake me up in the morning"
"wake me up every day at 6"
"wake me up at 6 tomorrow morning"
"set an alarm on Monday at 9am"

Named parts of the day resolve to a time: morning 07:00, breakfast 08:00, noon 12:00, lunch 13:00, afternoon 15:00, evening 19:00, dinner 20:00, night 22:00.

This is the one place the parser uses a default, which sits against the no-guessing rule the rest of the file keeps. The rule survives in the only way that matters: an assumed time is always spoken back concretely — "Alarm set for 7:00 am tomorrow. I have taken morning as 7:00 AM — say a different time if you would rather" — so a wrong default costs one sentence rather than being discovered at 7am.

Two details that are easy to get backwards:

• A day-part beside a number settles the meridiem. "Wake me at 6 tomorrow morning" is 06:00. Without this it takes the roll-forward branch and lands on 18:00, the one reading the sentence rules out.
• "Wake me" implies AM between 4 and 11. Bare hours roll forward everywhere else, but waking is a morning act; "wake me at seven" said at 9am means 07:00 tomorrow, not 19:00 tonight.

Recurrence — every day, daily, every weekday, every Monday — is carried on the alarm and re-armed by the scheduler after each firing. A repeat advances by whole calendar days rather than by 86,400,000 milliseconds, because those differ across a DST change and a 7am alarm quietly becoming a 6am alarm twice a year is a bug discovered at 6am.

The daily routine

"wake me up at 7 every day"           ·   "move lunch to two"
"remind me every day at 8 to have breakfast"
"what's left today"                   ·   "what is my routine"
"no dinner reminder"                  ·   "set my morning walk for 7"

Named slots — wake, walk, breakfast, study, lunch, dinner — each firing daily on its own. Seeded with a starting template on first run, because a routine that starts empty needs six spoken commands before it does anything.

• The routine owns only its own alarms. Entries are tagged source: 'routine', so editing lunch never cancels the alarm you set for a meeting.
• "What's left today" reads the live schedule, not the template, so it stays true after you move a slot mid-morning.
• Slots are matched longest-alias-first, so "set my morning walk" edits the walk slot rather than the wake slot.

[!IMPORTANT] A wake-up alarm belongs on your phone, not here. The desktop scheduler runs in the renderer: if JARVIS is closed at 6am, nothing fires, and nobody leaves a desktop awake and unlocked overnight to be woken by it.

With the companion paired, every alarm is mirrored to Android and scheduled through AlarmManager.setAlarmClock(), which survives a closed app, a locked screen and Doze. Both copies ring; two alarms is a far better failure than none. See Android companion.

Diagram: Waking someone up →

Calendar and meetings

"connect my calendar"
"what are my meetings"        ·   "when is my next meeting"
"schedule a meeting"
"create a meet room"
"I know about the meeting"

Scheduling is a conversation. JARVIS asks what to call it, what it is about, when, how long, and who to invite, then reads the whole thing back before creating anything:

You    schedule a meeting
JARVIS Of course, Sir. What should I call this meeting?
You    project review
JARVIS "Project review". And what is this meeting about?
You    Q3 roadmap and hiring
JARVIS Noted. When should I schedule it, Sir?
You    tomorrow at 3 pm
JARVIS 3:00 PM tomorrow. How long should it run?
You    an hour
JARVIS Anyone to invite? Say their email addresses, or "no one".
You    no one
JARVIS "Project review", 3:00 PM tomorrow for 1 hour, ending 4:00 PM.
       About: Q3 roadmap and hiring. Shall I create it, Sir?
You    yes
JARVIS Creating it now, Sir.

• Background awareness escalates rather than repeats: one warning at 30 minutes, then 10, 5 and 1, each phrased differently. Eight identical five-minute reminders train you to ignore the one that matters
• The calendar is fetched every five minutes, but checked against local clocks every twenty seconds, so alert timing does not depend on when a network call landed
• A failed poll keeps the last known schedule. Announcing "no meetings today" because one fetch failed would be a fabrication
• Spoken email addresses work — "john at example dot com" — and an address that cannot be parsed is refused rather than quietly inviting nobody
• The model is asked for exactly one thing: a better title when you gave a generic one, which you then confirm. Nothing that lands in your calendar is model-decided

[!IMPORTANT] Creating a Google Meet link requires a paid Google Workspace account. On a personal Gmail the event is created without one — the API returns no link and no error — and JARVIS says so rather than implying a link exists.

Running in the background

"start with windows"     ·   "don't start with windows"
"hide yourself"          ·   click the tray icon to bring it back

Closing the window hides JARVIS to the tray rather than quitting, so alarms still fire and meetings are still watched. Only Quit Jarvis from the tray menu actually exits.

• Autostart is registered through the OS login-items API, and starts hidden — someone who wanted a window on every boot would not need autostart
• A single-instance lock is claimed before anything else. This is not a nicety: JARVIS spawns a speech server, a TTS server, a vision server and Ollama on fixed ports, and runs a microphone listener and an alarm scheduler. Autostart plus a manual launch is the ordinary case on day one, and a second instance would fight the first for all of it. Launching again surfaces the running window instead
• Autostart can only be registered from an installed build. In a development run process.execPath is electron.exe, and registering there would put a bare Electron runtime in your startup that launches and shows nothing. JARVIS refuses and says so, rather than leaving an entry that looks installed and does nothing

What JARVIS can reach

JARVIS runs as you, not as Administrator, and that is deliberate.

Unelevated already covers everything you do day to day: your files, launching programs, reading the process list and network state, the microphone and the screen. What it does not cover is writing to Program Files, the Windows directory, or another user's data — none of which JARVIS has a reason to touch.

Running it elevated would mean a misheard word, or anything that reached the renderer through a web result, inherits Administrator. JARVIS acts on speech recognition, which mishears; that is the whole reason the file commands are rule-parsed and the write path is allowlisted in the first place.

To widen its reach, name the directories:

JARVIS_EXTRA_ROOTS=D:\Projects;C:\Work

Absolute paths, semicolon-separated. A relative entry is dropped with a warning rather than resolved against the working directory, because a typo'd Work becoming <cwd>/Work would grant a directory nobody chose. Point it at a drive root if you genuinely want that — the difference that matters is that it is your decision, written down, rather than an implicit consequence of enabling autostart.

Screen and documents

• Screen reading through Gemma 3 vision, fully offline. The captured question is passed through, so "what error is showing" reaches the model intact
• Optional Unlimited-OCR server for dense text
• Downloads are watched, OCR'd, and ingested into memory automatically
• Your Android screen, live on the desktop with touch and keyboard control — see Screen mirror. "take a phone screenshot" grabs the current frame and describes it through the same local vision path

Knowledge

• Hybrid retrieval over local memory, detailed below
• Keyless web search with a three-provider failover chain — DuckDuckGo HTML, DuckDuckGo Instant Answer, then Wikipedia — injected as cited context for search-shaped queries. The chain exists because the HTML endpoint starts serving a captcha once an IP is flagged, which silently emptied every result
• Finance watchlist with crossing alerts. Read-only by design: no order placement code exists anywhere in the project

Markets and quantitative analysis

Every number here is computed by tested code from measured data. The language model is never asked to calculate a financial figure, because it cannot be trusted with one and a wrong figure stated confidently is worse than no answer.

• Live quotes with day change, resolved name to ticker
• Single security — src/js/services/quant.js: annualised return, volatility, Sharpe, Sortino, maximum drawdown, beta and alpha against a benchmark, correlation, R², historical VaR, expected shortfall, information ratio, tracking error, up/down capture, and Black-Scholes pricing with greeks
• A book of holdings — src/js/services/portfolio.js: covariance, risk contribution, risk parity, minimum variance, maximum Sharpe, diversification ratio, and portfolio-level VaR and expected shortfall
• A name against its peers — sectorMove.js: how much of a move the sector explains and how much belongs to the company
• SEC filings through a pinned fetch guard, edgarGuard.js
• Disclosure venues beyond EDGAR — see below
• Headlines from Google News with Bing failover, keyless

Three choices in here are load-bearing rather than incidental.

VaR is historical simulation, never parametric. The Gaussian form (mu - z*sigma) is one line shorter and wrong in exactly the situation the number exists for: return distributions have fat tails, so it understates the 99th percentile precisely when that matters. The implementation reads the quantile off the observed returns and refuses fewer than 30 observations rather than resting a 99% loss estimate on a single bad day.

R² gates the interpretation of beta and alpha. Measured against the S&P 500 over 250 sessions, Micron's R² is 0.283 — the benchmark explains 28% of its variance, so its beta of 3.29 and alpha of 170% are weakly determined and are reported as such. Without that gate the assistant would speak a 170% alpha as though it meant something.

Dollar weight is not risk weight. A 60/40 book is roughly a 94% equity-risk book, which is arithmetic rather than opinion: it is what riskContributions() returns for those weights, and it is checked against that case in the tests.

Peer and portfolio analysis

Two questions the single-security metrics cannot answer, each with its own module and voice intent.

SECTOR_QUERY — "decompose Micron's move", "break down the memory sector". The move is split into the part the peer group explains (beta times the sector's move) and the part that belongs to the company. Measured 29 July 2026: Micron fell 9.94% while its peer group fell 4.96%; with a beta of 0.91 that leaves -5.40% as its own, while Western Digital's flat day was +4.77% of relative strength. Group mode ranks every member by that residual, because the largest faller is often just the highest beta.

PORTFOLIO_QUERY — "how risky is my watchlist", "what would risk parity do", "minimum variance weights for MU, SNDK, WDC". Holdings are aligned by date, never by index, because two venues do not share a trading calendar. The covariance inverse refuses a singular matrix rather than returning the enormous offsetting weights that a collinear book produces, and a short position in the minimum-variance solution is surfaced rather than clipped — "hold none" and "sell short" are different instructions.

Both modules state their limits instead of implying them with a null. A holding that only listed three weeks ago truncates every other series to match; the analysis names it and says that dropping it would widen the window.

Disclosure venues beyond EDGAR

The memory industry is mostly not American, so an EDGAR-only assistant answers "no filings" for half of it and is wrong every time. edgarGuard.js carries a venue registry naming where each issuer's filings actually live, and every entry records the date it was checked rather than a permanent claim.

Issuer	Venue	Reachable how
Micron, Sandisk, Western Digital	SEC EDGAR	Atom feeds, keyless, declared User-Agent
SK hynix	Both — SEC since 9 Jul 2026, and DART	6-K and 424B4 on EDGAR; business reports on DART
Samsung Electronics	Korea's DART	Open API, free key required
CXMT	Shanghai STAR Market since 27 Jul 2026	HTML announcements only
YMTC	None — privately held	No public filings of any kind

Probed live on 30 July 2026, because pasted endpoint lists have been wrong repeatedly in this project:

• HKEX publishes real RSS. Two feeds are wired and return 25 parsed items each. They are exchange-wide, not per-company.
• DART advertises no RSS anywhere. Its Open API is real and returns clean JSON, but rejects unregistered callers with {"status":"010"}. The key is free, so this follows the Alchemy and Helius pattern: dormant without one, and it says so rather than failing obscurely.
• SSE serves announcements as HTML only — no feed, no public API.
• SZSE does not complete a fetch from here at all.

The registry exists because of a specific failure. Its first version asserted that CXMT was "a private Chinese DRAM maker with no US listing." That was true when written and false three days later: CXMT listed on the STAR Market on 27 July 2026, rose 466% on debut, and now trades as 688825.SS. A registry that hardcodes a company's status will state a falsehood the moment that status changes, so entries carry a venue and a checked-on date instead.

On-chain intelligence

Read-only by construction. Only a hard allowlist of JSON-RPC read methods is ever sent; there is no signing code, no transaction construction, and no private key handling anywhere in the project.

The governing rule is the same one the quant engine follows: the chain is the source of truth, and anything the chain cannot prove is not claimed. An address with no ENS name stays an address. No exchange or entity is ever named from a guess.

Address and contract reads

• Native and ERC-20 balances, gas, nonce, across Ethereum, Arbitrum, Base, Optimism, Polygon and BNB Chain
• ENS forward and reverse resolution, implemented from a pure keccak-256 in src/js/services/keccak.js and verified against public vectors
• Transaction decode: status, native value, and every ERC-20/721 Transfer in the receipt, resolved to symbols and exact decimal amounts
• Contract classification through ERC-165 supportsInterface and an ERC-20 probe. Classification only; this is not a vulnerability auditor
• Cross-chain portfolio. With an Alchemy key this returns everything a wallet holds, priced; without one it falls back to scanning known tokens per chain
• Solana wallet assets and recent activity through Helius, including native SOL balance and USDC/USDT supply

Real-time whale stream

A websocket subscription to new block headers. Each confirmed block is scanned for large movements, and everything announced is a fact read out of that block.

Diagram: Real-time whale stream →

• Token flows, not just native. Most large value on Ethereum moves as stablecoins. Sampled over five live blocks: 0-2 native ETH whales versus 16 token movements
• Token decimals are verified on chain with a decimals() call before any amount is decoded. Reading a 6-decimal token as 18 turns $4M into $4
• One transaction is one movement. An arbitrage route through several pools emits the same tokens repeatedly; a live drill caught the same 14,050 WETH being announced three times. Transfers are now grouped per transaction, the source is the address that only sends, the destination the one that only receives, and the hop count and any round trip are stated
• Ranked across assets by measured USD. 100 ETH has more raw units than 4,000,000 USDC, so unit ordering picks the wrong headline
• Stablecoin issuance. A mint is a Transfer from the zero address and a burn is one to it, so supply changes need no label database. Live-verified against mainnet: a 5,414,317 USDC mint, and in one hour DAI net +6.9M against USDC net -6.0M
• Address context on both ends: ENS name, contract or wallet via eth_getCode, transactions sent, ETH held. The display carries full addresses and the transaction hash; speech carries the readable form
• Operational hardening: exponential backoff with jitter, 30s heartbeat and 90s silence detection, gap detection with in-order backfill through the same code path as live blocks, and bounded dedup so memory stays flat

What is deliberately not built

Asked for	Why not
Exchange labels ("from Binance")	Not on-chain data. It requires a proprietary attribution database; naming a wallet on a guess is the one thing that would make these alerts untrustworthy. Arkham is supported with your own key, and its labels are spoken attributed
Wallet classification by the model	A 4B model producing "institutional accumulator" is a confabulated verdict, not analysis
Mempool alerts	Pending transactions get dropped and replaced. An alert about a transaction that never lands is misinformation
Global Solana whale scanning	Measured: the Helius socket delivers over 200 token-program events in 15 seconds. Filtering that firehose is not something this machine does while also running voice
Bitcoin monitoring	A different data source entirely, and none is connected

Provider keys

All optional. JARVIS runs keyless and degrades honestly, saying which chains it can read and why one is missing.

Key	Unlocks	Without it
ALCHEMY_API_KEY	Full wallet holdings with prices, faster RPC, keyed websocket	Public endpoints, known-token scanning only
HELIUS_API_KEY	Solana wallets, activity, stablecoin supply	No Solana
DUNE_API_KEY	Aggregate analytics: top holders, USD-priced flows	Those queries state the key is needed
ARKHAM_API_KEY	Entity labels, spoken with attribution	Addresses stay addresses
GOOGLE_MAPS_API_KEY	Globe: country/state/street geocoding, place photos, weather, air quality	Cities only, from the bundled gazetteer; Nominatim, Wikipedia and USGS still work
LUMA_API_KEY	Globe: events from one Luma calendar (needs Luma Plus)	No events layer; every other globe feature is unaffected
AVIATIONSTACK_API_KEY	Globe: real flight routes with origin and destination	Live aircraft still shown from OpenSky, but a route query reports traffic over the corridor rather than named flights
WINDY_WEBCAMS_API_KEY	Globe: ~70,000 opt-in public webcams worldwide	Camera layer still covers London and Singapore road cameras
PARSE_API_KEY	Globe: refreshing the company ranking, and per-company revenue history. Metered — one credit per uncached call	The 11,222-company crawl in data/ still draws in full; it is a snapshot rather than today's prices

Networks are discovered, not assumed: each candidate endpoint must return the chain ID it claims before it is used. On the free Alchemy tier this correctly rejects Optimism and Polygon, which answer 403, rather than failing later with a confusing error.

Measured provider limits that shape the design: Alchemy's free tier caps eth_getLogs at 10 blocks, 1rpc at 50, and drpc handles a few hundred but refuses under load. Wide-range log queries are therefore chunked at 50 blocks across the keyless pool, and any chunk that fails is reported rather than silently dropped — "nothing happened this hour" and "I could only read half the hour" are different answers.

Fund tracing

src/js/services/tracer.js implements the deterministic Approximate Personalized PageRank from the TRacer paper, with its tracing-tendency and weighted-pollution strategies, plus structural pattern detection (amount consistency, cycles, consistent chains). It reports pattern presence, never a verdict. The algorithm is tested and works; live tracing needs address history, which public RPC cannot enumerate, so it awaits an Etherscan-family key.

Web search

webSearch.js (main process) and src/js/services/webSearchIntent.js (renderer) answer questions from the live internet. Split along the process boundary, not by topic: the renderer cannot fetch these origins because CORS blocks it, and Rollup cannot take named imports from a CommonJS module.

Why it exists

There was no web search. search about elon musk was classified TYPE_TEXT — the dictation intent — so asking for a search typed the words into whatever window had focus. Anything that instead reached AI_COMMAND was answered by the local model, which has no network access. It did not decline; it invented:

"search about elon musk"       -> "...recognized as a trillionaire in US dollars ."
"list latest vulnerabilities"  -> "According to OpenCVE, Google released Chrome 151
                                   with patches for 382 vulnerabilities"
"latest cve number of chrome"  -> "According to Google's Chrome Releases,
                                   CVE-2026-15905 is the latest critical vulnerability"

Those citations are fabricated. A fabricated CVE number is worse than a refusal.

Pipeline

Diagram: Pipeline →

Providers are measured, not assumed

HTML scraping was tried first and does not work:

Endpoint	Result
html.duckduckgo.com	HTTP 202 + challenge page, 0 results
lite.duckduckgo.com	HTTP 202 + challenge page, 0 results
mojeek.com	HTTP 200, body is an altcha CAPTCHA
searx.be	HTTP 200, JSON output disabled

The first DuckDuckGo query of a session usually succeeds, which makes this especially deceptive: it looks like it works until it is used twice.

Keyless general open-web search is not available in 2026. Google's Custom Search JSON API closed to new signups in 2025 and shuts down on 1 Jan 2027; Bing's Search APIs were retired on 11 Aug 2025; Brave withdrew its free tier. So the providers below are the keyless endpoints that are official, each measured before being added:

Provider	Measured	Intent
DuckDuckGo Instant Answer	361 ms	general (sourced abstract)
Wikipedia	541 ms	general (encyclopedic)
Google News RSS	642 ms	news, anything current
Hacker News (Algolia)	831 ms	discuss
crates.io	1147 ms	code (Rust)
Open Library	1259 ms	book
NVD	1462 ms	security
GitHub repos	1523 ms	code
Stack Overflow	1555 ms	code, discuss
arXiv	1973 ms	academic
npm	2078 ms	code (JS)
Brave	—	general, only with BRAVE_API_KEY

Probed and rejected: GitHub code search (HTTP 401, needs auth), Semantic Scholar (HTTP 429), Reddit (HTTP 403 to datacentre traffic).

Gather, don't race

Providers here are complementary rather than interchangeable — a Rust question wants the crates.io entry and the GitHub repo and the Stack Overflow thread — so gatherAll collects everything that arrives inside the budget instead of resolving on the first success.

The early exit counts providers, not results. Counting results was tried first and silently destroyed the feature: Google News alone returns six, which satisfied a result quota instantly and ended the query before any other source replied — measured as answered 1: google-news on every single query, a first-wins race wearing a gather's clothes.

rrfFuse merges the ranked lists by position only (k=60), because GitHub stars, Stack Overflow votes and news recency cannot be normalised against each other. Provider weights are derived from the query, after plain RRF put npm's uniffi-bindgen-react-native first for "best rust crate for async runtime" — an off-target index's rank-1 beating a relevant index's rank-2.

Query understanding

Spelling and entity correction run concurrently with the search, so they cost nothing when nothing needs correcting, and the corrected query is only re-run when the original returned fewer than three results — and only kept if it did better. A bad suggestion cannot make results worse.

Jarvis auto-corrects your spelling. Mistype a word and it recognises what you meant, fixes it, and shows you what it searched for — so a typo never silently returns nothing.

There is no hardcoded dictionary. Building one from the local corpus was tried and measured useless: 721 feed items yield 2652 "entities" that are almost entirely filing boilerplate (Filer, Filed, AccNo), and the result knew none of the terms people actually mistype. Wikipedia's search API knows all of them, live, and stays current for free.

Two kinds of correction, handled differently:

Kind	Example	Behaviour
Spelling	situtational → situational	corrected silently, shown on screen
Entity	a misspelt name → the right one	corrected and spoken aloud

The difference matters. Reading "showing results for situational awareness" aloud after a one-letter typo is noise. But a misheard name resolves to a different person entirely, and answering about someone else without saying so is indistinguishable from being wrong — so entity corrections are always announced.

Suggestions are never applied blindly. The decision is made locally on Damerau-Levenshtein distance relative to word length, and a correction is only kept if it actually returned better results than the original. A bad suggestion cannot make things worse.

Latency

Search returns in 46–956 ms against 31–51 s for the old path, which ran retrieval plus local generation. Repeat queries are 0 ms (cached).

Connection warmth was measured rather than assumed. Node 22's default dispatcher holds pooled connections for at least 120 s idle — far longer than the ~4 s commonly quoted — so no custom undici dispatcher is needed and none is added:

cold (first ever fan-out)   7812 ms
after   0s idle              664 ms
after  60s idle              690 ms
after 120s idle              564 ms

Only the cold start is worth removing, so the three general origins are warmed once, 3 s after launch. The eight specialised providers are intent-gated and left cold. There is no repeating warmer: warmth already survives a session, and periodic warming would be unsolicited traffic to third parties.

Retrieval engine

src/js/services/ragService.js implements hybrid retrieval. Design choices are evidence-driven and each is traceable to a measurement or a paper.

Diagram: Retrieval engine →

Components

Stage	Implementation	Rationale
Sparse	BM25 over a persistent inverted index, incremental on ingest	Re-tokenising the corpus per query measured 104.8ms at 5k chunks on the render thread
Dense	nomic-embed-text through Ollama, cosine similarity	Degrades to BM25-only when no embedder is present
Fusion	Reciprocal Rank Fusion, k=60	Derived from PubHealthBench, where hybrid beat both single-retriever modes. This did not reproduce locally — see Retrieval accuracy
Expansion	PRF: top 4 chunks, top 6 non-query terms, fused as a separate list at weight 0.5	Kept separate so a poor feedback pool can dilute but not corrupt the original ranking
Entities	Normalised Levenshtein, threshold 0.25, after exact-match miss	Input is speech-to-text, so names arrive mangled
Selection	Late sentence selection, IDF-weighted overlap with lead bias, budget 10	LongEval's winning system paired plain passages with late sentence selection
Reranking	Ambiguity-gated LLM rerank, opt-in	See below

Retrieval accuracy

Speed was measured long before accuracy was, which is backwards: a fast ranker that ranks the wrong passage first is worse than a slow one that does not. eval/ now carries a labelled benchmark — 29 questions over 30 documents, driving the shipped module through ablation switches rather than a reimplementation of it. Full numbers and method in eval/RESULTS.md.

Configuration	P@1	P@3	MRR	ms/query
lexical only (BM25)	69.0%	79.3%	0.737	<1
dense only	89.7%	100%	0.948	60
hybrid, as shipped	72.4%	93.1%	0.825	61
hybrid + rerank	72.4%	93.1%	0.825	3,243

Dense-only beats the shipped hybrid by 17 points at rank 1. The rationale for hybrid fusion came from the literature and did not reproduce on this corpus with this embedding model. Lexical retrieval is in the stack to catch rare proper nouns; dense matched it there (5/5) and beat it on every other question type, so its weight in the fusion is diluting a better ranking rather than protecting against a weakness.

The default has not been changed on that basis, for reasons stated in full in the results: the benchmark's author also wrote its questions, 29 questions makes anything under ~7 points a single labelling choice, and BM25 is the only thing that still works when the embedder is down. But the shipped weighting is currently unsupported by the only measurement that exists, and saying so is more useful than citing the paper it came from.

Reranking changed no answer on this set while costing 3.2 seconds per query.

Memory accuracy

The belief store's claims, measured by replaying 12 scripted observations over 6 simulated days (node eval/memory-eval.mjs):

Claim	Result
A repeated genuine preference becomes durable	3/3 held
A one-off speech mangling never does	0/2 admitted
A changed fact replaces the old value	VS Code durable, Sublime archived
Confidence bounded and reported	83% after 3 observations
Provenance retained	3 records, sources voice and text

This exercises the state machine — corroboration, decay, competition, revision. It does not measure how well a 4B model distils facts from real conversation, nor whether durable beliefs improve the final answer. Both need labelled real data, and neither is claimed here.

Measured performance

Inverted index against the previous implementation, top-10 rankings verified bit-identical at every size:

Corpus	Before	After	Speedup
100 chunks	1.87 ms	0.008 ms	223x
500 chunks	8.71 ms	0.037 ms	238x
2,000 chunks	37.1 ms	0.116 ms	319x
5,000 chunks	104.8 ms	0.456 ms	230x

Late sentence selection, measured end to end on real document text:

Metric	Result
Context size reduction	81 percent, 11,396 to 2,192 characters
Correct evidence position	ranks 1 to 3
Determinism across repeated calls	byte-identical

On reranking

Ollama exposes no /api/rerank endpoint, so a conventional cross-encoder is not available. Gemma 3 can rerank correctly, scoring 3 of 3 top-1 on labelled passages, but a single call costs roughly 3 seconds.

Reranking is therefore gated on ambiguity and is opt-in rather than default:

Path	Frequency	Latency
Gate skips, top-1 clearly dominant	~50 percent	~90 ms
Gate fires, candidates close	~50 percent	~4,800 ms

Typed input opts in. Voice input does not, because roughly 5 seconds of added silence is unacceptable on the spoken path. Any timeout or malformed response falls back to lexical order, so reranking is an enhancement and never a dependency.

Why not agentic retrieval

A-RAG style agentic retrieval was evaluated and deliberately not adopted. Benchmarked on this hardware, Gemma 3 routes queries to the correct source with 92 percent accuracy, which is sufficient. The blocker is latency: a single planning call costs about 3 seconds, and the published agent loops use 5 to 20 steps. That is 15 to 60 seconds of silence before the first word, which does not work for a voice interface.

Evaluation

node eval/retrieval-eval.mjs   # ranking accuracy across configurations
node eval/memory-eval.mjs      # belief store: corroboration, revision, garble rejection

Both harnesses drive the shipped modules. Results, method, and the caveats that bound them are in eval/RESULTS.md; the headline numbers are in Retrieval accuracy and Memory accuracy above.

The benchmark corpus is synthetic and labelled. It supports comparison between configurations, since each sees identical data; it does not predict accuracy on a real user's memory, and it is not presented as doing so.

What is still unmeasured, and should be: whether retrieved context and durable beliefs improve the final answer, as opposed to the ranking. That needs answer-level labels and a judge. The rankings are now measured; the answers are not, and no claim is made about them.

Android companion

companion/ contains a Kotlin application that mirrors the visualizer to a phone and exposes device control back to the desktop.

The visualizer is copied, not reimplemented

Asset	Origin	State
visualizerModes.js	src/js/visualizerModes.js	byte-identical, SHA-256 verified
three.module.js	three@0.158.0	byte-identical
Vertex and fragment shaders	src/index.html	verbatim
visualizer.js	src/js/scripts.js	renderer, uniforms, and FFT blend preserved

visualizerModes.js still carries import * as THREE from 'three'. Rather than edit the copy, an import map in the host page resolves the bare specifier, so the file stays identical to the desktop original.

Assets are served through WebViewAssetLoader on a virtual https origin rather than file://. WebView blocks ES module scripts from file:// because the origin is opaque, which presents as a silent black screen.

Audio bridge

The desktop fills window.jarvisFrequencyData from a WebAudio AnalyserNode. A WebView cannot obtain microphone access that way, so AudioFft.kt reads AudioRecord, applies a Hann window, runs a radix-2 FFT, and writes the same 64 bins natively. Bins use WebAudio's decibel mapping, minus 100 to minus 30 dB onto 0 to 255. Linear magnitude was tried and leaves the orb nearly static at speaking volume.

Pairing

Diagram: Pairing →

The phone always dials outward, which avoids Doze restrictions and handset address churn. Pairing retries every 10 seconds while unpaired, because the window is usually opened after discovery has already resolved.

/pair and /apk return 403 once the window closes. That window is the only thing standing between a network neighbour and the bridge token, so it is short and user-initiated.

Capability negotiation

On connect the phone reports what it can actually do, probed rather than assumed:

{"open_app":true,"list_apps":true,"clipboard":true,"battery":true,
 "tts":true,"flashlight":true,"volume":true,"alarms":true,
 "ui_automation":false,"screenshot":false,"read_screen":false,
 "silent_install":false}

The desktop reasons about the device instead of firing commands blindly. A request needing accessibility explains how to enable it rather than failing opaquely. alarms reports whether exact alarms are currently permitted — it is user-revocable on Android 12 and 13, and an assistant that keeps promising wake-ups it cannot schedule is the failure this project refuses everywhere else.

Control tiers

Tier	Requires	Commands
1	Nothing beyond install	ping, device_info, battery, clipboard_get, clipboard_set, tts, list_apps, open_app_by_name, flashlight, volume, capabilities, schedule_alarm, cancel_alarm, list_alarms
2	AccessibilityService enabled	get_layout, click, long_press, swipe, input_text, global, screenshot, read_screen
3	Wireless Debugging enabled	Desktop-side ADB: brightness, volume, keyevents, package management, file transfer, screenrecord

Tier 3 runs entirely on the desktop through adbService.js. The APK is not involved. All ADB invocations pass argument arrays, never concatenated strings, and raw shell passthrough is disabled at the IPC boundary.

Alarms that actually wake you

The desktop's scheduler cannot, and says so. The companion schedules through AlarmManager.setAlarmClock() — the only API the platform treats as a real alarm clock, exempt from every battery-saver relaxation and special-cased by OEM battery layers. setExactAndAllowWhileIdle is rate-limited to roughly one firing per app per nine minutes in Doze; a wake-up that is "usually within a few minutes" is not a wake-up.

• Parsing stays on the desktop, so both surfaces agree on what "tomorrow morning" means. Only the scheduling happens on the phone.
• Re-armed on boot. AlarmManager forgets everything across a restart, and an alarm app that skips this works perfectly until the first reboot and then fails silently overnight. Xiaomi's QUICKBOOT_POWERON is handled alongside the AOSP broadcast.
• The ring screen shows over the lock screen, says what the alarm was for out loud, and offers a nine-minute snooze that stays one-shot even on a repeating alarm — copying the repeat rule would walk a 7am wake-up into the afternoon over a week of snoozing.
• Rings on STREAM_ALARM, which survives silent mode and Do Not Disturb, and raises the alarm volume off zero rather than playing nothing.

The floating orb

OverlayService keeps JARVIS on screen after the app is closed — a TYPE_APPLICATION_OVERLAY window you can drag, which snaps to the nearer edge and opens the conversation when tapped.

Drawn on a Canvas rather than reusing the WebView visualizer: the overlay persists for the life of the phone, and a full Chromium renderer held that long is what OEM battery managers kill first. It stops animating entirely when idle.

The permanent notification is not removable. An overlay added from a background process dies with the process, so the foreground service is what makes the orb outlive the activity at all.

Continuous listening

Diagram: The wake-word gate →

Opt-in always-on listening, gated by the wake word. The gate is closed by default: nothing becomes a turn until "Jarvis" opens it.

This is a direct response to what the desktop interaction log shows. Across 682 recorded turns roughly a third were empty or failed, because the microphone was open and everything it heard became a command — a keynote about AI infrastructure, a financial-advice podcast, "see you in the next video". One session recorded a name and biography as a fact about the user, from mis-transcribed background audio. A phone is in the room more often than a desktop is, so always-on capture is only defensible behind a gate.

• Wake variants are deliberately few. "travis" and "service" are not among them: both occur in ordinary speech, and each added variant trades a missed wake for a false wake.
• Interim transcripts never fire a command. They are routinely revised, and acting on one means acting on something never said.
• The microphone is torn down while the assistant speaks. Speaker and mic inches apart means it hears its own text-to-speech as a command — every reply, not an edge case.
• Recognition runs on-device via createOnDeviceSpeechRecognizer where the language pack is present. The notification states which mode is live, because "local-first" is false if a live microphone is being streamed to a server and nobody said so.
• After six consecutive hard failures it stops rather than looping. An assistant that appears to be listening and is not is worse than one that admits it stopped.

The gate is pure logic with no Android imports and is covered by 18 JVM unit tests whose negative cases are real transcripts from the log.

Reading the screen on the phone

read_screen merges two sources:

• The accessibility node tree — the real strings apps handed the framework, with no transcription step and so no transcription errors.
• On-device OCR over a screenshot, for what the tree cannot see: text baked into images, video frames, canvas and OpenGL UI.

The tree wins on conflicts and OCR duplicates are dropped, or the assistant says everything twice, slightly differently the second time. OCR alone would be the obvious build and it would be worse — a fuzzy reading of text that was available exactly, with the structure lost.

No MediaProjection anywhere: PROJECT_MEDIA is a signature permission an app cannot hold, and the user-facing flow demands a consent dialog per session, which for "what's on my screen" means a system prompt every single time. AccessibilityService.takeScreenshot() needs a grant the user gives once.

Structured phone tools

The desktop reasons; the phone executes. Commands travel as structured intents, never free-form text:

"open settings on my phone"
  -> routePhoneCommand()
  -> {tool: "phone.open_app", parameters: {name: "settings"}}
  -> companion: open_app_by_name
  -> {"package": "com.android.settings", "label": "Settings"}
  -> "Settings is now open on your phone, Sir."

Every spoken confirmation is built from what the phone returned. The language model is deliberately absent from this path, because earlier logs showed it inventing outcomes when it had no execution feedback.

Screen mirror

Full write-up: docs/SCREEN-MIRROR.md · Diagram: Screen mirror →

Your Android screen on the desktop, with touch and keyboard control, from one spoken sentence.

"mirror my phone"          -> panel slides in, phone appears
"stop mirroring"           -> session ends, nothing left on the device
"take a phone screenshot"  -> grabs the current frame and describes it locally

Alt+Shift+M closes it too, as does the ✕ on the panel.

USB works out of the box; Wi-Fi works once the phone has been paired over Wireless Debugging. Nothing is installed on the phone — the scrcpy server jar is pushed to /data/local/tmp for the session and removed when it ends.

How it is split

phone  --H.264 over adb-->  main process  --IPC-->  renderer  --WebGL-->  canvas
                                 ^                     |
                                 +---- touch/keys -----+

Piece	File	Runs in
Voice routing, coordinate and key mapping	src/js/services/mirrorIntent.js	pure, no I/O
scrcpy session, control injection	mirrorService.js	main
IPC wire	electron.js, preload.js	main / bridge
Decode, draw, input relay	src/js/components/mirrorPanel.js	renderer

The session lives in main because it needs a TCP socket to the local ADB server on 127.0.0.1:5037, which the renderer cannot open. The decode lives in the renderer because WebCodecs hands frames to WebGL without them entering JavaScript memory. Decoding in main would mean shipping raw frames over IPC — 1920×1080×4 bytes at 60 fps is about 500 MB/s. What crosses IPC instead is the compressed elementary stream, roughly 1 MB/s.

Decisions worth knowing

• The server jar is pinned to scrcpy 3.3.3, not "latest". The client implements the protocol up to 3.3.3 and scrcpy compares version strings exactly, so dropping a 4.x jar in produces a session that dies at handshake rather than an upgrade. The SHA-256 is checked on every start and asserted in tests, because a substituted jar otherwise just hangs.
• maxSize defaults to 0, meaning device native. Measured on a 1080×2400 handset, maxSize caps the longer edge — the obvious-looking 1920 produced 864×1920, narrower than 1080. At native size the stream measured 4.1 Mbps against an 8 Mbps ceiling, so the downscale bought nothing.
• The panel is sized from the phone, not the other way round. Width is derived from the stage's measured height times the device aspect ratio, so there are zero letterbox bars in either direction, and rotation refits both axes. A 20:9 phone in an 800px-tall window is 360px wide — that is the phone's real shape, and the only way to a bigger mirror is a taller Jarvis window.
• Printable keys are sent as text, not keycodes. A keycode replays a physical key and is resolved through the device's layout, so on a phone set to anything but the host layout the wrong character appears. Enter, Backspace, arrows and modifiers have no text and must be keycodes.
• Audio is raw PCM, and mutable. Phone audio out of the speakers can be transcribed back as a user turn, so it ships with a mute button and buildMirrorOptions({audio:false}). Renderer playback goes through Chromium's render path and is seen by the echo canceller, unlike the SAPI voice that bypassed it. Transport is verified; audible content is not — see the doc.
• Right-click is Back, Escape is Back. scrcpy convention, and it is what makes the mirror usable.

Measured — Xiaomi M2101K6P (Android 16), USB, 2 Aug 2026

Driven through the shipped modules, not a harness reimplementation.

Resolution	1080×2400, device native
Handshake	1078 ms cold, 629 ms warm
First frame	1289 ms cold, 799 ms warm
Frame rate	60.5 fps received; 49 fps presented on a live screen
Bitrate	4.11 Mbps at native size (ceiling 8 Mbps)
Control round trip	back / home / recents / notifications / rotate, ≤1 ms each

Picture verified by pixel statistics rather than by looking: mean luma tracked the device — 60 → 39 when the notification shade was opened over the control channel, back to 60 on collapse. A frozen surface cannot do that. Cleanup confirmed after stop(): no app_process running, jar removed, nothing installed.

The LAG badge is not glass-to-glass latency and does not claim to be. The device's capture clock and the host's clock have no shared origin, so their difference is an unknown constant. What the badge shows is arrival delay above the smallest value seen this session — queueing on top of the fastest path actually observed. The label says LAG rather than latency for that reason.

Failure messages

Each is produced by mirrorService, not the model, and names the thing you control:

Message	Fix
no Android device is connected over USB or Wi-Fi ADB	plug it in, or adb connect
your phone has not authorised this computer	accept the USB debugging prompt
N devices are connected — say which one, or unplug the others	ambiguity is an error here, never a coin flip
the ADB server is not reachable on port 5037	adb start-server (tried automatically first)
does not match the pinned v3.3.3 build	resources/scrcpy-server.jar was replaced

Globe

Press F3, or ask for a place, and the orb becomes a command centre: a dark sphere with a glowing amber vector network, a blue pin on the target, labelled landmarks on leader lines, and live seismic ripples.

show me Japan on map
show me Karnataka
take me to Tokyo
show me MG Road Bengaluru
show me what's happening in San Francisco
show me AI companies in San Francisco

Country, state, city, street and building all resolve. Press L for the layer switchboard and T to cycle themes. Full reference: docs/GLOBE.md.

Vectors, not satellite imagery

The plan called for 16 MB of NASA Blue Marble textures. Vectors won on all three axes: 2.2 MB of public-domain Natural Earth GeoJSON ships in the repo and works offline, lines stay sharp at city zoom where an 8K equirectangular is ~2 km per pixel, and photoreal reads as Google Earth rather than as a command centre.

The globe is a Group inside the orb's existing scene, driven from the existing render loop. One WebGL context, one camera, one requestAnimationFrame — a second renderer would double GPU buffers for a view only one of which is visible.

Finding a place

Three tiers, cheapest first.

Tier	Covers	Cost
Bundled gazetteer	~1,250 cities, 162 KB, offline	free
Google Geocoding v4	country → state → city → street → building	billed
Nominatim	keyless fallback	free

Matching is forgiving because speech-to-text is the real input — sanfrancisco and san fransico both resolve — but it is bounded. Without a length floor on prefix matching, map matched Maputo and ku matched Kuwait City, both at the exact confidence the intent parser acts on, so "show me the map" flew the camera to Mozambique. A prefix now needs four characters and half the name, and the fuzzy edit budget scales with query length.

Framing is measured, not tabulated

There is no table mapping country → zoom level. Google's v4 response carries a viewport; the camera distance is derived from the extent it reports — Japan measures 3,331 km across, a street 2 km — through one continuous curve. The same measurement scales the landmark ring, because ten kilometres around Japan finds one suburb of Tokyo and calls it the country.

Ground truth, and photographs

On arrival, five parallel lookups report what is actually true there:

Delhi: 19:23 local · 28.6°C, light rain · 237 m elevation
       AQI 48 (Moderate) · street view 2012-11

A field that did not answer is omitted, never printed as a zero. Real photographs come from Wikipedia and Wikimedia Commons first — both keyless — falling back to Places and Street View only where free sources cannot answer, with Street View always gated behind its free metadata check. Every image carries its attribution, and one whose attribution did not survive the parse is dropped rather than shown bare. If nothing has a picture, nothing is shown.

Live layers

Feed	Key needed	Ships on
USGS earthquakes	none	✅
OpenSky flights	none	✅
Satellites (CelesTrak SGP4)	none	✅
Aurora / Kp (NOAA)	none	✅
Road cameras (TfL, Singapore)	none	✅
NASA EONET (volcanoes, storms, ice)	none	✅
Company head offices (11,222)	none — crawled to data/	✅
OSM campus shapes (Nominatim)	none	✅
Windy webcams (worldwide)	free key	needs key
NASA FIRMS wildfires	free MAP_KEY	needs key
Companies at a place (Places)	Google	needs key
Luma events	Luma Plus	needs key
Flight routes (AviationStack)	free tier	needs key

Every layer is toggleable from a glass switchboard — press L, and cycle the amber / ghost / tactical themes with T. A layer switched off stops its polling and contributes nothing downstream; one that fails to load shows the reason on its row rather than an empty, silent map. A feed with no credentials reports itself unconfigured with the reason and never polls.

Three of these are worth a line. Satellites are propagated with SGP4 from live CelesTrak elements — the ISS is drawn where it actually is (~423 km, ~415 km/min), not at the sub-point of its element epoch. Cameras are public road cameras and opt-in Windy webcams, fetched entirely in the main process so no third-party camera URL ever reaches the renderer; nothing scans for private IP cameras. EONET deliberately fetches only the categories Jarvis lacks — its open feed is ~7,000 events of which ~6,950 are wildfires, which the FIRMS layer already draws from a better feed.

Luma events deserve one caution: their API is scoped to a single calendar and has no search endpoint — 66 endpoints, none of them discovery. It shows your events wherever they are, not the world's, and the feed is named "Luma (my calendar)" so nothing implies otherwise.

Companies, at their head offices

Switch the Companies layer on and the world's public companies appear where they actually are — sized by market capitalisation, coloured by the day's move, green up and red down. Click one for its price, rank, address and a photograph of the office.

This is a local database, not a live dependency. Two crawls, paid for once:

Step	Cost	Result
scripts/fetch-ranking.mjs	113 API credits	11,222 companies across 81 countries
scripts/resolve-hq.mjs	14,144 Places lookups, ≈ $453	10,995 resolved coordinates

Both ship in data/, are read from disk, and work offline. The globe must prefer them over resolving live — a launch that re-buys eleven thousand lookups already sitting in a file is the most expensive mistake this feature could make.

Every coordinate is validated, not assumed. A merely plausible coordinate is worse than none: it puts a real company at a real place that is the wrong place, and nothing downstream can tell. The ISO country must match the one the ranking recorded — this is what stops Reliance (ticker RS, an American steel distributor) being pinned to Mumbai — and the matched place name must share a meaningful token with the company. Failures are recorded with their reason, not silently dropped and not quietly kept:

10,995 resolved      10,959 building-level (Places) · 36 city-level (Wikidata)
   226 rejected      106 name mismatch · 78 no candidates · 42 wrong country

Photographs are fetched on demand — two billed requests each, so sweeping all 10,959 would be about $263 and 877 MB of pictures nobody asked for. One click is half a cent, and the second click is free because photos cache to disk.

A separate live search answers the other question, "what companies are here":

show me AI companies in San Francisco
find fintech companies in London

Nothing is baked. The query is biased to the target's coordinates and the bias radius scales with the place, so a country is not searched as a 5 km circle around its centroid. Sixty results is Google's hard ceiling per query, and the reply says "N on the map" rather than claiming to have found every company in the city. That layer is off by default because each navigation with it on is a billed search.

Shapes, not just points

Google returns Manyata Tech Park — a 120-hectare campus with fifty buildings — as one pin the same size as a coffee shop. OpenStreetMap has it as a polygon with its real boundary. So Google finds it and OSM shapes it, keylessly.

Built on Nominatim rather than Overpass: four Overpass endpoints were probed and all four returned 504 or timed out, and a globe that goes blank when a volunteer server is busy is not a globe. Nominatim's one-request-per-second policy is enforced by a serialised queue in code, so it cannot be violated by a caller who forgets it.

The key stays in the main process

GOOGLE_MAPS_API_KEY is optional — without it the globe runs entirely on keyless sources. When present it is read by googleMaps.js in the main process and never crosses the bridge; the renderer sends a whitelisted method name and receives data back. Enabling Google also obliges their logo on a non-Google map and forbids caching place content, both of which the implementation honours.

PARSE_API_KEY follows the identical rule in companiesMarketCap.js, and adds one of its own: because every uncached call costs a credit, nothing in that module paginates on its own. ranking() takes an explicit page, and a caller wanting ten pages has to ask for ten. Successes cache to disk for twelve hours; failures are never cached, because a failure was not charged and caching it would hide a transient problem for half a day.

osmGeometry.js needs no key at all — the boundary of a campus should not depend on anyone's billing account — but it is still main-process only, for the same reason: the renderer gets results, never the network surface.

Mineral intelligence

A global store of documented mineral deposits, and a ranking engine that finds ground which resembles them. src/js/services/mineral/.

The premise comes from Google DeepMind's AlphaEarth Foundations (Brown et al., arXiv:2507.22291): every 10 m of land surface has a published 64-dimensional embedding summarising a year of optical, radar, topographic and climate observation. Landscapes that look alike sit near each other in that space. Take the embeddings at deposits already documented, average them, and sweep for surface that matches. AEF-Transformer did exactly this on the Tuwu copper belt; Nakata et al. (arXiv:2604.14756) pushed the same representation at subsurface targets.

Asking it something

Build the database once. The app must not parse a 137 MB CSV at launch, and there is nothing to re-parse — USGS stopped updating MRDS in 2011.

curl -O https://mrdata.usgs.gov/mrds/mrds-csv.zip && unzip mrds-csv.zip
node scripts/build-mineral-db.mjs mrds.csv    # ~10 s -> cache/minerals.db

That is 304,613 deposits in 183 MB, ingested at 36,335 rows/s with a 20 MB peak heap because the CSV is streamed rather than loaded. It lands in cache/, not data/ — data/ is packaged into the installer and a locally built artifact that size has no business inside it.

Then just ask, by voice or text:

"show me gold in japan" "copper mines in chile" "where are the lithium deposits"

The answer is spoken, and the deposits are pinned on the globe — documented ones as solid commodity-coloured octahedra, predictions as hollow pulsing rings, because greyscale screenshots and colour-blind viewers preserve shape and can lose colour.

The renderer never opens the database. node:sqlite is unreachable from the sandboxed renderer, so the store stays in the main process and only the result crosses the bridge. It opens lazily on the first mineral question; someone who never asks never pays for it. If the database has not been built, the handler says so and names the script — it does not quietly answer from a web search instead, because the entire value of this path is that every number traces to a USGS record id.

Getting the question to the database at all

Worth its own heading, because this is where the feature was dead for a day while every test passed.

An utterance reaches minerals through a cheap shape test in jarvis.js, and that test's \b escapes were once written to disk as literal backspace bytes (U+0008). The regex then asked for a control character either side of the word and matched nothing, ever. Every mineral question fell through to web search while 304,613 local deposits sat one call away. 271 engine checks and 218 routing checks were green throughout; the engine was fine, and nothing tested the boundary between the utterance and the engine. routing.test.mjs now drives the real detectIntent for this path.

Two things that test pins, both of which have bitten:

• Word boundaries are load-bearing. Without them "ore" matches inside "more", "before", "store" and "score", and this branch runs early enough to hijack ordinary conversation.
• It has to cede. The branch sits ahead of ~500 lines of matchers so that "show me gold in japan" is not answered from the model. But a main-process notMineral: true falls through to web search rather than back into those matchers, so anything wrongly claimed is lost, not delayed. A commodity word inside a named subsystem's request — "search edgar for lithium supply" — is not a mineral question, and MINERAL_CEDES_TO keeps it out.

The one claim it refuses to make

A cosine similarity is a similarity. It is not a probability that ore is buried somewhere, and nothing here presents it as one.

• AlphaEarth observes the surface. Ore bodies are underground. The link is indirect and non-unique — Nakata et al. say so plainly — so a high score means "this landscape resembles landscapes that host known deposits", never "there is copper here".
• The score is ordinal. It ranks candidates against each other. It is not calibrated against drilled outcomes, because nothing in this repository has drilled anything. There is no scoreToProbability, and a test asserts its absence so nobody adds one by reflex.
• The field is named similarityScore, never prospectivity, in every result.

Facts and guesses cannot be mixed

Documented deposits and model output share one table, and a CHECK constraint makes the dangerous row unrepresentable rather than merely discouraged:

prospectivity REAL CHECK (
  prospectivity IS NULL
  OR classification IN ('predicted_prospect','exploration_target'))

A documented mine carrying a model score is how a prediction gets laundered into a fact. The database rejects it. Searches return documented records only unless the caller passes includePredictions explicitly, and every GeoJSON feature carries isPrediction so a renderer cannot draw the two alike by omission.

Storage

This is the only service in the repository with a real database. The others hold thousands of records and a JSON file is right for them; this holds millions — MRDS alone is ~300k sites — and "copper within 100 km of Bengaluru" cannot be answered by parsing the planet into memory first.

It adds no native dependency. node:sqlite ships inside Node 22 and inside the Node that Electron 39 embeds, so ragService.js's zero-native-deps promise survives intact.

Proximity is served by a 1° grid: every record carries an indexed integer cell, a radius query reads only the cells the circle touches, then haversine gives true distance. A bounding-box filter alone over-reports by about 27% at 100 km, because the corners of a square are not inside a circle.

Place names

Exploration targets are in empty country — that is why nobody has drilled them. So mineralGeocoder.js returns a place name and how far away it is. "Antofagasta" on a target 380 km from Antofagasta is a lie of omission.

It reuses what the repo already ships: the 243-place Natural Earth index, and the 3,269-entry IATA airport index, which is thirteen times denser and dense in exactly the remote districts the cities file leaves blank. Past 150 km, placeName is null and only the bearing is returned. A coordinate with nothing near it is a normal, reportable state; inventing a locality for it is the failure groundingGuard.js exists to stop.

Country names for airports are derived from the places file's iso_a2 → adm0name pairs rather than kept as a lookup table, and codes no bundled place covers stay null.

Measured, on live data

npm run eval:mineral — run against live USGS MRDS across five real mining districts on four continents, 2,314 deposits:

Parse fidelity	100.0%
Coordinate validity	100.0%
Spatial index recall	100.0% (100,683 hits, 0 false positives)
Idempotency	100.0%
Status classification	96.8%
Geocoder country accuracy	97.9%
Pipeline integrity	99.2%

That composite covers 43% of the intended evaluation. The rest — AlphaEarth similarity, prospectivity accuracy, calibration, spatial block CV — is reported UNAVAILABLE and excluded, never defaulted to a pass.

Discovery accuracy is not established, and the harness refuses to estimate one. It needs Earth Engine credentials and drilled outcomes; neither exists. 99.2% says the plumbing is correct — parsing, storage, indexing, geography. It says nothing about whether a ranked target contains ore.

The run also found that the ground truth is not clean: USGS labels a deposit in Karnataka as being in Peru, 16,263 km from the nearest Peruvian anchor. The eval adjudicates disagreements by distance rather than trusting the reference, because scoring against a source with continental-scale errors measures agreement, not accuracy.

A published AUC of 0.98 is not what it looks like

Two 2026 papers apply AlphaEarth embeddings to prospectivity and report macro-AUC > 0.98 (random split) and 0.912 (spatial blocked CV). The gap is the validation protocol, and node eval/mineral-leakage.mjs measures it on real MRDS deposits using a null model that contains no geology at all — a nearest-neighbour distance lookup:

Commodity	Random split	Spatial block CV	Inflation
Copper	0.9732	0.6563	+0.317
Gold	0.9717	0.6905	+0.281
Silver	0.9612	0.6685	+0.293
Lead	0.9729	0.5842	+0.389

A model that knows nothing scores 0.96–0.97 under a random split. Deposits cluster, so a random split puts one in train and its neighbour 800 m away in test. That does not prove the published model is wrong — it proves the reported number cannot separate geology from spatial memorisation. The 0.912 under spatial blocking is the stronger evidence, because the null model reaches only 0.58–0.69 on that protocol.

Jarvis defaults to spatial blocking. The random-split path exists only as randomFoldsForLeakageComparison, named so it is hard to use by accident.

MRDS is frozen at 2011

USGS ceased systematic MRDS updates in 2011, and the evaluation confirms it from the records rather than taking the notice on trust. Across all 2,314 deposits the most recent update_date is 2011, and 97.4% were last touched in 2003 — the freshest record is fifteen years old.

That single fact explains the rest: only 31.2% of records have a name, 47.0% a commodity, and the Karnataka-in-Peru error is still there because no correction pass exists. MRDS is an excellent historical record of where mineralisation has been documented — ore bodies do not move, so it is exactly the right input for a prototype — but status must never be reported as present-day fact. The eval prints the year distribution on every run and deliberately does not score it: staleness is a property of the source, and folding it into a quality percentage would imply the pipeline could fix it.

Data

Full reference: docs/MINERAL-INTELLIGENCE.md. Two sources are wired and verified; four more are licence-checked and staged. See docs/DATA_SOURCES.md for every licence, attribution and redistribution term — notably GEM's active-fault database, which is CC-BY-SA-4.0 and therefore fetched at runtime rather than bundled.

Source	Role	Licence	Status
USGS MRDS	documented deposits	public domain	live
Google AlphaEarth	surface embeddings	GEE terms	needs your credentials
World Mining Monitor	production / operators	MIT	staged
MinMod	critical-minerals graph	MIT	staged
Mindat	localities	API terms, key required	staged
NASA EMIT	surface mineralogy	Apache-2.0 / NASA open	staged
GEM active faults	structural features	CC-BY-SA-4.0	staged, not bundled

Earth Engine access is optional and never bundled. Without credentials the service throws naming the missing piece; it does not fall back to demo data, because a mineral target that came from nowhere is the most expensive fabrication this repository could produce — the user's next move is to go look at ground.

Four corrections to the published guidance

Worth recording, because each fails silently:

• The annual asset covers 2017–2024, not "2017–present". assertYear rejects 2025 rather than letting Earth Engine fail later on a null image.
• ee.Image.constant(vector) yields bands constant_0…63; the asset's are A00…A63. Multiplying images with disjoint band names does not do what it looks like. .rename(BAND_NAMES) is the fix and it is not optional.
• MRDS wants f=json. The res=json form returns XML, which parses to zero deposits and reads as "no data in this area".
• .first() on the collection is correct for a point and wrong for any area — it returns one tile and the rest of the map reads as "no targets". Use .mosaic().

Routing

Commands used to be matched by pattern, and every new ability meant another regex that only recognised the phrasings someone thought of. "latest trending meme coin search" reached the local model — which correctly answered that it cannot search — because the verb was at the end, and no pattern anticipated that. Adding a pattern fixes that sentence and not the next one.

Capabilities now describe themselves, and the router matches meaning.

Measured, not assumed

Held-out phrasings, none of which appear in the capability manifests, scored against the live nomic-embed-text embedder:

Router	Accuracy
As shipped — deterministic, then semantic	24/24 · 100%
Semantic router alone	23/24 · 95.8%
Regex baseline	18/24 · 75.0%

eval/routing-eval.mjs. The comparison exists because a smarter architecture is a hypothesis until it has a number — an earlier causal-graph retrieval layer in this project was obviously better in principle and measured worse than plain retrieval, 56.6% against 68.8%.

Blast radius decides the router

"empty the recycle bin"
"don't empty the recycle bin"
"what happens if I empty the recycle bin"

These are neighbours in embedding space. Cosine similarity has no reliable signal for negation or interrogation — the content words dominate — while a regex with a question guard separates them exactly.

A wrong retrieval costs a wasted search. A wrong destructive action costs files. So capabilities declare their effects, and only read-only ones are reachable by similarity; anything that writes or destroys needs a deterministic parse. Tests route the negated and interrogative forms and assert they cannot reach anything destructive.

This is not a rejection of semantic routing. It is semantic routing where being approximately right is good enough, and deterministic parsing kept where it is not.

Freshness is the lever

STATIC · DYNAMIC · REALTIME. A local model answering a REALTIME question is not recalling — the answer postdates its training, so it is generating, which is the path that produced the fabricated citations this project already guards against. Freshness catches it before the model is asked.

Deliberately lexical rather than model-judged: it runs on every utterance and an Ollama round trip would slow the fast path; a 4B model is the least reliable possible judge of its own knowledge age, because it does not know what it does not know; and a function can be tested against a labelled set where an opinion can only be spot-checked. Unrecognised questions fall to DYNAMIC, which prefers the network — over-searching costs a request, under-searching costs a fabricated answer.

A nearest neighbour always exists, so there is a floor (0.55) and a margin (0.04): below the floor, or within the margin of the runner-up, nothing is chosen. "thank you" routes to nothing. No embedder means no opinion, never a random capability.

Music

Real-time track resolution through the Spotify Web API, then handed to whatever can actually play it.

"play starboy by the weeknd"     -> track:starboy artist:the weeknd
                                 -> Starboy — The Weeknd, Daft Punk
"play bohemian rhapsody on spotify"
"pause" · "skip this song" · "what's this song"

"X by Y" becomes a fielded query — track:X artist:Y — because the plain string lets a popular artist name outrank the requested track: "play hello by adele" can otherwise return an Adele song that is not Hello.

What actually plays it

Search is genuinely live and verified. Playback depends on what is installed, and Jarvis says which of the three happened rather than reporting "playing" for all of them:

Route	Needs	Result
Web API playback	Premium + a running Spotify device	audio starts, nothing opens
spotify: desktop URI	Spotify desktop installed	plays in the app, no browser
open.spotify.com	nothing	opens a tab, off by default

The browser fallback is disabled by default, because a tab is not background playback — it is a window appearing, with still no music until someone presses play. With no player present Jarvis reports that it found the track and has nothing to play it with.

Free accounts: the Web API playback endpoint is Premium-only, but the desktop URI path works on Free. Installing the Spotify desktop app is what turns this into background playback with no browser.

Not a workaround: playing the web player inside a hidden Electron window would need Widevine, and Electron ships no CDM — it loads and fails on DRM. Measured, not assumed.

Tokens live in the same safeStorage vault as every other secret (DPAPI on Windows). None is written to the repository.

Installation

Download a build

Prebuilt installers for every tagged release are on the Releases page.

Platform	Download	Notes
Windows	Jarvis-Setup-<version>-x64.exe	Installer. Jarvis-Portable-*.exe needs no install.
macOS	Jarvis-<version>-universal.dmg	One universal build for Apple Silicon and Intel
Linux	Jarvis-<version>-x64.AppImage	chmod +x and run. .deb, .rpm and .tar.gz are also published.

Verify what you downloaded:

sha256sum -c SHA256SUMS --ignore-missing

Builds are unsigned unless signing certificates are configured for the repository. Windows SmartScreen will warn on first run (More info → Run anyway); macOS needs right-click → Open the first time, or xattr -dr com.apple.quarantine /Applications/Jarvis.app.

The app checks for updates a minute after launch and every six hours after that. It never downloads or installs on its own — a voice assistant should not restart itself mid-sentence. See docs/RELEASE.md for the full release process, signing and notarization.

Prerequisites

Requirement	Version	Purpose
Node.js	18 or higher	Runtime and build
Ollama	any current	Local model serving
uv	any current	Isolated Python environment for the neural TTS server. No longer needed for speech to text — that runs on Node now
JDK	17 or higher	Companion app only
Android SDK	platform 35, build-tools 35	Companion app only

Desktop

npm install

ollama pull gemma3:4b
ollama pull nomic-embed-text

npm run build

Ollama does not need to be running. JARVIS starts it if the port is idle and preloads the model so the first question does not pay a cold-load penalty.

Companion app

cd companion
./gradlew assembleDebug

The APK is written to app/build/outputs/apk/debug/app-debug.apk and served automatically during pairing.

Running

npm run electron

This runs the production build from dist/. For live reload while editing the renderer, run the Vite server and the development launcher in separate shells:

npm run dev
npm run electron:dev

Do not use electron:dev without npm run dev running, as it expects a server on port 5173.

Scripts

Command	Effect
npm test	Every suite, printing the total check count
npm run eval	Retrieval and memory benchmarks (needs Ollama)
npm run eval:mineral	Pipeline integrity against live USGS MRDS (needs network)
node scripts/build-mineral-db.mjs mrds.csv	Build cache/minerals.db — 304,613 deposits, ~10 s
npm run dev	Vite dev server on port 5173
npm run build	Production bundle into dist/
npm run electron	Launch against dist/
npm run electron:dev	Launch against the Vite server
npm run icon	Regenerate build/icon.png, the source for every platform icon
npm run dist	Package for the current platform into release/
npm run dist:win	Windows: NSIS installer, portable exe, zip
npm run dist:mac	macOS: universal DMG and zip
npm run dist:linux	Linux: AppImage, deb, rpm, tar.gz
npm run checksums	Write release/SHA256SUMS
npm run checksums:verify	Re-hash artifacts and fail on any mismatch
npm run smoke	Launch the packaged app and assert it starts
npm run electron:build	Legacy alias for a plain electron-builder run

The data crawls

Not npm scripts, and deliberately so: these spend money, and the results are already committed to data/. Nobody needs to run them to use the globe — they exist so the database can be refreshed, and so its provenance is reproducible rather than asserted.

Command	Cost	Effect
node scripts/fetch-ranking.mjs --status	none	What is already crawled
node scripts/fetch-ranking.mjs --pages 3	3 credits	Three pages, 300 companies
node scripts/fetch-ranking.mjs --all --max-credits 120	≤ 120 credits	The full 11,222-company ranking
node scripts/resolve-hq.mjs --status	none	Resolution progress and spend so far
node scripts/resolve-hq.mjs --top 500	≈ $16	Head offices for the richest 500
node scripts/resolve-hq.mjs --all	≈ $360	Every remaining company
node scripts/resolve-hq-wikidata.mjs --dry-run	none	City-level fallback for what Places refused

Every one of them defaults to the cheap behaviour. resolve-hq.mjs stops at --max-lookups 200 unless told otherwise, because the expensive default is the one that gets run by accident. All are resumable — results are written as they land, keyed by ticker, so an interrupt at company 9,000 does not re-buy 9,000 lookups — and all print the running spend in dollars, so stopping is an informed decision rather than a guess.

Configuration

Settings live in browser local storage and are seeded from src/js/settings.js. Relevant defaults:

Key	Default	Meaning
llmProvider	gemma-local	Local inference through Ollama
localOllamaUrl	http://localhost:11434	Ollama endpoint
localModel	gemma3:4b	Generation and vision model
micPreference	auto	headset, internal, or auto
echoCancellation	true	Stops JARVIS hearing itself
noiseSuppression	true	Filters fans and keystrokes
autoGainControl	true	Required for quiet microphones
ocrProvider	auto	Local OCR server when available

Environment overrides

The main process cannot read renderer local storage at boot, so these are available as environment variables:

Variable	Default
JARVIS_OLLAMA_URL	http://localhost:11434
JARVIS_LOCAL_MODEL	gemma3:4b
JARVIS_OCR_URL	http://127.0.0.1:10000
JARVIS_ADB_PATH	auto-detected
JARVIS_ETH_WS	keyed endpoint if available, else wss://ethereum-rpc.publicnode.com

Provider keys and .env

Copy .env.example to .env and fill in whatever you have. Every key is optional; see Provider keys for what each unlocks. The file is git-ignored, values are never logged (only the key names appear at startup), and a real environment variable always wins over the file.

cp .env.example .env

ALCHEMY_API_KEY=      # EVM RPC, portfolio, prices
HELIUS_API_KEY=       # Solana RPC, assets, activity
# DUNE_API_KEY=       # aggregate analytics
# ARKHAM_API_KEY=     # entity labels, spoken with attribution

GOOGLE_CLIENT_ID=     # Calendar and Meet
GOOGLE_CLIENT_SECRET=

GOOGLE_MAPS_API_KEY=  # globe: geocoding, places, weather, air quality
LUMA_API_KEY=         # globe: events from ONE Luma calendar (Luma Plus)
AVIATIONSTACK_API_KEY= # globe: real flight routes (free tier ~100-500/month)
WINDY_WEBCAMS_API_KEY= # globe: ~70k opt-in webcams worldwide (free key)
PARSE_API_KEY=        # globe: refresh the company ranking (metered, 1 credit/call)

GOOGLE_MAPS_API_KEY and PARSE_API_KEY are read only in the main process and never cross the IPC bridge — see Globe. Without either, the globe runs on the bundled gazetteer, Nominatim, Wikipedia, USGS, NASA EONET and the crawled company database in data/, all keyless and all offline.

Connecting Google Calendar

One-time setup, then "connect my calendar" does the rest.

• Google Cloud Console -> new project
• APIs & Services -> Library -> enable Google Calendar API (and Google Meet API if you want instant Meet rooms)
• OAuth consent screen -> External -> add yourself as a test user
• Credentials -> Create credentials -> OAuth client ID -> Application type: Desktop app
• Put the client ID and secret in .env

Then say "connect my calendar". Your system browser opens Google's real consent screen; JARVIS receives the code on a loopback port and stores a refresh token in the app's user-data directory at mode 0600. The renderer never sees a token.

The client "secret" is not secret for a desktop app — Google's own installed-app flow puts it in the binary — which is why the exchange also uses PKCE. See OAuth 2.0 for native apps.

To revoke: "disconnect my calendar", or remove the app at myaccount.google.com/permissions.

[!IMPORTANT] Programmatic Google Meet link creation requires a paid Google Workspace account. With a personal Gmail, events are created normally but without a Meet link, and JARVIS tells you so rather than pretending one exists.

At startup the log states exactly what was found and what it can reach:

[env] loaded keys: ALCHEMY_API_KEY, HELIUS_API_KEY
[chain] Alchemy verified in 394ms: arbitrum, ethereum, base, bsc | unavailable: optimism, polygon

Credentials

Secrets are held in an Electron safeStorage vault backed by Windows DPAPI. The renderer can set, list, and delete entries but can never read raw values. The typed command store key <name> <value> bypasses the model and conversation memory entirely. Provider keys can live here instead of .env, and the environment is checked first.

Network ports

All listeners bind locally or to the LAN. None are exposed to the internet.

Port	Service	Bind	Authentication
8765	Phone bridge HTTP	0.0.0.0	Bearer token, except the pairing routes
8766	Companion WebSocket	0.0.0.0	Token in X-Jarvis-Token, constant-time compare
8770	Whisper STT (transformers.js)	127.0.0.1	Loopback only
8771	Local TTS	127.0.0.1	Loopback only
8772	Vision llama-server, optional	127.0.0.1	Loopback only
11434	Ollama	127.0.0.1	Loopback only
10000	Unlimited-OCR, optional	127.0.0.1	Loopback only
5173	Vite dev server	127.0.0.1	Development only

One port is connected to rather than listened on: 127.0.0.1:5037, the ADB server, used by Tier 3 phone control and by the screen mirror. JARVIS does not bind it — adb owns it, and JARVIS starts the server if it is not already running.

Troubleshooting

JARVIS does not speak

Speech is suppressed while a cloud Live session is connected. Without a key that never happens, so text-to-speech should be active. If the voice list loaded late the selected voice may be null; check for the onvoiceschanged race.

The microphone stops working mid-session

Usually a Bluetooth profile switch. Speaking to earbuds forces Windows from A2DP to HFP, which tears down the capture device. The recovery path retries indefinitely with backoff, watches for track.onended, and runs a 5s watchdog that forces a restart when no frames arrive for 15 seconds.

The stable configuration is laptop microphone for input with earbuds for output. Full duplex over Bluetooth is inherently fragile on Windows.

JARVIS transcribes its own voice

The echo guard compares each transcript against recently spoken text by word overlap and drops matches above 60 percent. If self-talk still appears, confirm the selected microphone is not a loopback device such as Stereo Mix.

The STT server will not start

It needs no Python and no uv. JARVIS spawns server/stt-server.mjs on Electron's own binary with ELECTRON_RUN_AS_NODE=1. To see what it is actually doing — the app spawns it with stdio: 'ignore', so its log goes nowhere — run it by hand:

node server/stt-server.mjs

A healthy start prints three lines:

loading Whisper 'onnx-community/whisper-base.en' (q8) — first run downloads it...
model ready in 1.2s
listening on ws://127.0.0.1:8770

If it hangs on the first line, it is downloading the weights, which needs network access once. After that it is fully offline.

If port 8770 is already held, something else is serving it — usually a second JARVIS, or an orphaned server that survived a force-kill (before-quit does not run on a force-kill, so the child is not reaped). This is not an error: JARVIS reuses whatever answers on the port and logs STT server: port 8770 already served — not respawning. If that holder later dies, the 30-second watchdog spawns a replacement.

Override the model or port with JARVIS_STT_MODEL, JARVIS_STT_PORT and JARVIS_STT_DTYPE.

Companion shows OFFLINE

Confirm both devices share a subnet. A common failure is mDNS advertising a virtual adapter such as VirtualBox host-only at 192.168.56.1, which the phone cannot route to. Interface ranking now deprioritises virtual, Docker, WSL, and link-local adapters, and the phone tries every advertised address.

Pairing retries every 10 seconds, so opening the window after launching the app is fine.

Gradle fails to start

JAVA_HOME may point at a stale path. Set it explicitly:

$env:JAVA_HOME = "C:\Program Files\Microsoft\jdk-21.0.9.10-hotspot"

The alarm did not go off

Check where it was scheduled. A desktop-only alarm needs JARVIS running at the time — say "what alarms do I have" and, if the phone is paired, the confirmation will have said "Also set on your phone."

On the phone, in order of likelihood:

• Exact alarms revoked. Settings → Apps → JARVIS → Alarms & reminders. capabilities.alarms reports false when this is the cause, and schedule_alarm refuses rather than accepting an alarm it cannot ring.
• Battery optimisation killed the app. On MIUI, Settings → Apps → JARVIS → Battery saver → No restrictions, and lock the app in Recents.
• Rebooted and never re-armed. BootReceiver handles this, but it needs the app to have been opened at least once after install before the first reboot.
• Notifications denied. The ring screen is launched by a full-screen intent; without POST_NOTIFICATIONS on Android 13+ only the direct activity launch remains, and that is the path background-launch rules can block.

JARVIS answers things nobody said to it

Ambient audio — a video, a podcast, someone else in the room — being transcribed and answered. On the phone this is what the wake-word gate prevents: nothing becomes a turn until "Jarvis" is heard. If it is still happening, continuous listening is not the path in use; the in-app microphone button bypasses the gate by design, because a deliberate press is itself the wake signal.

Continuous listening stops on its own

The service gives up after six consecutive hard recognition failures and says so in its notification. Usual causes are the OEM restricting background SpeechRecognizer (MIUI), or the microphone being held by another app. Sitting in a restart loop while appearing to listen would be worse, so it stops.

The floating orb vanishes

SYSTEM_ALERT_WINDOW was revoked, or the process was killed without the foreground notification being visible. The permission cannot be requested with a dialog — Settings → Apps → JARVIS → Display over other apps.

The mirror opens black, or will not start

The messages in Screen mirror name the cause directly — they come from mirrorService, not the model, so treat them literally. The two that are not self-explanatory:

• does not match the pinned v3.3.3 build means resources/scrcpy-server.jar was replaced. A newer scrcpy jar is not an upgrade; the client speaks the 3.3.3 protocol and the handshake compares version strings exactly.
• A black panel with a healthy-looking session is the decoder never receiving scrcpy's configuration packet, which carries SPS/PPS. Packets are queued from before the handshake and replayed once the decoder exists, so this should not recur — if it does, check that the video subscription is opened before start().

Answers ignore stored memory

Check stats() on the retrieval service. Chunks stored while Ollama was unavailable have a null vector and are invisible to dense search. Backfill runs automatically on load once an embedder is reachable.

Known limits

These are deliberate or platform-imposed, not defects.

• No barge-in. The microphone is gated while speaking. Synthesised audio bypasses Chromium's echo cancellation and would otherwise be transcribed as user input.
• Video listening lags by a window, and cannot not. Whisper cannot begin until it holds 5000 ms of audio, so the floor on how far behind the source JARVIS can be is one window plus transcription — measured at 1.06–1.41s, landing end-to-end around two to three seconds. Shorter windows cut the lag and cost accuracy, because Whisper starts guessing at half-heard words.
• Video listening repeats, it does not summarise. No model rewrites the transcript on its way to the speakers; the dedup pass is string comparison with fixed rules. What JARVIS says is what Whisper heard, minus the overlap repetition. Summarising is a separate thing you have to ask for.
• Speech to text is offline; neural text to speech is not. stt-server.mjs runs Whisper locally with no key. The neural voice in tts-server.py uses edge-tts, which streams from Microsoft's endpoint — the one remaining outbound leg of the voice loop. The system SAPI voice is the offline fallback.
• The connection graph only knows what EDGAR and the feeds put in it. It is small and honest by construction. "I have never heard of this entity" and "these two are in the graph and nothing connects them" are different answers and are kept different, because collapsing them sounds identical to a listener and means the opposite.
• No general open-web index. Web search federates official keyless APIs plus a BM25 pass over already-crawled feeds. It is not a crawler and does not try to be: Google indexes hundreds of billions of pages, and a personal crawler would spend its time re-fetching what the live providers already return. Where a personal index genuinely wins is the narrow set the user tracks, which is what the feed poller already collects.
• Keyless web search has no general provider. Google's Custom Search JSON API shuts down 1 Jan 2027, Bing's Search APIs were retired 11 Aug 2025, and Brave dropped its free tier. Without BRAVE_API_KEY, coverage is DuckDuckGo's abstracts, Wikipedia, Google News and the intent-gated specialised indexes — strong on entities, current events, code, papers and CVEs; weak on arbitrary open-web pages.
• Search answers are extractive, never generated. A spoken answer is a sentence lifted from a fetched page and checked against it before speaking. Nothing is summarised by a model, because a model summarising search results is how the fabricated citations above got in.
• Radio toggles need administrator rights. Wi-Fi scanning and connecting to saved profiles work at user level; enabling the adapter does not. JARVIS opens the relevant Settings page and says so plainly.
• Silent APK install is impossible. Android reserves it for device-owner applications. Google Play policy separately prohibits self-updating outside Play. Delta patching would cut transfer size but cannot remove the install prompt. capabilities.silent_install reports false accordingly.
• The companion is sideload-only. Google restricts accessibility APIs to genuine accessibility use, so Tier 2 would not survive Play review.
• Desktop alarms need JARVIS running; phone alarms do not. The desktop scheduler lives in the renderer, so a closed app fires nothing. This is not fixable on the desktop — it is what a renderer-side timer is. Pair the phone for anything that has to wake you.
• Exact alarms are user-revocable. On Android 12 and 13 the user can turn off SCHEDULE_EXACT_ALARM, after which nothing here can schedule a reliable wake-up. capabilities.alarms reports the live answer and the phone refuses the command with the Settings path rather than accepting an alarm it will not ring.
• Continuous listening costs battery, visibly. Recognition runs whenever the service is up, which is why it is opt-in and carries a permanent notification with a Stop action. There is no configuration that hides that notification.
• On-device speech recognition needs the language pack. Without it Android falls back to Google's server path and audio leaves the phone. The notification says which mode is running; it is not silently degraded.
• Some OEM builds restrict background SpeechRecognizer. MIUI in particular. The service reports repeated failures and stops rather than looping while appearing to listen.
• The floating orb cannot be requested with a dialog. SYSTEM_ALERT_WINDOW is a special permission granted only through a Settings page, so the app can route you there but cannot prompt.
• Bundled OCR costs about 40 MB of APK. ML Kit's bundled model was chosen over the Play Services variant so a sideloaded install needs no Google download and the screen never leaves the device — at the price of a much larger APK, which the offline Wi-Fi Direct share then has to carry.
• LAN traffic is cleartext. The bridge is plain HTTP and WebSocket on a DHCP address that cannot be pinned by CIDR, so network_security_config.xml permits cleartext. Authentication is the shared bridge token. Do not run this on an untrusted network.
• The orb is cropped in portrait. The desktop camera sits at z=14, which assumes a landscape aspect ratio.
• Nothing vibrates on a desktop. navigator.vibrate is callable in Electron and moves nothing, because there is no motor. Feedback is animation plus a short synthesized click; the vibration channel reports itself unavailable rather than pretending. See docs/FEEDBACK.md.
• The mirror needs USB debugging, and one device. There is no wireless fallback that skips ADB, and with several devices connected JARVIS asks which one rather than picking. Ambiguity is an error here, never a coin flip.
• The scrcpy server jar is pinned, not tracked. It is upgraded when the client library implements a newer protocol, not when scrcpy releases. A version bump is a code change with a hash change, by design.
• Glass-to-glass mirror latency is not measurable from inside the app. The device and host clocks share no origin. The LAG badge reports arrival delay above the session's best observed path, which is a real measurement of a different thing.
• No order placement, no signing. The finance and on-chain modules are read-only by construction. No code path anywhere in the project can place a trade, sign a transaction, or handle a private key.
• No entity attribution. JARVIS will not tell you a wallet belongs to Binance or Coinbase, because that fact is not on-chain. It comes from a proprietary database, and guessing it is how alerts become untrustworthy. With your own Arkham key, labels are used and spoken with attribution.
• The whale stream is Ethereum only. Arbitrum's sub-second blocks and Solana's event rate — over 200 token-program events in 15 seconds, measured — are firehoses this machine cannot filter while also running voice.
• Address history needs an indexer. Public RPC cannot enumerate the transactions of an address, so the fund tracer is tested against synthetic graphs and awaits an Etherscan-family key for live use.
• Historical log windows are chunked and may be partial. Free RPC endpoints cap eth_getLogs ranges between 10 and 50 blocks and rate-limit under load. Coverage is reported rather than assumed.

Developed by Ashutosh Kumar Singh (Ashutosh0x)