Natural Language User Interface
![ForTheBadge built-with-swag](http://ForTheBadge.com/images/badges/built-with-swag.svg)
![Python](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)
Platform Supported
![Generic badge](https://img.shields.io/badge/Platform-Linux%7CMacOS%7CWindows-1f425f.svg)
Language Stats
![Code coverage](https://img.shields.io/github/languages/top/thevickypedia/Jarvis)
Repo Stats
![GitHub Repo watchers](https://img.shields.io/github/watchers/thevickypedia/Jarvis)
![LOC](https://img.shields.io/tokei/lines/github/thevickypedia/Jarvis)
![GitHub Repo pr](https://img.shields.io/github/issues-pr-raw/thevickypedia/Jarvis)
Code Stats
![Make](https://img.shields.io/github/search/thevickypedia/Jarvis/Makefile)
Deployments
![pypi](https://github.com/thevickypedia/Jarvis/actions/workflows/python-publish.yml/badge.svg)
![sourcerank](https://img.shields.io/librariesio/sourcerank/pypi/jarvis-ironman)
Activity
![GitHub last release](https://img.shields.io/github/release-date/thevickypedia/Jarvis)
Development and Maintenance
![Maintainer](https://img.shields.io/badge/Maintained%20By-Vignesh%20Sivanandha%20Rao-blue.svg)
Reach Out
![Ask Me | Anything](https://img.shields.io/badge/Ask%20me-Anything-1abc9c.svg)
Kick off
Install
python -m pip install jarvis-ironman
Initiate
import jarvis
if __name__ == '__main__':
jarvis.start()
Prerequisites
-
MacOS
Tested on macOS High Sierra, Mojave, Catalina, Big Sur, Monterey and Ventura*
System Preferences
→ Security & Privacy
→ Privacy
- Click
+
sign and add the preferred IDE
and Terminal
in the following sections in left pane.
Microphone
- Required to listen and respond.Accessibility
- Required to use key combinations for brightness and volume controls.Camera
- [Optional] Required only during face recognition/detection.Automation
- Required to control System Events
and other apps like Outlook and Calendar.Files and Folders
[OR] Full Disk Access
- Required for all FileIO
operations.
:warning: Known Issue with <a href=https://pypi.org/project/pyttsx3/>pyttsx3 module on <a href=https://www.apple.com/macos/ventura/> macOS Ventura 13.0: This version of macOS does not hold the attribute VoiceAge
. <a href=https://github.com/nateshmbhat/pyttsx3/pull/247>Workaround has been raised as a PR
-
Linux
Tested on Ubuntu 22.04 LTS
- Store the host machine's password as the env var:
ROOT_PASSWORD
- Unlike macOS and Windows,
Ubuntu
does not have app specific permissions.
-
Windows
Tested on Windows 10
Settings
→ Privacy
Microphone
- Required to listen and respond.Camera
- [Optional] Required only during face recognition/detection.- Unlike
macOS
, Windows
pops a confirmation window to Allow or Deny access to files and folders.
- Install Anaconda or Miniconda, VisualStudio C++ BuildTools, and Git
- Make sure C++ build tools are installed completely and restart
- Add anaconda/miniconda scripts location to
PATH
in Environment Variables
Test peripherals
Camera
from jarvis.modules.camera import camera
if __name__ == '__main__':
cam_object = camera.Camera()
print(cam_object.get_index())
print(cam_object.list_cameras())
Text to Speech
from jarvis.modules.speaker import speak
if __name__ == '__main__':
speak_object = speak.Speaker()
speak_object.run()
print(list(speak_object.get_all_voices()))
Speech to Text
import asyncio
from jarvis.modules.microphone import recognizer
if __name__ == '__main__':
asyncio.run(recognizer.main())
ENV Variables
Environment variables are loaded from a .env
file and validated using pydantic
More on Environment variables
-
ROOT_PASSWORD - System password to get the system vitals and run other sudo
commands. Mandatory for Linux
-
NAME - Name which Jarvis should address the user by. Defaults to Vignesh
-
TITLE - Title which Jarvis should address the user by. Defaults to sir
-
PLOT_MIC - Boolean value whether to show microphone usage in realtime. Defaults to True
-
LOG_RETENTION - Number of days to store the logs. Defaults to 10
-
WAKE_WORDS - List of wake words to initiate Jarvis' listener. Defaults to ['jarvis']
(Defaults to ['alexa']
in legacy macOS)
:warning: Jarvis has limitations on the wake words as it relies on ML libraries for wake word detection.
-
VOICE_NAME - Name of the voice supported by the OperatingSystem. Defaults to the author's favorite.
-
VOICE_RATE - Speed/rate at which the text should be spoken. Defaults to the value from pyttsx3
module. Typically 200
To add more voices
macOS:
- System Preferences → Accessibility → Spoken Content → System voice → Manage Voices...
Windows:
- Settings → Time & Language → Speech → Manage voices → Add voices
-
SENSITIVITY - Hot word detection sensitivity. Allowed range: [0-1] Defaults to 0.5
-
TIMEOUT - Timeout in seconds until which the listener should wait for speech. Defaults to 3
-
PHRASE_LIMIT - Timeout in seconds until which the listener will remain active. Defaults to None
-
LIMITED - Boolean flag to run only the main version of Jarvis
skipping background processes. Defaults to False
Enforced based on the number of CPU cores.
-
DEBUG - Boolean flag to enable debug level for logging. Defaults to False
-
RECOGNIZER_SETTINGS - A JSON object that has with customized speech recognition settings.
Custom settings for speech recognition
These are customized according to the author's voice pitch.
Please use recognizer.py to figure out the suitable values on a trial and error basis.
These settings are added (optionally), to avoid the hard coded PHRASE_LIMIT
Cons in using hard coded PHRASE_LIMIT
:
- Disables the listener after the set limit even the speaker is actively talking.
- Listener will be active until the set limit even after the speaker has stopped talking.
Sample settings (formatted as JSON object)
RECOGNIZER_SETTINGS
: '{"energy_threshold": 1100, "dynamic_energy_threshold": false, "pause_threshold": 2, "phrase_threshold": 0.1, "non_speaking_duration": 2}'
Description
energy_threshold
: Minimum audio energy to consider for recording. Greater the value, louder the speech should be.dynamic_energy_threshold
: Change considerable audio energy threshold dynamically.pause_threshold
: Seconds of non-speaking audio before a phrase is considered complete.phrase_threshold
: Minimum seconds of speaking audio before it can be considered a phrase - values below this are ignored. This helps to filter out clicks and pops.non_speaking_duration
: Seconds of non-speaking audio to keep on both sides of the recording.
Peripherals
- CAMERA_INDEX - Camera index that has to be used. Run camera.py to get the index value of each camera.
- SPEAKER_INDEX - Speaker index that has to be used. Run peripherals.py to get the index value of each speaker.
- MICROPHONE_INDEX - Microphone index that has to be used. Run peripherals.py to get the index value of each microphone.
Features
- GIT_USER - GitHub Username
- GIT_PASS - GitHub Token
- WEATHER_API - API Key from openweathermap
- NEWS_API - API Key from newsapi
- MAPS_API - API Key for maps from Google
- BIRTHDAY - Birth date in the format DD-MM - Example:
24-April
- WOLFRAM_API_KEY - API Key from wolfram alpha.
Email/SMS notifications
- GMAIL_USER - Gmail account username to send and read emails.
- GMAIL_PASS - Gmail account password to send and read emails.
- ALT_GMAIL_USER - Alternate gmail account username to send an SMS. (
GMAIL_USER
can be re-used) - ALT_GMAIL_PASS - Alternate gmail account password to send an SMS. (
GMAIL_PASS
can be re-used) - RECIPIENT - Email address to which the emails from jarvis have to be received.
iOS integrations
- ICLOUD_USER - iCloud account username/email.
- ICLOUD_PASS - iCloud account password.
- ICLOUD_RECOVERY - Recovery phone number to activate lost mode on a target device - Example:
+11234567890
- PHONE_NUMBER - To send SMS from Jarvis - Example:
+11234567890
Calendar/Meeting integrations
- ICS_URL - Shared calendar URL to get meetings information from. Should end with
.ics
- EVENT_APP - To read events from
outlook
or calendar
application in macOS
. Defaults to calendar
:bulb: When calender
is used, the name of the calendar within the Calendar.app
should be Jarvis
Background scans [Defaults to 1 hour]
- SYNC_MEETINGS - Interval in seconds to generate
meetings
information using an ics
URL. - SYNC_EVENTS - Interval in seconds to generate
events
information using calendar
or outlook
application.
Wi-Fi Controls
- WIFI_SSID - SSID of the wireless connection.
- WIFI_PASSWORD - Password for the wireless connection.
- CONNECTION_RETRY - Frequency in seconds to check for an active internet connection. Defaults to 10 seconds.
VPNServer integration
- VPN_USERNAME - Username to create vpn-server. Defaults to profile username.
- VPN_PASSWORD - Password to authenticate vpn-server. Defaults to profile password.
- VPN_DOMAIN - Domain name for the hosted zone.
- VPN_RECORD_NAME - Alias record name to access VPN server.
Car Controls - Applies only for JLR vehicles subscribed to InControl
application.
- CAR_EMAIL - Email address to log in to InControl API.
- CAR_PASS - Password to authenticate InControl API.
- CAR_PIN - InControl PIN.
Garage Controls - Applies only for garages using MyQ garage controller.
- MYQ_USERNAME - Email address to log in to MyQ API.
- MYQ_PASSWORD - Password to authenticate MyQ API.
Telegram Bot integration
- BOT_TOKEN - Telegram BOT token.
- BOT_CHAT_IDS - UserID/ChatID for a particular user.
- BOT_USERS - Usernames that should have access to Jarvis.
OS Agnostic Voice Model
-
SPEECH_SYNTHESIS_TIMEOUT - Timeout to connect to the docker container that processes text to speech requests.
-
SPEECH_SYNTHESIS_VOICE - Voice for the speech synthesis model. Defaults to author's favorite.
-
SPEECH_SYNTHESIS_QUALITY - Quality of speech synthesis conversion. Defaults to medium
.
To enable independent speech-synthesis
docker run \
-it \
-p 5002:5002 \
-e "HOME=${HOME}" \
-v "$HOME:${HOME}" \
-v /usr/share/ca-certificates:/usr/share/ca-certificates \
-v /etc/ssl/certs:/etc/ssl/certs \
-w "${PWD}" \
--user "$(id -u):$(id -g)" \
rhasspy/larynx
:bulb: Speech Synthesis can run on a docker container for better voices but, response might be negligibly slower. If you don't have docker installed or simply don't want to use it, set the SPEECH_SYNTHESIS_TIMEOUT
env var to 0. This is also done automatically if failed to launch a docker container upon startup.
Offline communicator
- OFFLINE_PORT - Port number to initiate offline communicator. Defaults to
4483
- OFFLINE_PASS - Secure phrase to authenticate offline requests. Defaults to
OfflineComm
- WORKERS - Number of uvicorn workers (processes) to spin up. Defaults to
1
Stock Portfolio
- ROBINHOOD_USER - Robinhood account username.
- ROBINHOOD_PASS - Robinhood account password.
- ROBINHOOD_QR - Robinhood login QR code
API Features
- ROBINHOOD_ENDPOINT_AUTH - Authentication token to access the robinhood portfolio which is generated every hour.
- SURVEILLANCE_ENDPOINT_AUTH - Token to access webcam live feed via Jarvis API.
- SURVEILLANCE_SESSION_TIMEOUT - Session time out for
/surveillance
. Defaults to 300 seconds.
Background Tasks
There are two options to run background tasks on Jarvis.
-
Jarvis can run internal tasks (offline communicator compatible) at certain intervals using a background_tasks.yaml
file stored in fileio
directory.
Setup Instructions
This is the sample content of background_tasks.yaml
- seconds: 1_800
task: just turn off all lights
- seconds: 10_800
task: remind me to drink water
ignore_hours:
- 21
- 22
- 23
- 0
- 1
- 2
- 3
- 4
- 5
- 6
-
CRONTAB - Runs external tasks using cron expressions. Needs to be stored as env var.
Sample value
[
"0 0 * * 1-5/2 find /var/log -delete",
"0 5 * * 1 tar -zcf /var/backups/home.tgz /home/"
]
Contacts
Jarvis can send on demand notifications using a contacts.yaml
file stored in fileio
directory. Uses gmail-connector for SMS and email notifications.
Setup Instructions
Note: Jarvis currently supports sending emails only when the contacts.yaml
file is present, however phone numbers can be used directly.
phone:
Tony: 0123456789
Thor: 1234567890
email:
Eddard: ned@gmail.com
Aegon: egg@yahoo.com
Smart Devices
A source file smart_devices.yaml
is used to store smart devices' hostnames.
Jarvis supports MagicHome
for lights, LGWebOS
and Roku
for TVs.
-
TV hostnames should include the brand name [LG
/Roku
] to distinguish the modules accordingly.
- This will be set by default, if yours doesn't include the brand name change it in the TV settings.
-
To wake up Roku
TVs using MAC address, make sure the Bandwidth saver
feature is turned off under,
Settings/Network/Bandwidth saver >> Off
-
For first time users on LGWebOS
TVs, there will be a prompt on the TV to accept the connection request.
- Once the connection request is accepted a client key will be generated and logged.
- Please make sure to store this in
smart_devices.yaml
file to avoid repeated connection prompt.
Setup Instructions
- TV identifiers should have the word
tv
to distinguish between lights and tv. - The name used in the keys (for both lights and tv) will be the identifier when an action is requested.
- Lights should be a dictionary of identifier and a list of hostnames.
- TVs should be a nested dictionary of multiple parameters.
- The source file (
smart_devices.yaml
) should be as following:
bedroom:
- 'HOSTNAMES'
hallway:
- 'HOSTNAMES'
hallway basement:
- 'HOSTNAMES'
kitchen:
- 'HOSTNAMES'
living room:
- 'HOSTNAMES'
party mode:
- 'HOSTNAMES'
living room tv:
hostname: 'HOSTNAME'
client_key: 'CLIENT_KEY'
mac_address:
- 'WIRED_MAC_ADDRESS'
- 'WIRELESS_MAC_ADDRESS'
bedroom tv:
hostname: 'HOSTNAME'
mac_address: 'MAC_ADDRESS'
Automation Setup [Optional]
Jarvis can execute offline compatible tasks
at pre-defined times without any user interaction. Uses an automation.yaml
file as source which should be stored
within the directory fileio
Setup Instructions
The YAML file should be a dictionary within a dictionary that looks like the below.
OPTIONAL: The key, day
can be a list
of days, or a str
of a specific day or simply a str
saying weekday
or
weekend
when the particular automation should be executed.
Not having the key day
will run the automation daily.
Date format should match exactly as described below.
06:00 AM:
day: weekday
task: set my bedroom lights to 50%
06:30 AM:
day:
- Monday
- wednesday
- FRIDAY
task: set my bedroom lights to 100%
08:00 AM:
day: weekend
task: set my bedroom lights to 100%
09:00 PM:
task: set my bedroom lights to 5%
12:00 AM:
task: restart all background processes
Simulation Setup [Optional]
Jarvis can execute offline compatible tasks
as a simulation to test the required functions and send an email with the results. Uses a simulation.yaml
file as source which should be stored
within the directory fileio
Setup Instructions
The YAML file should be a list of phrases within a dictionary that looks like the below.
meeting_event:
- get me the events from my calendar
- what meetings do I have today
Guide
Please refer to the wiki page for API usage, access controls, env variables, features' overview and demo videos.
FAQs
Please refer to the FAQs section of the wiki.
Coding Standards
Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8
and
isort
Requirement
python -m pip install changelog-generator
Usage
changelog reverse -f release_notes.rst -t 'Release Notes'
Linting
PreCommit
will ensure linting, and the doc creation are run on every commit.
Requirement
pip install sphinx==5.1.1 pre-commit recommonmark
Usage
pre-commit run --all-files
Pypi Package
![pypi-module](https://img.shields.io/badge/Software%20Repository-pypi-1f425f.svg)
https://pypi.org/project/jarvis-ironman/
Runbook
![made-with-sphinx-doc](https://img.shields.io/badge/Code%20Docs-Sphinx-1f425f.svg)
https://thevickypedia.github.io/Jarvis/
License & copyright
© Vignesh Sivanandha Rao
Licensed under the MIT License