Voice Interface

Usejarvis includes a full voice interface: speak your requests using the wake word or the pebble, and hear responses streamed back as speech in real time. Text-to-speech requires no API key by default. Speech-to-text supports cloud and local providers.

Voice, TTS, and STT settings live in the dashboard (Settings → Channels and Voice). They are stored in the settings database; a legacy stt:/tts: block in config.yaml is imported once at boot, but the file has no ongoing authority. The YAML snippets below show the shape of each setting.

Overview

You speak  →  16 kHz PCM (AudioWorklet)  →  WAV  →  STT  →  text  →  Agent
Agent response  →  sentence split  →  TTS  →  MP3 per sentence  →  speaker

Audio travels over the existing WebSocket connection as binary frames alongside the normal JSON message stream. The browser captures the microphone, plays audio, and manages voice state. Each TTS chunk is a complete MP3 per sentence, which is what makes low-latency playback possible.

Text-to-Speech

TTS is off by default (tts.enabled: false). Turn it on in Settings.

Default Provider: Edge

The default provider is edge, which uses Microsoft Edge’s neural voices. Completely free: no API key, no account.

tts:
  enabled: true
  provider: edge
  voice: en-US-AriaNeural    # Female, US English
  rate: "+0%"                # "-20%" (slower) to "+30%" (faster)
  volume: "+0%"

The dashboard voice picker offers Aria, Guy, Sonia, Natasha, Jenny, and Davis.

ElevenLabs

For premium voices:

tts:
  provider: elevenlabs
  elevenlabs:
    api_key: "..."
    voice_id: "21m00Tcm4TlvDq8ikWAM"   # Rachel (default)
    model: "eleven_flash_v2_5"
    stability: 0.5                      # 0.0 to 1.0
    similarity_boost: 0.75              # 0.0 to 1.0

Get an API key at elevenlabs.io. The first-run dashboard setup can configure ElevenLabs, and the dashboard can list your ElevenLabs voices. A malformed voice_id silently falls back to Rachel.

Sarvam

Sarvam AI is also supported as a TTS provider (default model bulbul:v3), with language and speaker options. Configure it in Settings.

Streaming Playback

Usejarvis does not wait for the full response before playing audio. Responses are split into sentences, and each sentence is synthesized and streamed to the browser as it is produced. This minimizes perceived latency.

When a spoken sentence contains the word “Jarvis”, the daemon flags it so the browser suppresses the wake-word listener while that audio plays. The assistant does not wake itself up.

Speech-to-Text

Four STT providers are supported. Configuration is nested per provider:

stt:
  provider: openai            # openai | groq | local | sarvam
  openai:
    api_key: "sk-..."
    model: whisper-1

• OpenAI: Whisper via the OpenAI API.
• Groq (default model whisper-large-v3-turbo): fast Whisper hosting.
• Local: point Usejarvis at a whisper.cpp server or any OpenAI-compatible transcription endpoint you run yourself: local.endpoint: http://localhost:8080, local.server_type: whisper_cpp. The daemon POSTs the recorded WAV to the endpoint. Nothing leaves your machines.
• Sarvam AI (default model saaras:v3), with a language option.

Transcription for OpenAI, Groq, and local endpoints is currently pinned to English.

STT also powers voice messages on Telegram and Discord.

Wake Word

The wake phrase is “Hey Jarvis” (or just “Jarvis”). It comes from the shipped detection model and is not renameable today.

Detection runs entirely in the browser via ONNX Runtime Web (WASM), so no audio leaves your machine while listening. The only real setting is the engine:

voice:
  wake_engine: openwakeword   # openwakeword | webspeech | auto

• openwakeword (default): local ONNX models, private.
• webspeech: the browser’s speech recognition, which may stream audio to the browser vendor’s cloud.
• auto: prefer webspeech, fall back to openwakeword.

Sensitivity (0.3), wake cooldown (2 s), and the 1.5 s silence auto-stop are fixed values today, not user-configurable.

Voice States

idle → wake_detected → recording → processing → speaking → idle

Errors are recoverable and return to idle. You can also start a turn manually: tap the pebble.

Realtime Voice (GPT Realtime)

An opt-in premium mode replaces the STT → LLM → TTS pipeline with a single OpenAI Realtime speech-to-speech connection:

voice:
  realtime:
    enabled: false            # off by default
    model: gpt-realtime-2
    max_session_minutes: 10
    monthly_budget_usd: 0     # 0 = no budget cap

It reuses your OpenAI key from Settings → LLM. In realtime mode, tool calls auto-approve except for blocked categories (payments, deletes, shell execution, installs, settings changes, agent termination), which are always refused in-session.

Troubleshooting

Wake word not triggering

• Check the mic permission in your browser
• Try switching voice.wake_engine
• Say the shipped phrase: “Hey Jarvis”

No audio playback

• Confirm tts.enabled: true in Settings
• Interact with the page once first: browsers block autoplay before a user gesture

Transcription fails

• Run jarvis doctor to check STT configuration
• For local STT, confirm your whisper.cpp server is reachable from the daemon, not from your browser