Jexa-S AI
Local-first voice assistant and life coach. Wake-name listening, Whisper speech-to-text, persistent memory in Chroma, coaching playbooks and an Android companion. Runs on Ollama on your own PC; phone actions are drafts you confirm.
Python, Ollama, Whisper, ChromaDB, SQLite, WebSocket, Kotlin
The problem
Cloud assistants send everything you say to a server and can act on your behalf without asking. Jexa-S is built to run on your own PC, remember what you tell it, coach you through goals and conversations, and keep every outgoing phone action behind a manual confirmation.
My role
Responsibilities
Sole developer: the voice pipeline, memory layer, tool router, coaching playbooks, proactive check-ins, the Android companion and the evaluation harness.
Main workflows
- 1.You say the wake name, and the follow-up window keeps the conversation open without repeating it
- 2.Speech goes through voice-activity detection and Whisper, then a router decides between a tool call, a coaching playbook or a plain answer
- 3.Ollama answers with retrieved memory and playbooks in context, and the reply is spoken back with Edge TTS
- 4.Proactive mode sends a wake-up alarm, morning and evening check-ins, and late-night or long-session nudges
- 5.Calls, SMS and WhatsApp messages are drafted on the PC and only sent when you tap send on the phone
Architecture
- Python main loop with a router in front of the model: mic, VAD, Whisper, router, Ollama (with memory, playbooks and tools), then TTS
- Local models through Ollama (llama3.2:1b for chat, nomic-embed-text for embeddings); a cloud model is optional and off by default
- Persistent memory in Chroma with fact extraction, and SQLite for reminders, goals and people to stay in touch with
- Kotlin Android companion connected over a WebSocket bridge for notifications, calls, SMS and share-to-Jexa-S
Security and deployment
- Opening apps or links asks yes or no first, and only allowlisted apps can be opened
- Every outgoing phone action is a draft; an audit trail is written to a local log
- Contacting people, posting or buying without confirmation is deliberately not built
- The phone bridge uses a shared token and is meant for a trusted local network
Results
Evaluation baseline from Oct 17: unit tests 14/14 and routing 14/14. The hard routing set is 0/6 and is the open problem.