PyAI
default- STT
- pyai-hear (WS stream)
- Correction
- gpt-5.6-sol
- Wire format
- Anthropic-style
Native stable/active split — the contract every other adapter normalizes to.
back to the short version
Stats for nerds
The architecture, the measured numbers, and the parts still marked “pending”. This is the honest version.
~590ms
to first partial
mic → streaming STT
<1s
correction after pause
segment → animated diff
16ms
partial → on screen
local render budget
live
as spoken
Layer 1 never blocks
Architecture
Two layers, moving at different speeds
A self-correction can't be detected until it happens — “8 pm” only becomes wrong once “no no make it 9 pm” arrives. So raw words stream continuously, and correction fires at pause boundaries.
Layer 1
Instant, verbatim
Never waits on the model. What you said appears as you say it.
Mic
16 kHz PCM, ~20 ms frames
cpal
Streaming STT
WebSocket partials
pyai-hear
Live text
stable solid · active dim
~590 ms
on final segment ↓
Layer 2
Explained corrections
Lands under a second after you pause — and shows exactly what it removed.
Segment
pause / VAD boundary
500–800 ms
LLM pass
compact edit ops
JSON
Animated diff
strike-through → fade
reason labels
Inject
AX write · paste fallback
focused field
Latency budget
Target vs. measured
Layer 1 is independent of the model, so the live feel is guaranteed. The correction pass is the number we're still driving down with compact edit-ops.
| Stage | Target | Actual | Status |
|---|---|---|---|
| Mic → first partial | ~300 ms | ~590 ms | measured |
| Partial → on screen | < 16 ms | local render | ok |
| Pause → segment final | 500–800 ms | VAD timeout | pending |
| Segment → correction | 300–800 ms | compact-ops | tuning |
| Correction animation | 200–400 ms | CSS | ok |
Adapters
Vendor-agnostic, all the way down
Speech-to-text and correction are two separate interfaces. Pick a vendor for each — Deepgram for listening and Anthropic for cleanup is a perfectly normal setup. Adding a vendor is one file.
Deepgram
STT only- STT
- streaming WS
- Correction
- —
- Wire format
- Deepgram
Interim/final events; stable and active text computed from finals.
OpenAI
both roles- STT
- Whisper / Realtime
- Correction
- GPT via chat completions
- Wire format
- OpenAI
Covers speech-to-text and the correction pass on its own.
Anthropic
correction only- STT
- —
- Correction
- Claude via /v1/messages
- Wire format
- Anthropic
Native tool-use makes structured edit ops especially clean.
STT_PROVIDER and CORRECTION_PROVIDER resolve through a small registry — mix and match freely, no core changes.
Security
A gate, not a milestone
Every PR passes secret-scan, SAST and dependency audit from M0 onward. The assets we're protecting are your audio, your transcripts and your vendor keys.
Secret scanning
gitleaks on every push and PR, plus a full-history scan and pre-commit hooks.
SAST + dependency audit
CodeQL on TypeScript and Python, npm audit and pip-audit gating merges, Dependabot on.
Signed builds
Pinned lockfiles, reproducible releases and signed binaries before anything ships.
Keys never travel
OS keychain storage, redacted logs, and no key in the renderer or client bundle. Ever.
packages/coreapps/widgetapps/backenddocs/
Roadmap
M0 → M6
Every milestone ends in something you can run or demo. The two scariest unknowns — correction latency and macOS injection — are closed early, in M1 and M3.
M0shipped
Foundation & de-risking
PyAI mapped, streaming protocol decoded, vendor-neutral core, CI gates.
M1in progress
Core pipeline, headless
Audio in, live partials + validated edit ops out. Closes the latency risk.
M2in progress
Live web demo
Speak in a browser, watch it transcribe and correct itself end to end.
M3planned
Desktop widget (macOS)
Hotkey, non-focusable overlay, injection into the focused field.
M4planned
Multi-vendor + config
Deepgram, OpenAI, Anthropic adapters. Keys in the OS keychain.
M5planned
Quality & polish
Custom vocabulary, undo, formatting modes, daily-driver reliability.
M6planned
Open-source v1.0
Windows support, signed builds, public repo, final security review.
Each arrow is a hard gate — the left side must be demoable before the right side starts.
Read the whole thing yourself.
Interfaces, correction prompt, reconstructor and diff logic — all public, all MIT.