Skip to Content
Mélodium 0.10.3 is now available!
DocsExamplesAI Voice Assistant

AI Voice Assistant

Source: showcase/ai_voice_assistant See in Playground

This combines several ml capabilities from across the library into one demo, without pausing to explain every underlying concept; those are covered individually in the tutorial track. Two entrypoints: chat, an HTTP server that streams a remote LLM’s response token by token, and voice, a full loop from microphone to spoken reply.

Note

Requires real API keys (an LLM provider for both entrypoints, plus ElevenLabs for voice) and, for voice, a working microphone. voice also downloads a small Whisper model from the HuggingFace Hub on first run and caches it locally.

Running

chat, an HTTP server streaming a remote LLM’s response back token by token:

cd showcase/ai_voice_assistant
melodium run Compo.toml chat --api_key sk-...
 
curl -X POST http://127.0.0.1:8080/chat -d "What is Mélodium?"

voice, microphone to local Whisper (speech-to-text) to remote LLM (streamed reply) to remote text-to-speech to answer.mp3:

cd showcase/ai_voice_assistant
melodium run Compo.toml voice --llm_api_key sk-... --tts_api_key el-...

Speech-to-text runs locally (a small Whisper model, fetched once from HuggingFace Hub and cached), so raw audio never leaves the machine: only the transcribed text is sent to the LLM, and only the LLM’s reply text is sent to the TTS provider.

Optional: add --api-report and an API token (MELODIUM_API_TOKEN) to see this run’s full trace on Cadence.CI .

How it works

chat: streaming an LLM response over HTTP

chat starts an HttpServer and, exactly as established for the HTTP server tutorial step, gates the response on connection.started rather than on the body stream starting:

treatment respond[llm: RemoteLlm]()
  input  data: Stream<byte>
  output data: Stream<byte>
{
    decoded: decode()
    reply:   stream[llm=llm]()
    encoded: encode()
    Self.data -> decoded.data,text -> reply.prompt,token -> encoded.text,data -> Self.data
 
    llmErrors: logErrors(label="llm")
    reply.error -> llmErrors.messages
}

voice: microphone to spoken reply

voice first fetches and loads a Whisper model from an HfHub repository, then records from the microphone with recordMono and transcribes locally:

model WhisperHub() : HfHub   { repo_id = "openai/whisper-tiny" }
model Asr()        : Whisper {}
fetchAsr: fetch[hub=whisperHub]()
loadAsr:  load[whisper=asr]()
 
startup.trigger      -> fetchAsr.trigger
fetchAsr.safetensors -> loadAsr.safetensors
fetchAsr.tokenizer   -> loadAsr.tokenizer
 
record:    recordMono(device=_, sample_rate=_)
asrDecode: whisperDecode[whisper=asr]()
loadAsr.loaded -> record.trigger
loadAsr.loaded -> asrDecode.ready
record.signal  -> asrDecode.audio

The transcribed text feeds a RemoteLlm stream call, and each token is both logged and sent on to a RemoteTts synthesize call configured for ElevenLabs, whose audio is written locally:

model Voice(const tts_api_key: string, const voice_id: string) : RemoteTts {
    backend = "elevenlabs"
    api_key = |wrap<string>(tts_api_key)
    model   = "eleven_multilingual_v2"
    voice   = voice_id
}
reply: stream[llm=llm]()
asrDecode.transcribed -> reply.prompt
 
speak: synthesize[tts=tts]()
reply.token -> speak.text
 
write: writeLocal(path=output)
speak.audio -> write.data

Dependencies

[dependencies]
std      = "0.10.3"  # core flows, logging, data structures
http     = "0.10.3"  # HTTP client and server
net      = "0.10.3"  # IP address helpers
encoding = "0.10.3"  # UTF-8 encode / decode
fs       = "0.10.3"  # local file I/O
record   = "0.10.3"  # microphone recording
ml       = "0.10.3"  # LLM, STT, TTS and local model inference
Smart LLM RouterCI Pipeline