Smart LLM Router

Source: showcase/smart_llm_router See in Playground

A JavaScript decision function reads each incoming prompt, estimates how complex it actually is, and routes it to one of three pre-configured RemoteLlm tiers, instead of always paying for the biggest model and the largest response budget regardless of what was actually asked.

Note

Requires a real LLM provider API key. Optionally add --api-report together with a Mélodium Services API token (MELODIUM_API_TOKEN) to follow the run on Cadence.CI .

Running


cd showcase/smart_llm_router
melodium run Compo.toml --api_key sk-...
 
curl -X POST http://127.0.0.1:8080/chat -d "What year is it?"
curl -X POST http://127.0.0.1:8080/chat -d "Can you summarize the main differences between REST and GraphQL APIs?"
curl -X POST http://127.0.0.1:8080/chat -d "Explain, step by step, how a hash map resizes, and compare it to a B-tree's rebalancing cost."

The first request (short, factual) is routed to the economy tier; the second (moderate length, one complexity signal) to standard; the third (three distinct complexity signals: “explain”, “step by step”, “compare”) to premium. The server log prints the router’s decision and its reasoning for every request:


{"complexity_score":-1.0,"estimated_input_tokens":4.0,"reason":"short, simple request","tier":"economy","word_count":4}
{"complexity_score":1.0,"estimated_input_tokens":18.0,"reason":"moderate length or complexity","tier":"standard","word_count":11}
{"complexity_score":3.0,"estimated_input_tokens":24.0,"reason":"long or explicitly complex request","tier":"premium","word_count":17}

complexity_score and estimated_input_tokens (built with arithmetic: +=, Math.min, Math.ceil) print with a trailing .0, even for whole numbers; word_count (read straight off Array.length) does not. JavaScript has no separate integer type, so this comes from how the embedded JS engine happens to represent each value internally, not from any rounding choice on Mélodium’s side. Do not assume every numeric field here parses cleanly as an integer downstream without checking.

How it works

Three RemoteLlm models are declared as fixed (model, max_tokens) presets, since RemoteLlm sets its model and max_tokens once per model instance, not per request. “Optimise the token budget for this request” therefore cannot mean “compute an arbitrary number every time”: it means “pick the right one of a few pre-defined tiers”:


model EconomyLlm(const api_key: string, const model: string) : RemoteLlm {
    backend     = "anthropic"
    api_key     = |wrap<string>(api_key)
    base_url    = ""
    model       = model
    system      = "You are a fast, concise assistant. Answer as briefly and directly as accuracy allows."
    max_tokens  = |wrap<u64>(200)
    temperature = |wrap<f32>(1.0)
    top_p       = _
    timeout     = _
}

temperature is set explicitly to |wrap<f32>(1.0) rather than left as _. On a Rust-declared model like RemoteLlm, _ does not omit the parameter the way it would on a .mel-declared one: it sends the value 0, which some providers reject outright for their newest models. top_p, which has no override here, stays _ and is correctly omitted.

StandardLlm and PremiumLlm follow the same shape, with 600 and 1500 tokens respectively, and progressively more thorough system prompts.

Scoring the prompt, then reading back the tier

A JavaScriptEngine model holds a decide() function that scores each prompt with a few heuristics (keyword matches, question marks, code-like content, word count) and returns a tier/reason/word_count object, plus a tiny pickTier() helper that projects just the tier field back out:


model Router() : JavaScriptEngine {
    code = ${{function decide(text) {
        var trimmed = text.trim();
        var words = trimmed.length ? trimmed.split(/\s+/) : [];
        var wordCount = words.length;
 
        // Rough, backend-agnostic estimate: about 4 characters per token.
        var estimatedInputTokens = Math.ceil(text.length / 4);
 
        var complexityKeywords = /\b(explain\w*|analyz\w*|analys\w*|compar\w*|design\w*|architecture\w*|prov\w*|deriv\w*|debug\w*|refactor\w*|summar\w*|step by step|in detail|pros and cons)\b/gi;
        var keywordMatches = (text.match(complexityKeywords) || []).length;
        var looksLikeCode = /```|function\s*\(|class\s+\w+|SELECT\s+.+FROM/i.test(text);
        var questionMarks = (text.match(/\?/g) || []).length;
 
        var score = 0;
        score += Math.min(keywordMatches, 3);
        if (looksLikeCode) score += 2;
        if (questionMarks > 1) score += 1;
        if (wordCount > 60) score += 1;
        if (wordCount <= 8 && !looksLikeCode) score -= 1;
 
        var tier, reason;
        if (score >= 3 || wordCount > 120) {
            tier = "premium";
            reason = "long or explicitly complex request";
        } else if (score >= 1 || wordCount > 25) {
            tier = "standard";
            reason = "moderate length or complexity";
        } else {
            tier = "economy";
            reason = "short, simple request";
        }
 
        return {
            tier: tier,
            word_count: wordCount,
            complexity_score: score,
            estimated_input_tokens: estimatedInputTokens,
            reason: reason
        };
    }
 
    function pickTier(decision) {
        return decision.tier;
    }
}}
}

Since there is still no field-by-field access into a parsed Json value outside JavaScript, chaining a second process call on the first call’s own output is how one field is read out of a result already computed in JS:


decideCall: process[engine=router](code="decide(value)")
asJson.json -> decideCall.value
decisionOpt: unwrapOr<Json>(default=|null())
decideCall.result -> decisionOpt.option


asStream2: stream<Json>()
decisionLast.last -> asStream2.block
pickTierCall: process[engine=router](code="pickTier(value)")
asStream2.stream -> pickTierCall.value

Routing with `equalTo` and `filterBlock`

The prompt is routed with three equalTo plus filterBlock gates, one per tier, the block-level counterpart of the filter pattern used on streams elsewhere:


isEconomy: equalTo<string>(value="economy")
tierLast.last -> isEconomy.data
gateEconomy: filterBlock<string>()
promptLast.last -> gateEconomy.value
isEconomy.result -> gateEconomy.select

Exactly one gate’s accepted output actually carries the prompt; the other two close empty. Because an LLM stream treatment fed an empty prompt stream never calls the provider at all, the two tiers not chosen cost nothing, not even a request:


economyPrompt: stream<string>()
gateEconomy.accepted -> economyPrompt.block
economyReply: llmStream[llm=economyLlm]()
economyPrompt.stream -> economyReply.prompt

The three (mostly empty) token streams are combined with two merges into one; since only one branch ever produced anything, the merged stream is just that branch’s output.

Dependencies


[dependencies]
std        = "0.10.3"  # core flows, logging, data structures
http       = "0.10.3"  # HTTP client and server
net        = "0.10.3"  # IP address helpers
encoding   = "0.10.3"  # UTF-8 encode / decode
json       = "0.10.3"  # JSON parsing and serialisation
javascript = "0.10.3"  # embedded JavaScript engine
ml         = "0.10.3"  # LLM, STT, TTS and local model inference