CI Failure Triage

Source: showcase/ci_failure_triage See in Playground

Two CI steps run on provisioned containers, and each step’s raw output is deterministically classified by a small JavaScript function before anything else happens. Only the step the classifier actually flagged as failed is handed to a remote LLM for a plain-language diagnosis and a suggested fix; the step the classifier considers fine costs nothing beyond the classification itself, no LLM request is made for it at all.

Note

Requires a Mélodium Services API token and an LLM provider API key. Set MELODIUM_API_TOKEN in the environment and run with --api-report; see Cadence.CI to obtain a token and follow execution.

Running


cd showcase/ci_failure_triage
export MELODIUM_API_TOKEN="my-melodium-services-token"
melodium run --api-report Compo.toml --api_key sk-...

Runs two steps on the stock python:3.13-slim image, nothing to clone or install:

unit_tests: three trivial, always-true assertions. Always passes.
integration_check: reads a config dictionary by a misspelled key ('tiemout' instead of 'timeout'). Always fails with a KeyError.

Both are deliberately deterministic so this example runs the same way every time, independent of any external repository’s current state. ci_failure_report.md ends up looking like this:


## unit_tests
 
Status: **passed**
Classification: `{"category":"none","status":"passed","summary":"no known failure signature found in the step output"}`
 
No issues detected by the deterministic classifier; no AI analysis was requested for this step.
 
## integration_check
 
Status: **failed**
Classification: `{"category":"key_error","status":"failed","summary":"A dictionary key does not exist: tiemout"}`
 
## Root Cause
 
The script contains a typo in the dictionary key lookup: `config['tiemout']` instead of `config['timeout']`. Python raises a `KeyError` because `'tiemout'` does not exist as a key in the `config` dictionary. The first `print` succeeds (retries prints correctly), but execution halts on the second `print` when it tries to access the misspelled key.
 
## Suggested Fix
 
Correct the typo in the key name from `'tiemout'` to `'timeout'`:
 
```python
# Before (broken)
print('effective timeout:', config['tiemout'])
 
# After (fixed)
print('effective timeout:', config['timeout'])
```
 
This is a one-character transposition (`ie` to `ei`) and is the only change needed.

How it works

A JavaScriptEngine model holds the deterministic classifier, matching a step’s captured output against a short list of known failure signatures:


model Classifier() : JavaScriptEngine {
    code = ${{function classify(log) {
        var signatures = [
            { re: /KeyError:\s*'([^']*)'/,             category: "key_error",    summary: "A dictionary key does not exist" },
            { re: /TypeError:\s*(.+)/,                 category: "type_error",   summary: "A type error occurred" },
            { re: /ModuleNotFoundError|ImportError/,   category: "dependency",   summary: "A required module or dependency is missing" },
            { re: /SyntaxError:\s*(.+)/,               category: "syntax_error", summary: "A syntax error was found" },
            { re: /AssertionError/,                    category: "assertion",    summary: "An assertion failed" },
            { re: /Traceback \(most recent call last\)/, category: "runtime_error", summary: "An unhandled exception occurred" }
        ];
        for (var i = 0; i < signatures.length; i++) {
            var match = log.match(signatures[i].re);
            if (match) {
                var detail = match[1] ? (": " + match[1]) : "";
                return {
                    status: "failed",
                    category: signatures[i].category,
                    summary: signatures[i].summary + detail
                };
            }
        }
        return {
            status: "passed",
            category: "none",
            summary: "no known failure signature found in the step output"
        };
    }
 
    function statusOf(decision) {
        return decision.status;
    }
    }}
}

A single RemoteLlm, Analyst, is used to explain and suggest a fix, only for a step the classifier already flagged as failed. Unlike showcase/smart_llm_router, there is only one tier here, since this example’s cost-saving lever is not “which model” but “whether to call a model at all”.

Capturing output regardless of exit code

Each step’s real command redirects to a file and ends with ; true, so the container always exits 0 and the log is always streamed back as data; whether the step actually succeeded is then something this treatment decides for itself from the captured text, not something it trusts the container’s own exit code for:


unitTests: ciStepAnalysis[dispatcher=dispatcher, classifier=classifier, analyst=analyst](
    step_name = "unit_tests",
    image     = "python:3.13-slim",
    commands  = [|command("sh", ["-c", "python3 -c \"assert 1 + 1 == 2; assert 'melodium'.upper() == 'MELODIUM'; assert len([1, 2, 3]) == 3; print('all checks passed')\" > /mnt/data/log.txt 2>&1; true"])]
)
 
integrationCheck: ciStepAnalysis[dispatcher=dispatcher, classifier=classifier, analyst=analyst](
    step_name = "integration_check",
    image     = "python:3.13-slim",
    commands  = [|command("sh", ["-c", "python3 -c \"config = {'timeout': 30, 'retries': 3}; print('effective retries:', config['retries']); print('effective timeout:', config['tiemout'])\" > /mnt/data/log.txt 2>&1; true"])]
)

Classify before you spend a request

The captured log is decoded, collapsed to a Block<string>, wrapped as Json, and passed to the JS classify() function; a second, tiny JS call, statusOf(decision), projects just the status field back out, the same two-step JS pipeline as showcase/smart_llm_router:


classifyCall: process[engine=classifier](code="classify(value)")
asJson.json -> classifyCall.value
decisionOpt: unwrapOr<Json>(default=|null())
classifyCall.result -> decisionOpt.option

Two equalTo plus filterBlock gates then route on that status: failed gates the raw log into the LLM’s prompt; passed gates a fixed, free message instead:


isFailed: equalTo<string>(value="failed")
statusLast.last -> isFailed.data
gateFailedLog: filterBlock<string>()
logLast.last -> gateFailedLog.value
isFailed.result -> gateFailedLog.select
 
diagnosisPrompt: stream<string>()
gateFailedLog.accepted -> diagnosisPrompt.block
diagnosis: llmChat[llm=analyst]()
diagnosisPrompt.stream -> diagnosis.prompt


isPassed: equalTo<string>(value="passed")
statusLast.last -> isPassed.data
passedMessage: emit<string>(value="No issues detected by the deterministic classifier; no AI analysis was requested for this step.")
Self.trigger -> passedMessage.trigger
gatePassed: filterBlock<string>()
passedMessage.emit -> gatePassed.value
isPassed.result -> gatePassed.select

Exactly one gate ever carries anything, so exactly one of “call the LLM” or “say nothing needed” ever happens, and merging the two (mostly empty) branches just yields whichever one fired. chat, not stream, is used here since it returns one complete response per prompt rather than tokens as they arrive, the right choice when the result is going into a report rather than out over a live connection.

Assembling the report

Each step’s section (name, status, the full classification, and the analysis or the fixed message) is assembled with blockEntry/blockInsert/format, and the two steps’ sections are combined the same way once more at the top level before being written to ci_failure_report.md:


sectionEntries: stream<StringMap>()
sectionText:    format(format="## {step}\n\nStatus: **{status}**\nClassification: `{classification}`\n\n{analysis}\n\n")
sectionLast:    trigger<string>()
 
withAnalysis.map -> sectionEntries.block,stream -> sectionText.entries,formatted -> sectionLast.stream
sectionLast.last -> Self.report

Dependencies


[dependencies]
std        = "0.10.3"  # core flows, logging, data structures
process    = "0.10.3"  # external process execution
work       = "0.10.3"  # cloud runner provisioning
cicd       = "0.10.3"  # CI/CD step dispatch and orchestration
encoding   = "0.10.3"  # UTF-8 encode / decode
json       = "0.10.3"  # JSON parsing and serialisation
javascript = "0.10.3"  # embedded JavaScript engine
ml         = "0.10.3"  # LLM, STT, TTS and local model inference
fs         = "0.10.3"  # local file I/O