CI Pipeline

Source: showcase/ci_pipeline See in Playground

A three-stage CI pipeline that runs entirely on provisioned cloud containers, combining cicd, work, and process in one scenario. build and test run concurrently; package only starts once both have finished, and only proceeds if neither failed.

Running

cd showcase/ci_pipeline
export MELODIUM_API_TOKEN="my-api-token"
melodium run --api-report Compo.toml

repo_url defaults to sharkdp/hexyl, a small, real, permissively-licensed Rust binary crate with no external dependencies at build or test time, chosen specifically so this example is copy-paste runnable. Pass --repo_url to build something else instead:

melodium run --api-report Compo.toml \
  --repo_url "https://github.com/your-org/your-project.git"

• Stage 1, build: rust:1.90-slim container; cargo build --release; the binary’s name is read back from the cloned repo’s own Cargo.toml (works for any ordinary single-binary crate, not just the default one), and it is streamed back locally.
• Stage 2, test: rust:1.90-slim plus a postgres:16 service container; a psql connectivity check against DATABASE_URL proves the sidecar is actually reachable, then cargo test runs the repo’s own test suite. hexyl’s own tests are database-independent, so the psql check, not cargo test, is what this stage actually demonstrates about service_containers; point repo_url at a project with database-backed integration tests to exercise the sidecar more fully.
• Stage 3, package: debian:bookworm-slim; bundles stage 1’s binary into a .tar.gz, streamed back and written to disk.

How it works

A single CicdDispatchEngine model handles all infrastructure concerns, and each stage is a sub-treatment wrapping simpleStep or simpleStepWithInput, which abstract provisioning, execution, file I/O, and cleanup into a single call:

treatment main(
    const repo_url: string = "https://github.com/sharkdp/hexyl.git",
    const output:   string = "artifact.tar.gz"
)
  model dispatcher: CicdDispatchEngine()
{
    ...
    build[dispatcher=dispatcher](repo_url=repo_url)
    startup.trigger -> build.trigger
 
    test[dispatcher=dispatcher](repo_url=repo_url)
    startup.trigger -> test.trigger
    ...
}

Both stages are explicitly pinned to the same arch (|wrap<Arch>(|arm64())): a service container’s arch has no “unspecified” option the way a step’s own does, so the two must agree explicitly or dispatch is rejected outright, before either container starts.

Stage 1: build, with the binary name read from Cargo.toml

build wraps simpleStep in a rust:1.90-slim container, clones the repository, and compiles a release binary. out_file = "binary" tells simpleStep to stream /mnt/data/binary back as data: Stream<byte>:

commands     = [
    |command("sh", ["-c", "apt-get update -qq && apt-get install -y --no-install-recommends git"]),
    |command("sh", ["-c", "git clone --depth 1 ${REPO_URL} /workspace"]),
    |command("sh", ["-c", "cd /workspace && cargo build --release 2>&1"]),
    |command("sh", ["-c", "BIN=$(grep -m1 '^name' /workspace/Cargo.toml | cut -d '\"' -f2) && cp \"/workspace/target/release/$BIN\" /mnt/data/binary"])
],
variables    = |wrap<StringMap>(|map([|entry("REPO_URL", repo_url)])),
out_file     = "binary"

commands runs each entry as a direct exec, never through a shell: ${REPO_URL} is a real OS environment variable for the process (from variables), but nothing expands ${...} syntax unless the command running is itself a shell. Every entry that needs shell features (variable expansion, &&, cd, 2>&1, $(...)) is wrapped in an explicit sh -c.

Stage 2: test, with a Postgres sidecar

test runs the same image, but attaches a Postgres 16 |service_container reachable by the hostname "postgres". A psql -c "SELECT 1;" step proves the sidecar is genuinely reachable via DATABASE_URL before cargo test runs:

service_containers = [
    |service_container(
        "postgres",
        2048,
        2000,
        4096,
        |arm64(),
        [],
        "postgres:16",
        _,
        |wrap<StringMap>(|map([
            |entry("POSTGRES_USER",     "ci"),
            |entry("POSTGRES_PASSWORD", "ci"),
            |entry("POSTGRES_DB",       "ci_test")
        ])),
        _
    )
],
commands     = [
    |command("sh", ["-c", "apt-get update -qq && apt-get install -y --no-install-recommends git postgresql-client"]),
    |command("sh", ["-c", "git clone --depth 1 ${REPO_URL} /workspace"]),
    |command("sh", ["-c", "psql \"${DATABASE_URL}\" -c \"SELECT 1;\""]),
    |command("sh", ["-c", "cd /workspace && cargo test 2>&1"])
],

hexyl’s own test suite does not touch the database, so this psql check, not cargo test, is what actually proves the service_containers wiring: cargo test here would pass identically whether or not DATABASE_URL pointed at a real, reachable database.

Stage 3: package, fed directly by stage 1’s binary

package wraps simpleStepWithInput, receiving the binary bytes as data: Stream<byte>, streamed straight from build.data with no local disk involved, writing them to /mnt/data/binary via in_file = "binary" on a debian:bookworm-slim container, then archiving:

package[dispatcher=dispatcher](output=output)
packageGate: filterBlock<void>()
startup.trigger  -> packageGate.value
bothOkLast.last  -> packageGate.select
packageGate.accepted -> package.trigger
build.data           -> package.data

One container’s filesystem output becomes another container’s filesystem input, without ever touching the machine running this program in between.

A true AND-gate on an optional signal

success/error/failed are mutually exclusive, but only one is guaranteed to actually fire depending on what happened. flock<T>() waits for both of its Block inputs to resolve, value or empty close, and forwards whichever ones actually carried a value. Feeding it success directly would let one stage’s success alone leak through even after the other failed, since the other’s success input would simply close empty and still count as “resolved”.

Instead, each stage’s three outcomes are turned into one guaranteed Block<bool> first, using two chained one<bool>() (true from success, false from either error or failed), and only then combined with and():

buildOkOrError: one<bool>()
buildSuccessTrue.emit -> buildOkOrError.a
buildErrorFalse.emit  -> buildOkOrError.b
buildOk: one<bool>()
buildOkOrError.value  -> buildOk.a
buildFailedFalse.emit -> buildOk.b

bothOk: and<bool>()
buildOkStream.stream -> bothOk.a
testOkStream.stream  -> bothOk.b

Stage 3 is gated on that combined Block<bool> via filterBlock, so it only ever dispatches when both stages truly succeeded.

Error merging

Two chained one<void>() merge the four failure/error signals (build.failed, test.failed, build.error, test.error) into a single oneAnyFailed signal for logging, without hand-writing the check four times:

failedOrError1: one<void>()
failedOrError2: one<void>()
build.failed -> failedOrError1.a
test.failed  -> failedOrError1.b
build.error  -> failedOrError2.a
test.error   -> failedOrError2.b
 
oneAnyFailed: one<void>()
failedOrError1.value -> oneAnyFailed.a
failedOrError2.value -> oneAnyFailed.b
 
logAbort: logErrorMessage(label="ci", message="pipeline aborted: skipping package stage")
oneAnyFailed.value -> logAbort.trigger

Key patterns

• simpleStep / simpleStepWithInput: the difference is exactly what it sounds like, whether the step needs data streamed in (as a file at /mnt/data/<in_file>) in addition to whatever it streams out.
• service_containers: sidecar containers that live alongside a step’s main container for its duration; |service_container(...)’s parameters follow its documented order (name, memory, cpu, storage, arch, mounts, image, pull_secret, env, command).
• |raw_commands tokenises each command string literally, exactly as written: shell features like ${VAR} expansion, &&, cd, or redirection only work through an explicit |command("sh", ["-c", "..."]).

Dependencies

[dependencies]
std     = "0.10.4"  # core flows, logging, data structures
fs      = "0.10.4"  # local file I/O
process = "0.10.4"  # external process execution
work    = "0.10.4"  # cloud runner provisioning
distrib = "0.10.4"  # stream distribution across runners
cicd    = "0.10.4"  # CI/CD step dispatch and orchestration