CI Pipeline

Source: 16_ci_pipeline

A three-stage CI pipeline running entirely on provisioned cloud containers: build (Rust release binary), test (cargo test with a Postgres sidecar), and package (tar.gz archive). build and test run concurrently; package starts only after both complete.

Running


melodium run 16_ci_pipeline/Compo.toml \
  --api_token "my-api-token" \
  --repo_url  "https://github.com/my-org/my-project.git"

Note

api_token here authenticates against a Mélodium Services API, such as Cadence.CI .


[…] info: ci: pipeline started
[…] info: build: build succeeded
[…] info: test: all tests passed
[…] info: ci: pipeline complete, artifact written

How it works

A single CicdDispatchEngine model handles all infrastructure concerns, wrapping DistantEngine and DistributionEngine provisioning under one API:


model Dispatcher(const api_token: string) : CicdDispatchEngine {
    api_token = |wrap<string>(api_token)
}

Each stage is a sub-treatment wrapping simpleStep or simpleStepWithInput, which abstract provisioning, execution, file I/O, and cleanup into a single call. No manual distant / start / distribute wiring is needed.

Concurrent stages with fork-join

build and test both start from startup.trigger:


startup.trigger -> build.trigger
startup.trigger -> test.trigger

package waits for both to finish using flock<void>(), which collects two Block<void> signals and emits a Stream<void> once both arrive. A trigger<void>() converts that stream to a single Block<void> gate for package:


bothFinished: flock<void>()
build.finished -> bothFinished.a
test.finished  -> bothFinished.b
 
bothTrigger: trigger<void>()
bothFinished.stream -> bothTrigger.stream
 
package[dispatcher=dispatcher](output=output)
bothTrigger.start -> package.trigger
build.data        -> package.data

main treatment diagram See in Compositeur Studio

Stage 1: build

build wraps simpleStep in a rust:1.82-slim container, cloning the repo and compiling a release binary. out_file = "binary" tells simpleStep to stream /mnt/data/binary back as data: Stream<byte>:


treatment build[dispatcher: CicdDispatchEngine](const repo_url: string)
  input  trigger:  Block<void>
  output started:  Block<void>
  output success:  Block<void>
  output error:    Block<void>
  output failed:   Block<void>
  output finished: Block<void>
  output data:     Stream<byte>
{
    simpleStep[dispatcher=dispatcher](
        name         = "build",
        image        = "rust:1.82-slim",
        cpu          = 2000, // millicores
        memory       = 4096, // MB
        storage      = 8192, // MB
        max_duration = 1800,
        commands     = |raw_commands([
            "apt-get update -qq && apt-get install -y --no-install-recommends git",
            "git clone --depth 1 ${REPO_URL} /workspace",
            "cd /workspace && cargo build --release 2>&1",
            "cp /workspace/target/release/app /mnt/data/binary"
        ]),
        variables    = |wrap<StringMap>(|map([|entry<string>("REPO_URL", repo_url)])),
        out_file     = "binary"
    )
 
    Self.trigger -> simpleStep.trigger
    simpleStep.data     -> Self.data
    simpleStep.finished -> Self.finished
}

Stage 2: test, with a service container sidecar

test runs the same image, but attaches a Postgres 16 |service_container reachable by the hostname "postgres". DATABASE_URL in the main container’s variables points at it:


treatment test[dispatcher: CicdDispatchEngine](const repo_url: string)
  input  trigger:  Block<void>
  output started:  Block<void>
  output success:  Block<void>
  output error:    Block<void>
  output failed:   Block<void>
  output finished: Block<void>
{
    simpleStep[dispatcher=dispatcher](
        name         = "test",
        image        = "rust:1.82-slim",
        cpu          = 2000, // millicores
        memory       = 4096, // MB
        storage      = 8192, // MB
        max_duration = 1800,
        service_containers = [
            |service_container(
                "postgres",
                512,  // memory, MB
                1000, // cpu, millicores
                2048, // storage, MB
                |amd64(),
                [],
                "postgres:16",
                _,
                |wrap<StringMap>(|map([
                    |entry<string>("POSTGRES_USER",     "ci"),
                    |entry<string>("POSTGRES_PASSWORD", "ci"),
                    |entry<string>("POSTGRES_DB",       "ci_test")
                ])),
                _
            )
        ],
        commands     = |raw_commands([
            "apt-get update -qq && apt-get install -y --no-install-recommends git",
            "git clone --depth 1 ${REPO_URL} /workspace",
            "cd /workspace && cargo test 2>&1"
        ]),
        variables    = |wrap<StringMap>(|map([
            |entry<string>("REPO_URL",     repo_url),
            |entry<string>("DATABASE_URL", "postgres://ci:ci@postgres/ci_test")
        ]))
    )
 
    Self.trigger -> simpleStep.trigger
    simpleStep.finished -> Self.finished
}

Stage 3: package, fed by stage 1’s binary

package wraps simpleStepWithInput, receiving the binary bytes as data: Stream<byte> (streamed straight from build.data, no local disk involved), writing them to /mnt/data/binary via in_file = "binary" on a debian:bookworm-slim container, then archiving:


treatment package[dispatcher: CicdDispatchEngine](const output: string)
  input  trigger:  Block<void>
  input  data:     Stream<byte>
  output started:  Block<void>
  output success:  Block<void>
  output error:    Block<void>
  output failed:   Block<void>
  output finished: Block<void>
  output data:     Stream<byte>
{
    simpleStepWithInput[dispatcher=dispatcher](
        name         = "package",
        image        = "debian:bookworm-slim",
        cpu          = 500, // millicores
        memory       = 512, // MB
        storage      = 2048, // MB
        max_duration = 300,
        in_file      = "binary",
        commands     = |raw_commands([
            "chmod +x /mnt/data/binary",
            "tar -czf /mnt/data/artifact.tar.gz -C /mnt/data binary"
        ]),
        out_file     = "artifact.tar.gz"
    )
 
    Self.trigger -> simpleStepWithInput.trigger
    Self.data    -> simpleStepWithInput.data
    simpleStepWithInput.data -> Self.data
}

build.data -> package.data streams the compiled binary directly from the build container into the package container’s input.

Error merging

Two chained one<void>() merge the four failure/error signals (build.failed, test.failed, build.error, test.error) into a single oneAnyFailed signal for logging:


failedOrError1: one<void>()
failedOrError2: one<void>()
build.failed -> failedOrError1.a
test.failed  -> failedOrError1.b
build.error  -> failedOrError2.a
test.error   -> failedOrError2.b
 
oneAnyFailed: one<void>()
failedOrError1.value -> oneAnyFailed.a
failedOrError2.value -> oneAnyFailed.b
 
logAbort: logErrorMessage(label="ci", message="pipeline aborted — skipping package stage")
oneAnyFailed.value -> logAbort.trigger

package still triggers on bothFinished regardless of build/test outcome; if build failed, package will error when simpleStepWithInput tries to extract the missing in_file.

Dependencies


[dependencies]
std     = "0.10.1"  # core flows, logging, data structures
fs      = "0.10.1"  # local file I/O
process = "0.10.1"  # shell command execution
work    = "0.10.1"  # cloud runner provisioning
distrib = "0.10.1"  # stream distribution across runners
cicd    = "0.10.1"  # CI/CD step dispatch and orchestration