CI Pipeline

Source: 16_ci_pipeline

A three-stage CI pipeline running entirely on provisioned cloud containers: build (Rust release binary), test (cargo test with a Postgres sidecar), package (tar.gz archive). Stages 1 and 2 run concurrently; stage 3 starts only after both complete.

Running

melodium run 16_ci_pipeline/Compo.toml \
  --api_token "my-api-token" \
  --repo_url  "https://github.com/my-org/my-project.git"

[…] info: ci: pipeline started
[…] info: build: build succeeded
[…] info: test: all tests passed
[…] info: ci: pipeline complete — artifact written

How it works

A single CicdDispatchEngine model handles all infrastructure concerns:

model Dispatcher(const api_token: string) : CicdDispatchEngine {
    api_token = |wrap<string>(api_token)
}

Each stage is a sub-treatment wrapping simpleStep or simpleStepWithInput, which abstract provisioning, execution, file I/O, and cleanup into a single call.

Concurrent stages with fork-join

Stages 1 and 2 both start from startup.trigger:

startup.trigger -> build.trigger
startup.trigger -> test.trigger

Stage 3 waits for both to finish using flock<void>(), which collects two Block<void> signals and emits a Stream<void> once both arrive. A trigger<void>() converts that stream to a single Block<void> gate for package:

bothFinished: flock<void>()
build.finished -> bothFinished.a
test.finished  -> bothFinished.b
 
bothTrigger: trigger<void>()
bothFinished.stream -> bothTrigger.stream
bothTrigger.start   -> package.trigger

Binary piped between stages

The compiled binary produced by stage 1 is piped directly into stage 3 without touching local disk:

build.data -> package.data

simpleStepWithInput streams the received bytes into /mnt/data/binary on the remote container before executing commands.

Service container sidecar

The test stage attaches a Postgres 16 container as a sidecar reachable by the hostname postgres:

service_containers = [
  |service_container(
    "postgres", 512, 1000, 2048, |amd64(), [],
    "postgres:16", _,
    |wrap<StringMap>(|map([
      |entry<string>("POSTGRES_USER",     "ci"),
      |entry<string>("POSTGRES_PASSWORD", "ci"),
      |entry<string>("POSTGRES_DB",       "ci_test")
    ])), _
  )
]

DATABASE_URL is set to postgres://ci:ci@postgres/ci_test in the test container’s environment.

Error merging

If either stage 1 or stage 2 fails or errors, a log message is emitted and stage 3 still triggers (on bothFinished) — allowing the pipeline to continue to the packaging attempt. To abort instead, connect oneAnyFailed.value to a treatment that cancels the pipeline explicitly.

Dependencies

[dependencies]
std     = "0.10.1"  # core flows, logging, data structures
fs      = "0.10.1"  # local file I/O
process = "0.10.1"  # shell command execution
work    = "0.10.1"  # cloud runner provisioning
distrib = "0.10.1"  # stream distribution across runners
cicd    = "0.10.1"  # CI/CD step dispatch and orchestration