JSON Toolkit

Source: tutorial/04_json_toolkit See in Playground

Reads one record per line (a mix of valid JSON values and garbage), separates the invalid lines, classifies the valid ones as JSON objects or plain scalars (strings, numbers, booleans, arrays), and writes a small JSON summary of the counts.

Running

cd tutorial/04_json_toolkit
melodium run Compo.toml --input_file records.txt

With the shipped records.txt (a string, a number, a boolean, two objects with non-ASCII names such as Saša and Örség, one array, and one line of garbage), it logs each classified record and writes summary.json:

{"invalid":"1","objects":"2","scalars":"4"}

Optional: add --api-report and an API token (MELODIUM_API_TOKEN) to see this run’s full trace on Cadence.CI .

How it works

This example uses no models: JSON parsing and validation are stateless treatments.

Data flow

Lines are extracted with split + flatten + trim, and blank lines are dropped.

Separating garbage before parsing

validate checks each line without parsing it; filter splits the stream into valid text (accepted) and garbage (rejected, logged as-is):

isValid: validate()
split:   filter<string>()
nonBlank.accepted -> isValid.text
nonBlank.accepted -> split.value
isValid.is_json   -> split.select
 
logInvalid: logInfos(label="invalid")
split.rejected -> logInvalid.messages

Parsing and unwrapping

Only the valid text reaches toJson, so parsing never fails here. But toJson still returns Stream<Option<Json>> by design (it has no way to know from the type alone that every input is valid), so unwrapOr is used to get a plain Stream<Json>:

parsed:  toJson()
asJson:  unwrapOr<Json>(default=|null())
split.accepted -> parsed.text,json -> asJson.option,value -> classify.value

Checking validity first, with validate, avoids ever having to handle a parse failure downstream: the Option returned by toJson still has to be unwrapped, but it is guaranteed to always be some.

Classifying objects vs scalars

isObject classifies each Json value; a second filter splits objects from scalars (strings, numbers, booleans, arrays: anything that is not a JSON object). Most “does this satisfy X” library treatments, isObject included, are designed to plug directly into filter.select:

classify:  isObject()
partition: filter<Json>()
asJson.value      -> partition.value
classify.is_object -> partition.select

Building the summary

Three totals (objects, scalars, invalid lines) are computed with a small local treatment, finalCount<T>, used here at two different types (Json and string) in the same file without changing a single line of its body. They are combined into one StringMap, converted to a JSON object with fromStringMap, serialised with toString<Json>, and written to summary.json:

withObjects: blockEntry(key="objects")
withScalars: blockInsert(key="scalars")
withInvalid: blockInsert(key="invalid")
objTotal.total     -> withObjects.value
withObjects.map     -> withScalars.base
scalarTotal.total   -> withScalars.value
withScalars.map     -> withInvalid.base
invalidTotal.total  -> withInvalid.value
 
summaryStream: stream<StringMap>()
summaryJson:   fromStringMap()
summaryText:   toString<Json>()
write:         writeTextLocal(path=output)
logDone:       logInfoMessage(label="json", message="summary written")
 
withInvalid.map -> summaryStream.block,stream -> summaryJson.value,json -> summaryText.value,into -> write.text
write.finished  -> logDone.trigger

fromStringMap is the direct way to build a JSON object out of Mélodium data: every value becomes a JSON string, which is enough for a summary report. For richer JSON, with numbers or nested objects, build it with the individual json/value::from* functions and treatments instead.

Dependencies

[dependencies]
std  = "0.10.3"  # core flows, logging, data structures
fs   = "0.10.3"  # local file I/O
json = "0.10.3"  # JSON parsing and serialisation