Portable pipeline data boundaries
finos/morphir-engine/data supplies registered codecs for a typed IRUnit and
engine-owned Ion metadata. It is callable from a memory snapshot on native, JS,
WASM and WASM-GC. It performs no filesystem, network, process or compiler effects.
Morphir JSON remains supported. morphir-json reads the existing versioned JSON
schema and explicitly migrates it to current IR. ion-text and ion-binary read
the profile below into the same typed contract. Registry::checkpoint defaults
to Ion binary. Selecting JSON as a source does not change that default.
morphir-yaml reads the bounded normative YAML profile into the same typed IR.
Use the closed ModelFormat domain at file and command boundaries:
ModelFormat::from_control, infer_path, codec_id and is_binary.
decode_model(format, bytes, unit_id~, limits~) validates UTF-8 and dispatches
through shared codecs. decode_model_tree(files, unit_id~, limits~) accepts
relative paths and captured bytes for qualified v4/v3.1 trees. It bounds the
whole capture and ignores binary sidecars outside the physical tree profile.
Adapters check model_tree_file_limit before inserting captured files.
See the YAML package for resource budgets.
YAML encoding is not implemented; retain original bytes or export normalized JSON.
let registry = @data.Registry::new()
let unit = registry.decode("morphir-json", {
unit_id: "Pricing",
content: Text(json_source),
})
let checkpoint = registry.checkpoint(unit) // Binary(Bytes), never UTF-8 text
let restored = registry.decode("ion-binary", {
unit_id: "Pricing",
content: checkpoint,
})
Registry::open_checkpoint(codec, content) opens a self-contained saved model.
For Ion it adopts the embedded logical unit ID, then applies the same strict
profile, metadata and budget validation as decode. JSON has no embedded unit
identity and receives the local default workbench. This logical identity grants
no filesystem or provider authority. Use decode when the caller already knows
and needs to enforce the expected unit ID.
Registry::inspect_checkpoint(codec, content) also returns the input checkpoint's
original_format_version as a validated FormatVersion, alongside the migrated
unit. Both opening methods use the same profile, metadata and resource checks.
For Ion, the source format is read from the embedded IR before migration. It is
independent of the Ion encoding, envelope profile and current semantic version.
The caller must retain original bytes separately. This result does not identify
a provider response as an original file or grant a retained-model handle.
The module depends on the exact registry release moonrockz/ion@0.6.0, using its
synchronous annotated value, text and binary packages. Async streaming is not
part of this portable callback contract.
The Ion readers reject containers nested deeper than 128 levels, even when the
registry's max_depth is higher. This reader ceiling is separate from the
registry's tree budget. Both decode and open_checkpoint report the reader's
LimitExceeded error as data.tree_limit. The library's stack-safe walkers and
writers do not provide a memory bound; the registry still checks its own budgets.
IRUnit::validate(limits?) validates rich source-frontend output before processing.
It checks the envelope profile, duplicate metadata, tree depth/value budgets,
metadata ownership against the distribution's stable node IDs, and the canonical
binary Ion byte budget. These are validation limits; serialization allocates before
the byte check and does not provide a streaming memory bound. Codec import/export
behavior remains governed by the selected registry format.
IR-unit profile
Conceptual graph adapter
IRUnit::with_conceptual_graph(Graph) attaches the complete draft conceptual
graph at unit:<id> using the existing metadata entry. It refuses an occupied
owner with conceptual.metadata_owner_conflict and leaves the input unchanged.
IRUnit::conceptual_graph() reads that owner; graph-free legacy units return
None. A recognized malformed or unsupported conceptual profile raises an error.
Graph entity IDs stay inside the graph and never become IR metadata owners.
IRUnit::conceptual_specification() extracts a Library's public signatures,
retaining the entire graph, annotations, provenance and valid sidecar metadata.
It hides private constructors and excludes private declarations. A public
declaration without a representable signature or a sidecar whose owner disappears
causes data.lossy_conversion; the adapter does not drop that data. Existing
Specs pass through. Application projection requires a separate linking policy
and is unsupported here. Projection does not lower conceptual declarations or
interpret their traits, and retained projectionGaps remain visible.
These methods use the existing morphir-pipeline-v1 carrier and registered codecs.
Both Ion codecs retain the rich graph. Bare morphir-json export continues to
refuse metadata loss. They add no MEP tuple or model envelope. See the
model API and limits.
Wire shape
The initial morphir-pipeline-v1 contract is ir-unit. It keeps the profile
version separate from the nested IR format version and Ion 1.0 encoding. Text and
binary carry the same value model. The source snapshot supplies an expected
logical unit ID; an envelope containing another ID is refused.
morphir_pipeline::{
profile: "morphir-pipeline-v1",
contract: "ir-unit",
unit_id: "Pricing",
ir: {
formatVersion: 4,
distribution: {
Library: {
packageName: "pricing",
dependencies: {},
def: {modules: {}}
}
}
},
metadata: [
{node_id: "value:Pricing.quote/body", data: hint::12.340d0}
]
}
An envelope must contain the morphir_pipeline annotation. Additional ordered
annotations are retained. Schema records reject duplicate/unknown fields.
Metadata records have unique, nonempty node IDs and an unrestricted Ion data
value, subject to budgets. That value retains annotations, duplicate struct
fields, exact decimals, typed nulls, timestamps, symbols, blobs/clobs and other
Ion kinds. IDs are caller-supplied semantic identities, not array positions or
filesystem paths. This slice supports identity processing. Structural rewrites
need the separately planned lineage and metadata policy contract.
The nested ir follows the supported Morphir IR wire shape. It reuses the existing
validated versioned IR codecs through an explicit schema mapping. Rich metadata
never goes through a JSON projection or the JSON-shaped Data extension model.
Native Ion integers retain arbitrary magnitude. A numeric JSON lexeme whose
spelling could change, including -0, decimal scale or exponents, has the explicit
representation morphir_json_number::"1e-400". It is validated as a JSON number
before semantic decoding. Canonical integers use native Ion integers. Decimal
literal strings retain their original spelling under the existing IR schema.
Unsupported annotations or richer kinds inside the semantic ir subtree are
refused, rather than silently discarded. Put rich engine metadata in metadata.
This first profile does not automatically extract annotations from arbitrary IR
nodes or extend the current semantic IR vocabulary. JSON object field order is
not a semantic identity; the Ion semantic subtree is emitted in sorted field
order for deterministic checkpoints.
Selection and diagnostics
Registry::register adds a named codec with typed decode/encode callbacks.
Duplicate IDs fail. A binary codec requires Binary(Bytes) and a text codec
requires Text(String); there is no implicit encoding or format fallback.
Callbacks are trusted local components and receive the configured limits.
Registry checks also apply to their accepted results and output byte sizes.
morphir-json writes standard current Morphir JSON. It refuses units containing
engine metadata or additional envelope annotations with data.lossy_conversion.
A JSON-only component uses this semantic representation while the engine retains
the native Ion sidecar through its component/lineage adapter. The project CLI
selects these boundaries explicitly.
Errors have a stable code and detail in BoundaryError::Invalid. Existing IR
codec/version diagnostics are retained, including migration failures. Boundary
codes include data.profile_mismatch, data.unit_mismatch,
data.encoding_mismatch, data.duplicate_field, data.profile_value_unsupported,
data.symbol_catalog_missing, data.unknown_codec, and data.codec_conflict.
Unresolved shared-symbol imports require a catalog and fail explicitly in this
self-contained profile. Invalid Unicode is refused rather than replaced on write.
Budgets and acceptance
Defaults are 4 MiB per source/output, depth 128, and 100,000 values/annotations. Byte limits run before decoding and after encoding. Tree budgets apply to accepted envelopes and before Ion emission. The synchronous Ion parsers construct a value tree before its depth/value validation; these budgets do not promise bounded parser allocation, interruption, or a streaming security sandbox. Parser-level depth/number limits and host streaming are separate work. Public typed IR is trusted application data and uses the existing IR traversal/codec rules.
The focused tests verify equivalent JSON/Ion input, migration, semantic IR and exact-number preservation, rich metadata text/binary round trips, loss refusal, registration conflicts, malformed input, identity checks, and byte/tree budgets on all four portable targets:
mise exec -- moon test -p finos/morphir-engine/data --target all
An in-memory example is runnable with:
mise exec -- moon run pkgs/morphir-engine/data/examples --target native
The engine and CLI now support filesystem byte I/O, configured checkpoints, component lineage, generated projects and explicit build validation. Arbitrary annotated semantic IR imports remain outside this envelope profile.
Typed components retain the engine's Ion metadata sidecar. A JSON-only component
receives the explicit semantic projection from semantic_json, which is a
standard current Morphir JSON file. It returns semantic IR and optional lineage;
it never receives a JSON encoding of the rich sidecar. Whole-unit JSON export
still refuses metadata, additional annotations or rewrite provenance loss.
Structural rewrites require complete lineage. Every source identity is retained
or explicitly removed; every output identity is retained or derived from named
source identities. Retained metadata preserves the original Ion value. Derived
metadata uses a registered policy. The default retain policy refuses inherited
metadata on derived nodes; copy-single-origin copies one annotated origin and
refuses implicit merges. Ordered unknown annotations remain intact. Removed
metadata is recorded in an Ion provenance event instead of disappearing.
fill_lineage(before, after) builds the lineage of a rewrite that changes only
attributes and the variables of before that carry the unknown marker, such as
an inference fill. It joins nodes by logical path
(@identities.logical_nodes), so a node keeps its identity when its compact
JSON form becomes expanded:
- A node with the same logical path in both files is retained. A replaced marker variable is retained, because its replacement has its logical path.
- A node only in
afteris derived from the nearest node above it thatbeforehas: a new attribute node from its owner, a node under a replaced marker from the marker. - A node only in
beforeis removed.
Any other difference raises data.not_attribute_only with the logical path.
MetadataPolicy::no_derived() attaches no metadata to a derived node. Inference
uses it, because a type that a fill writes has no metadata of its own; the
retain policy would refuse the derived nodes of an annotated owner.
The optional provenance list in the pipeline profile preserves native Ion
rewrite records through checkpoints and into generated symbol files. Existing
profile files without it remain supported. finos/morphir-ir/identities supplies
stable declaration IDs and JSON-pointer child suffixes; pair-array indices denote
positions, so reordering requires declared lineage. Identity traversal is bounded
to depth 128 and 100,000 identities.