# Morphir Rust > Rust-based CLI tools and libraries for the Morphir ecosystem. Morphir enables functional domain modeling where you write business logic once and consume it through visualizations, code generation, type checking, and execution. Morphir Rust provides tools for working with Morphir IR (Intermediate Representation), including format migration, validation, and code generation. It supports language bindings for Gleam and includes an extension system for adding new languages and targets. --- # Getting Started ## Getting Started Source: https://finos.github.io/morphir-rust/getting-started # Getting Started This guide will help you get up and running with Morphir Rust. ## Prerequisites - A terminal (bash, zsh, PowerShell) - Internet connection (for downloading binaries) - Optional: [Rust](https://www.rust-lang.org/tools/install) if building from source ## Installation The quickest way to install morphir: **Linux / macOS:** ```bash curl -fsSL https://raw.githubusercontent.com/finos/morphir-rust/main/scripts/install.sh | bash ``` **Windows (PowerShell):** ```powershell irm https://raw.githubusercontent.com/finos/morphir-rust/main/scripts/install.ps1 | iex ``` For more installation options, see the [Installation Guide](install). ## Your First Command After installation, verify morphir is working: ```bash morphir --version ``` ## Basic Usage ### Migrating Morphir IR The most common use case is migrating Morphir IR between format versions. Morphir IR has evolved through several versions: - **Classic (V1-V3)**: Original format used by morphir-elm - **V4**: New format with improved structure Convert a local file to V4 format: ```bash morphir ir migrate --input ./morphir-ir.json --output ./morphir-ir-v4.json ``` Convert from a remote URL: ```bash morphir ir migrate \ --input https://lcr-interactive.finos.org/server/morphir-ir.json \ --output ./lcr-v4.json ``` Convert from GitHub: ```bash morphir ir migrate \ --input github:finos/morphir-examples@main/examples/basic/morphir-ir.json \ --output ./example-v4.json ``` ### Generating JSON Schema Generate a JSON Schema for validating Morphir IR files: ```bash # Output to stdout morphir schema # Output to file morphir schema --output ./morphir-ir-schema.json ``` ### Getting Help ```bash # Show help morphir --help # Show help including experimental commands morphir --help-all # Get help for a specific command morphir ir migrate --help ``` ## Version Management The morphir launcher supports automatic version management, similar to tools like rustup or nvm. ### Using a Specific Version Override the version for a single command: ```bash morphir +0.1.0 ir migrate --input ./ir.json --output ./v4.json ``` ### Pinning Version for a Project Create a `.morphir-version` file in your project root: ```bash echo "0.1.0" > .morphir-version ``` Or add to your `morphir.toml`: ```toml version = "0.1.0" ``` ### Managing Versions ```bash # Upgrade to latest version morphir self upgrade # List installed versions morphir self list # Show which version will be used morphir self which ``` ### Dev Mode For developing and testing morphir itself, you can run from a local source checkout: ```bash # One-time dev mode morphir --dev ir migrate --input ./ir.json # Check dev mode status morphir self dev ``` See the [Installation Guide](install#dev-mode) for more details. ## Next Steps - [Installation Guide](install) - Detailed installation options - [CLI Reference](cli/) - Complete command documentation - [IR Migrate](ir-migrate) - Detailed migration documentation --- # CLI Reference ## CLI Reference Source: https://finos.github.io/morphir-rust/cli/ # `morphir` **Usage**: `morphir [--help-all] [-V --version] ` **Version**: 0.2.0 - **Usage**: `morphir [--help-all] [-V --version] ` ## Flags ### `--help-all` Print help including experimental commands ### `-V --version` Print version ## Subcommands - [`morphir compile [FLAGS]`](/compile.md) - [`morphir generate [FLAGS]`](/generate.md) - [`morphir tool `](/tool.md) - [`morphir tool install [-v --version ] `](/tool/install.md) - [`morphir tool list`](/tool/list.md) - [`morphir tool update [-v --version ] `](/tool/update.md) - [`morphir tool uninstall `](/tool/uninstall.md) - [`morphir dist `](/dist.md) - [`morphir dist install [-v --version ] `](/dist/install.md) - [`morphir dist list`](/dist/list.md) - [`morphir dist update [-v --version ] `](/dist/update.md) - [`morphir dist uninstall `](/dist/uninstall.md) - [`morphir extension `](/extension.md) - [`morphir extension install [-v --version ] `](/extension/install.md) - [`morphir extension list`](/extension/list.md) - [`morphir extension update [-v --version ] `](/extension/update.md) - [`morphir extension uninstall `](/extension/uninstall.md) - [`morphir ir `](/ir.md) - [`morphir ir migrate [FLAGS] `](/ir/migrate.md) - [`morphir gleam [--json] [--json-lines] `](/gleam.md) - [`morphir gleam compile [FLAGS]`](/gleam/compile.md) - [`morphir gleam generate [FLAGS]`](/gleam/generate.md) - [`morphir gleam roundtrip [FLAGS]`](/gleam/roundtrip.md) - [`morphir schema [-o --output ]`](/schema.md) - [`morphir version [--json]`](/version.md) --- ## morphir self Source: https://finos.github.io/morphir-rust/cli/self # `morphir self` **Usage**: `morphir self ` Manage the morphir installation, including version management and dev mode. The `self` command provides utilities for managing your morphir installation, switching between versions, and enabling development mode. ## Commands ### `morphir self upgrade` Download and install the latest version of morphir. ```bash morphir self upgrade ``` This clears the version cache and fetches the latest release from GitHub. ### `morphir self list` List all installed versions of morphir. ```bash morphir self list ``` **Example output:** ``` Installed versions: 0.1.0 0.2.0 ``` ### `morphir self which` Show which version of morphir would be used based on current configuration. ```bash morphir self which ``` **Example output:** ``` Version: 0.1.0 Binary: /home/user/.morphir/versions/0.1.0/morphir-bin Status: installed Backend: github ``` ### `morphir self install ` Install a specific version of morphir. ```bash morphir self install 0.1.0 ``` ### `morphir self prune` Remove old versions of morphir, keeping only the currently active version. ```bash morphir self prune ``` ### `morphir self update` Update the morphir launcher script itself (not the morphir binary). ```bash morphir self update ``` This downloads the latest launcher script from the repository. ### `morphir self dev` Show dev mode status and configuration. ```bash morphir self dev ``` **Example output:** ``` info: Dev mode status: MORPHIR_DEV env: not set .morphir-version: not found morphir.toml: dev_mode not set MORPHIR_DEV_PATH: not set (will auto-detect) Source directory: /home/user/code/morphir-rust Debug binary: /home/user/code/morphir-rust/target/debug/morphir (available) Release binary: not built Dev mode is DISABLED To enable dev mode, use one of: - morphir --dev (one-time) - export MORPHIR_DEV=1 (session) - echo 'local-dev' > .morphir-version (project) - Add 'dev_mode = true' to morphir.toml [morphir] section ``` ## Version Override Run morphir with a specific version using the `+` syntax: ```bash morphir +0.1.0 ``` This downloads and uses the specified version without changing your default configuration. ## Dev Mode Flag Run morphir from local source using the `--dev` flag: ```bash morphir --dev ``` This builds and runs morphir from a local source checkout. See [Dev Mode](#dev-mode) for details on how the source directory is detected. ## Dev Mode Dev mode runs morphir from a local source checkout instead of a downloaded binary. ### Enabling Dev Mode | Method | Scope | Example | |--------|-------|---------| | `--dev` flag | One command | `morphir --dev ir migrate ...` | | `MORPHIR_DEV=1` | Session | `export MORPHIR_DEV=1` | | `.morphir-version` | Project | `echo "local-dev" > .morphir-version` | | `morphir.toml` | Project | `dev_mode = true` in `[morphir]` section | ### Source Detection When dev mode is enabled, the source directory is found by checking: 1. `MORPHIR_DEV_PATH` environment variable 2. CI environment variables (GitHub Actions, GitLab CI, Jenkins, etc.) 3. Current directory and parent directories (walks up to find `Cargo.toml` with workspace) 4. Common development locations: - `~/code/morphir-rust` - `~/dev/morphir-rust` - `~/src/morphir-rust` - `~/projects/morphir-rust` ### Build Behavior In dev mode: 1. If a debug binary exists and is newer than source files, it's used directly 2. Otherwise, `cargo run` is invoked to build and run ## Environment Variables | Variable | Description | |----------|-------------| | `MORPHIR_VERSION` | Override version to use | | `MORPHIR_HOME` | Override home directory (default: `~/.morphir`) | | `MORPHIR_BACKEND` | Force backend: `mise`, `binstall`, `github`, `cargo` | | `MORPHIR_DEV` | Set to `1` to enable dev mode | | `MORPHIR_DEV_PATH` | Path to morphir-rust source directory | ## Installation Backends The launcher supports multiple backends for downloading morphir: | Backend | Description | |---------|-------------| | `mise` | Uses [mise](https://mise.jdx.dev/) version manager (preferred if available) | | `binstall` | Uses [cargo-binstall](https://github.com/cargo-bins/cargo-binstall) for pre-built binaries | | `github` | Downloads directly from GitHub Releases (default fallback) | | `cargo` | Compiles from source using `cargo install` | Backend is auto-detected based on available tools, or can be forced with `MORPHIR_BACKEND`. ## See Also - [Installation](../install) - Installation guide with detailed options - [Getting Started](../getting-started) - Basic usage guide --- ## tool Source: https://finos.github.io/morphir-rust/cli/tool # `morphir tool` - **Usage**: `morphir tool ` Manage Morphir tools, distributions, and extensions ## Subcommands - [`morphir tool install [-v --version ] `](/tool/install.md) - [`morphir tool list`](/tool/list.md) - [`morphir tool uninstall `](/tool/uninstall.md) - [`morphir tool update [-v --version ] `](/tool/update.md) --- ## extension Source: https://finos.github.io/morphir-rust/cli/extension # `morphir extension` - **Usage**: `morphir extension ` Manage Morphir extensions ## Subcommands - [`morphir extension install [-v --version ] `](/extension/install.md) - [`morphir extension list`](/extension/list.md) - [`morphir extension uninstall `](/extension/uninstall.md) - [`morphir extension update [-v --version ] `](/extension/update.md) --- ## ir Source: https://finos.github.io/morphir-rust/cli/ir # `morphir ir` - **Usage**: `morphir ir ` Manage Morphir IR ## Subcommands - [`morphir ir migrate [FLAGS] `](/ir/migrate.md) --- ## compile Source: https://finos.github.io/morphir-rust/cli/compile # `morphir compile` - **Usage**: `morphir compile [FLAGS]` Compile source code to Morphir IR ## Flags ### `-l --language ` Source language (e.g., gleam, elm) ### `-i --input ` Input source directory or file ### `-o --output ` Output directory ### `--package-name ` Package name override ### `--config ` Explicit config file path ### `--project ` Project name (for workspaces) ### `--json` Output as JSON ### `--json-lines` Output as JSON Lines (streaming) --- ## generate Source: https://finos.github.io/morphir-rust/cli/generate # `morphir generate` - **Usage**: `morphir generate [FLAGS]` Generate code from Morphir IR ## Flags ### `-t --target ` Target language or format ### `-i --input ` Path to the Morphir IR file or directory ### `-o --output ` Output directory ### `--config ` Explicit config file path ### `--project ` Project name (for workspaces) ### `--json` Output as JSON ### `--json-lines` Output as JSON Lines (streaming) --- ## gleam Source: https://finos.github.io/morphir-rust/cli/gleam # `morphir gleam` - **Usage**: `morphir gleam [--json] [--json-lines] ` Gleam language binding commands ## Flags ### `--json` Output as JSON ### `--json-lines` Output as JSON Lines (streaming) ## Subcommands - [`morphir gleam compile [FLAGS]`](/gleam/compile.md) - [`morphir gleam generate [FLAGS]`](/gleam/generate.md) - [`morphir gleam roundtrip [FLAGS]`](/gleam/roundtrip.md) --- ## schema Source: https://finos.github.io/morphir-rust/cli/schema # `morphir schema` - **Usage**: `morphir schema [-o --output ]` Generate JSON Schema for Morphir IR ## Flags ### `-o --output ` Output file path (optional) --- ## version Source: https://finos.github.io/morphir-rust/cli/version # `morphir version` - **Usage**: `morphir version [--json]` Print version information ## Flags ### `--json` Output version info as JSON --- ## dist Source: https://finos.github.io/morphir-rust/cli/dist # `morphir dist` - **Usage**: `morphir dist ` Manage Morphir distributions ## Subcommands - [`morphir dist install [-v --version ] `](/dist/install.md) - [`morphir dist list`](/dist/list.md) - [`morphir dist uninstall `](/dist/uninstall.md) - [`morphir dist update [-v --version ] `](/dist/update.md) --- ## dist install Source: https://finos.github.io/morphir-rust/cli/dist/install # `morphir dist install` - **Usage**: `morphir dist install [-v --version ] ` Install a Morphir distribution ## Arguments ### `` Name of the distribution to install ## Flags ### `-v --version ` Version to install (defaults to latest) --- ## dist list Source: https://finos.github.io/morphir-rust/cli/dist/list # `morphir dist list` - **Usage**: `morphir dist list` List installed Morphir distributions --- ## dist uninstall Source: https://finos.github.io/morphir-rust/cli/dist/uninstall # `morphir dist uninstall` - **Usage**: `morphir dist uninstall ` Uninstall a Morphir distribution ## Arguments ### `` Name of the distribution to uninstall --- ## dist update Source: https://finos.github.io/morphir-rust/cli/dist/update # `morphir dist update` - **Usage**: `morphir dist update [-v --version ] ` Update an installed Morphir distribution ## Arguments ### `` Name of the distribution to update ## Flags ### `-v --version ` Version to update to (defaults to latest) --- ## extension install Source: https://finos.github.io/morphir-rust/cli/extension/install # `morphir extension install` - **Usage**: `morphir extension install [-v --version ] ` Install a Morphir extension ## Arguments ### `` Name of the extension to install ## Flags ### `-v --version ` Version to install (defaults to latest) --- ## extension list Source: https://finos.github.io/morphir-rust/cli/extension/list # `morphir extension list` - **Usage**: `morphir extension list` List installed Morphir extensions --- ## extension uninstall Source: https://finos.github.io/morphir-rust/cli/extension/uninstall # `morphir extension uninstall` - **Usage**: `morphir extension uninstall ` Uninstall a Morphir extension ## Arguments ### `` Name of the extension to uninstall --- ## extension update Source: https://finos.github.io/morphir-rust/cli/extension/update # `morphir extension update` - **Usage**: `morphir extension update [-v --version ] ` Update an installed Morphir extension ## Arguments ### `` Name of the extension to update ## Flags ### `-v --version ` Version to update to (defaults to latest) --- ## gleam compile Source: https://finos.github.io/morphir-rust/cli/gleam/compile # `morphir gleam compile` - **Usage**: `morphir gleam compile [FLAGS]` Compile Gleam source to Morphir IR ## Flags ### `-i --input ` Input source directory or file ### `-o --output ` Output directory ### `--package-name ` Package name override ### `--config ` Explicit config file path ### `--project ` Project name (for workspaces) --- ## gleam generate Source: https://finos.github.io/morphir-rust/cli/gleam/generate # `morphir gleam generate` - **Usage**: `morphir gleam generate [FLAGS]` Generate Gleam code from Morphir IR ## Flags ### `-i --input ` Path to the Morphir IR file or directory ### `-o --output ` Output directory ### `--config ` Explicit config file path ### `--project ` Project name (for workspaces) --- ## gleam roundtrip Source: https://finos.github.io/morphir-rust/cli/gleam/roundtrip # `morphir gleam roundtrip` - **Usage**: `morphir gleam roundtrip [FLAGS]` Roundtrip: compile then generate (for testing) ## Flags ### `-i --input ` Input source directory or file ### `-o --output ` Output directory ### `--package-name ` Package name override ### `--config ` Explicit config file path ### `--project ` Project name (for workspaces) --- ## ir migrate Source: https://finos.github.io/morphir-rust/cli/ir/migrate # `morphir ir migrate` - **Usage**: `morphir ir migrate [FLAGS] ` Migrate IR between versions Converts Morphir IR between Classic (V1-V3) and V4 formats. Supports local files, URLs, and GitHub shorthand sources. **Examples:** ```bash # Migrate to file morphir ir migrate ./morphir-ir.json -o ./morphir-ir-v4.json --target-version v4 # Migrate from URL morphir ir migrate https://lcr-interactive.finos.org/server/morphir-ir.json -o ./lcr-v4.json # Display in console with syntax highlighting (no -o) morphir ir migrate ./morphir-ir.json # Downgrade V4 to Classic morphir ir migrate ./morphir-ir-v4.json -o ./morphir-ir-classic.json --target-version classic ``` See the [IR Migration Guide](/ir-migrate/) for detailed real-world examples including the US Federal Reserve FR 2052a regulation model. ## Arguments ### `` Input file, directory, or remote source (e.g., github:owner/repo, URL) ## Flags ### `-o --output ` Output file or directory (if omitted, displays in console with syntax highlighting) ### `--target-version ` Target version: latest, v4/4, classic, v3/3, v2/2, v1/1 (default: latest) **Default:** `latest` ### `--force-refresh` Force refresh cached remote sources ### `--no-cache` Skip cache entirely for remote sources ### `--json` Output result as JSON (for scripting) ### `--expanded` Use expanded (non-compact) format for V4 output --- ## tool install Source: https://finos.github.io/morphir-rust/cli/tool/install # `morphir tool install` - **Usage**: `morphir tool install [-v --version ] ` Install a Morphir tool or extension ## Arguments ### `` Name of the tool to install ## Flags ### `-v --version ` Version to install (defaults to latest) --- ## tool list Source: https://finos.github.io/morphir-rust/cli/tool/list # `morphir tool list` - **Usage**: `morphir tool list` List installed Morphir tools --- ## tool uninstall Source: https://finos.github.io/morphir-rust/cli/tool/uninstall # `morphir tool uninstall` - **Usage**: `morphir tool uninstall ` Uninstall a Morphir tool ## Arguments ### `` Name of the tool to uninstall --- ## tool update Source: https://finos.github.io/morphir-rust/cli/tool/update # `morphir tool update` - **Usage**: `morphir tool update [-v --version ] ` Update an installed Morphir tool ## Arguments ### `` Name of the tool to update ## Flags ### `-v --version ` Version to update to (defaults to latest) --- # Tutorials ## Getting Started Tutorial Source: https://finos.github.io/morphir-rust/tutorials/getting-started-tutorial # Getting Started Tutorial This tutorial will walk you through setting up your first Morphir project and using the CLI to compile and generate code. ## Prerequisites - Morphir CLI installed (see [Installation Guide](../install)) - A terminal - Basic familiarity with command-line tools ## Step 1: Create a Project Create a new directory for your project: ```bash mkdir my-morphir-project cd my-morphir-project ``` ## Step 2: Create Source Files Create a `src` directory and add some Gleam source files: ```bash mkdir src ``` Create `src/main.gleam`: ```gleam pub fn hello() { "world" } pub fn add(x: Int, y: Int) -> Int { x + y } ``` ## Step 3: Create Configuration Create a `morphir.toml` file in the project root: ```toml [project] name = "my-package" version = "0.1.0" source_directory = "src" [frontend] language = "gleam" ``` ## Step 4: Compile to Morphir IR Compile your Gleam source to Morphir IR: ```bash morphir gleam compile ``` Or using the language-agnostic command: ```bash morphir compile --language gleam --input src/ ``` This will: - Parse your Gleam source files - Convert them to Morphir IR V4 - Write the IR to `.morphir/out/my-package/compile/gleam/` ## Step 5: Inspect the Output The compiled IR is stored in a document tree structure: ``` .morphir/ └── out/ └── my-package/ └── compile/ └── gleam/ ├── format.json └── modules/ └── my-package/ └── main/ ├── module.json ├── types/ └── values/ ``` You can view the `format.json` file to see the IR structure. ## Step 6: Generate Code (Optional) Generate code from the IR: ```bash morphir gleam generate ``` This will: - Read the IR from `.morphir/out/my-package/compile/gleam/` - Generate Gleam source code - Write it to `.morphir/out/my-package/generate/gleam/` ## Step 7: Roundtrip Testing Test the complete pipeline: ```bash morphir gleam roundtrip --input src/ ``` This compiles your source to IR, then generates code back, allowing you to verify the roundtrip preserves semantics. ## Understanding the .morphir/ Folder Structure Morphir uses a Mill-inspired folder structure: - `.morphir/out//compile//` - Compiled IR output - `.morphir/out//generate//` - Generated code output - `.morphir/out//dist/` - Distribution files - `.morphir/test/` - Test fixtures and scenarios (version controlled) - `.morphir/logs/` - Log files - `.morphir/cache/` - Cache files The `out/` directory should be added to `.gitignore`: ```gitignore .morphir/out/ .morphir/logs/ .morphir/cache/ ``` ## Next Steps - Learn about [Gleam Language Binding](gleam-tutorial) - Explore [Configuration Options](configuration-guide) - See [Complete Workflow Examples](complete-workflow) --- ## Gleam Language Binding Tutorial Source: https://finos.github.io/morphir-rust/tutorials/gleam-tutorial # Gleam Language Binding Tutorial This tutorial covers using Morphir with Gleam, including compilation, code generation, and roundtrip testing. ## Overview The Gleam binding provides: - **Frontend**: Parse Gleam source → Morphir IR V4 - **Backend**: Morphir IR V4 → Generate Gleam source ## Setting Up a Gleam Project ### Option 1: New Gleam Project If you're starting fresh: ```bash mkdir my-gleam-project cd my-gleam-project mkdir src ``` Create `src/main.gleam`: ```gleam pub fn main() { "Hello, Morphir!" } ``` ### Option 2: Existing Gleam Project If you have an existing Gleam project, just add a `morphir.toml` file to the root. ## Configuration Create `morphir.toml`: ```toml [project] name = "my-gleam-package" version = "0.1.0" source_directory = "src" [frontend] language = "gleam" ``` ## Compiling Gleam to Morphir IR ### Basic Compilation ```bash morphir gleam compile ``` This uses the configuration from `morphir.toml` to: - Find source files in `src/` - Compile to Morphir IR V4 - Output to `.morphir/out/my-gleam-package/compile/gleam/` ### Custom Input/Output ```bash morphir gleam compile --input src/ --output custom-output/ ``` ## Generating Gleam Code from IR ### Basic Generation ```bash morphir gleam generate ``` This reads from the default compile output and generates to `.morphir/out/my-gleam-package/generate/gleam/`. ## Roundtrip Testing Roundtrip testing verifies that compiling and generating preserves semantics: ```bash morphir gleam roundtrip --input src/ ``` ## JSON Output For programmatic use: ```bash morphir gleam compile --json morphir gleam compile --json-lines ``` ## Next Steps - Learn about [Configuration](configuration-guide) - See [Complete Workflow](complete-workflow) --- ## Configuration Guide Source: https://finos.github.io/morphir-rust/tutorials/configuration-guide # Configuration Guide Morphir uses `morphir.toml` (or `morphir.json`) for project and workspace configuration. ## Configuration Discovery Morphir automatically discovers configuration files by walking up the directory tree. ## Project Configuration ```toml [project] name = "my-package" version = "0.1.0" source_directory = "src" ``` ## Workspace Configuration ```toml [workspace] members = ["project-a", "project-b"] default_member = "project-a" ``` ## Configuration Merging Configuration is merged in this order: 1. Workspace config 2. Project config 3. CLI arguments (highest priority) ## Next Steps - See [Complete Workflow](complete-workflow) --- ## Complete Workflow Source: https://finos.github.io/morphir-rust/tutorials/complete-workflow # Complete Workflow This guide walks through a complete end-to-end workflow using Morphir with Gleam. ## Scenario: Building a Business Logic Library Let's build a simple business logic library for a financial application. ## Step 1: Project Setup ```bash mkdir finance-library cd finance-library mkdir src ``` Create `morphir.toml`: ```toml [project] name = "Finance.Library" version = "0.1.0" description = "Financial calculations library" source_directory = "src" ``` ## Step 2: Write Business Logic Create `src/calculations.gleam`: ```gleam pub fn calculate_interest(principal: Float, rate: Float, years: Int) -> Float { principal * (1.0 + rate) ** float(years) } pub fn calculate_payment(principal: Float, rate: Float, periods: Int) -> Float { let monthly_rate = rate / 12.0 principal * (monthly_rate * (1.0 + monthly_rate) ** float(periods)) / ((1.0 + monthly_rate) ** float(periods) - 1.0) } ``` Create `src/types.gleam`: ```gleam pub type Account { Account(account_number: String, balance: Float, interest_rate: Float) } pub fn get_balance(account: Account) -> Float { case account { Account(_, balance, _) -> balance } } ``` ## Step 3: Compile to Morphir IR ```bash morphir gleam compile ``` Output: ``` Compilation successful! Output: .morphir/out/Finance.Library/compile/gleam/ ``` Inspect the IR: ```bash cat .morphir/out/Finance.Library/compile/gleam/format.json ``` ## Step 4: Verify IR Structure The IR is stored as a document tree: ``` .morphir/out/Finance.Library/compile/gleam/ ├── format.json └── modules/ └── Finance.Library/ ├── calculations/ │ ├── module.json │ └── values/ │ ├── calculate_interest.json │ └── calculate_payment.json └── types/ ├── module.json ├── types/ │ └── Account.json └── values/ └── get_balance.json ``` ## Step 5: Generate Code (Roundtrip) Generate Gleam code from the IR: ```bash morphir gleam generate ``` This creates: ``` .morphir/out/Finance.Library/generate/gleam/ ├── calculations.gleam └── types.gleam ``` ## Step 6: Verify Roundtrip Compare original and generated code: ```bash # Original cat src/calculations.gleam # Generated cat .morphir/out/Finance.Library/generate/gleam/calculations.gleam ``` The generated code should be semantically equivalent. ## Step 7: Use JSON Output for Automation For CI/CD or automation: ```bash morphir gleam compile --json > compile-result.json ``` Parse the result: ```bash jq '.success' compile-result.json jq '.modules[]' compile-result.json ``` ## Step 8: Workspace Setup (Optional) If you have multiple projects, set up a workspace: ```toml # morphir.toml (workspace root) [workspace] members = ["finance-library", "reporting-service"] default_member = "finance-library" [project] name = "workspace" ``` Compile a specific project: ```bash morphir gleam compile --project finance-library ``` ## Step 9: Integration with Build System ### Makefile Example ```makefile .PHONY: compile generate roundtrip compile: morphir gleam compile generate: morphir gleam generate roundtrip: compile generate @echo "Roundtrip complete" test: roundtrip @echo "Comparing original and generated code..." diff -r src/ .morphir/out/Finance.Library/generate/gleam/ || true ``` ### CI/CD Example (GitHub Actions) ```yaml name: Morphir Build on: [push, pull_request] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Install Morphir run: curl -fsSL https://raw.githubusercontent.com/finos/morphir-rust/main/scripts/install.sh | bash - name: Compile to IR run: morphir gleam compile --json > compile-result.json - name: Check compilation run: | if [ "$(jq -r '.success' compile-result.json)" != "true" ]; then echo "Compilation failed" exit 1 fi - name: Generate code run: morphir gleam generate - name: Upload artifacts uses: actions/upload-artifact@v3 with: name: morphir-ir path: .morphir/out/ ``` ## Step 10: Debugging ### Enable Verbose Output ```bash RUST_LOG=debug morphir gleam compile ``` ### Check Diagnostics ```bash morphir gleam compile --json | jq '.diagnostics' ``` ### View Error Details Errors are reported with source locations: ``` Error: Compilation failed ┌─ src/calculations.gleam:3:5 │ 3 │ principal * (1.0 + rate) ** float(years) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ Type mismatch: expected Float, found Int ``` ## Best Practices 1. **Version Control**: Add `.morphir/out/` to `.gitignore`, but keep `.morphir/test/` versioned 2. **Configuration**: Use `morphir.toml` for project settings 3. **Roundtrip Testing**: Regularly run roundtrip to verify semantic preservation 4. **JSON Output**: Use `--json` for automation and CI/CD 5. **Workspaces**: Use workspaces for multi-project setups ## Troubleshooting ### Common Issues **Issue**: "No morphir.toml found" - **Solution**: Create `morphir.toml` in project root or use `--config` **Issue**: "Extension not found" - **Solution**: Ensure Gleam binding is built and bundled **Issue**: "Path does not exist" - **Solution**: Check `source_directory` in config matches your project structure ## Next Steps - Explore [Extension Development](../contributors/extension-tutorial) - Read [Architecture Documentation](../contributors/architecture) - Check [CLI Reference](../cli/index) --- ## Tutorials Source: https://finos.github.io/morphir-rust/tutorials/ # Tutorials Step-by-step guides for using Morphir Rust. ## Getting Started - [Getting Started Tutorial](getting-started-tutorial) - Your first Morphir project ## Language Bindings - [Gleam Language Binding](gleam-tutorial) - Using Morphir with Gleam ## Configuration - [Configuration Guide](configuration-guide) - Understanding morphir.toml ## Workflows - [Complete Workflow](complete-workflow) - End-to-end examples ## Extension Development - [Extension Development Tutorial](extension-tutorial) - Creating your own extensions --- ## Extension Development Tutorial Source: https://finos.github.io/morphir-rust/tutorials/extension-tutorial # Extension Development Tutorial Learn how to create Morphir extensions for new languages or targets. ## Overview Morphir extensions are WASM modules that implement the `Frontend` and/or `Backend` traits. Extensions can be: - **Builtin**: Bundled with the CLI (like Gleam) - **Registry**: Installed from a registry - **Local**: Referenced by path in `morphir.toml` ## Extension Types ### Frontend Extension Converts source code → Morphir IR V4 ```rust impl Frontend for MyExtension { fn compile(&self, request: CompileRequest) -> Result { // Parse source files // Convert to Morphir IR V4 // Return CompileResult } } ``` ### Backend Extension Converts Morphir IR V4 → Generated code ```rust impl Backend for MyExtension { fn generate(&self, request: GenerateRequest) -> Result { // Read Morphir IR // Generate target language code // Return GenerateResult } } ``` ## Creating a Simple Extension ### Step 1: Create Extension Crate ```bash cargo new --lib my-morphir-extension cd my-morphir-extension ``` ### Step 2: Configure for WASM In `Cargo.toml`: ```toml [lib] crate-type = ["cdylib"] [dependencies] morphir-extension-sdk = { path = "../../morphir-extension-sdk" } extism-pdk = "1.2" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" ``` ### Step 3: Implement Extension In `src/lib.rs`: ```rust use morphir_extension_sdk::prelude::*; #[derive(Default)] pub struct MyExtension; impl Extension for MyExtension { fn info() -> ExtensionInfo { ExtensionInfo { id: "my-extension".into(), name: "My Extension".into(), version: env!("CARGO_PKG_VERSION").into(), types: vec![ExtensionType::Frontend], // ... } } } impl Frontend for MyExtension { fn compile(&self, request: CompileRequest) -> Result { // Implementation Ok(CompileResult { success: true, ir: vec![], diagnostics: vec![], }) } } ``` ### Step 4: Build WASM ```bash cargo build --target wasm32-unknown-unknown --release ``` ### Step 5: Use Extension Add to `morphir.toml`: ```toml [extensions.my-extension] path = "./extensions/my-extension.wasm" enabled = true ``` ## Builtin Extensions Builtin extensions (like Gleam) are: - Compiled as part of the CLI build - Bundled in the CLI binary or resources - Automatically discovered To make an extension builtin: 1. Add to `morphir-design/src/extensions.rs` 2. Bundle WASM in build process 3. Update extension discovery ## Extension Discovery Extensions are discovered in this order: 1. Builtin extensions (highest priority) 2. Extensions from `morphir.toml` 3. Registry extensions ## Extension Configuration Extensions can have custom configuration: ```toml [extensions.my-extension] path = "./extensions/my-extension.wasm" enabled = true [extensions.my-extension.config] custom_setting = "value" ``` Access in extension: ```rust let custom_setting = request.options .get("custom_setting") .and_then(|v| v.as_str()); ``` ## Testing Extensions ### Unit Tests ```rust #[cfg(test)] mod tests { use super::*; #[test] fn test_compile() { let ext = MyExtension; let request = CompileRequest { sources: vec![], options: Default::default(), }; let result = ext.compile(request).unwrap(); assert!(result.success); } } ``` ### Integration Tests Use the CLI to test extensions: ```bash morphir compile --language my-language --input src/ ``` ## Best Practices 1. **Error Handling**: Provide clear diagnostics with source locations 2. **Performance**: Optimize parsing and conversion for large codebases 3. **Testing**: Include comprehensive test coverage 4. **Documentation**: Document language-specific features and limitations ## Next Steps - Read [Extension System Design](../contributors/extension-system) - See [Architecture Overview](../contributors/architecture) - Check [Development Guide](../contributors/development) --- # For Contributors ## Architecture Overview Source: https://finos.github.io/morphir-rust/contributors/architecture # Architecture Overview This document provides an overview of the Morphir Rust architecture. ## System Architecture The system consists of: - **CLI (morphir)**: User-facing commands - **Design-Time (morphir-design)**: Configuration and extension discovery - **Common (morphir-common)**: Shared infrastructure - **Daemon (morphir-daemon)**: Runtime extension execution - **Extensions**: Language-specific implementations ## Crate Responsibilities Each crate has clear responsibilities for separation of concerns. ## Next Steps - Read [Extension System Design](extension-system) - See [CLI Architecture](cli-architecture) --- ## Morphir Daemon Source: https://finos.github.io/morphir-rust/contributors/design/daemon/README # Morphir Daemon The Morphir Daemon is a long-running service that manages workspaces, projects, builds, and provides IDE integration. ## Tracking | Type | References | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Beads** | morphir-l75 (caching), morphir-n6b (analyzer), morphir-369 (SQLite VFS) | | **GitHub Issues** | [#392](https://github.com/finos/morphir/issues/392) (pipeline types), [#393](https://github.com/finos/morphir/issues/393) (diagnostics), [#394](https://github.com/finos/morphir/issues/394) (JSON output), [#400](https://github.com/finos/morphir/issues/400) (analyzer), [#401](https://github.com/finos/morphir/issues/401) (caching) | | **Discussions** | [#88](https://github.com/finos/morphir/discussions/88) (package manager) | ## Overview The daemon provides: - **Workspace Management**: Multi-project development with shared dependencies - **Build Orchestration**: Coordinated builds in dependency order - **File Watching**: Automatic recompilation on source changes - **IDE Integration**: Language server protocol support - **Package Publishing**: Pack and publish to registries ## Documents | Document | Status | Description | | ----------------------------------------- | ------ | --------------------------------------------- | | [Lifecycle](./lifecycle.md) | Draft | Workspace creation, opening, closing | | [Projects](./projects.md) | Draft | Project management within a workspace | | [Dependencies](./dependencies.md) | Draft | Dependency resolution and caching | | [Build](./build.md) | Draft | Build orchestration and diagnostics | | [Watching](./watching.md) | Draft | File system watching for incremental builds | | [Packages](./packages.md) | Draft | Package format, registry backends, publishing | | [Configuration](./configuration.md) | Draft | morphir.toml system overview | | [Workspace Config](./workspace-config.md) | Draft | Multi-project workspace configuration | | [CLI Interaction](./cli-interaction.md) | Draft | CLI-daemon communication and lifecycle | ## Architecture ``` workspace-root/ ├── morphir.toml # Workspace configuration ├── .morphir/ # Workspace-level cache and state │ ├── deps/ # Resolved dependencies (shared) │ └── cache/ # Build cache ├── packages/ │ ├── core/ # Project: my-org/core │ │ ├── morphir.toml │ │ └── src/ │ ├── domain/ # Project: my-org/domain │ │ ├── morphir.toml │ │ └── src/ │ └── api/ # Project: my-org/api │ ├── morphir.toml │ └── src/ ``` ## Key Concepts ### Workspace vs Project | Concept | Scope | Configuration | | ------------- | ----------------- | ----------------------------------------- | | **Workspace** | Multiple projects | `morphir.toml` with `[workspace]` section | | **Project** | Single package | `morphir.toml` with `[project]` section | Both use the same `morphir.toml` file format. The presence of `[workspace]` section enables workspace mode. ### Workspace States | State | Description | | -------------- | ---------------------------------- | | `closed` | Workspace is not active | | `initializing` | Workspace is being loaded | | `open` | Workspace is ready for operations | | `error` | Workspace has unrecoverable errors | ### Project States | State | Description | | ---------- | ----------------------------------------- | | `unloaded` | Project metadata loaded, IR not compiled | | `loading` | Project is being compiled | | `ready` | Project IR is loaded and valid | | `stale` | Source files changed, needs recompilation | | `error` | Project has compilation errors | ## JSON-RPC Protocol Daemon operations are exposed via JSON-RPC for client communication: ``` workspace/create, workspace/open, workspace/close workspace/addProject, workspace/removeProject, workspace/listProjects workspace/buildAll, workspace/clean, workspace/watch daemon/health, daemon/capabilities ``` See [CLI Interaction](./cli-interaction.md) for connection modes, transport options, and CLI-to-daemon communication details. See [IR v4](../ir/README.md) for full protocol and type specifications. ## CLI Commands ```bash morphir workspace init # Create new workspace morphir workspace add # Add project to workspace morphir workspace build # Build all projects morphir workspace watch # Watch and rebuild on changes morphir pack # Create distributable package morphir publish # Publish to registry ``` ## Design Principles 1. **Lazy Loading**: Projects are not compiled until explicitly needed 2. **Incremental**: Only recompile what changed 3. **Shared Resolution**: Dependencies resolved once at workspace level 4. **Isolation**: Project failures don't break the workspace 5. **Observable**: Rich state and diagnostic information ## Related ### Morphir Rust Design Documents - **[Extensions](../extensions/)** - WASM components and task system - **[Wasm Extension Architecture (Interactive Session)](../extensions/actor-based/12-wasm-extension-architecture-session.md)** - Hidden Extism, envelope protocol, TEA runtime, and actors - **[Design Documents](../)** - All design documents ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Morphir Extensions Source: https://finos.github.io/morphir-rust/contributors/design/extensions/README # Morphir Extensions The extension architecture for adding capabilities to Morphir via WASM components and the task system. ## Tracking | Type | References | | ----------------- | ---------------------------------------------------------------------------------- | | **Beads** | morphir-go-772 (task execution), morphir-010 (CLI extensions) | | **GitHub Issues** | [#399](https://github.com/finos/morphir/issues/399) (task/target execution engine) | ## Overview Morphir Extensions enable: - **Custom Code Generators**: Add new backend targets (Spark, Scala, etc.) - **Custom Frontends**: Support new source languages - **Additional Tasks**: Register new intrinsic tasks - **Build Automation**: Pre/post hooks for built-in commands - **Protocol Integration**: JSON-RPC based communication ## Documents ### Actor-Based Extension System Design The **[Actor-Based Extension System Design](actor-based/)** documents describe an actor-based, multi-protocol architecture for extending Morphir functionality. These documents explore: - **Multi-Protocol Support**: JSON-RPC, gRPC, stdio, and WASM protocols - **Actor Framework**: Kameo-based architecture for isolation and concurrency - **Extension Hosts**: Protocol-specific implementations for different communication methods - **Security and Isolation**: Sandboxing and capability-based permissions - **Protocol Specifications**: Detailed wire protocol documentation **Status**: Draft - These are brainstorming documents that will be refined and integrated with the main extension design. **Key Documents:** - [Actor-Based Design Overview](actor-based/00-overview.md) - System overview and goals - [Architecture](actor-based/01-architecture.md) - Detailed component architecture - [Wasm Extension Architecture (Interactive Session)](actor-based/12-wasm-extension-architecture-session.md) - Hidden Extism, envelope protocol, TEA runtime, and Kameo actors - [Extension Manager](actor-based/08-extension-manager.md) - Central coordinator actor - [Protocol Implementations](actor-based/) - JSON-RPC, gRPC, stdio, WASM hosts ### Core Extension Design | Document | Status | Description | | -------------------------------------- | ------ | ---------------------------------------------- | | [WASM Components](./wasm-component.md) | Draft | Component model integration and WIT interfaces | | [Tasks](./tasks.md) | Draft | Task system, dependencies, and hooks | ## Getting Started with Extensions Extension development starts with a minimal "info" extension that verifies connectivity before adding features. ### Minimal Extension (Hello World) Every extension must implement the `info` interface - this is the only required interface: ```wit package morphir:extension@0.4.0; /// Required interface - all extensions must implement this interface info { /// Extension metadata record extension-info { /// Unique identifier (e.g., "spark-codegen") id: string, /// Human-readable name name: string, /// Version (semver) version: string, /// Description description: string, /// Author/maintainer author: option, /// Homepage/repository URL homepage: option, /// License identifier (SPDX) license: option, } /// Return extension metadata get-info: func() -> extension-info; /// Health check - return true if extension is ready ping: func() -> bool; } ``` **Minimal Extension Implementation:**
Rust ```rust use morphir_extension::info::{ExtensionInfo, Info}; struct MyExtension; impl Info for MyExtension { fn get_info() -> ExtensionInfo { ExtensionInfo { id: "my-extension".to_string(), name: "My First Extension".to_string(), version: "0.1.0".to_string(), description: "A minimal Morphir extension".to_string(), author: Some("My Name".to_string()), homepage: Some("https://github.com/me/my-extension".to_string()), license: Some("Apache-2.0".to_string()), } } fn ping() -> bool { true // Extension is ready } } ```
Go ```go package main import "github.com/morphir/extension" type MyExtension struct{} func (e *MyExtension) GetInfo() extension.ExtensionInfo { return extension.ExtensionInfo{ ID: "my-extension", Name: "My First Extension", Version: "0.1.0", Description: "A minimal Morphir extension", Author: stringPtr("My Name"), Homepage: stringPtr("https://github.com/me/my-extension"), License: stringPtr("Apache-2.0"), } } func (e *MyExtension) Ping() bool { return true // Extension is ready } func stringPtr(s string) *string { return &s } ```
TypeScript ```typescript import { ExtensionInfo, Info } from "@morphir/extension"; class MyExtension implements Info { getInfo(): ExtensionInfo { return { id: "my-extension", name: "My First Extension", version: "0.1.0", description: "A minimal Morphir extension", author: "My Name", homepage: "https://github.com/me/my-extension", license: "Apache-2.0", }; } ping(): boolean { return true; // Extension is ready } } export default new MyExtension(); ```
### Extension Discovery The CLI can list and inspect all registered extensions: ```bash # List all extensions morphir extension list # Output: # NAME VERSION TYPE CAPABILITIES # spark-codegen 1.2.0 codegen generate, streaming, incremental # elm-frontend 0.19.1 frontend compile, diagnostics # my-extension 0.1.0 unknown (info only) # Detailed info about an extension morphir extension info spark-codegen # Output: # spark-codegen v1.2.0 # Type: codegen # Description: Generate Apache Spark DataFrame code from Morphir IR # Author: Morphir Contributors # Homepage: https://github.com/finos/morphir-spark # License: Apache-2.0 # # Capabilities: # ✓ codegen/generate # ✓ codegen/generate-streaming # ✓ codegen/generate-incremental # ✓ codegen/generate-module # ✓ codegen/options-schema # # Targets: spark # # Options: # spark_version string "3.5" Spark version to target # scala_version string "2.13" Scala version to target # Verify extension connectivity morphir extension ping spark-codegen # spark-codegen: OK (2ms) # Ping all extensions morphir extension ping --all # spark-codegen: OK (2ms) # elm-frontend: OK (1ms) # my-extension: OK (1ms) ``` ### JSON-RPC Methods **List Extensions:** ```json { "jsonrpc": "2.0", "id": "list-001", "method": "extension/list", "params": {} } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "list-001", "result": { "extensions": [ { "id": "spark-codegen", "name": "Spark Code Generator", "version": "1.2.0", "type": "codegen", "source": { "path": "./extensions/spark-codegen.wasm" }, "capabilities": [ "generate", "generate-streaming", "generate-incremental" ] }, { "id": "my-extension", "name": "My First Extension", "version": "0.1.0", "type": null, "source": { "path": "./extensions/my-extension.wasm" }, "capabilities": [] } ] } } ``` **Get Extension Info:** ```json { "jsonrpc": "2.0", "id": "info-001", "method": "extension/info", "params": { "extension": "spark-codegen" } } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "info-001", "result": { "id": "spark-codegen", "name": "Spark Code Generator", "version": "1.2.0", "description": "Generate Apache Spark DataFrame code from Morphir IR", "author": "Morphir Contributors", "homepage": "https://github.com/finos/morphir-spark", "license": "Apache-2.0", "type": "codegen", "capabilities": { "codegen/generate": true, "codegen/generate-streaming": true, "codegen/generate-incremental": true, "codegen/generate-module": true, "codegen/options-schema": true }, "targets": ["spark"], "options": { "spark_version": { "type": "string", "default": "3.5", "description": "Spark version to target" }, "scala_version": { "type": "string", "default": "2.13", "description": "Scala version to target" } } } } ``` **Ping Extension:** ```json { "jsonrpc": "2.0", "id": "ping-001", "method": "extension/ping", "params": { "extension": "spark-codegen" } } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "ping-001", "result": { "extension": "spark-codegen", "status": "ok", "latency_ms": 2 } } ``` ### Extension Development Workflow 1. **Start minimal**: Implement only `info` interface 2. **Verify connectivity**: `morphir extension ping my-extension` 3. **Check registration**: `morphir extension list` 4. **Add capabilities incrementally**: One interface at a time 5. **Test each capability**: Verify with CLI before adding more ## Extension Formats Extensions can be distributed in several formats: | Format | Extension | Description | | ------------------ | ------------------ | --------------------------------- | | **WASM Component** | `.wasm` | WebAssembly component (sandboxed) | | **Executable** | Platform-specific | JSON-RPC over stdio executable | | **Package** | `.morphir-ext.tgz` | Tar gzipped bundle with manifest | | **Directory** | `*/extension.toml` | Unpacked extension with manifest | ### WASM Component Extensions Sandboxed WebAssembly components using the Component Model: ``` extensions/ └── spark-codegen.wasm # Self-contained WASM component ``` ### Executable Extensions (JSON-RPC) Native executables that communicate via JSON-RPC over stdio: ``` extensions/ ├── my-backend # Unix executable ├── my-backend.exe # Windows executable └── my-frontend/ ├── extension.toml └── bin/ ├── frontend-linux-amd64 # Linux ├── frontend-darwin-amd64 # macOS Intel ├── frontend-darwin-arm64 # macOS Apple Silicon └── frontend-windows-amd64.exe ``` **Platform Resolution:** The daemon selects the appropriate executable based on OS and architecture: | OS | Architecture | Search Pattern | | ------- | ------------ | ----------------------------------------- | | Linux | amd64 | `*-linux-amd64`, `*-linux-x86_64`, `*` | | Linux | arm64 | `*-linux-arm64`, `*-linux-aarch64`, `*` | | macOS | amd64 | `*-darwin-amd64`, `*-darwin-x86_64`, `*` | | macOS | arm64 | `*-darwin-arm64`, `*-darwin-aarch64`, `*` | | Windows | amd64 | `*.exe`, `*-windows-amd64.exe` | **Executable Manifest:** ```toml # extension.toml [extension] id = "my-backend" name = "My Backend" version = "1.0.0" type = "codegen" # Executable configuration [extension.executable] # Platform-specific binaries [extension.executable.bin] "linux-amd64" = "bin/backend-linux-amd64" "linux-arm64" = "bin/backend-linux-arm64" "darwin-amd64" = "bin/backend-darwin-amd64" "darwin-arm64" = "bin/backend-darwin-arm64" "windows-amd64" = "bin/backend-windows-amd64.exe" # Or single cross-platform binary (e.g., Go, Java) # command = "bin/backend" # Arguments passed to executable args = ["--mode", "jsonrpc"] # Environment variables [extension.executable.env] LOG_LEVEL = "info" ``` ### Package Format (`.morphir-ext.tgz`) Distributable tar gzipped packages: ```bash # Package structure (when extracted) spark-codegen-1.2.0/ ├── extension.toml # Required: manifest ├── codegen.wasm # WASM component ├── README.md # Documentation ├── LICENSE └── examples/ └── basic.elm ``` **Creating a Package:** ```bash # Package an extension morphir extension pack ./spark-codegen/ # → spark-codegen-1.2.0.morphir-ext.tgz # Package with specific output morphir extension pack ./spark-codegen/ -o dist/ ``` **Installing a Package:** ```bash # Install from package file morphir extension install spark-codegen-1.2.0.morphir-ext.tgz # Extracts to: .morphir/extensions/spark-codegen/ # Install from URL morphir extension install https://example.com/spark-codegen-1.2.0.morphir-ext.tgz ``` **Package Manifest:** ```toml # extension.toml in package [extension] id = "spark-codegen" name = "Spark Code Generator" version = "1.2.0" description = "Generate Apache Spark DataFrame code from Morphir IR" author = "Morphir Contributors" license = "Apache-2.0" homepage = "https://github.com/finos/morphir-spark" # Component type and file type = "codegen" component = "codegen.wasm" # WASM component # OR # executable = "bin/codegen" # Native executable targets = ["spark"] # Dependencies on other extensions (optional) [extension.dependencies] morphir-ir = "^4.0.0" # Configuration schema [extension.options] spark_version = { type = "string", default = "3.5", description = "Spark version" } scala_version = { type = "string", default = "2.13", description = "Scala version" } ``` ## Extension Discovery Locations Extensions are discovered from multiple locations, in order of precedence: ### Discovery Order 1. **Explicit configuration** (`morphir.toml`) 2. **Workspace extensions** (`.morphir/extensions/`) 3. **User extensions** (`$XDG_DATA_HOME/morphir/extensions/`) 4. **System extensions** (`/usr/share/morphir/extensions/` or platform equivalent) ### Workspace Extensions ``` my-workspace/ ├── morphir.toml ├── .morphir/ │ ├── extensions/ # Auto-discovered │ │ ├── spark-codegen.wasm # WASM component │ │ ├── my-backend # Executable (Unix) │ │ ├── my-backend.exe # Executable (Windows) │ │ ├── flink-codegen/ # Directory extension │ │ │ ├── extension.toml │ │ │ └── codegen.wasm │ │ └── custom-frontend/ # Executable extension │ │ ├── extension.toml │ │ └── bin/ │ │ ├── frontend-linux-amd64 │ │ └── frontend-darwin-arm64 │ ├── cache/ │ └── deps/ ``` ### User Extensions (Global) ```bash # Linux/macOS $XDG_DATA_HOME/morphir/extensions/ ~/.local/share/morphir/extensions/ # Fallback # macOS alternative ~/Library/Application Support/morphir/extensions/ # Windows %LOCALAPPDATA%\morphir\extensions\ ``` ### System Extensions ```bash # Linux /usr/share/morphir/extensions/ /usr/local/share/morphir/extensions/ # macOS /Library/Application Support/morphir/extensions/ # Windows %PROGRAMDATA%\morphir\extensions\ ``` ### Explicit Configuration Override or supplement auto-discovery in `morphir.toml`: ```toml [extensions] # WASM component (explicit path) spark-codegen = { path = "./custom/spark-codegen.wasm" } # Executable with command my-backend = { command = "./bin/my-backend", args = ["--mode", "jsonrpc"] } # URL (downloaded and cached) flink-codegen = { url = "https://extensions.morphir.dev/flink-codegen-1.0.0.morphir-ext.tgz" } # Disable auto-discovered extension legacy-ext = { enabled = false } # Override options for auto-discovered extension [extensions.spark-codegen.config] spark_version = "3.4" ``` ### Discovery Resolution ```bash # Show where each extension was discovered from morphir extension list --show-source # Output: # NAME VERSION FORMAT SOURCE # spark-codegen 1.2.0 wasm .morphir/extensions/spark-codegen.wasm # my-backend 1.0.0 executable .morphir/extensions/my-backend/ # flink-codegen 1.0.0 package morphir.toml (url → cached) # elm-frontend 0.19.1 wasm ~/.local/share/morphir/extensions/ ``` ### JSON-RPC Extension Info (with Location) **Request:** ```json { "jsonrpc": "2.0", "id": "info-001", "method": "extension/info", "params": { "extension": "my-backend" } } ``` **Response (Executable Extension):** ```json { "jsonrpc": "2.0", "id": "info-001", "result": { "id": "my-backend", "name": "My Backend", "version": "1.0.0", "type": "codegen", "format": "executable", "source": { "type": "workspace", "path": ".morphir/extensions/my-backend/", "manifest": ".morphir/extensions/my-backend/extension.toml" }, "executable": { "resolved": ".morphir/extensions/my-backend/bin/backend-darwin-arm64", "platform": "darwin-arm64", "args": ["--mode", "jsonrpc"] }, "capabilities": { "codegen/generate": true, "codegen/generate-streaming": true } } } ``` **Response (WASM Extension):** ```json { "jsonrpc": "2.0", "id": "info-002", "result": { "id": "spark-codegen", "name": "Spark Code Generator", "version": "1.2.0", "type": "codegen", "format": "wasm", "source": { "type": "workspace", "path": ".morphir/extensions/spark-codegen.wasm" }, "component": { "path": ".morphir/extensions/spark-codegen.wasm", "size": 245760 }, "capabilities": { "codegen/generate": true, "codegen/generate-streaming": true, "codegen/generate-incremental": true } } } ``` ### Extension Installation ```bash # Install WASM component (to workspace) morphir extension install spark-codegen.wasm # → .morphir/extensions/spark-codegen.wasm # Install package (extracts) morphir extension install spark-codegen-1.2.0.morphir-ext.tgz # → .morphir/extensions/spark-codegen/ # Install globally morphir extension install --global spark-codegen.wasm # → ~/.local/share/morphir/extensions/spark-codegen.wasm # Install from URL morphir extension install https://releases.example.com/spark-codegen-1.2.0.morphir-ext.tgz # Downloads, verifies, extracts to .morphir/extensions/spark-codegen/ ``` ## Extension Types ### WASM Components Extensions implemented as WASM components using the Component Model: ```wit package morphir:extension@0.4.0; interface codegen { /// Generate code for a target language generate: func(ir: distribution, options: codegen-options) -> result; } ``` **Benefits:** - Language agnostic (Rust, Go, C, etc.) - Sandboxed execution - Capability-based permissions - Hot-reloadable ### Tasks User-defined or extension-provided commands: ```toml # Built-in tasks work automatically # Extensions can register additional intrinsic tasks [tasks.ci] description = "Run CI pipeline" depends = ["check", "test", "build"] [tasks."post:build"] run = "prettier --write .morphir-dist/" ``` **Task Types:** - **Built-in**: `build`, `test`, `check`, `codegen`, `pack`, `publish` - **Extension-provided**: Registered by WASM components - **User-defined**: Shell commands in `[tasks]` ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Morphir CLI/Daemon │ ├─────────────────────────────────────────────────────────┤ │ Extension Host │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ WASM Runtime│ │ Task Runner │ │ JSON-RPC │ │ │ │ (wasmtime) │ │ │ │ Protocol │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ └─────────┼────────────────┼────────────────┼─────────────┘ │ │ │ ▼ ▼ ▼ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ Codegen │ │ Custom │ │ External │ │ Extension │ │ Tasks │ │ Process │ └───────────┘ └───────────┘ └───────────┘ ``` ## Extension Points | Point | Mechanism | Use Case | | --------------- | --------------- | ------------------------ | | Code Generation | WASM component | Custom backend targets | | Frontend | WASM component | New source languages | | Validation | WASM component | Custom analyzers | | Tasks | Task definition | Build automation | | Hooks | Pre/post tasks | Extend built-in commands | ## Task System ### Built-in Tasks These work automatically without configuration: | Task | Description | | --------- | ---------------------------- | | `build` | Compile project to IR | | `test` | Run tests | | `check` | Lint and validate | | `codegen` | Generate code for targets | | `pack` | Create distributable package | | `publish` | Publish to registry | ### Pre/Post Hooks Extend built-in tasks with hooks: ```toml [tasks."pre:build"] run = "echo 'Starting build...'" [tasks."post:build"] run = "prettier --write .morphir-dist/" [tasks."post:codegen"] run = "./scripts/post-codegen.sh" ``` ### Task Dependencies Chain tasks together: ```toml [tasks.ci] description = "Full CI pipeline" depends = ["check", "test", "build", "pack"] [tasks.release] depends = ["ci"] run = "morphir publish --backend github" ``` ## Configuration ### Registering Extensions ```toml # morphir.toml [extensions] # WASM component extensions codegen-spark = { path = "./extensions/spark-codegen.wasm" } codegen-scala = { url = "https://extensions.morphir.dev/scala-codegen-1.0.0.wasm" } # Extension configuration [extensions.codegen-spark.config] spark_version = "3.5" ``` ### Extension Capabilities Extensions are **capability-based** - they declare which features they implement, allowing incremental development. An extension doesn't need to implement everything; it exports only what it supports. ```wit world codegen-extension { // Required imports (what the extension needs) import morphir:ir/types; import morphir:ir/values; // Provided exports (what the extension offers) export morphir:extension/codegen; // Optional exports (implement incrementally) // export morphir:extension/codegen-streaming; // export morphir:extension/codegen-incremental; } ``` ## Capability-Based Design Extensions implement features incrementally through optional interfaces: ### Frontend Capabilities | Capability | Interface | Description | | --------------- | ------------------------------ | ------------------------------- | | **Basic** | `frontend/compile` | Compile source to IR (required) | | **Streaming** | `frontend/compile-streaming` | Stream module-by-module results | | **Incremental** | `frontend/compile-incremental` | Recompile only changed files | | **Fragment** | `frontend/compile-fragment` | Compile code fragments (IDE) | | **Diagnostics** | `frontend/diagnostics` | Rich error messages with fixes | **Minimal Frontend:** ```wit world minimal-frontend { import morphir:ir/types; export frontend/compile; // Only basic compilation } ``` **Full-Featured Frontend:** ```wit world full-frontend { import morphir:ir/types; export frontend/compile; export frontend/compile-streaming; export frontend/compile-incremental; export frontend/compile-fragment; export frontend/diagnostics; } ``` ### Backend/Codegen Capabilities | Capability | Interface | Description | | ---------------- | ------------------------------ | ----------------------------------- | | **Basic** | `codegen/generate` | Generate code for target (required) | | **Streaming** | `codegen/generate-streaming` | Stream file-by-file output | | **Incremental** | `codegen/generate-incremental` | Regenerate only changed modules | | **Module-level** | `codegen/generate-module` | Generate single module | | **Options** | `codegen/options-schema` | Declare configurable options | **Minimal Backend:** ```wit world minimal-backend { import morphir:ir/distributions; export codegen/generate; // Only basic generation } ``` ### Capability Negotiation The daemon queries extension capabilities at load time: **JSON-RPC:** ```json { "jsonrpc": "2.0", "id": "caps-001", "method": "extension/capabilities", "params": { "extension": "codegen-spark" } } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "caps-001", "result": { "extension": "codegen-spark", "type": "codegen", "capabilities": { "codegen/generate": true, "codegen/generate-streaming": true, "codegen/generate-incremental": true, "codegen/generate-module": true, "codegen/options-schema": true }, "targets": ["spark"], "options": { "spark_version": { "type": "string", "default": "3.5" }, "scala_version": { "type": "string", "default": "2.13" } } } } ``` ### Graceful Degradation When an extension lacks a capability, Morphir falls back gracefully: | Missing Capability | Fallback Behavior | | --------------------- | ----------------------------------------- | | `compile-streaming` | Compile all at once, return single result | | `compile-incremental` | Full recompilation on every change | | `generate-streaming` | Generate all files, return at end | | `generate-module` | Generate full distribution | **CLI Feedback:** ```bash $ morphir codegen --target spark --stream Warning: spark-codegen does not support streaming, generating all at once... ``` ### Incremental Implementation Path Recommended order for implementing extension capabilities: **Frontend:** 1. `compile` - Basic compilation (MVP) 2. `diagnostics` - Better error messages 3. `compile-incremental` - Watch mode support 4. `compile-streaming` - Large project support 5. `compile-fragment` - IDE integration **Backend:** 1. `generate` - Basic codegen (MVP) 2. `options-schema` - Configurable output 3. `generate-module` - Granular generation 4. `generate-incremental` - Efficient rebuilds 5. `generate-streaming` - Large project support ## Extension Development Playbook A step-by-step guide for building Morphir extensions from scratch. ### Phase 1: Hello World (Day 1) **Goal:** Verify your development setup and basic connectivity. 1. **Create project structure:** ```bash mkdir my-morphir-extension && cd my-morphir-extension cargo new --lib . cargo add wit-bindgen ``` ```bash mkdir my-morphir-extension && cd my-morphir-extension go mod init my-morphir-extension go get github.com/bytecodealliance/wasm-tools-go ``` ```bash mkdir my-morphir-extension && cd my-morphir-extension npm init -y npm install @morphir/extension-sdk npm install -D typescript @aspect-build/component ``` 2. **Implement minimal info interface:** ```rust // src/lib.rs use morphir_extension::info::{ExtensionInfo, Info}; struct MyExtension; impl Info for MyExtension { fn get_info() -> ExtensionInfo { ExtensionInfo { id: "my-extension".into(), name: "My Extension".into(), version: "0.1.0".into(), description: "My first Morphir extension".into(), author: Some("My Name".into()), homepage: None, license: Some("Apache-2.0".into()), types: vec![], // Will add later } } fn ping() -> bool { true } } ``` ```go // main.go package main import "github.com/morphir/extension" type MyExtension struct{} func (e *MyExtension) GetInfo() extension.ExtensionInfo { return extension.ExtensionInfo{ ID: "my-extension", Name: "My Extension", Version: "0.1.0", Description: "My first Morphir extension", Author: stringPtr("My Name"), Homepage: nil, License: stringPtr("Apache-2.0"), Types: []extension.ExtensionType{}, // Will add later } } func (e *MyExtension) Ping() bool { return true } func stringPtr(s string) *string { return &s } ``` ```typescript // src/index.ts import { ExtensionInfo, Info, ExtensionType } from "@morphir/extension-sdk"; class MyExtension implements Info { getInfo(): ExtensionInfo { return { id: "my-extension", name: "My Extension", version: "0.1.0", description: "My first Morphir extension", author: "My Name", homepage: undefined, license: "Apache-2.0", types: [], // Will add later }; } ping(): boolean { return true; } } export default new MyExtension(); ``` 3. **Build and install:** ```bash # Build WASM component cargo build --target wasm32-unknown-unknown --release wasm-tools component new target/wasm32-unknown-unknown/release/my_extension.wasm \ -o my-extension.wasm # Install to workspace cp my-extension.wasm /path/to/workspace/.morphir/extensions/ ``` ```bash # Build WASM component GOOS=wasip1 GOARCH=wasm go build -o my-extension.wasm . # Install to workspace cp my-extension.wasm /path/to/workspace/.morphir/extensions/ ``` ```bash # Build WASM component npx tsc npx componentize -o my-extension.wasm dist/index.js # Install to workspace cp my-extension.wasm /path/to/workspace/.morphir/extensions/ ``` 4. **Verify installation:** ```bash morphir extension list # Should show: my-extension 0.1.0 unknown (info only) morphir extension ping my-extension # Should show: my-extension: OK (Xms) morphir extension info my-extension # Should show full extension metadata ``` **Checkpoint:** Extension appears in list, responds to ping. ### Phase 2: Basic Functionality (Week 1) **Goal:** Implement core functionality (compile or generate). **For a Backend (Code Generator):** 1. **Add generator interface:** ```rust use morphir_backend::generator::{Generator, GeneratorInfo, GenerationResult, Artifact}; impl Generator for MyExtension { fn info() -> GeneratorInfo { GeneratorInfo { name: "my-generator".into(), description: "Generates MyLang from Morphir IR".into(), version: "0.1.0".into(), target: TargetLanguage::Custom, custom_target: Some("mylang".into()), supported_granularities: vec![Granularity::Distribution], } } fn generate_distribution(dist: Distribution, options: GenerationOptions) -> GenerationResult { let artifacts = vec![ Artifact { path: "output.mylang".into(), content: transform_to_mylang(&dist), source_map: None, } ]; GenerationResult::Ok(artifacts) } } ``` ```go func (e *MyExtension) GeneratorInfo() generator.GeneratorInfo { return generator.GeneratorInfo{ Name: "my-generator", Description: "Generates MyLang from Morphir IR", Version: "0.1.0", Target: generator.TargetLanguageCustom, CustomTarget: stringPtr("mylang"), SupportedGranularities: []generator.Granularity{generator.GranularityDistribution}, } } func (e *MyExtension) GenerateDistribution(dist ir.Distribution, opts generator.GenerationOptions) generator.GenerationResult { artifacts := []generator.Artifact{ { Path: "output.mylang", Content: transformToMyLang(dist), SourceMap: nil, }, } return generator.GenerationResultOk(artifacts) } ``` ```typescript import { Generator, GeneratorInfo, GenerationResult, Artifact, } from "@morphir/extension-sdk"; class MyExtension implements Generator { generatorInfo(): GeneratorInfo { return { name: "my-generator", description: "Generates MyLang from Morphir IR", version: "0.1.0", target: "custom", customTarget: "mylang", supportedGranularities: ["distribution"], }; } generateDistribution( dist: Distribution, options: GenerationOptions, ): GenerationResult { const artifacts: Artifact[] = [ { path: "output.mylang", content: transformToMyLang(dist), sourceMap: undefined, }, ]; return { ok: artifacts }; } } ``` 2. **Update extension type:** ```rust fn get_info() -> ExtensionInfo { ExtensionInfo { // ... other fields types: vec![ExtensionType::Codegen], } } ``` ```go func (e *MyExtension) GetInfo() extension.ExtensionInfo { return extension.ExtensionInfo{ // ... other fields Types: []extension.ExtensionType{extension.ExtensionTypeCodegen}, } } ``` ```typescript getInfo(): ExtensionInfo { return { // ... other fields types: ["codegen"], }; } ``` 3. **Test with real IR:** ```bash # Create test IR morphir build /path/to/test-project # Run your generator morphir codegen --target mylang # Check output ls .morphir-dist/ ``` **Checkpoint:** Generator produces valid output files. ### Phase 3: Production Quality (Month 1) **Goal:** Add robustness features. 1. **Add capability discovery:** ```rust impl Capabilities for MyExtension { fn list_capabilities() -> Vec { vec![ CapabilityInfo { id: "codegen/generate".into(), description: "Basic code generation".into(), available: true, }, CapabilityInfo { id: "codegen/options-schema".into(), description: "Configurable options".into(), available: true, }, ] } fn get_targets() -> Vec { vec!["mylang".into()] } fn get_options_schema() -> Vec { vec![ OptionSchema { name: "indent".into(), option_type: OptionType::String, default_value: Some("\" \"".into()), description: "Indentation string".into(), required: false, }, ] } } ``` ```go func (e *MyExtension) ListCapabilities() []extension.CapabilityInfo { return []extension.CapabilityInfo{ { ID: "codegen/generate", Description: "Basic code generation", Available: true, }, { ID: "codegen/options-schema", Description: "Configurable options", Available: true, }, } } func (e *MyExtension) GetTargets() []string { return []string{"mylang"} } func (e *MyExtension) GetOptionsSchema() []extension.OptionSchema { return []extension.OptionSchema{ { Name: "indent", OptionType: extension.OptionTypeString, DefaultValue: stringPtr(" "), Description: "Indentation string", Required: false, }, } } ``` ```typescript listCapabilities(): CapabilityInfo[] { return [ { id: "codegen/generate", description: "Basic code generation", available: true, }, { id: "codegen/options-schema", description: "Configurable options", available: true, }, ]; } getTargets(): string[] { return ["mylang"]; } getOptionsSchema(): OptionSchema[] { return [ { name: "indent", optionType: "string", defaultValue: " ", description: "Indentation string", required: false, }, ]; } ``` 2. **Add error handling:** ```rust fn generate_distribution(dist: Distribution, options: GenerationOptions) -> GenerationResult { match try_generate(&dist, &options) { Ok(artifacts) => GenerationResult::Ok(artifacts), Err(e) => GenerationResult::Failed(vec![ Diagnostic { severity: Severity::Error, code: "GEN001".into(), message: format!("Generation failed: {}", e), location: None, } ]), } } ``` ```go func (e *MyExtension) GenerateDistribution(dist ir.Distribution, opts generator.Options) generator.Result { artifacts, err := tryGenerate(dist, opts) if err != nil { return generator.ResultFailed([]generator.Diagnostic{ { Severity: generator.SeverityError, Code: "GEN001", Message: fmt.Sprintf("Generation failed: %v", err), Location: nil, }, }) } return generator.ResultOk(artifacts) } ``` ```typescript generateDistribution(dist: Distribution, options: GenerationOptions): GenerationResult { try { const artifacts = this.tryGenerate(dist, options); return { ok: artifacts }; } catch (e) { return { failed: [ { severity: "error", code: "GEN001", message: `Generation failed: ${e}`, location: undefined, }, ], }; } } ``` 3. **Add module-level generation:** ```rust fn generate_module(path: ModulePath, module: ModuleDefinition, ...) -> GenerationResult { // Generate for single module } ``` ```go func (e *MyExtension) GenerateModule(path naming.ModulePath, module modules.Definition, ...) generator.Result { // Generate for single module } ``` ```typescript generateModule(path: ModulePath, module: ModuleDefinition, ...): GenerationResult { // Generate for single module } ``` **Checkpoint:** Extension handles errors gracefully, shows in `morphir extension info`. ### Phase 4: Advanced Features (Month 2+) **Goal:** Add streaming and incremental support. 1. **Add streaming generation:** ```rust impl backend::Streaming for MyExtension { fn generate_streaming(dist: Distribution, options: GenerationOptions) -> StreamHandle { // Start background generation let handle = start_generation_thread(dist, options); handle.id } fn poll_result(handle: StreamHandle) -> Option { // Return next completed module or None } } ``` 2. **Add incremental generation:** ```rust impl backend::Incremental for MyExtension { fn generate_incremental( dist: Distribution, changed_modules: Vec, options: GenerationOptions, ) -> IncrementalGenerationResult { // Only regenerate affected modules } } ``` 3. **Update capabilities:** ```rust fn list_capabilities() -> Vec { vec![ // ... existing CapabilityInfo { id: "codegen/generate-streaming".into(), description: "Stream results as modules complete".into(), available: true, }, CapabilityInfo { id: "codegen/generate-incremental".into(), description: "Regenerate only changed modules".into(), available: true, }, ] } ``` **Checkpoint:** `morphir codegen --target mylang --stream` shows incremental progress. ### Phase 5: Distribution (Month 3+) **Goal:** Package and distribute your extension. 1. **Create package manifest:** ```toml # extension.toml [extension] id = "my-extension" name = "My Extension" version = "1.0.0" description = "Generate MyLang from Morphir IR" author = "My Name" license = "Apache-2.0" homepage = "https://github.com/me/my-extension" type = "codegen" component = "my-extension.wasm" targets = ["mylang"] [extension.options] indent = { type = "string", default = " ", description = "Indentation" } ``` 2. **Create distributable package:** ```bash morphir extension pack ./ # Creates: my-extension-1.0.0.morphir-ext.tgz ``` 3. **Test installation:** ```bash # In a new workspace morphir extension install my-extension-1.0.0.morphir-ext.tgz morphir extension list morphir codegen --target mylang ``` **Checkpoint:** Package installs cleanly on other machines. ### Development Tips | Tip | Description | | ---------------------- | ----------------------------------------------------- | | **Start minimal** | Get `info` working before adding features | | **Test early** | Use `morphir extension ping` after each change | | **Check capabilities** | `morphir extension info ` shows what's detected | | **Watch logs** | `morphir daemon logs` shows extension loading | | **Iterate fast** | Hot-reload works with WASM components | ### Common Issues | Issue | Cause | Fix | | --------------------- | ----------------- | ------------------------------------------- | | Extension not found | Wrong location | Check `.morphir/extensions/` | | Ping fails | Missing info impl | Implement `info` interface | | No capabilities | Not exported | Export `capabilities` interface | | Codegen not triggered | Wrong target | Check `get_targets()` returns correct value | | Missing options | No schema | Implement `get_options_schema()` | ### Testing Checklist - [ ] `morphir extension list` shows your extension - [ ] `morphir extension ping ` returns OK - [ ] `morphir extension info ` shows correct metadata - [ ] Capabilities appear in info output - [ ] Target/language appears in info output - [ ] Options schema appears in info output - [ ] Basic generation produces valid output - [ ] Errors produce helpful diagnostics - [ ] Streaming shows incremental progress (if implemented) - [ ] Incremental skips unchanged modules (if implemented) ## Future: Alternative Extension Runtimes Beyond WASM components and native executables, several alternative extension runtimes are worth exploring for different trade-offs in performance, ease of authoring, and sandboxing. ### QuickJS [QuickJS](https://bellard.org/quickjs/) is a small, embeddable JavaScript engine that could enable JavaScript/TypeScript extensions without WASM compilation. **Potential Benefits:** - Familiar language for web developers - No compilation step (interpret directly) - Small runtime footprint (~210KB) - ES2020 support **Example:** ```javascript // extensions/my-codegen.js export function getInfo() { return { id: "my-codegen", name: "My Code Generator", version: "1.0.0", type: "codegen", }; } export function generate(ir, options) { // Transform IR to target code return { files: [{ path: "output.ts", content: generateTypeScript(ir) }], }; } ``` **Discovery:** ``` .morphir/extensions/ └── my-codegen.js # Detected as QuickJS extension ``` ### Javy (JavaScript → WASM) [Javy](https://github.com/bytecodealliance/javy) compiles JavaScript to WASM, combining JavaScript's ease of authoring with WASM's sandboxing. **Potential Benefits:** - Write in JavaScript, run as sandboxed WASM - Leverage existing JS ecosystem - Same security model as WASM components - AOT compilation for better performance than interpretation **Workflow:** ```bash # Author in JavaScript cat > my-extension.js << 'EOF' export function generate(ir) { ... } EOF # Compile to WASM javy compile my-extension.js -o my-extension.wasm # Install as normal WASM extension morphir extension install my-extension.wasm ``` ### Lua [Lua](https://www.lua.org/) is a lightweight, embeddable scripting language often used for game scripting and configuration. **Potential Benefits:** - Extremely lightweight (~300KB) - Fast startup time - Simple, learnable syntax - Battle-tested embedding API - Good for simple transformations **Example:** ```lua -- extensions/my-transform.lua function get_info() return { id = "my-transform", name = "My Transform", version = "1.0.0", type = "validator" } end function validate(ir) -- Check IR constraints local errors = {} for _, module in ipairs(ir.modules) do if not module.doc then table.insert(errors, { module = module.path, message = "Missing module documentation" }) end end return { valid = #errors == 0, errors = errors } end ``` ### Morphir IR Extensions (Self-Hosting) The most interesting possibility: **extensions written in Morphir itself**, compiled to IR, and interpreted or compiled by the daemon. **Potential Benefits:** - Dogfooding: Use Morphir to extend Morphir - Type-safe extension authoring - Extensions benefit from Morphir's guarantees - Can generate extensions to multiple targets - Ultimate validation of Morphir's expressiveness **Example:** ```elm -- extensions/MyCodegen.elm module MyCodegen exposing (generate) import Morphir.IR.Distribution exposing (Distribution) import Morphir.Extension exposing (GeneratedFile) generate : Distribution -> List GeneratedFile generate distribution = distribution.modules |> List.concatMap generateModule generateModule : Module -> List GeneratedFile generateModule mod = [ { path = modulePath mod ++ ".scala" , content = moduleToScala mod } ] ``` **Compilation:** ```bash # Compile extension to IR morphir build extensions/my-codegen/ # The IR itself becomes the extension # Daemon interprets or JIT-compiles the IR ``` **Architecture:** ``` ┌─────────────────────────────────────────────────────┐ │ Morphir Daemon │ ├─────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ WASM Runtime│ │ IR Interp. │ │ Native │ │ │ │ (wasmtime) │ │ (Morphir IR)│ │ (exec) │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐ │ │ │ .wasm │ │ .mir │ │ bin/* │ │ │ │ codegen │ │ codegen │ │ codegen │ │ │ └─────────┘ └─────────┘ └─────────┘ │ └─────────────────────────────────────────────────────┘ ``` **Considerations:** - Requires IR interpreter or JIT in the daemon - Bootstrap problem: need initial extensions to build Morphir extensions - Performance may vary (interpreted vs compiled) - Could compile IR extensions to WASM for production ### Runtime Comparison | Runtime | Sandboxed | Startup | Authoring | Performance | Size | | --------------------- | ---------- | ------- | --------- | ----------- | ------ | | **WASM Component** | ✓ Strong | Medium | Rust/Go/C | High | Medium | | **Native Executable** | ✗ Process | Fast | Any | Highest | Varies | | **QuickJS** | ✓ Embedded | Fast | JS/TS | Medium | Small | | **Javy** | ✓ WASM | Medium | JS/TS | Medium-High | Medium | | **Lua** | ✓ Embedded | Fastest | Lua | Medium | Tiny | | **Morphir IR** | ✓ Semantic | Varies | Morphir | Varies | Small | ### Exploration Status | Runtime | Status | Priority | | ----------------- | ------------- | --------------------- | | WASM Component | **Supported** | Primary | | Native Executable | **Supported** | Primary | | QuickJS | Future | Medium | | Javy | Future | Medium | | Lua | Future | Low | | Morphir IR | Future | **High** (dogfooding) | > **Note:** Alternative runtimes are documented for future exploration. The current implementation focuses on WASM components and native executables. Morphir IR extensions are particularly interesting as they would validate Morphir's expressiveness and enable self-hosting. ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](../daemon/)** - Workspace and build management - **[WASM Components](./wasm-component.md)** - Component model integration - **[Tasks](./tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification - [WIT Type Mapping](https://morphir.finos.org/docs/user-guides/cli-tools/wit-type-mapping/) - WIT to Morphir IR mapping --- ## Extension System Design Source: https://finos.github.io/morphir-rust/contributors/extension-system # Extension System Design This document describes the Morphir extension system, including current implementation and future vision. ## Overview The Morphir extension system allows adding support for new languages and targets through WASM-based extensions. Extensions implement the `Frontend` and/or `Backend` traits to provide language-specific functionality. ## Current Implementation ### Architecture The current implementation uses: - **Extism**: WASM runtime for loading and executing extensions - **Extension Registry**: Manages loaded extensions - **Extension Container**: Wraps Extism plugin with Morphir-specific functionality ### Extension Types Extensions can implement: - **Frontend**: Converts source code → Morphir IR V4 - **Backend**: Converts Morphir IR V4 → Generated code - **Transform**: IR-to-IR transformations - **Validator**: IR validation ### Extension Discovery Extensions are discovered in this order: 1. **Builtin Extensions**: Bundled with CLI (e.g., Gleam) 2. **Config Extensions**: Specified in `morphir.toml` 3. **Registry Extensions**: Installed from extension registry ### Extension Loading ```rust // Discover extension let extension = registry.find_extension_by_language("gleam").await?; // Call extension method let result: serde_json::Value = extension .call("morphir.frontend.compile", params) .await?; ``` ## Future Vision: Actor Model The design documents describe a sophisticated actor-based architecture: ### Extension Manager Central coordinator as a singleton Kameo actor: - Manages extension lifecycle - Routes extension requests - Coordinates extension hosts ### Extension Hosts One actor per protocol type: - **JSON-RPC Host**: JSON-RPC protocol extensions - **gRPC Host**: gRPC protocol extensions - **Stdio Host**: Standard I/O extensions - **Extism WASM Host**: Current WASM extensions - **WASM Component Host**: Future WASM Component Model extensions ### Benefits of Actor Model - **Isolation**: Each component is an independent actor - **Concurrency**: Parallel processing across actors - **Supervision**: Automatic failure recovery - **Scalability**: Easy to add new protocol types ## Migration Path The current implementation is designed to be compatible with the future actor model: 1. **Current**: Direct `ExtensionRegistry` usage 2. **Future**: Extension Manager actor routes requests 3. **Compatibility**: Current API can be wrapped in actor messages ## Extension Development See [Extension Development Tutorial](../tutorials/extension-tutorial) for how to create extensions. ## Next Steps - Read [Extension Host Interface](design/extension-host-interface) - See [Extism WASM Host](design/extism-wasm-host) - Check [Extension Manager](design/extension-manager) --- ## CLI Architecture Source: https://finos.github.io/morphir-rust/contributors/cli-architecture # CLI Architecture This document describes the architecture of the Morphir CLI, including command structure, configuration handling, and extension integration. ## Command Structure The CLI uses `clap` for argument parsing and `starbase` for application lifecycle: ``` morphir ├── compile # Language-agnostic compilation ├── generate # Language-agnostic code generation ├── gleam # Gleam-specific commands │ ├── compile │ ├── generate │ └── roundtrip ├── extension # Extension management ├── ir # IR operations └── ... ``` ## Command Execution Flow ### Compile Command ``` 1. Parse CLI arguments 2. Discover configuration (morphir-design) 3. Load config context (workspace/project detection) 4. Merge CLI args with config (CLI overrides) 5. Resolve paths (.morphir/out//compile//) 6. Discover extension by language (morphir-design → morphir-daemon) 7. Load extension WASM (morphir-daemon) 8. Collect source files 9. Call extension.frontend.compile() 10. Write IR to output directory 11. Format output (human/JSON/JSON Lines) ``` ### Generate Command ``` 1. Parse CLI arguments 2. Discover configuration 3. Load config context 4. Resolve IR input path 5. Discover extension by target (morphir-design → morphir-daemon) 6. Load extension WASM 7. Load Morphir IR (detect format) 8. Call extension.backend.generate() 9. Write generated code to output directory 10. Format output ``` ## Configuration Handling ### Discovery Configuration is discovered by walking up the directory tree: ```rust let config_path = discover_config(¤t_dir)?; let ctx = load_config_context(&config_path)?; ``` ### Merging Configuration is merged in priority order: 1. Workspace config (if in workspace) 2. Project config 3. CLI arguments (highest priority) ### Path Resolution Paths are resolved relative to: - Config file location (for relative paths) - Workspace root (for workspace paths) - Project root (for project paths) ## Extension Integration ### Discovery ```rust // Design-time discovery let builtins = morphir_design::discover_builtin_extensions(); // Daemon registry let registry = ExtensionRegistry::new(workspace_root, output_dir)?; // Register builtins for builtin in builtins { registry.register_builtin(&builtin.id, builtin.path).await?; } // Find extension let extension = registry.find_extension_by_language("gleam").await?; ``` ### Execution ```rust // Call extension method let params = serde_json::json!({ "input": input_path, "output": output_path, "package_name": package_name, }); let result: serde_json::Value = extension .call("morphir.frontend.compile", params) .await?; ``` ## Output Formatting ### Human-Readable Default format with progress messages and diagnostics. ### JSON Single JSON object with structured data: ```json { "success": true, "ir": {...}, "diagnostics": [...], "modules": [...] } ``` ### JSON Lines Streaming format (one JSON object per line): ```jsonl {"type": "progress", "message": "Compiling..."} {"type": "result", "success": true, "ir": {...}} ``` ## Error Handling Errors are handled using `miette` for rich diagnostics: - **Human mode**: Pretty-printed errors with source spans - **JSON mode**: Structured error objects ## Next Steps - See [Design-Time Crate](design-time-crate) - Read [Extension System Design](extension-system) - Check [Development Guide](development) --- ## Design-Time Crate Source: https://finos.github.io/morphir-rust/contributors/design-time-crate # Design-Time Crate The `morphir-design` crate provides reusable design-time tooling that can be used by the CLI, IDEs, build tools, and other Morphir tooling. ## Purpose The design-time crate separates concerns: - **CLI**: User-facing commands and output formatting - **Design-Time**: Configuration and extension discovery (reusable) - **Common**: Shared data structures and utilities - **Daemon**: Runtime extension execution This allows IDEs and other tools to use design-time functionality without CLI dependencies. ## Key Functionality ### Configuration Discovery ```rust // Walk up directory tree to find morphir.toml let config_path = discover_config(&start_dir)?; // Load config with workspace/project context let ctx = load_config_context(&config_path)?; ``` ### Extension Discovery ```rust // Discover builtin extensions let builtins = discover_builtin_extensions(); // Get builtin extension path let path = get_builtin_extension_path("gleam")?; ``` ### Path Resolution ```rust // Resolve compile output path let compile_path = resolve_compile_output( "My.Project", "gleam", &morphir_dir ); // Resolve generate output path let generate_path = resolve_generate_output( "My.Project", "gleam", &morphir_dir ); ``` ## Usage in IDEs IDEs can use the design-time crate to: - Discover project configuration - Resolve paths for build outputs - Find available extensions - Determine workspace/project context Example: ```rust use morphir_design::{discover_config, load_config_context}; // In IDE plugin let ctx = load_config_context(&discover_config(&project_dir)?)?; let compile_path = resolve_compile_output( &ctx.config.project.as_ref().unwrap().name, "gleam", &ctx.morphir_dir ); ``` ## Usage in Build Tools Build tools can integrate Morphir compilation: ```rust use morphir_design::{discover_config, ensure_morphir_structure}; // In build script let ctx = load_config_context(&discover_config(&project_dir)?)?; ensure_morphir_structure(&ctx.morphir_dir)?; // ... trigger compilation ``` ## API Stability The design-time crate API is designed to be stable and reusable. Changes should maintain backward compatibility when possible. ## Next Steps - See [Architecture Overview](architecture) - Read [CLI Architecture](cli-architecture) - Check [Development Guide](development) --- ## Design Documents Source: https://finos.github.io/morphir-rust/contributors/design/README # Morphir Design Documents This directory contains design documents for the Morphir daemon and extension system, covering architecture, protocols, and implementation details. ## Overview This directory is organized into two main areas: 1. **Daemon Design** - Workspace management, build orchestration, file watching, and CLI-daemon interaction 2. **Extension Design** - WASM component model, extension interfaces, and task system ## Daemon Design Documents The daemon design documents describe the Morphir daemon architecture for managing workspaces, projects, builds, and IDE integration: ### Core Documents - **[README](daemon/README.md)** - Daemon overview and architecture - **[Lifecycle](daemon/lifecycle.md)** - Workspace creation, opening, closing - **[Projects](daemon/projects.md)** - Project management within a workspace - **[Dependencies](daemon/dependencies.md)** - Dependency resolution and caching - **[Build](daemon/build.md)** - Build orchestration and diagnostics - **[Watching](daemon/watching.md)** - File system watching for incremental builds - **[Packages](daemon/packages.md)** - Package format, registry backends, publishing - **[Configuration](daemon/configuration.md)** - morphir.toml system overview - **[Workspace Config](daemon/workspace-config.md)** - Multi-project workspace configuration - **[CLI Interaction](daemon/cli-interaction.md)** - CLI-daemon communication and lifecycle ### Configuration Documents - **[morphir.toml](daemon/morphir-toml.md)** - Complete configuration file specification - **[Merge Rules](daemon/merge-rules.md)** - Configuration inheritance and merge behavior - **[Environment](daemon/environment.md)** - Environment variables and runtime overrides ## Extension Design Documents The extension design documents describe the architecture for adding capabilities to Morphir via WASM components and the task system: ### Core Documents - **[README](extensions/README.md)** - Extension system overview and getting started - **[WASM Components](extensions/wasm-component.md)** - Component model integration and WIT interfaces - **[Tasks](extensions/tasks.md)** - Task system, dependencies, and hooks ## Key Concepts ### Daemon The Morphir daemon is a long-running service that: - Manages workspaces and projects - Orchestrates builds in dependency order - Watches files for automatic recompilation - Provides IDE integration via JSON-RPC - Hosts extensions for design-time and runtime use ### Extensions Morphir extensions enable: - Custom code generators (new backend targets) - Custom frontends (new source languages) - Additional tasks (build automation) - Protocol integration (JSON-RPC based communication) Extensions are implemented as: - **WASM Components** - Sandboxed WebAssembly components using the Component Model - **Native Executables** - JSON-RPC over stdio executables - **Packages** - Distributable bundles with manifest ## Design Status All documents in this directory are marked as **draft** status and represent the evolving design for the Morphir daemon and extension system. These documents guide implementation work in the morphir-rust repository. ## References ### Morphir Rust Documentation - [Extension System Design](../extension-system) - Current implementation overview - [Architecture Overview](../architecture) - System architecture - [CLI Architecture](../cli-architecture) - CLI integration ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index for LLMs - [Morphir Design Documents](https://github.com/finos/morphir/tree/main/docs/design/draft) - Source repository for these design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents ### Related Design Documents - [Daemon Design (Main Repo)](https://morphir.finos.org/docs/design/draft/daemon/) - Original daemon design documents - [Extension Design (Main Repo)](https://morphir.finos.org/docs/design/draft/extensions/) - Original extension design documents --- ## Development Guide Source: https://finos.github.io/morphir-rust/contributors/development # Development Guide This guide helps developers contribute to Morphir Rust. ## Setting Up Development Environment ### Prerequisites - Rust toolchain (1.70+) - `wasm32-unknown-unknown` target for extension development - Git ### Clone and Build ```bash git clone https://github.com/finos/morphir-rust.git cd morphir-rust cargo build ``` ### Run Tests ```bash # All tests cargo test # Specific crate cargo test -p morphir # BDD acceptance tests cd crates/morphir-gleam-binding cargo test --test acceptance ``` ## Project Structure ``` morphir-rust/ ├── crates/ │ ├── morphir/ # CLI │ ├── morphir-design/ # Design-time tooling │ ├── morphir-common/ # Shared infrastructure │ ├── morphir-daemon/ # Runtime services │ ├── morphir-gleam-binding/ # Gleam extension │ └── ... ├── docs/ # Documentation site └── ... ``` ## Code Organization ### CLI Commands Commands are in `crates/morphir/src/commands/`: - `compile.rs` - Compile command - `generate.rs` - Generate command - `gleam.rs` - Gleam subcommands ### Design-Time Design-time functionality in `crates/morphir-design/`: - `config.rs` - Configuration discovery - `extensions.rs` - Extension discovery ### Common Utilities Shared code in `crates/morphir-common/`: - `config/model.rs` - Configuration models - `pipeline/` - Pipeline framework - `vfs/` - Virtual file system ## Adding a New Command 1. Create command module in `crates/morphir/src/commands/` 2. Add to `Commands` enum in `main.rs` 3. Wire up execution in `MorphirSession::execute()` 4. Add help text in `help.rs` 5. Add tests in `tests/cli_integration.rs` ## Adding a New Extension 1. Create extension crate 2. Implement `Extension`, `Frontend`, and/or `Backend` traits 3. Build as WASM: `cargo build --target wasm32-unknown-unknown` 4. Add to builtin extensions (if applicable) 5. Update extension discovery ## Testing ### Unit Tests ```rust #[cfg(test)] mod tests { #[test] fn test_feature() { // Test implementation } } ``` ### Integration Tests Integration tests in `tests/` directory: ```rust #[tokio::test] async fn test_command() { // Test command execution } ``` ### BDD Tests BDD tests use Cucumber/Gherkin: ```gherkin Feature: My Feature Scenario: Test case Given setup When action Then assertion ``` ## Code Style - Follow Rust standard formatting (`cargo fmt`) - Use `clippy` for linting (`cargo clippy`) - Document public APIs - Write tests for new features ## Contributing 1. Create a feature branch 2. Make changes 3. Add tests 4. Update documentation 5. Submit pull request ## Next Steps - Read [Architecture Overview](architecture) - See [Extension System Design](extension-system) - Check [CLI Architecture](cli-architecture) --- ## For Contributors Source: https://finos.github.io/morphir-rust/contributors/ # For Contributors Documentation for developers contributing to Morphir Rust. ## Architecture - [Architecture Overview](architecture) - System architecture and crate responsibilities - [Extension System Design](extension-system) - Extension system architecture - [CLI Architecture](cli-architecture) - CLI command structure and execution flow - [Design-Time Crate](design-time-crate) - Reusable design-time tooling ## Development - [Development Guide](development) - Setting up and contributing ## Design Documents - [Design Documents](design/) - Extension system design documents --- ## Extension Development Reference Source: https://finos.github.io/morphir-rust/extensions/DEVELOPMENT # Morphir Extension Development Guide This guide describes how to develop Morphir extensions using various programming languages. Morphir extensions are WebAssembly (Wasm) components that follow the The Elm Architecture (TEA) pattern. ## Prerequisites Regardless of the language you choose, you will need the following tools: - [Wasmtime](https://wasmtime.dev/) (Runtime) - [wit-bindgen](https://github.com/bytecodealliance/wit-bindgen) (Binding generator) - [wit-component](https://github.com/bytecodealliance/wit-component) (Component encoder) The WIT definitions for Morphir extensions are located in `crates/morphir-ext-core/wit`. ## WASI Compatibility and the Component Model Morphir uses a hybrid architecture to balance modern features with toolchain compatibility: - **Host (Preview 2)**: The Morphir Daemon provides a **WASI Preview 2 (P2)** environment. This is based on the WebAssembly Component Model, which allows us to use WIT for high-level interfaces. - **Guests (Preview 1 / Unknown)**: - Most language toolchains (Rust `wasm32-wasip1`, TinyGo) still generate **WASI Preview 1 (P1)** code. These can be run in our host using a "compatibility adapter" (shim). - For extensions that don't need system access (like pure logic or string transformations), we recommend the `wasm32-unknown-unknown` target. This produces a "pure" Wasm module with no system dependencies, making it faster to load and easier to distribute. --- ## 1. Rust Rust provides the most mature support for Wasm Components. ### Setup Add `wit-bindgen` to your `Cargo.toml`: ```toml [dependencies] wit-bindgen = "0.35.0" ``` ### Implementation Use the `bindgen!` macro to generate host and guest bindings. ```rust wit_bindgen::generate!({ world: "extension", path: "wit", // Path to the .wit files }); struct MyExtension; impl Guest for MyExtension { fn init(init_data: Envelope) -> (Envelope, Envelope) { // ... implementation } fn update(msg: Envelope, model: Envelope) -> (Envelope, Envelope) { // ... implementation } // ... other TEA methods } export!(MyExtension); ``` ### Build Target `wasm32-wasip1` or `wasm32-unknown-unknown`, then encode into a component. --- ## 2. TypeScript / JavaScript TypeScript extensions are built using `jco` and `componentize-js`. ### Setup Install the necessary tools: ```bash npm install -g @bytecodealliance/jco @bytecodealliance/componentize-js ``` ### Implementation Create a module that matches the exported `program` interface. ```typescript export const program = { init(initData) { const initialModel = { /* ... */ }; return [initialModel, []]; }, update(msg, model) { // ... implementation return [newModel, commands]; }, subscriptions(model) { return []; }, info() { return { content: "My TypeScript Extension" }; } }; ``` ### Build Componentize the JavaScript using `jco`: ```bash jco componentize app.js --wit wit --world extension -o extension.wasm ``` --- ## 3. Python Python extensions are built using `componentize-py`. ### Setup Install `componentize-py`: ```bash pip install componentize-py ``` ### Implementation Implement the `Extension` world using Python class-based or module-based exports. ```python import extension class MyExtension(extension.Extension): def init(self, init_data: extension.Envelope) -> (extension.Envelope, extension.Envelope): # ... return (model, commands) def update(self, msg: extension.Envelope, model: extension.Envelope) -> (extension.Envelope, extension.Envelope): # ... return (new_model, commands) ``` ### Build ```bash componentize-py -d wit -w extension componentize my_app -o extension.wasm ``` --- ## 4. Go Go extensions use `TinyGo` for small Wasm binaries and `wit-bindgen-go`. ### Setup Install [TinyGo](https://tinygo.org/) and [wit-bindgen-go](https://github.com/bytecodealliance/wit-bindgen-go). ### Implementation Generate bindings and implement the methods. ```go package main import ( "morphir/ext/program" "morphir/ext/envelope" ) type MyExtension struct{} func (e *MyExtension) Init(initData envelope.Envelope) (envelope.Envelope, envelope.Envelope) { // ... } func (e *MyExtension) Update(msg envelope.Envelope, model envelope.Envelope) (envelope.Envelope, envelope.Envelope) { // ... } func main() { program.SetProgram(&MyExtension{}) } ``` ### Build ```bash tinygo build -o extension.wasm -target=wasi main.go # Then use wit-component to encode ``` --- ## Appendix: Low-Level ABI Specification If you are developing for a language that doesn't yet have high-level Component Model toolchains, you can implement the **Core Wasm ABI** directly. ### Memory Convention All structured data (strings, bytes, JSON) is passed using a `(pointer, length)` pair where both values are `i32`. ### Guest Exports The extension must export the following functions: | Function | Signature | Purpose | | :--- | :--- | :--- | | `init` | `(hdr_p, hdr_l, ct_p, ct_l, m_p, m_l) -> i32` | Initializes the extension. Returns a program ID. | | `update` | `(id, hdr_p, hdr_l, ct_p, ct_l, m_p, m_l) -> void` | Sends a message to a specific program instance. | ### Host Imports The host provides the following functions in the `env` module: | Function | Signature | Purpose | | :--- | :--- | :--- | | `log` | `(level, ptr, len) -> void` | Logs a message. Level: 0=Trace, 1=Debug, 2=Info, 3=Warn, 4=Error. | | `get_env_var`| `(k_p, k_l, o_p, o_l) -> void` | Retrieves an environment variable as JSON. | | `set_env_var`| `(k_p, k_l, v_p, v_l) -> void` | Stores an environment variable as JSON. | ### Data Structures - **Header**: Passed as a JSON string: `{"seqnum": u64, "session_id": "uuid", "kind": "hint"}`. - **Envelope Content**: Passed as raw bytes. - **Environment Value**: Passed as a JSON serialized `EnvValue` enum. For further details, explore the implementation in `crates/morphir-ext-core/src/abi.rs`. --- # Other ## Morphir Extension System Design Documents Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/README # Morphir Extension System Design Documents This directory contains the complete design documentation for the Morphir Extension System, an actor-based, multi-protocol architecture for extending Morphir functionality. ## Document Index ### Core Architecture - **[00-overview.md](00-overview.md)** - System overview, goals, and key concepts - **[01-architecture.md](01-architecture.md)** - Detailed architecture, components, and data flow - **[02-extension-host-interface.md](02-extension-host-interface.md)** - Common interface for all extension hosts ### Protocol Implementations - **[03-jsonrpc-host.md](03-jsonrpc-host.md)** - JSON-RPC 2.0 over HTTP - **[04-grpc-host.md](04-grpc-host.md)** - gRPC with Protocol Buffers - **[05-stdio-host.md](05-stdio-host.md)** - JSON Lines over stdin/stdout - **[06-extism-wasm-host.md](06-extism-wasm-host.md)** - Extism WASM runtime - **[07-wasm-component-host.md](07-wasm-component-host.md)** - WASM Component Model ### Integration & Operations - **[08-extension-manager.md](08-extension-manager.md)** - Central coordinator actor - **[09-security-and-isolation.md](09-security-and-isolation.md)** - Security model and sandboxing - **[10-protocol-specifications.md](10-protocol-specifications.md)** - Wire protocol details - **[11-examples-and-recipes.md](11-examples-and-recipes.md)** - Practical examples and patterns ## Quick Start For a quick understanding of the system: 1. Read [00-overview.md](00-overview.md) for the big picture 2. Read [01-architecture.md](01-architecture.md) for component details 3. Choose your protocol (likely [05-stdio-host.md](05-stdio-host.md) for simplicity) 4. Follow examples in [11-examples-and-recipes.md](11-examples-and-recipes.md) ## Key Technologies - **Actor Framework**: Kameo (Rust) - **Protocols**: JSON-RPC 2.0, gRPC, Stdio (JSON Lines), WASM (Extism, Component Model) - **Concurrency**: Tokio async runtime - **Security**: Process isolation, WASM sandboxing, capability-based permissions ## Design Status **Status**: Draft **Version**: 0.1.0 **Last Updated**: 2025-01-23 These documents are design drafts intended for incorporation into the FINOS Morphir project documentation. ## Contributing When implementing these designs: 1. Maintain actor-based isolation 2. Follow the `ExtensionHost` trait contract 3. Implement all protocol lifecycle methods 4. Add comprehensive tests 5. Update security documentation as needed ## Related Resources - [FINOS Morphir Project](https://github.com/finos/morphir) - [Kameo Actor Framework](https://github.com/tqwewe/kameo) - [Extism WASM Framework](https://extism.org) - [WASM Component Model](https://component-model.bytecodealliance.org) --- ## Morphir Extension System - Overview Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/00-overview # Morphir Extension System - Overview **Status:** Draft **Version:** 0.1.0 **Last Updated:** 2025-01-23 ## Purpose The Morphir Extension System provides a pluggable architecture for extending Morphir's capabilities through multiple protocols and runtime environments. This enables developers to write extensions in their language of choice while maintaining type safety, security, and performance. ## Goals 1. **Multi-Protocol Support**: Enable extensions via JSON-RPC, gRPC, stdio, and WASM 2. **Language Agnostic**: Allow extensions in any language (TypeScript, Python, Go, Gleam, Rust, etc.) 3. **Type Safety**: Maintain strong typing at protocol boundaries 4. **Isolation**: Ensure extension failures don't crash the core system 5. **Performance**: Support both high-throughput (gRPC) and lightweight (stdio) options 6. **Security**: Sandbox untrusted extensions (WASM) 7. **Developer Experience**: Make extension development simple and well-documented ## Non-Goals 1. Dynamic code loading of unsafe native code 2. Backwards compatibility with pre-1.0 Morphir APIs (clean break) 3. Supporting every possible IPC mechanism (focus on proven protocols) ## Key Concepts ### Extension A piece of code that extends Morphir functionality. Extensions can: - Transform IR (custom backends, optimizers) - Validate IR (custom linting, business rules) - Generate code (new target languages) - Provide tooling (formatters, analyzers) - Integrate external services (databases, APIs) ### Extension Host An actor that manages extensions using a specific protocol. Each host: - Spawns/manages extension processes or runtimes - Translates between Morphir's internal API and the protocol - Handles failures and restarts - Reports health and metrics ### Extension Manager The central coordinator that: - Routes requests to appropriate hosts - Maintains extension registry - Handles extension lifecycle - Provides unified API to Morphir core ## Supported Protocols | Protocol | Use Case | Languages | Performance | Isolation | | ---------------------- | --------------------------- | ----------------------- | ----------- | ------------ | | **JSON-RPC 2.0** | Web services, microservices | Any with HTTP | Medium | Process | | **gRPC** | High-performance, typed | Any with protobuf | High | Process | | **Stdio (JSON Lines)** | Simple tools, scripts | Any | Low-Medium | Process | | **Extism WASM** | Sandboxed, portable | Many via Extism PDK | Medium | WASM sandbox | | **WASM Component** | Future-proof, portable | Rust, C, others growing | Medium-High | WASM sandbox | ## Architecture Principles 1. **Actor-Based Isolation**: Each extension host runs as an independent Kameo actor 2. **Protocol Abstraction**: Common `ExtensionHost` trait hides protocol details 3. **Fail-Safe**: Extension failures are isolated and recoverable 4. **Observable**: All extension operations emit metrics and logs 5. **Configurable**: Extensions declared in configuration files ## Document Organization - **01-architecture.md**: Overall system architecture and component relationships - **02-extension-host-interface.md**: Common trait and interfaces - **03-08**: Individual host implementations - **09**: Security model and sandboxing - **10**: Wire protocol specifications - **11**: Usage examples and recipes ## Related Documents - [Morphir IR Specification](../ir-spec.md) - [Morphir CLI Architecture](../cli-architecture.md) - [Configuration Schema](../config-schema.md) --- ## Tasks Source: https://finos.github.io/morphir-rust/contributors/design/extensions/tasks # Tasks This document defines the task system for Morphir, modeled after [mise tasks](https://mise.jdx.dev/tasks/). ## Overview Morphir provides **built-in tasks** for common operations and allows **user-defined tasks** for custom workflows. Tasks are run via `morphir run `. ### Built-in Tasks Morphir and its extensions provide intrinsic tasks that work out of the box: | Task | Description | Equivalent Command | | --------- | ---------------------------- | ------------------ | | `build` | Compile project to IR | `morphir build` | | `test` | Run tests | `morphir test` | | `check` | Lint and validate | `morphir check` | | `codegen` | Generate code for targets | `morphir codegen` | | `clean` | Remove build artifacts | `morphir clean` | | `pack` | Create distributable package | `morphir pack` | | `publish` | Publish to registry | `morphir publish` | ```bash # These all work without any configuration morphir run build morphir run test morphir run codegen ``` ### User-Defined Tasks Users can define custom tasks or override built-in tasks: ```toml # morphir.toml [tasks] # Custom task integration = "./scripts/integration-tests.sh" # Override built-in task test = "morphir test && ./scripts/integration-tests.sh" ``` ```bash morphir run integration # Custom task morphir run test # Runs overridden test task ``` ### Extension-Provided Tasks Extensions (via WASM components) can register additional intrinsic tasks: ```bash # Tasks provided by a TypeScript codegen extension morphir run codegen:typescript # Tasks provided by a Scala codegen extension morphir run codegen:scala ``` See [WASM Components](./wasm-component.md) for how extensions register tasks. ## Task Definition ### Inline Tasks The simplest form is an inline command string: ```toml [tasks] lint = "morphir check --strict" clean = "rm -rf .morphir dist/" ``` ### Detailed Tasks For more control, use the detailed table syntax: ```toml [tasks.build] description = "Build the project and generate TypeScript" run = "morphir build && morphir codegen typescript" [tasks.test] description = "Run all tests" run = [ "morphir build", "morphir test", "./scripts/integration-tests.sh" ] depends = ["lint"] ``` ### Task Options | Option | Type | Description | | ------------- | ------------------ | ----------------------------------------- | | `run` | string or string[] | Command(s) to execute | | `description` | string | Description shown in `morphir run --list` | | `depends` | string[] | Tasks to run before this task | | `env` | table | Environment variables for the task | | `dir` | string | Working directory (default: project root) | | `sources` | string[] | File patterns that trigger rebuild | | `outputs` | string[] | Expected output files | | `hide` | bool | Hide from task listings | ## Task Dependencies Tasks can depend on other tasks: ```toml [tasks.lint] run = "morphir check" [tasks.test] run = "morphir test" depends = ["lint"] [tasks.ci] description = "Run full CI pipeline" depends = ["lint", "test", "build"] run = "echo 'CI complete'" ``` Dependencies run in order. If multiple dependencies have no interdependencies, they may run in parallel. ## Environment Variables Set environment variables for tasks: ```toml [tasks.test] run = "morphir test" env = { MORPHIR_LOG_LEVEL = "debug", CI = "true" } [tasks.deploy] run = "./scripts/deploy.sh" env = { ENVIRONMENT = "production" } ``` ## File-Based Tasks For complex tasks, use external scripts in a `tasks/` or `.morphir/tasks/` directory: ``` my-project/ ├── morphir.toml ├── tasks/ │ ├── build.sh │ ├── deploy.sh │ └── test.py ``` File-based tasks are automatically discovered and can include metadata: ```bash #!/usr/bin/env bash #MISE description="Build and package for release" #MISE depends=["lint", "test"] set -euo pipefail morphir build --release morphir pack ``` Reference file tasks in configuration: ```toml [tasks.build] file = "tasks/build.sh" [tasks.deploy] file = "tasks/deploy.sh" env = { ENVIRONMENT = "production" } ``` ## Pre/Post Hooks Conventional `pre:` and `post:` task prefixes allow extending built-in Morphir commands: ```toml [tasks."pre:build"] description = "Run before morphir build" run = "echo 'Starting build...'" [tasks."post:build"] description = "Run after morphir build" run = [ "cp .morphir-dist/morphir-ir.json dist/", "echo 'Build complete'" ] [tasks."pre:test"] run = "morphir build" [tasks."post:codegen"] run = "prettier --write generated/" ``` ### Hook Execution Order When running `morphir build`: 1. `pre:build` task (if defined) 2. Built-in `morphir build` command 3. `post:build` task (if defined) ### Available Hooks | Hook | Triggered By | | ------------------------------ | ----------------- | | `pre:build` / `post:build` | `morphir build` | | `pre:test` / `post:test` | `morphir test` | | `pre:codegen` / `post:codegen` | `morphir codegen` | | `pre:pack` / `post:pack` | `morphir pack` | | `pre:publish` / `post:publish` | `morphir publish` | | `pre:clean` / `post:clean` | `morphir clean` | ### Disabling Hooks ```bash # Skip hooks for a single command morphir build --no-hooks # Skip specific hook morphir build --skip-hook=pre:build ``` ## Workspace Tasks In workspaces, tasks can be defined at workspace or project level: ```toml # workspace/morphir.toml [workspace] members = ["packages/*"] [tasks] # Workspace-level tasks available from anywhere build-all = "morphir workspace build" test-all = "morphir workspace test" ``` ```toml # workspace/packages/core/morphir.toml [project] name = "my-org/core" [tasks] # Project-specific tasks benchmark = "./scripts/benchmark.sh" ``` ### Running Workspace Tasks ```bash # From workspace root morphir run build-all # From project directory (runs project task) cd packages/core morphir run benchmark # Run workspace task from project directory morphir run --workspace build-all ``` ## Built-in Variables Tasks have access to these environment variables: | Variable | Description | | ------------------------ | --------------------------------- | | `MORPHIR_PROJECT_ROOT` | Project root directory | | `MORPHIR_WORKSPACE_ROOT` | Workspace root (if in workspace) | | `MORPHIR_PROJECT_NAME` | Current project name | | `MORPHIR_TASK_NAME` | Name of the current task | | `MORPHIR_CONFIG_DIR` | Directory containing morphir.toml | ```toml [tasks.info] run = "echo Building $MORPHIR_PROJECT_NAME from $MORPHIR_PROJECT_ROOT" ``` ## Incremental Tasks Use `sources` and `outputs` for incremental execution: ```toml [tasks.codegen] description = "Generate TypeScript (incremental)" run = "morphir codegen typescript --output generated/" sources = ["src/**/*.morphir", ".morphir-dist/**/*.json"] outputs = ["generated/**/*.ts"] ``` The task only runs if sources are newer than outputs. ## CLI Reference ```bash # List available tasks morphir run --list # Run a task morphir run # Run with arguments (passed to task) morphir run test -- --verbose # Run multiple tasks morphir run lint test build # Dry run (show what would execute) morphir run --dry-run build # Force run (ignore incremental check) morphir run --force codegen ``` ## Examples ### CI Pipeline with Hooks Since `build`, `test`, and `check` are built-in tasks, you only need to configure custom behavior: ```toml # Use pre/post hooks to extend built-in tasks [tasks."pre:test"] description = "Ensure build is fresh before testing" run = "morphir build" [tasks."post:test"] description = "Generate coverage report" run = "./scripts/coverage-report.sh" # Custom CI task that chains built-in tasks [tasks.ci] description = "Full CI pipeline" depends = ["check", "test", "build", "pack"] run = "echo 'CI passed'" ``` ```bash # Built-in tasks work immediately morphir run build # Runs pre:build -> build -> post:build morphir run test # Runs pre:test -> test -> post:test morphir run ci # Runs the full pipeline ``` ### Development Workflow ```toml [tasks.dev] description = "Start development with watch mode" run = "morphir workspace watch" [tasks."post:build"] description = "Auto-format generated code" run = "prettier --write .morphir-dist/" [tasks."post:codegen"] description = "Format generated TypeScript" run = "prettier --write generated/" [tasks.release] description = "Create a release" depends = ["test", "build", "pack"] run = "morphir publish --backend github" env = { MORPHIR_RELEASE = "true" } ``` ### Monorepo Tasks ```toml # workspace/morphir.toml [tasks.bootstrap] description = "Initialize workspace after clone" run = [ "morphir deps resolve", "morphir workspace build" ] [tasks.release-all] description = "Release all packages" depends = ["test"] # Built-in test runs for all projects run = "./scripts/release-all.sh" ``` ### Custom Integration Tests ```toml # Add integration tests after the built-in test task [tasks."post:test"] run = "./scripts/integration-tests.sh" # Or define a separate task [tasks.integration] description = "Run integration tests" depends = ["build"] run = "./scripts/integration-tests.sh" ``` ## Migration from Toolchain Config If you previously used `[toolchain]` configuration, migrate to tasks: ```toml # Before (deprecated) [toolchain.morphir-elm] enabled = true [toolchain.morphir-elm.tasks.make] exec = "morphir-elm" args = ["make"] # After (tasks) [tasks.elm-make] description = "Build with morphir-elm" run = "morphir-elm make" ``` ## Design Notes 1. **Simplicity**: Tasks are just shell commands, no special DSL 2. **Composability**: Dependencies allow building complex workflows from simple tasks 3. **Convention**: Pre/post hooks follow predictable naming 4. **Compatibility**: File-based tasks work with any scripting language 5. **Incremental**: Source/output tracking avoids unnecessary work For extension points beyond tasks (custom commands, protocol extensions), see [WASM Components](./wasm-component.md) for the WASM Component Model approach. ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](./README.md)** - Extension system overview - **[WASM Components](./wasm-component.md)** - Component model integration - **[Morphir Daemon](../daemon/)** - Workspace and build management ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Workspace Lifecycle Source: https://finos.github.io/morphir-rust/contributors/design/daemon/lifecycle # Workspace Lifecycle This document defines the lifecycle operations for Morphir workspaces: creation, opening, closing, and configuration management. ## Overview Workspace lifecycle operations manage the overall state of a workspace session. A workspace must be opened before any project operations can be performed. ## State Machine ``` ┌─────────────────┐ │ (none) │ └────────┬────────┘ │ create / open ▼ ┌─────────────────┐ │ initializing │ └────────┬────────┘ │ ┌──────────────┴──────────────┐ │ success │ failure ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ open │ │ error │ └────────┬────────┘ └────────┬────────┘ │ close │ close └──────────────┬──────────────┘ ▼ ┌─────────────────┐ │ closed │ └─────────────────┘ ``` ## Types ### WorkspaceState ```gleam /// Workspace lifecycle state pub type WorkspaceState { /// Workspace is not active Closed /// Workspace is being initialized Initializing /// Workspace is ready for operations Open /// Workspace has unrecoverable errors Error } ``` ### WorkspaceInfo ```gleam /// Complete workspace information pub type WorkspaceInfo { WorkspaceInfo( /// Absolute path to workspace root root: String, /// Workspace name (from config or derived from root) name: String, /// Current lifecycle state state: WorkspaceState, /// Projects in this workspace projects: List(ProjectInfo), /// Workspace-level configuration config: Option(Document), ) } ``` ### WorkspaceError ```gleam /// Errors that can occur during workspace operations pub type WorkspaceError { /// Workspace directory not found NotFound(path: String) /// Workspace already exists at path AlreadyExists(path: String) /// No workspace is currently open NotOpen /// Invalid workspace configuration InvalidConfig(message: String) /// IO error during operation IoError(message: String) } ``` ## Operations ### Create Workspace Creates a new workspace at the specified path. #### Behavior 1. Verify path does not already contain a workspace 2. Create `morphir.toml` with initial configuration 3. Create `.morphir/` directory for cache and state 4. Transition to `Open` state 5. Return workspace info #### WIT Interface ```wit /// Create a new workspace create-workspace: func( /// Workspace root path (must not exist or be empty) root: string, /// Initial configuration (optional) config: option, ) -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/create", "params": { "root": "/path/to/workspace", "config": { "name": "my-workspace" } } } ``` **Response:** ```json { "result": { "root": "/path/to/workspace", "name": "my-workspace", "state": "open", "projects": [] } } ``` #### CLI ```bash morphir workspace init [path] morphir workspace init --name my-workspace ``` ### Open Workspace Opens an existing workspace for operations. #### Behavior 1. Locate `morphir.toml` (search upward if needed) 2. Parse workspace configuration 3. Discover projects in workspace 4. Load project metadata (not full IR) 5. Transition to `Open` state 6. Return workspace info #### WIT Interface ```wit /// Open an existing workspace open-workspace: func( /// Workspace root path root: string, ) -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/open", "params": { "root": "/path/to/workspace" } } ``` **Response:** ```json { "result": { "root": "/path/to/workspace", "name": "my-workspace", "state": "open", "projects": [ { "name": "my-org/core", "version": "1.0.0", "path": "packages/core", "state": "unloaded", "sourceDir": "src" } ] } } ``` #### CLI ```bash morphir workspace open [path] cd /path/to/workspace && morphir workspace open ``` ### Close Workspace Closes the current workspace, releasing resources. #### Behavior 1. Stop file watching if active 2. Unload all projects 3. Flush any pending state 4. Transition to `Closed` state #### WIT Interface ```wit /// Close the current workspace close-workspace: func() -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/close", "params": {} } ``` **Response:** ```json { "result": null } ``` ### Get Workspace Info Returns current workspace state and information. #### WIT Interface ```wit /// Get current workspace info get-workspace-info: func() -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/info", "params": {} } ``` ### Update Configuration Updates workspace-level configuration. #### Behavior 1. Validate new configuration 2. Merge with existing configuration 3. Write updated `morphir.toml` 4. Notify affected projects if needed #### WIT Interface ```wit /// Update workspace configuration update-workspace-config: func( config: document-value, ) -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/updateConfig", "params": { "config": { "defaultSourceDir": "lib", "outputDir": "dist" } } } ``` ## Configuration See [Configuration System](./configuration.md) for full configuration documentation. ### morphir.toml ```toml [morphir] version = "^4.0.0" [workspace] # Output directory for workspace-level artifacts output_dir = ".morphir" # Glob patterns for discovering member projects members = ["packages/*"] # Patterns to exclude from discovery exclude = ["packages/deprecated-*"] # Default member when no project specified default_member = "packages/core" # Shared dependencies (resolved once for all projects) [workspace.dependencies] "morphir/sdk" = "3.0.0" # Default settings inherited by all projects [ir] format_version = 4 [codegen] targets = ["typescript"] ``` ## Error Handling | Error | Cause | Recovery | | --------------- | -------------------------------------- | -------------------- | | `NotFound` | Path doesn't exist or has no workspace | Use `create` instead | | `AlreadyExists` | Workspace already at path | Use `open` instead | | `NotOpen` | Operation requires open workspace | Call `open` first | | `InvalidConfig` | Malformed configuration file | Fix configuration | | `IoError` | File system error | Check permissions | ## Best Practices 1. **Single Active Workspace**: Only one workspace should be open per daemon instance 2. **Workspace Discovery**: Search upward for `morphir.toml` to support nested directories 3. **Graceful Degradation**: If workspace config is missing, treat directory as single-project workspace 4. **State Persistence**: Save workspace state to `.morphir/state.json` for session recovery ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Projects](./projects.md)** - Project management within a workspace - **[Configuration](./configuration.md)** - Configuration system overview ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Installation Source: https://finos.github.io/morphir-rust/install # Installation There are several ways to install the Morphir CLI. ## Quick Install (Recommended) The easiest way to install morphir is using the installer script, which sets up automatic version management. ### Linux / macOS ```bash curl -fsSL https://raw.githubusercontent.com/finos/morphir-rust/main/scripts/install.sh | bash ``` ### Windows (PowerShell) ```powershell irm https://raw.githubusercontent.com/finos/morphir-rust/main/scripts/install.ps1 | iex ``` The installer: 1. Creates `~/.morphir/bin/` directory 2. Downloads the morphir launcher script 3. Adds the directory to your PATH After installation, restart your terminal or run: ```bash source ~/.bashrc # or ~/.zshrc ``` ## Alternative Methods ### Using mise [mise](https://mise.jdx.dev/) is a polyglot version manager. If you use mise: ```bash mise install github:finos/morphir-rust@v0.1.0 mise use github:finos/morphir-rust@v0.1.0 ``` ### Using cargo-binstall [cargo-binstall](https://github.com/cargo-bins/cargo-binstall) downloads pre-built binaries: ```bash cargo binstall --git https://github.com/finos/morphir-rust morphir ``` ### Manual Download Download the appropriate binary from the [GitHub Releases](https://github.com/finos/morphir-rust/releases) page. Available platforms: - `morphir-{version}-x86_64-apple-darwin.tgz` - macOS Intel - `morphir-{version}-aarch64-apple-darwin.tgz` - macOS Apple Silicon - `morphir-{version}-x86_64-unknown-linux-gnu.tgz` - Linux x86_64 - `morphir-{version}-aarch64-unknown-linux-gnu.tgz` - Linux ARM64 - `morphir-{version}-x86_64-pc-windows-msvc.zip` - Windows x86_64 ### Build from Source Requires [Rust](https://www.rust-lang.org/tools/install) (latest stable): ```bash git clone https://github.com/finos/morphir-rust.git cd morphir-rust cargo install --path crates/morphir ``` ## Version Management The morphir launcher automatically manages versions for you. ### Version Resolution Version is resolved in this order: 1. **Command line override**: `morphir +0.1.0 ` 2. **Environment variable**: `MORPHIR_VERSION=0.1.0` 3. **Project file**: `.morphir-version` in current or parent directory 4. **Config file**: `version = "..."` in `morphir.toml` 5. **Latest**: Fetches latest release from GitHub ### Pinning a Version For reproducible builds, pin the version in your project: ```bash # Option 1: .morphir-version file echo "0.1.0" > .morphir-version # Option 2: morphir.toml cat > morphir.toml << 'EOF' version = "0.1.0" EOF ``` ### Self Commands ```bash # Upgrade to latest version morphir self upgrade # List installed versions morphir self list # Show which version will be used morphir self which # Install a specific version morphir self install 0.1.0 # Remove old versions morphir self prune # Update the launcher script itself morphir self update # Show dev mode status morphir self dev ``` ## Dev Mode Dev mode allows you to run morphir from a local source checkout instead of a downloaded binary. This is useful for: - Developing and testing changes to morphir - Debugging issues with local modifications - CI/CD pipelines that build from source ### Enabling Dev Mode There are several ways to enable dev mode: ```bash # One-time: use --dev flag morphir --dev # Session: set environment variable export MORPHIR_DEV=1 # Project: create .morphir-version with "local-dev" echo "local-dev" > .morphir-version # Config: add to morphir.toml cat >> morphir.toml << 'EOF' [morphir] dev_mode = true EOF ``` ### Source Directory Detection When dev mode is enabled, the launcher automatically searches for the morphir-rust source directory in this order: 1. `MORPHIR_DEV_PATH` environment variable 2. CI environment variables (`GITHUB_WORKSPACE`, `CI_PROJECT_DIR`, etc.) 3. Current directory and parent directories 4. Common development locations (`~/code/morphir-rust`, `~/dev/morphir-rust`, etc.) ### Dev Mode Status Check your dev mode configuration: ```bash morphir self dev ``` This shows: - Whether dev mode is enabled and why - The detected source directory - Available debug/release binaries ## Environment Variables | Variable | Description | |----------|-------------| | `MORPHIR_VERSION` | Override version to use | | `MORPHIR_HOME` | Override home directory (default: `~/.morphir`) | | `MORPHIR_BACKEND` | Force specific backend: `mise`, `binstall`, `github`, `cargo` | | `MORPHIR_DEV` | Set to `1` or `true` to enable dev mode | | `MORPHIR_DEV_PATH` | Path to morphir-rust source directory for dev mode | ## Uninstalling ### Launcher Installation ```bash rm -rf ~/.morphir # Remove the PATH entry from your shell rc file ``` ### Cargo Installation ```bash cargo uninstall morphir ``` ### mise Installation ```bash mise uninstall github:finos/morphir-rust ``` ## Troubleshooting ### Command not found Ensure `~/.morphir/bin` is in your PATH: ```bash export PATH="$HOME/.morphir/bin:$PATH" ``` Add this line to your `~/.bashrc` or `~/.zshrc`. ### Permission denied Make the launcher executable: ```bash chmod +x ~/.morphir/bin/morphir ``` ### Version not downloading Check your internet connection and try: ```bash MORPHIR_BACKEND=github morphir self upgrade ``` ## Next Steps - [Getting Started](getting-started) - Basic usage guide - [CLI Reference](cli/) - Complete command documentation --- ## Morphir Extension System - Architecture Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/01-architecture # Morphir Extension System - Architecture **Status:** Draft **Version:** 0.1.0 ## System Overview ``` ┌─────────────────────────────────────────────────────────────┐ │ Morphir Core │ │ ┌─────────────┐ ┌──────────┐ ┌────────────┐ │ │ │ Compiler │ │ CLI │ │ Codegen │ │ │ └──────┬──────┘ └────┬─────┘ └─────┬──────┘ │ │ │ │ │ │ │ └──────────────┴───────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────┐ │ │ │ Extension Manager │ │ │ │ (Kameo Actor) │ │ │ └──────────┬───────────────┘ │ │ │ │ └────────────────────┼────────────────────────────────────────┘ │ ┌────────────┼────────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ JSON-RPC │ │ gRPC │ │ Stdio │ │ Host Actor │ │ Host Actor │ │ Host Actor │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ ▼ ▼ ▼ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ Extension │ │ Extension │ │ Extension │ │ (HTTP) │ │ (gRPC) │ │ (Process) │ └────────────┘ └────────────┘ └────────────┘ ┌──────────────┐ ┌──────────────┐ │ Extism WASM │ │ Component │ │ Host Actor │ │ Host Actor │ └──────┬───────┘ └──────┬───────┘ │ │ ▼ ▼ ┌────────────┐ ┌────────────┐ │ Extension │ │ Extension │ │ (WASM) │ │ (Component)│ └────────────┘ └────────────┘ ``` ## Component Responsibilities ### Extension Manager **Type**: Kameo Actor **Lifecycle**: Singleton, lives for application duration **Responsibilities:** - Register and manage extension hosts - Route extension calls to appropriate host - Maintain extension registry (name → ID → host) - Handle extension discovery and loading - Provide unified query interface (list, inspect, etc.) - Emit metrics and events **State:** ```rust struct ExtensionManager { hosts: HashMap>, extensions: HashMap, registry: HashMap, event_bus: ActorRef, } ``` ### Extension Host **Type**: Kameo Actor (one per protocol type) **Lifecycle**: Created on first use, persists until shutdown **Responsibilities:** - Manage lifecycle of extensions using specific protocol - Translate Morphir operations to protocol operations - Handle protocol-specific failures and retries - Monitor extension health - Report metrics **Common Interface:** ```rust #[async_trait] pub trait ExtensionHost: Send + 'static { fn protocol(&self) -> &str; async fn initialize(&mut self) -> Result<(), ExtensionError>; async fn load_extension(&mut self, config: ExtensionConfig) -> Result; async fn call(&mut self, id: ExtensionId, method: &str, params: Value) -> Result; async fn unload_extension(&mut self, id: ExtensionId) -> Result<(), ExtensionError>; async fn capabilities(&self, id: ExtensionId) -> Result, ExtensionError>; } ``` ### Extension **Type**: External process, WASM module, or network service **Lifecycle**: Managed by host **Responsibilities:** - Implement extension protocol - Respond to initialization - Report capabilities - Handle method calls - Clean up resources on shutdown ## Data Flow ### Extension Loading ``` User/Config → Manager.load_extension(config) ↓ Manager determines protocol from config.source ↓ Manager routes to appropriate Host ↓ Host spawns/connects to extension ↓ Host sends initialize(config) ↓ Extension responds with status ↓ Host queries capabilities ↓ Extension responds with capability list ↓ Host returns ExtensionId to Manager ↓ Manager stores metadata and returns ID ``` ### Extension Invocation ``` Core calls Manager.call_extension(name, method, params) ↓ Manager resolves name → ExtensionId ↓ Manager looks up metadata to find protocol ↓ Manager routes to appropriate Host ↓ Host finds extension by ID ↓ Host translates call to protocol format ↓ Extension executes method ↓ Extension returns result ↓ Host translates response back to Value ↓ Manager returns result to Core ``` ## Concurrency Model ### Actor-Based Isolation Each component is an independent Kameo actor: - **No shared mutable state** between components - **Message passing** for all communication - **Sequential processing** within each actor (no locks needed) - **Concurrent execution** across actors ### Mailbox Configuration - **Manager**: Bounded mailbox (capacity: 256) for backpressure - **Hosts**: Bounded mailbox (capacity: 64) per host - **Extension tasks**: Unbounded for flexibility ### Parallelism - Multiple extensions can execute **concurrently** - Multiple hosts can process **independently** - Extension calls within same host are **sequential** (per extension) - Cross-extension calls are **concurrent** ## Error Handling ### Failure Domains 1. **Extension Failure**: Extension process crashes or returns error - Host detects failure - Host marks extension as unhealthy - Host attempts restart based on policy - Returns error to caller 2. **Host Failure**: Host actor panics - Supervisor restarts host actor - Extensions in that host need reload - Manager maintains registry state 3. **Manager Failure**: Manager actor panics - System-level supervisor restarts manager - Hosts remain operational - Registry reconstructed from hosts ### Restart Strategies ```rust pub enum RestartStrategy { /// Never restart on failure Never, /// Restart immediately, up to N times Immediate { max_retries: u32 }, /// Restart with exponential backoff Exponential { initial_delay: Duration, max_delay: Duration, max_retries: u32, }, } ``` ## Configuration ### Extension Configuration Schema ```toml [[extensions]] name = "typescript-generator" enabled = true protocol = "jsonrpc" source = { type = "http", url = "http://localhost:3000" } permissions = { network = false, filesystem = ["./output"] } restart = { strategy = "exponential", max_retries = 3 } [[extensions]] name = "ir-validator" enabled = true protocol = "stdio" source = { type = "process", command = "python3", args = ["./validator.py"] } permissions = { network = false, filesystem = [] } [[extensions]] name = "wasm-optimizer" enabled = true protocol = "extism" source = { type = "wasm", path = "./optimizer.wasm" } permissions = { max_memory = "10MB", max_execution_time = "5s" } ``` ## Performance Considerations ### Latency Characteristics | Protocol | Typical Latency | Best For | | ---------------- | --------------- | -------------------------------- | | WASM (Extism) | < 1ms | CPU-bound transformations | | WASM (Component) | < 1ms | CPU-bound with complex types | | gRPC | 1-5ms | High-throughput, many calls | | Stdio | 5-20ms | Simple scripts, infrequent calls | | JSON-RPC | 10-50ms | Networked services, web APIs | ### Throughput - **WASM**: 10,000+ calls/sec per extension - **gRPC**: 1,000+ calls/sec per extension - **Stdio**: 50-100 calls/sec per extension - **JSON-RPC**: 20-100 calls/sec per extension ### Memory - **Manager**: ~1MB base - **Each Host**: 1-5MB base - **Stdio Extension**: 10-100MB (process overhead) - **WASM Extension**: 1-10MB (sandbox + module) - **JSON-RPC/gRPC**: Network client overhead (~500KB) ## Observability ### Metrics All components emit metrics via OpenTelemetry: - `morphir.extension.load.duration_ms` - `morphir.extension.call.duration_ms` - `morphir.extension.call.count` (by extension, method, status) - `morphir.extension.failure.count` (by extension, reason) - `morphir.host.active_extensions` (by protocol) - `morphir.host.mailbox.depth` (by host) ### Logging Structured logging using `tracing`: ```rust #[instrument(skip(self))] async fn load_extension(&mut self, config: ExtensionConfig) -> Result { info!("Loading extension", extension = %config.name); // ... } ``` ### Events Published via PubSub actor: - `extension.loaded` - Extension successfully loaded - `extension.failed` - Extension failed to load - `extension.unloaded` - Extension unloaded - `extension.call.started` - Method call started - `extension.call.completed` - Method call completed - `extension.call.failed` - Method call failed ## Security Model See [09-security-and-isolation.md](./09-security-and-isolation.md) for details. **Summary:** - **Process Isolation**: Each extension in separate process (except WASM) - **Capability-Based**: Extensions declare required capabilities - **Sandboxing**: WASM extensions run in strict sandbox - **Resource Limits**: CPU, memory, and execution time limits - **Principle of Least Privilege**: Extensions only get what they need ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Project Management Source: https://finos.github.io/morphir-rust/contributors/design/daemon/projects # Project Management This document defines operations for managing projects within a Morphir workspace. ## Overview A **project** is a single Morphir package with its own `morphir.toml` configuration. Projects within a workspace can depend on each other and share resolved dependencies. ## Project States ``` ┌─────────────────┐ │ unloaded │◄──────────────┐ └────────┬────────┘ │ │ load │ unload ▼ │ ┌─────────────────┐ │ │ loading │ │ └────────┬────────┘ │ │ │ ┌──────────────┴──────────────┐ │ │ success │ failure │ ▼ ▼ │ ┌─────────────────┐ ┌─────────────────┐ │ ready │ │ error │ └────────┬────────┘ └─────────────────┘ │ source changed ▼ ┌─────────────────┐ │ stale │ └────────┬────────┘ │ reload └─────────────► loading ``` ## Types ### ProjectState ```gleam /// Project lifecycle state within a workspace pub type ProjectState { /// Project metadata loaded, IR not compiled Unloaded /// Project is being compiled Loading /// Project IR is loaded and valid Ready /// Source files changed, needs recompilation Stale /// Project has compilation errors Error } ``` ### ProjectInfo ```gleam /// Project information and status pub type ProjectInfo { ProjectInfo( /// Package name (e.g., "my-org/domain") name: PackagePath, /// Semantic version version: SemVer, /// Path relative to workspace root path: String, /// Current state state: ProjectState, /// Source directory within project source_dir: String, /// Project dependencies dependencies: List(DependencyInfo), ) } ``` ## Operations ### Add Project Adds an existing project directory to the workspace. #### Behavior 1. Verify project path exists and contains `morphir.toml` 2. Parse project configuration 3. Register project in workspace 4. Update `morphir.toml` 5. Return project info (state: `Unloaded`) #### WIT Interface ```wit /// Add a project to the workspace add-project: func( /// Project name (package path) name: package-path, /// Project path (relative to workspace root) path: string, /// Initial version version: semver, /// Source directory source-dir: string, ) -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/addProject", "params": { "name": "my-org/new-service", "path": "packages/new-service", "version": "0.1.0", "sourceDir": "src" } } ``` **Response:** ```json { "result": { "name": "my-org/new-service", "version": "0.1.0", "path": "packages/new-service", "state": "unloaded", "sourceDir": "src", "dependencies": [] } } ``` #### CLI ```bash morphir workspace add packages/new-service morphir workspace add --name my-org/new-service --path packages/new-service ``` ### Remove Project Removes a project from the workspace (does not delete files). #### Behavior 1. Verify project exists in workspace 2. Check for dependents (warn if other projects depend on it) 3. Unload project if loaded 4. Remove from workspace registry 5. Update `morphir.toml` #### WIT Interface ```wit /// Remove a project from the workspace remove-project: func( name: package-path, ) -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/removeProject", "params": { "name": "my-org/old-service" } } ``` #### CLI ```bash morphir workspace remove my-org/old-service ``` ### Get Project Info Returns detailed information about a specific project. #### WIT Interface ```wit /// Get project info get-project-info: func( name: package-path, ) -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/projectInfo", "params": { "name": "my-org/domain" } } ``` ### List Projects Lists all projects in the workspace. #### WIT Interface ```wit /// List all projects list-projects: func() -> result, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/listProjects", "params": {} } ``` **Response:** ```json { "result": [ { "name": "my-org/core", "version": "1.0.0", "path": "packages/core", "state": "ready", "sourceDir": "src", "dependencies": [] }, { "name": "my-org/domain", "version": "2.0.0", "path": "packages/domain", "state": "stale", "sourceDir": "src", "dependencies": [ { "name": "my-org/core", "version": "1.0.0", "resolved": true } ] } ] } ``` ### Load Project Compiles a project and loads its IR into memory. #### Behavior 1. Transition to `Loading` state 2. Resolve project dependencies 3. Compile source files to IR 4. Validate IR 5. Transition to `Ready` or `Error` state 6. Return distribution #### WIT Interface ```wit /// Load a project (parse and compile) load-project: func( name: package-path, ) -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/loadProject", "params": { "name": "my-org/domain" } } ``` **Response:** ```json { "result": { "distribution": { "Library": { "package": { "name": "my-org/domain", "version": "2.0.0" }, "definition": { "..." } } }, "diagnostics": [] } } ``` ### Unload Project Releases a project's IR from memory. #### Behavior 1. Free IR memory 2. Transition to `Unloaded` state 3. Retain project metadata #### WIT Interface ```wit /// Unload a project (free resources) unload-project: func( name: package-path, ) -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/unloadProject", "params": { "name": "my-org/domain" } } ``` ### Reload Project Recompiles a project (typically after source changes). #### Behavior 1. Same as `Load`, but preserves previous IR until new compilation succeeds 2. Supports incremental compilation if available #### WIT Interface ```wit /// Reload a project (recompile) reload-project: func( name: package-path, ) -> result; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/reloadProject", "params": { "name": "my-org/domain" } } ``` ## Project Discovery When a workspace is opened, projects are discovered by: 1. **Explicit listing** in `morphir.toml`: ```toml [[projects]] name = "my-org/core" path = "packages/core" ``` 2. **Glob patterns** for automatic discovery: ```toml [workspace] project-patterns = ["packages/*", "libs/*"] ``` 3. **Walking** the directory tree for `morphir.toml` files ## Dependency Order Projects are loaded in dependency order: ``` my-org/core (no deps) → load first my-org/domain (→ core) → load second my-org/api (→ domain) → load third ``` Circular dependencies are detected and reported as errors. ## Error Handling | Error | Cause | Recovery | | ---------------------- | ------------------------ | ------------------ | | `ProjectNotFound` | Project not in workspace | Check project name | | `ProjectAlreadyExists` | Duplicate project name | Use different name | | `InvalidConfig` | Bad `morphir.toml` | Fix configuration | | `CompilationError` | Source code errors | Fix source code | | `DependencyError` | Missing dependency | Add dependency | ## Best Practices 1. **Lazy Loading**: Only load projects when needed for operations 2. **Dependency Caching**: Cache resolved dependencies to speed up loads 3. **Partial Builds**: Allow building subset of projects 4. **State Recovery**: Persist project states for daemon restart ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Lifecycle](./lifecycle.md)** - Workspace lifecycle operations - **[Dependencies](./dependencies.md)** - Dependency resolution and caching - **[Build](./build.md)** - Build orchestration ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents --- ## WASM Component Model Source: https://finos.github.io/morphir-rust/contributors/design/extensions/wasm-component # WASM Component Model This document defines WIT (WebAssembly Interface Types) interfaces for implementing Morphir backends and extensions as portable, sandboxed WASM components. ## Design Principles - **Strongly Typed IR**: Full IR types modeled in WIT, not JSON strings - **Attributes as Document**: Since WIT lacks generics, attributes use our `document-value` AST type - **Full Pipeline**: Support both frontend (source → IR) and backend (IR → target) flows - **Multi-Granularity**: Support operations at distribution, module, and definition levels - **Capability-Based**: Components declare which interfaces they implement - **Sandboxed**: Components have no implicit access to filesystem or network ## Benefits | Benefit | Description | | ----------------- | ------------------------------------------------------------ | | Language-agnostic | Backends can be written in Rust, Go, C, AssemblyScript, etc. | | Sandboxed | Secure execution with explicit capability grants | | Portable | Run anywhere WASM runs (CLI, browser, edge) | | Hot-reloadable | Swap components without restarting the daemon | | Type-safe | Full IR types at component boundary | ## WIT Package Structure ``` wit/ ├── morphir-ir/ │ ├── document.wit # Document AST (for attributes) │ ├── naming.wit # Name, Path, FQName │ ├── types.wit # Type expressions │ ├── values.wit # Value expressions, literals, patterns │ ├── modules.wit # Module specs and defs │ ├── packages.wit # Package specs and defs │ └── distributions.wit # Distribution types ├── morphir-extension/ │ ├── info.wit # Required info interface (all extensions) │ └── capabilities.wit # Capability query interface ├── morphir-frontend/ │ ├── compiler.wit # Source → IR compilation (basic) │ ├── streaming.wit # Streaming compilation capability │ ├── incremental.wit # Incremental compilation capability │ └── fragment.wit # Fragment/REPL compilation capability ├── morphir-backend/ │ ├── generator.wit # IR → target code generation (basic) │ ├── streaming.wit # Streaming generation capability │ ├── incremental.wit # Incremental generation capability │ ├── validator.wit # Validation interface │ └── transform.wit # IR → IR transformation interface ├── morphir-vfs/ │ ├── reader.wit # VFS read access │ ├── writer.wit # VFS write access │ └── workspace.wit # Workspace management └── morphir-component/ └── worlds.wit # World definitions ``` ## Document Type (`document.wit`) ```wit package morphir:ir@0.4.0; /// Document AST for schema-less data and attributes interface document { /// Recursive document value (JSON-like AST) variant document-value { /// null doc-null, /// Boolean doc-bool(bool), /// Integer doc-int(s64), /// Float doc-float(float64), /// String doc-string(string), /// Array doc-array(list), /// Object (list of key-value pairs) doc-object(list>), } /// Empty document (convenience) empty: func() -> document-value; /// Create object from pairs object: func(pairs: list>) -> document-value; /// Get field from object get: func(doc: document-value, key: string) -> option; /// Merge two documents (second wins on conflict) merge: func(base: document-value, overlay: document-value) -> document-value; } ``` ## Naming Types (`naming.wit`) ```wit package morphir:ir@0.4.0; use document.{document-value}; /// Core naming types interface naming { /// Type-level attributes (optional metadata, no type info needed) /// Types don't carry type annotations on themselves variant type-attributes { /// No attributes none, /// Custom metadata only meta(document-value), } /// A single name segment in kebab-case /// Examples: "user", "order-id", "get-user-by-email" type name = string; /// A path is a list of names representing hierarchy /// Serialized as slash-separated: "main/domain/users" type path = string; /// Module path within a package type module-path = string; /// Package path (organization/project) type package-path = string; /// Qualified name: module path + local name /// Format: "module/path#local-name" record qname { module-path: module-path, local-name: name, } /// Fully qualified name: package + module + local name /// Format: "package:module#name" record fqname { package-path: package-path, module-path: module-path, local-name: name, } /// Type variable name type type-variable = string; /// Parse FQName from canonical string /// "my-org/project:domain/users#user" -> FQName fqname-from-string: func(s: string) -> option; /// Render FQName to canonical string fqname-to-string: func(fqn: fqname) -> string; } ``` ## Type Expressions (`types.wit`) ```wit package morphir:ir@0.4.0; use document.{document-value}; use naming.{type-attributes, name, fqname, type-variable}; /// Type system definitions interface types { /// Record field record field { field-name: name, field-type: ir-type, } /// Constructor for custom types record constructor { ctor-name: name, args: list>, } /// Type expression (uses type-attributes: none or metadata) variant ir-type { /// Type variable: `a`, `comparable` variable(tuple), /// Reference to named type: `String`, `List a` reference(tuple>), /// Tuple: `(Int, String)` %tuple(tuple>), /// Record: `{ name: String, age: Int }` record(tuple>), /// Extensible record: `{ a | name: String }` extensible-record(tuple>), /// Function: `Int -> String` %function(tuple), /// Unit type: `()` unit(type-attributes), } /// Value-level attributes (expressions carry their type) variant value-attributes { /// Just the inferred/checked type typed(ir-type), /// Type plus custom metadata typed-with-meta(tuple), } /// Access control enum access { public, private, } /// Access-controlled wrapper record access-controlled-constructors { access: access, constructors: list, } /// Type specification (public interface) variant type-specification { /// Type alias visible to consumers type-alias-specification(tuple, ir-type>), /// Opaque type (no structure visible) opaque-type-specification(list), /// Custom type with public constructors custom-type-specification(tuple, list>), /// Derived type with conversion functions derived-type-specification(tuple, derived-type-details>), } /// Details for derived types record derived-type-details { base-type: ir-type, from-base-type: fqname, to-base-type: fqname, } /// Hole reason for incomplete types variant hole-reason { unresolved-reference(fqname), deleted-during-refactor(string), type-mismatch(tuple), } /// Incompleteness marker variant incompleteness { hole(hole-reason), draft(option), } /// Type definition (implementation) variant type-definition { /// Custom type (sum type) custom-type-definition(tuple, access-controlled-constructors>), /// Type alias type-alias-definition(tuple, ir-type>), /// Incomplete type (v4) incomplete-type-definition(tuple, incompleteness, option>), } } ``` ## Literals (`values.wit` - Part 1) ```wit package morphir:ir@0.4.0; /// Literal values interface literals { /// Document value for schema-less data variant document-value { doc-null, doc-bool(bool), doc-int(s64), doc-float(float64), doc-string(string), doc-array(list), doc-object(list>), } /// Literal constant values variant literal { /// Boolean: true, false bool-literal(bool), /// Single character char-literal(string), /// Text string string-literal(string), /// Integer (includes negatives) integer-literal(s64), /// Floating-point float-literal(float64), /// Arbitrary-precision decimal (stored as string) decimal-literal(string), /// Document literal (schema-less JSON-like) document-literal(document-value), } } ``` ## Patterns (`values.wit` - Part 2) ```wit package morphir:ir@0.4.0; use naming.{attributes, name, fqname}; use literals.{literal}; /// Pattern matching interface patterns { use naming.{type-attributes}; /// Pattern for destructuring and matching /// Patterns use type-attributes (no type info, just optional metadata) variant pattern { /// Wildcard: `_` wildcard-pattern(type-attributes), /// As pattern: `x` or `(a, b) as pair` as-pattern(tuple), /// Tuple pattern: `(a, b, c)` tuple-pattern(tuple>), /// Constructor pattern: `Just x` constructor-pattern(tuple>), /// Empty list: `[]` empty-list-pattern(type-attributes), /// Head :: tail: `x :: xs` head-tail-pattern(tuple), /// Literal match: `42`, `"hello"` literal-pattern(tuple), /// Unit: `()` unit-pattern(type-attributes), } } ``` ## Value Expressions (`values.wit` - Part 3) ```wit package morphir:ir@0.4.0; use naming.{name, fqname}; use types.{ir-type, value-attributes, hole-reason, incompleteness}; use literals.{literal}; use patterns.{pattern}; /// Value expressions interface values { /// Native operation hint variant native-hint { arithmetic, comparison, string-op, collection-op, platform-specific(string), } /// Native operation info record native-info { hint: native-hint, description: option, } /// Value expression /// Values use value-attributes (carry their type, optionally with metadata) variant value { // === Literals & Data Construction === /// Literal constant %literal(tuple), /// Constructor reference: `Just` %constructor(tuple), /// Tuple: `(1, "hello")` %tuple(tuple>), /// List: `[1, 2, 3]` %list(tuple>), /// Record: `{ name = "Alice" }` %record(tuple>>), /// Unit: `()` %unit(value-attributes), // === References === /// Variable: `x` variable(tuple), /// Reference to defined value: `List.map` reference(tuple), // === Field Access === /// Field access: `record.field` field(tuple), /// Field function: `.field` field-function(tuple), // === Function Application === /// Apply: `f x` apply(tuple), /// Lambda: `\x -> x + 1` lambda(tuple), // === Let Bindings === /// Let: `let x = 1 in x + 1` let-definition(tuple), /// Recursive let: `let f = ... g ...; g = ... f ... in ...` let-recursion(tuple>, value>), /// Destructure: `let (a, b) = pair in a + b` destructure(tuple), // === Control Flow === /// If-then-else if-then-else(tuple), /// Pattern match: `case x of ...` pattern-match(tuple>>), // === Record Update === /// Update: `{ record | field = new }` update-record(tuple>>), // === Special (v4) === /// Incomplete/broken reference hole(tuple>), /// Native operation native(tuple), /// External FFI external(tuple), } /// Value definition body variant value-definition-body { /// Expression body expression-body(tuple>, ir-type, value>), /// Native body native-body(tuple>, ir-type, native-info>), /// External body external-body(tuple>, ir-type, string, string>), /// Incomplete body (v4) incomplete-body(tuple>, option, incompleteness, option>), } /// Value specification (signature only) record value-specification { inputs: list>, output: ir-type, } /// Access-controlled value definition record value-definition { access: types.access, body: value-definition-body, } } ``` ## Modules (`modules.wit`) ```wit package morphir:ir@0.4.0; use naming.{name, module-path}; use types.{type-specification, type-definition, access}; use values.{value-specification, value-definition}; /// Module definitions interface modules { /// Documentation (opaque string or list of strings) variant documentation { single-line(string), multi-line(list), } /// Documented wrapper record documented-type-spec { doc: option, value: type-specification, } /// Module specification (public interface) record module-specification { types: list>, values: list>, } /// Documented type definition with access record access-controlled-documented-type-def { access: access, doc: option, definition: type-definition, } /// Documented value definition with access record access-controlled-documented-value-def { access: access, doc: option, definition: value-definition, } /// Module definition (implementation) record module-definition { types: list>, values: list>, } /// Access-controlled module definition record access-controlled-module-definition { access: access, definition: module-definition, } } ``` ## Packages (`packages.wit`) ```wit package morphir:ir@0.4.0; use naming.{module-path}; use modules.{module-specification, access-controlled-module-definition}; /// Package definitions interface packages { /// Package specification (public interface) record package-specification { modules: list>, } /// Package definition (implementation) record package-definition { modules: list>, } } ``` ## Distributions (`distributions.wit`) ```wit package morphir:ir@0.4.0; use naming.{name, fqname, package-path}; use modules.{documentation}; use packages.{package-specification, package-definition}; /// Distribution types interface distributions { /// Semantic version record semver { major: u32, minor: u32, patch: u32, pre-release: option, build-metadata: option, } /// Package info record package-info { name: package-path, version: semver, } /// Entry point kind enum entry-point-kind { main, command, handler, job, policy, } /// Entry point record entry-point { target: fqname, kind: entry-point-kind, doc: option, } /// Library distribution record library-distribution { package: package-info, definition: package-definition, dependencies: list>, } /// Specs distribution record specs-distribution { package: package-info, specification: package-specification, dependencies: list>, } /// Application distribution record application-distribution { package: package-info, definition: package-definition, dependencies: list>, entry-points: list>, } /// Distribution variant variant distribution { library(library-distribution), specs(specs-distribution), application(application-distribution), } } ``` ## Extension Info Interface (`info.wit`) Every extension must implement this interface - it's the only required interface: ```wit package morphir:extension@0.4.0; /// Required interface - all extensions must implement this interface info { /// Extension type classification enum extension-type { /// Frontend/compiler (source → IR) frontend, /// Backend/code generator (IR → target) codegen, /// Validator/analyzer validator, /// IR transformer (IR → IR) transformer, /// General purpose general, } /// Extension metadata record extension-info { /// Unique identifier (e.g., "spark-codegen") id: string, /// Human-readable name name: string, /// Version (semver) version: string, /// Description description: string, /// Author/maintainer author: option, /// Homepage/repository URL homepage: option, /// License identifier (SPDX) license: option, /// Extension types (can fulfill multiple roles) types: list, } /// Return extension metadata get-info: func() -> extension-info; /// Health check - return true if extension is ready ping: func() -> bool; } /// Capability discovery interface interface capabilities { use info.{extension-type}; /// Capability identifier (e.g., "codegen/generate-streaming") type capability-id = string; /// Capability info record capability-info { /// Capability identifier id: capability-id, /// Human-readable description description: string, /// Whether this capability is available available: bool, } /// Options schema for configurable extensions record option-schema { /// Option name name: string, /// Option type option-type: option-type, /// Default value (as JSON string) default-value: option, /// Description description: string, /// Whether this option is required required: bool, } /// Option type enum option-type { %string, integer, float, boolean, array, object, } /// Query all capabilities this extension provides list-capabilities: func() -> list; /// Check if a specific capability is available has-capability: func(id: capability-id) -> bool; /// Get targets this extension supports (for codegen) get-targets: func() -> list; /// Get languages this extension supports (for frontend) get-languages: func() -> list; /// Get configurable options schema get-options-schema: func() -> list; } ``` ## Frontend Compiler Interface (`compiler.wit`) ```wit package morphir:frontend@0.4.0; use morphir:ir@0.4.0.{ naming.{name, module-path, package-path, fqname}, types.{type-definition}, values.{value-definition}, modules.{module-definition}, packages.{package-definition}, distributions.{distribution, semver}, }; /// Frontend compiler interface (source → IR) interface compiler { /// Source language enum source-language { elm, morphir-dsl, custom, } /// Compilation granularity capability flags compiler-capabilities { /// Can compile entire workspace workspace, /// Can compile single project project, /// Can compile single module module, /// Can compile individual files file, /// Can compile code fragments fragment, } /// Compiler metadata record compiler-info { name: string, description: string, version: string, source-language: source-language, custom-language: option, capabilities: compiler-capabilities, } /// Source file record source-file { /// File path (relative to project root) path: string, /// File content content: string, } /// Project configuration record project-config { /// Project name name: package-path, /// Project version version: semver, /// Source directory source-dir: string, /// Dependencies dependencies: list>, /// Custom configuration as document custom: option, } /// Workspace configuration record workspace-config { /// Workspace root path root: string, /// Projects in workspace projects: list, } /// Fragment context (for incremental/editor compilation) record fragment-context { /// Module this fragment belongs to module-path: module-path, /// Imports available in scope imports: list, /// Local bindings in scope locals: list>, } /// Diagnostic severity enum severity { error, warning, info, hint, } /// Source location record source-location { file: string, start-line: u32, start-col: u32, end-line: u32, end-col: u32, } /// Compiler diagnostic record diagnostic { severity: severity, code: string, message: string, location: option, hints: list, } /// Compilation result for workspace variant workspace-result { ok(list), partial(tuple, list>), failed(list), } /// Compilation result for project variant project-result { ok(distribution), partial(tuple>), failed(list), } /// Compilation result for files variant files-result { ok(package-definition), partial(tuple>), failed(list), } /// Compilation result for module variant module-result { ok(module-definition), partial(tuple>), failed(list), } /// Compilation result for fragment variant fragment-result { /// Compiled to type definition type-def(type-definition), /// Compiled to value definition value-def(value-definition), /// Compiled to expression (for REPL) expression(morphir:ir@0.4.0.values.value), /// Failed failed(list), } /// Get compiler metadata info: func() -> compiler-info; /// Compile entire workspace to IR compile-workspace: func( config: workspace-config, files: list, ) -> workspace-result; /// Compile single project to IR compile-project: func( config: project-config, files: list, ) -> project-result; /// Compile list of files to IR (incremental) compile-files: func( config: project-config, files: list, /// Existing IR to merge with (for incremental) existing: option, ) -> files-result; /// Compile a single module to IR compile-module: func( config: project-config, /// Module path within the project module-path: module-path, /// Source files for this module files: list, /// Existing module to merge with (for incremental) existing: option, ) -> module-result; /// Compile a code fragment (for editor/REPL) compile-fragment: func( source: string, context: fragment-context, ) -> fragment-result; /// Parse without full compilation (for syntax checking) parse-file: func( file: source-file, ) -> result<_, list>; /// Get completions at position (for editor integration) completions: func( file: source-file, line: u32, column: u32, context: fragment-context, ) -> list; /// Completion item record completion-item { label: string, kind: completion-kind, detail: option, insert-text: option, } /// Completion kind enum completion-kind { %function, variable, %type, %constructor, module, keyword, } } ``` ## Generator Interface (`generator.wit`) ```wit package morphir:backend@0.4.0; use morphir:ir@0.4.0.{ naming.{fqname, module-path}, types.{type-definition}, values.{value-definition}, modules.{module-definition}, packages.{package-specification}, distributions.{distribution}, }; /// Code generation interface interface generator { /// Target language enum target-language { typescript, scala, java, go, python, rust, elm, custom, } /// Granularity level enum granularity { distribution, module, definition, } /// Generator metadata record generator-info { name: string, description: string, version: string, target: target-language, custom-target: option, supported-granularities: list, } /// Generation options record generation-options { output-dir: string, indent: option, source-maps: bool, custom: option, } /// Generated artifact record artifact { path: string, content: string, source-map: option, } /// Diagnostic severity enum severity { error, warning, info, hint, } /// Source location record source-location { uri: string, start-line: u32, start-col: u32, end-line: u32, end-col: u32, } /// Diagnostic message record diagnostic { severity: severity, code: string, message: string, location: option, } /// Generation result variant generation-result { /// Success ok(list), /// Partial success with warnings degraded(tuple, list>), /// Failure failed(list), } /// Get generator metadata info: func() -> generator-info; /// Generate code for entire distribution generate-distribution: func( dist: distribution, options: generation-options, ) -> generation-result; /// Generate code for a single module generate-module: func( path: module-path, module: module-definition, deps: package-specification, options: generation-options, ) -> generation-result; /// Generate code for a single type generate-type: func( fqn: fqname, def: type-definition, deps: package-specification, options: generation-options, ) -> generation-result; /// Generate code for a single value generate-value: func( fqn: fqname, def: value-definition, deps: package-specification, options: generation-options, ) -> generation-result; } ``` ## Validator Interface (`validator.wit`) ```wit package morphir:backend@0.4.0; use morphir:ir@0.4.0.{ naming.{fqname, module-path}, types.{type-definition}, values.{value-definition}, modules.{module-definition}, packages.{package-specification}, distributions.{distribution}, }; use generator.{severity, diagnostic, granularity}; /// Validation interface interface validator { /// Validator metadata record validator-info { name: string, description: string, version: string, categories: list, supported-granularities: list, } /// Validation options record validation-options { min-severity: option, enabled-rules: list, disabled-rules: list, custom: option, } /// Validation result record validation-result { diagnostics: list, passed: bool, error-count: u32, warning-count: u32, } /// Get validator metadata info: func() -> validator-info; /// Validate entire distribution validate-distribution: func( dist: distribution, options: validation-options, ) -> validation-result; /// Validate a single module validate-module: func( path: module-path, module: module-definition, deps: package-specification, options: validation-options, ) -> validation-result; /// Validate a single type validate-type: func( fqn: fqname, def: type-definition, deps: package-specification, options: validation-options, ) -> validation-result; /// Validate a single value validate-value: func( fqn: fqname, def: value-definition, deps: package-specification, options: validation-options, ) -> validation-result; } ``` ## Frontend Streaming Interface (`frontend/streaming.wit`) Optional capability for streaming compilation results: ```wit package morphir:frontend@0.4.0; use compiler.{source-file, project-config, diagnostic}; use morphir:ir@0.4.0.modules.{module-definition}; use morphir:ir@0.4.0.naming.{module-path}; /// Streaming compilation interface (optional capability) interface streaming { /// Streaming module result record module-compile-result { /// Module path path: module-path, /// Compiled module (if successful) module: option, /// Diagnostics for this module diagnostics: list, } /// Compile project with streaming results /// Returns a stream handle for polling results compile-streaming: func( config: project-config, files: list, ) -> stream-handle; /// Stream handle for polling results type stream-handle = u64; /// Poll for next result (non-blocking) /// Returns none when stream is complete poll-result: func(handle: stream-handle) -> option; /// Check if stream is complete is-complete: func(handle: stream-handle) -> bool; /// Cancel streaming compilation cancel: func(handle: stream-handle); } ``` ## Frontend Incremental Interface (`frontend/incremental.wit`) Optional capability for incremental compilation: ```wit package morphir:frontend@0.4.0; use compiler.{source-file, project-config, diagnostic}; use morphir:ir@0.4.0.packages.{package-definition}; use morphir:ir@0.4.0.naming.{module-path}; /// Incremental compilation interface (optional capability) interface incremental { /// File change event variant file-change { /// File was created created(source-file), /// File was modified modified(source-file), /// File was deleted deleted(string), /// File was renamed renamed(tuple), } /// Incremental compilation result record incremental-result { /// Updated package definition package: package-definition, /// Modules that were recompiled recompiled-modules: list, /// Modules that were invalidated (dependents) invalidated-modules: list, /// Diagnostics from recompilation diagnostics: list, } /// Apply incremental changes to existing IR compile-incremental: func( config: project-config, /// Existing compiled IR existing: package-definition, /// File changes since last compilation changes: list, ) -> incremental-result; /// Get dependency graph for invalidation analysis get-module-dependencies: func( existing: package-definition, ) -> list>>; } ``` ## Backend Streaming Interface (`backend/streaming.wit`) Optional capability for streaming code generation: ```wit package morphir:backend@0.4.0; use generator.{generation-options, artifact, diagnostic}; use morphir:ir@0.4.0.distributions.{distribution}; use morphir:ir@0.4.0.naming.{module-path}; /// Streaming generation interface (optional capability) interface streaming { /// Module generation result record module-generation-result { /// Module path that was generated module-path: module-path, /// Generated artifacts for this module artifacts: list, /// Diagnostics for this module diagnostics: list, } /// Generate with streaming results generate-streaming: func( dist: distribution, options: generation-options, ) -> stream-handle; /// Stream handle for polling results type stream-handle = u64; /// Poll for next result (non-blocking) poll-result: func(handle: stream-handle) -> option; /// Check if stream is complete is-complete: func(handle: stream-handle) -> bool; /// Cancel streaming generation cancel: func(handle: stream-handle); } ``` ## Backend Incremental Interface (`backend/incremental.wit`) Optional capability for incremental code generation: ```wit package morphir:backend@0.4.0; use generator.{generation-options, artifact, diagnostic}; use morphir:ir@0.4.0.distributions.{distribution}; use morphir:ir@0.4.0.naming.{module-path}; /// Incremental generation interface (optional capability) interface incremental { /// Module change notification record module-change { /// Module path that changed path: module-path, /// Type of change change-type: change-type, } /// Change type enum change-type { /// Module was added added, /// Module was modified modified, /// Module was removed removed, } /// Incremental generation result record incremental-generation-result { /// Generated/updated artifacts artifacts: list, /// Artifacts to delete (paths) deleted-artifacts: list, /// Diagnostics diagnostics: list, } /// Generate incrementally for changed modules only generate-incremental: func( dist: distribution, changed-modules: list, options: generation-options, ) -> incremental-generation-result; } ``` ## Transform Interface (`backend/transform.wit`) Standalone interface for IR-to-IR transformations: ```wit package morphir:backend@0.4.0; use generator.{diagnostic}; use morphir:ir@0.4.0.{ naming.{fqname, module-path}, types.{type-definition}, values.{value-definition}, modules.{module-definition}, packages.{package-definition}, distributions.{distribution}, document.{document-value}, }; /// IR transformation interface interface transform { /// Transformation metadata record transform-info { /// Transform name name: string, /// Description description: string, /// Version version: string, /// Whether transform preserves semantics semantic-preserving: bool, } /// Transformation options record transform-options { /// Dry run (report changes without applying) dry-run: bool, /// Custom options (JSON-like) custom: option, } /// Transformation result record transform-result { /// Whether transformation succeeded success: bool, /// Number of definitions modified definitions-modified: u32, /// Diagnostics (warnings, info) diagnostics: list, } /// Get transform metadata info: func() -> transform-info; /// Transform entire distribution transform-distribution: func( dist: distribution, options: transform-options, ) -> tuple; /// Transform a single module transform-module: func( path: module-path, module: module-definition, options: transform-options, ) -> tuple; /// Transform a single type definition transform-type: func( fqn: fqname, def: type-definition, options: transform-options, ) -> tuple; /// Transform a single value definition transform-value: func( fqn: fqname, def: value-definition, options: transform-options, ) -> tuple; } ``` ## VFS Interfaces (`reader.wit` / `writer.wit`) ```wit package morphir:vfs@0.4.0; use morphir:ir@0.4.0.{ naming.{fqname, module-path, package-path}, types.{type-definition}, values.{value-definition}, modules.{module-definition}, distributions.{distribution}, }; use morphir:backend@0.4.0.generator.{diagnostic}; /// VFS read-only access interface reader { /// Read error variant read-error { not-found(string), permission-denied(string), parse-error(string), } /// Read a type definition read-type: func(fqn: fqname) -> result; /// Read a value definition read-value: func(fqn: fqname) -> result; /// Read a module definition read-module: func(path: module-path) -> result; /// Read entire distribution read-distribution: func() -> result; /// List modules in a package list-modules: func(pkg: package-path) -> result, read-error>; /// List types in a module list-types: func(path: module-path) -> result, read-error>; /// List values in a module list-values: func(path: module-path) -> result, read-error>; } /// VFS write access interface writer { /// Write error variant write-error { permission-denied(string), validation-failed(list), conflict(string), } /// Transaction handle type transaction = u64; /// Begin transaction begin-transaction: func() -> transaction; /// Commit transaction commit: func(tx: transaction) -> result<_, write-error>; /// Rollback transaction rollback: func(tx: transaction) -> result<_, write-error>; /// Write a type definition write-type: func( tx: transaction, fqn: fqname, def: type-definition, ) -> result<_, write-error>; /// Write a value definition write-value: func( tx: transaction, fqn: fqname, def: value-definition, ) -> result<_, write-error>; /// Delete a type delete-type: func(tx: transaction, fqn: fqname) -> result<_, write-error>; /// Delete a value delete-value: func(tx: transaction, fqn: fqname) -> result<_, write-error>; /// Rename a type rename-type: func( tx: transaction, old-fqn: fqname, new-fqn: fqname, ) -> result<_, write-error>; /// Rename a value rename-value: func( tx: transaction, old-fqn: fqname, new-fqn: fqname, ) -> result<_, write-error>; } /// Workspace management interface workspace { use morphir:ir@0.4.0.{ naming.{package-path}, distributions.{semver, distribution}, document.{document-value}, }; // ============================================================ // Types // ============================================================ /// Workspace state enum workspace-state { /// Workspace is closed closed, /// Workspace is open and ready open, /// Workspace is being initialized initializing, /// Workspace has errors error, } /// Project state within workspace enum project-state { /// Project not yet loaded unloaded, /// Project is loading loading, /// Project is loaded and ready ready, /// Project has compilation errors error, /// Project is stale (needs recompilation) stale, } /// Workspace info record workspace-info { /// Workspace root path root: string, /// Workspace name (derived from root or config) name: string, /// Current state state: workspace-state, /// Projects in workspace projects: list, /// Workspace-level configuration config: option, } /// Project info record project-info { /// Project name (package path) name: package-path, /// Project version version: semver, /// Project root path (relative to workspace) path: string, /// Current state state: project-state, /// Source directory source-dir: string, /// Dependencies dependencies: list, } /// Dependency info record dependency-info { /// Dependency name name: package-path, /// Required version version: semver, /// Whether dependency is resolved resolved: bool, } /// Workspace error variant workspace-error { /// Workspace not found not-found(string), /// Workspace already exists already-exists(string), /// Workspace is not open not-open, /// Project not found project-not-found(string), /// Project already exists project-already-exists(string), /// Invalid configuration invalid-config(string), /// IO error io-error(string), } /// Watch event type enum watch-event-type { /// File created created, /// File modified modified, /// File deleted deleted, /// File renamed renamed, } /// Watch event record watch-event { /// Event type event-type: watch-event-type, /// Affected path path: string, /// New path (for rename events) new-path: option, /// Affected project (if determinable) project: option, } // ============================================================ // Workspace Lifecycle // ============================================================ /// Create a new workspace create-workspace: func( /// Workspace root path root: string, /// Initial configuration config: option, ) -> result; /// Open an existing workspace open-workspace: func( /// Workspace root path root: string, ) -> result; /// Close the current workspace close-workspace: func() -> result<_, workspace-error>; /// Get current workspace info get-workspace-info: func() -> result; /// Update workspace configuration update-workspace-config: func( config: document-value, ) -> result<_, workspace-error>; // ============================================================ // Project Management // ============================================================ /// Add a project to the workspace add-project: func( /// Project name name: package-path, /// Project path (relative to workspace root) path: string, /// Initial version version: semver, /// Source directory source-dir: string, ) -> result; /// Remove a project from the workspace remove-project: func( name: package-path, ) -> result<_, workspace-error>; /// Get project info get-project-info: func( name: package-path, ) -> result; /// List all projects list-projects: func() -> result, workspace-error>; /// Load a project (parse and compile) load-project: func( name: package-path, ) -> result; /// Unload a project (free resources) unload-project: func( name: package-path, ) -> result<_, workspace-error>; /// Reload a project (recompile) reload-project: func( name: package-path, ) -> result; // ============================================================ // Dependency Management // ============================================================ /// Add a dependency to a project add-dependency: func( project: package-path, dependency: package-path, version: semver, ) -> result<_, workspace-error>; /// Remove a dependency from a project remove-dependency: func( project: package-path, dependency: package-path, ) -> result<_, workspace-error>; /// Resolve all dependencies for a project resolve-dependencies: func( project: package-path, ) -> result, workspace-error>; /// Resolve all dependencies for entire workspace resolve-all-dependencies: func() -> result>>, workspace-error>; // ============================================================ // File Watching // ============================================================ /// Start watching workspace for changes start-watching: func() -> result<_, workspace-error>; /// Stop watching workspace stop-watching: func() -> result<_, workspace-error>; /// Poll for watch events (non-blocking) poll-events: func() -> list; // ============================================================ // Workspace Operations // ============================================================ /// Build all projects in workspace build-all: func() -> result>, workspace-error>; /// Clean build artifacts clean: func( /// Specific project, or all if none project: option, ) -> result<_, workspace-error>; /// Get workspace-wide diagnostics get-diagnostics: func() -> list>>; } ``` ## World Definitions (`worlds.wit`) ```wit package morphir:component@0.4.0; use morphir:extension@0.4.0.{info, capabilities}; use morphir:frontend@0.4.0.{compiler}; use morphir:frontend@0.4.0 as frontend; use morphir:backend@0.4.0.{generator, validator, transform}; use morphir:backend@0.4.0 as backend; use morphir:vfs@0.4.0.{reader, writer, workspace}; // ============================================================ // MINIMAL EXTENSION (Info Only) // ============================================================ /// Minimal extension - just provides info and health check /// All extensions should start here before adding capabilities world info-component { export info; } /// Extension with capability discovery world discoverable-component { export info; export capabilities; } // ============================================================ // FRONTEND COMPONENTS (Source → IR) // ============================================================ /// Minimal frontend compiler component (basic compilation only) world minimal-compiler-component { export info; export compiler; } /// Frontend compiler component with capability discovery world compiler-component { export info; export capabilities; export compiler; } /// Frontend with VFS access (for reading dependencies) world compiler-with-vfs-component { import reader; export info; export compiler; } /// Frontend with streaming support world streaming-compiler-component { export info; export capabilities; export compiler; export frontend:streaming; } /// Frontend with incremental support world incremental-compiler-component { import reader; export info; export capabilities; export compiler; export frontend:incremental; } /// Full-featured frontend (all capabilities) world full-compiler-component { import reader; export info; export capabilities; export compiler; export frontend:streaming; export frontend:incremental; } // ============================================================ // BACKEND COMPONENTS (IR → Target) // ============================================================ /// Minimal code generator component world minimal-generator-component { export info; export generator; } /// Code generator component with capability discovery world generator-component { export info; export capabilities; export generator; } /// Generator with streaming support world streaming-generator-component { export info; export capabilities; export generator; export backend:streaming; } /// Generator with incremental support world incremental-generator-component { export info; export capabilities; export generator; export backend:incremental; } /// Full-featured generator (all capabilities) world full-generator-component { export info; export capabilities; export generator; export backend:streaming; export backend:incremental; } /// Validator component world validator-component { export info; export validator; } /// Full backend (generator + validator) world backend-component { export info; export capabilities; export generator; export validator; } // ============================================================ // TRANSFORMER COMPONENTS (IR → IR) // ============================================================ /// Standalone transformer (pure IR transformation) world transform-component { export info; export transform; } /// Transformer with VFS access world transformer-component { import reader; import writer; export info; export capabilities; export transform; } // ============================================================ // WORKSPACE COMPONENTS // ============================================================ /// Workspace management component (daemon-side) world workspace-manager-component { export info; export workspace; } /// Workspace-aware compiler world workspace-compiler-component { import workspace; import reader; export info; export capabilities; export compiler; } // ============================================================ // FULL PIPELINE COMPONENTS // ============================================================ /// Full toolchain component (frontend + backend) world toolchain-component { import reader; export info; export capabilities; export compiler; export generator; export validator; } /// Toolchain with workspace support world workspace-toolchain-component { import workspace; import reader; import writer; export info; export capabilities; export compiler; export generator; export validator; } /// Full plugin with all capabilities world plugin-component { import reader; import writer; import workspace; export info; export capabilities; export compiler; export generator; export validator; export transform; export frontend:streaming; export frontend:incremental; export backend:streaming; export backend:incremental; } ``` ## Component Manifest ```json { "name": "morphir-typescript-backend", "version": "1.0.0", "description": "TypeScript code generator for Morphir IR", "world": "morphir:component/backend-component@0.4.0", "exports": { "generator": { "target": "typescript", "granularities": ["distribution", "module", "definition"] }, "validator": { "categories": ["typescript-compat", "naming"] } }, "wasm": { "path": "morphir-typescript-backend.wasm", "sha256": "abc123..." } } ``` ## Security Model | World | Imports | Access Level | | --------------------------------- | --------------------- | --------------------------------- | | `info-component` | None | Metadata only (minimal) | | `discoverable-component` | None | Metadata + capability query | | `minimal-compiler-component` | None | Pure function (source → IR) | | `compiler-component` | None | Pure function with capabilities | | `compiler-with-vfs-component` | VFS reader | Read-only (for dependencies) | | `streaming-compiler-component` | None | Streaming compilation | | `incremental-compiler-component` | VFS reader | Incremental with dependency graph | | `full-compiler-component` | VFS reader | All frontend capabilities | | `minimal-generator-component` | None | Pure function (IR → target) | | `generator-component` | None | Pure function with capabilities | | `streaming-generator-component` | None | Streaming generation | | `incremental-generator-component` | None | Incremental generation | | `full-generator-component` | None | All codegen capabilities | | `validator-component` | None | Pure function | | `backend-component` | None | Generator + validator | | `transform-component` | None | Pure IR transformation | | `transformer-component` | VFS reader, writer | Scoped to distribution | | `workspace-manager-component` | None (exports only) | Workspace lifecycle management | | `workspace-compiler-component` | Workspace, VFS reader | Workspace-aware compilation | | `toolchain-component` | VFS reader | Full compilation pipeline | | `workspace-toolchain-component` | Workspace, VFS r/w | Full pipeline with workspace | | `plugin-component` | Full VFS + workspace | Maximum access (all capabilities) | Components are sandboxed: - No filesystem access outside VFS - No network access - No environment variables - Memory and CPU limits enforced ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](./README.md)** - Extension system overview - **[Tasks](./tasks.md)** - Task system definition - **[Morphir Daemon](../daemon/)** - Workspace and build management ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification - [WIT Type Mapping](https://morphir.finos.org/docs/user-guides/cli-tools/wit-type-mapping/) - WIT to Morphir IR mapping - [WIT Commands](https://morphir.finos.org/docs/user-guides/cli-tools/wit-commands/) - WIT command reference --- ## Dependency Management Source: https://finos.github.io/morphir-rust/contributors/design/daemon/dependencies # Dependency Management This document defines dependency resolution and management for Morphir workspaces. ## Overview Morphir supports three categories of dependency sources: | Source | Use Case | Status | | -------------- | ----------------------------------------------- | ------------- | | **Path** | Local development, workspace members, monorepos | **Available** | | **Repository** | Pre-release code, forks, private packages | **Available** | | **Registry** | Published packages, stable releases | **Planned** | > **Note**: A Morphir package registry is planned for future releases. Until then, path and repository dependencies are the recommended approaches for sharing Morphir packages. Dependencies can be declared at two levels: - **Workspace-level**: Shared versions defined once, inherited by members - **Project-level**: Specific to individual projects ## Dependency Sources ### Path Dependencies Path dependencies reference local directories containing Morphir projects. They are resolved directly from the filesystem without version negotiation. ```toml [dependencies] # Relative path (recommended for workspace members) "my-org/core" = { path = "../core" } # Absolute path (for external local projects) "my-org/shared" = { path = "/path/to/shared" } ``` **Use cases:** - Workspace member projects depending on each other - Local development of multiple related packages - Temporary overrides during development (similar to Go's `replace`) **Behavior:** - Always uses the current source from the path - No version constraint required (implicit "latest") - Changes to source are immediately visible - Must contain a valid `morphir.toml` ### Repository Dependencies Repository dependencies are fetched directly from git repositories, similar to Go modules. This is the **recommended approach** for sharing packages across organizations until a registry is available. ```toml [dependencies] # Git repository with tag/version "acme/experimental" = { git = "https://github.com/acme/experimental.git", tag = "v1.0.0" } # Git repository with branch "acme/feature" = { git = "https://github.com/acme/feature.git", branch = "main" } # Git repository with specific commit "acme/pinned" = { git = "https://github.com/acme/pinned.git", rev = "a8b3c5d" } # SSH URL (for private repositories) "private/internal" = { git = "git@github.com:private/internal.git", tag = "v2.0.0" } ``` **Reference types (mutually exclusive):** | Field | Description | Example | | -------- | ------------------------------------- | ------------- | | `tag` | Git tag (recommended for releases) | `"v1.0.0"` | | `branch` | Git branch (for tracking development) | `"main"` | | `rev` | Specific commit SHA (for pinning) | `"a8b3c5d82"` | **Use cases:** - Sharing packages across organizations - Pre-release packages - Forked versions with custom modifications - Private packages **Behavior:** - Repository is cloned/fetched to dependency cache - Specific ref is checked out - Treated as immutable once resolved (stored in lock file) ### Registry Dependencies (Planned) > **Status**: Registry dependencies are planned for a future release. Registry dependencies will be published packages fetched from a Morphir package registry: ```toml [dependencies] # Simple version constraint (planned) "morphir/sdk" = "3.0.0" # Semver range (planned) "some-org/utilities" = "^1.2.0" # Explicit registry specification (planned) "finos/morphir-json" = { version = "^1.0.0", registry = "https://registry.morphir.dev" } ``` Until registry support is available, use repository dependencies with git tags for similar functionality: ```toml # Current recommended approach for external packages "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } ``` ### Workspace Inheritance Workspace members can inherit dependency versions from the workspace root, similar to Cargo's workspace inheritance: ```toml # workspace/morphir.toml [workspace] members = ["packages/*"] [workspace.dependencies] "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } "acme/shared" = { git = "https://github.com/acme/shared.git", tag = "v2.0.0" } ``` ```toml # workspace/packages/domain/morphir.toml [project] name = "my-org/domain" [dependencies] # Inherit version from workspace "morphir/sdk" = { workspace = true } # Can still add project-specific dependencies "other/lib" = { path = "../other" } ``` **Benefits:** - Single source of truth for shared dependency versions - Consistent versions across all workspace members - Simplified updates (change once, apply everywhere) - Reduced configuration duplication ## Version Constraints Version constraints apply to git tags and future registry dependencies: | Syntax | Meaning | Example | | ------------------- | --------------------- | ----------------- | | `"1.2.3"` | Exact version | Only 1.2.3 | | `"^1.2.3"` | Compatible (caret) | `>=1.2.3, <2.0.0` | | `"~1.2.3"` | Approximately (tilde) | `>=1.2.3, <1.3.0` | | `">=1.2.0"` | Greater than or equal | `>=1.2.0` | | `"<2.0.0"` | Less than | `<2.0.0` | | `">=1.0.0, <2.0.0"` | Range | `>=1.0.0, <2.0.0` | **Notes:** - Path dependencies ignore version constraints - Git dependencies with `branch` track the branch head - Git dependencies with `rev` ignore semver entirely ## Types ### DependencySource ```gleam /// Source of a dependency pub type DependencySource { /// Local path dependency Path(path: String) /// Git repository dependency Repository( url: String, ref: GitRef, ) /// Registry dependency (planned) Registry( version: SemVer, registry: Option(String), ) /// Inherited from workspace Workspace } /// Git reference type pub type GitRef { Tag(String) Branch(String) Rev(String) } ``` ### DependencySpec ```gleam /// Dependency specification as declared in morphir.toml pub type DependencySpec { DependencySpec( /// Package name name: PackagePath, /// Dependency source source: DependencySource, ) } ``` ### DependencyInfo ```gleam /// Resolved dependency information pub type DependencyInfo { DependencyInfo( /// Package name name: PackagePath, /// Original source specification source: DependencySource, /// Whether dependency has been resolved resolved: Bool, /// Resolved location (path to resolved package) resolved_path: Option(String), /// Resolved version (from morphir.toml or git tag) resolved_version: Option(SemVer), ) } ``` ### DependencyGraph ```gleam /// Resolved dependency graph pub type DependencyGraph { DependencyGraph( /// Root projects being resolved roots: List(PackagePath), /// All resolved packages (topologically sorted) packages: List(ResolvedPackage), /// Resolution conflicts (if any) conflicts: List(DependencyConflict), ) } /// A resolved package in the graph pub type ResolvedPackage { ResolvedPackage( name: PackagePath, version: Option(SemVer), /// Direct dependencies dependencies: List(PackagePath), /// Resolved source information source: ResolvedSource, ) } /// Resolved source location pub type ResolvedSource { /// Local path (workspace member or path dep) Local(path: String) /// Cached from git repository Cached( cache_path: String, url: String, ref: String, ) /// Cached from registry (planned) RegistryCached( cache_path: String, registry: String, ) } ``` ## Operations ### Add Dependency Adds a dependency to a project. #### Behavior 1. Validate dependency source specification 2. Update project's `morphir.toml` 3. Mark dependency as unresolved 4. Optionally trigger resolution #### WIT Interface ```wit /// Dependency source specification variant dependency-source { /// Local path dependency path(string), /// Git repository dependency repository(git-dependency), /// Inherit from workspace workspace, } /// Git repository dependency record git-dependency { url: string, ref: git-ref, } /// Git reference variant git-ref { tag(string), branch(string), rev(string), } /// Add a dependency to a project add-dependency: func( project: package-path, dependency: package-path, source: dependency-source, ) -> result<_, workspace-error>; ``` #### JSON-RPC **Request (path dependency):** ```json { "method": "workspace/addDependency", "params": { "project": "my-org/api", "dependency": "my-org/core", "source": { "path": "../core" } } } ``` **Request (repository dependency):** ```json { "method": "workspace/addDependency", "params": { "project": "my-org/api", "dependency": "morphir/sdk", "source": { "git": "https://github.com/finos/morphir-sdk.git", "tag": "v3.0.0" } } } ``` **Request (workspace inheritance):** ```json { "method": "workspace/addDependency", "params": { "project": "my-org/api", "dependency": "morphir/sdk", "source": { "workspace": true } } } ``` #### CLI ```bash # Path dependency morphir deps add my-org/core --path ../core # Repository dependency with tag morphir deps add morphir/sdk --git https://github.com/finos/morphir-sdk.git --tag v3.0.0 # Repository dependency with branch morphir deps add acme/feature --git https://github.com/acme/feature.git --branch main # Repository dependency with commit morphir deps add acme/pinned --git https://github.com/acme/pinned.git --rev a8b3c5d # Workspace inheritance morphir deps add morphir/sdk --workspace # Specify target project morphir deps add my-org/core --path ../core --project my-org/api ``` ### Remove Dependency Removes a dependency from a project. #### Behavior 1. Verify dependency exists 2. Check if removal would break other projects 3. Update project's `morphir.toml` 4. Update lock file #### WIT Interface ```wit /// Remove a dependency from a project remove-dependency: func( project: package-path, dependency: package-path, ) -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/removeDependency", "params": { "project": "my-org/api", "dependency": "some/unused-lib" } } ``` #### CLI ```bash morphir deps remove some/unused-lib morphir deps remove some/unused-lib --project my-org/api ``` ### Resolve Dependencies Resolves all dependencies for a project or workspace. #### Behavior 1. Collect all dependency specifications 2. Build dependency graph 3. For path dependencies: verify path exists and contains valid project 4. For repository dependencies: clone/fetch to cache, checkout ref 5. Resolve any version conflicts 6. Update lock file 7. Return resolution result #### WIT Interface ```wit /// Resolve all dependencies for a project resolve-dependencies: func( project: package-path, ) -> result, workspace-error>; /// Resolve all dependencies for entire workspace resolve-all-dependencies: func() -> result>>, workspace-error>; ``` #### JSON-RPC **Request (single project):** ```json { "method": "workspace/resolveDependencies", "params": { "project": "my-org/api" } } ``` **Request (entire workspace):** ```json { "method": "workspace/resolveDependencies", "params": {} } ``` **Response:** ```json { "result": [ { "name": "morphir/sdk", "source": { "git": "https://github.com/finos/morphir-sdk.git", "tag": "v3.0.0" }, "resolved": true, "resolvedPath": ".morphir/deps/morphir/sdk/v3.0.0", "resolvedVersion": "3.0.0" }, { "name": "my-org/core", "source": { "path": "../core" }, "resolved": true, "resolvedPath": "/workspace/packages/core" }, { "name": "acme/shared", "source": { "git": "https://github.com/acme/shared.git", "branch": "main" }, "resolved": true, "resolvedPath": ".morphir/deps/acme/shared/main-a8b3c5d", "resolvedVersion": null } ] } ``` #### CLI ```bash morphir deps resolve morphir deps resolve --project my-org/api morphir deps list morphir deps list --resolved # Show resolved paths ``` ## Lock File The workspace maintains a `morphir.lock` file with resolved dependency information: ```toml # morphir.lock # This file is auto-generated. Do not edit. # It ensures reproducible builds by pinning exact dependency sources. [[package]] name = "morphir/sdk" source = "git" url = "https://github.com/finos/morphir-sdk.git" tag = "v3.0.0" rev = "abc123def456..." checksum = "sha256:abc123..." dependencies = [] [[package]] name = "my-org/core" source = "path" path = "packages/core" # Path dependencies are not version-locked; they use current source [[package]] name = "acme/shared" source = "git" url = "https://github.com/acme/shared.git" branch = "main" rev = "a8b3c5d82e..." # Pinned commit at resolution time checksum = "sha256:def456..." dependencies = ["morphir/sdk"] [[package]] name = "acme/pinned" source = "git" url = "https://github.com/acme/pinned.git" rev = "deadbeef..." checksum = "sha256:789abc..." dependencies = [] ``` ### Lock File Behavior | Source | Lock Behavior | | ------------ | ----------------------------------------------------- | | Path | Not locked; always uses current source | | Git (tag) | Locks tag + resolved commit SHA | | Git (branch) | Locks branch + resolved commit SHA at resolution time | | Git (rev) | Locks exact commit SHA | **Updating the lock file:** ```bash # Resolve and update lock file morphir deps resolve # Update a specific dependency to latest morphir deps update morphir/sdk # Update all dependencies morphir deps update --all # Update git branch dependencies to latest commit morphir deps update --branches ``` ## Resolution Algorithm The dependency resolution algorithm uses **Minimal Version Selection (MVS)**, similar to Go modules. This provides reproducible builds without a SAT solver. ### Algorithm Overview ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Collect │────►│ Resolve │────►│ Verify │ │ Dependencies │ │ Versions │ │ & Lock │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │ │ ▼ ▼ ▼ Direct deps Transitive Lock file from config resolution generation ``` ### 1. Dependency Collection Gather all dependency specifications from: - Workspace-level `[workspace.dependencies]` - Project-level `[dependencies]` - Transitive dependencies (from resolved packages) ### 2. Workspace Inheritance Resolution For dependencies marked `{ workspace = true }`: 1. Look up the dependency in `[workspace.dependencies]` 2. Replace with the workspace-defined source 3. Error if dependency not found in workspace ### 3. Graph Construction Build a dependency graph: ``` my-org/api ├── morphir/sdk (git: finos/morphir-sdk.git@v3.0.0) ├── my-org/domain (path: ../domain) │ ├── morphir/sdk (workspace -> git: finos/morphir-sdk.git@v3.0.0) │ └── my-org/core (path: ../core) │ └── morphir/sdk (workspace -> git: finos/morphir-sdk.git@v3.0.0) └── acme/shared (git: acme/shared.git@main) └── morphir/sdk (git: finos/morphir-sdk.git@v3.0.0) ``` ### 4. Source Resolution For each dependency in topological order: **Path dependencies:** 1. Resolve path relative to dependent project 2. Verify directory exists 3. Verify `morphir.toml` exists and is valid 4. Add to resolved graph **Repository dependencies:** 1. Check if already in cache (matching URL + ref) 2. If not cached: clone repository to `.morphir/deps/` 3. Checkout specified ref (tag/branch/rev) 4. For branches: record resolved commit SHA 5. Verify `morphir.toml` exists in repository root 6. Add to resolved graph with cache path ### 5. Transitive Dependency Resolution Transitive dependencies are resolved recursively. Each resolved package may declare its own dependencies. **Algorithm:** ``` function resolveTransitive(rootDeps): resolved = {} queue = rootDeps while queue is not empty: dep = queue.pop() if dep.name in resolved: # Already resolved - check for version conflict existing = resolved[dep.name] resolved[dep.name] = selectVersion(existing, dep) else: # New dependency - resolve and add transitives pkg = fetchAndResolve(dep) resolved[dep.name] = pkg queue.addAll(pkg.dependencies) return topologicalSort(resolved) ``` **Example:** ``` my-org/api ├── morphir/sdk@^3.0.0 ├── my-org/domain │ ├── morphir/sdk@^3.0.0 (transitive) │ └── morphir/json@^1.0.0 (transitive) └── acme/utils@^2.0.0 └── morphir/sdk@^3.1.0 (transitive - higher minimum) Resolution: morphir/sdk → 3.1.0 (highest minimum requested) morphir/json → 1.0.0 acme/utils → 2.0.0 my-org/domain → (path) ``` ### 6. Version Selection (MVS) Morphir uses **Minimal Version Selection (MVS)**: select the _minimum_ version that satisfies all constraints. **Rules:** | Scenario | Resolution | | --------------------------------------- | ------------------------- | | Single request | Use requested version | | Multiple requests, same constraint type | Use highest minimum | | Caret (`^`) constraint | Select minimum in range | | Exact version conflict | Error (user must resolve) | **Example - Caret Constraints:** ``` A requires morphir/sdk@^3.0.0 → needs ≥3.0.0, <4.0.0 B requires morphir/sdk@^3.2.0 → needs ≥3.2.0, <4.0.0 MVS selects: 3.2.0 (minimum that satisfies both) ``` **Example - Incompatible Constraints:** ``` A requires morphir/sdk@^3.0.0 → needs ≥3.0.0, <4.0.0 B requires morphir/sdk@^2.0.0 → needs ≥2.0.0, <3.0.0 Error: No version satisfies both constraints morphir/sdk@^3.0.0 (required by A) morphir/sdk@^2.0.0 (required by B) Resolution: Update B to use ^3.0.0, or use path override ``` ### 7. Diamond Dependency Resolution Diamond dependencies occur when two packages depend on different versions of the same transitive dependency. ``` my-org/api / \ / \ lib-a@1.0 lib-b@1.0 \ / \ / morphir/sdk@??? lib-a requires morphir/sdk@^3.0.0 lib-b requires morphir/sdk@^3.2.0 ``` **Resolution Strategy:** 1. **Compatible ranges**: Select highest minimum (MVS) ``` lib-a: ^3.0.0 (≥3.0.0, <4.0.0) lib-b: ^3.2.0 (≥3.2.0, <4.0.0) → Select 3.2.0 (satisfies both) ``` 2. **Incompatible ranges**: Report error with upgrade path ``` lib-a: ^2.0.0 (≥2.0.0, <3.0.0) lib-b: ^3.0.0 (≥3.0.0, <4.0.0) → Error: Cannot unify versions → Suggestion: Update lib-a to version that supports morphir/sdk@^3.0.0 ``` 3. **Path override**: User can force a specific version ```toml [dependencies] # Force specific version for development "morphir/sdk" = { path = "../morphir-sdk" } ``` ### 8. Conflict Detection and Reporting Conflicts are detected and reported clearly: **Source Conflicts:** ``` Error: Conflicting sources for morphir/sdk my-org/api requires: git@github.com/finos/morphir-sdk@v3.0.0 legacy/lib requires: git@github.com/fork/morphir-sdk@v3.0.0 Different repositories for same package name. Resolution options: 1. Update legacy/lib to use official repository 2. Rename fork to different package name 3. Use path override for local resolution ``` **Version Conflicts:** ``` Error: Incompatible versions for morphir/sdk my-org/api requires: ^3.0.0 (via direct dependency) legacy/lib requires: ^2.0.0 (via lib-old@1.0.0) No version satisfies both constraints. Resolution options: 1. Update legacy/lib to version supporting morphir/sdk@^3.0.0 2. Update lib-old to newer version 3. Use path override: "morphir/sdk" = { path = "..." } ``` **Cycle Detection:** ``` Error: Dependency cycle detected my-org/api → my-org/domain → my-org/core → my-org/api Cycles are not allowed. Refactor to break the cycle. ``` ### 10. Path Overrides For local development, path dependencies can override other sources: ```toml # morphir.toml [dependencies] # Development override - path takes precedence "morphir/sdk" = { path = "../morphir-sdk" } # This repository source is ignored when path is also specified: # "morphir/sdk" = { git = "...", tag = "v3.0.0" } ``` This is similar to Go's `replace` directive, allowing local modifications without changing the primary dependency specification. ## Dependency Cache Repository dependencies can be cached locally (per-workspace) or globally (user-level). Local caching is the default for workspace isolation, but global caching can reduce disk usage and download time across multiple workspaces. ### Cache Locations | Cache | Location | Use Case | | ------ | ------------------------------- | --------------------------------------------- | | Local | `.morphir/deps/` | Workspace isolation, offline builds | | Global | `$XDG_CACHE_HOME/morphir/deps/` | Shared across workspaces, reduced duplication | The global cache follows XDG Base Directory conventions: - Linux/macOS: `~/.cache/morphir/deps/` (or `$XDG_CACHE_HOME/morphir/deps/`) - Windows: `%LOCALAPPDATA%\morphir\cache\deps\` ### Local Cache (Default) ``` workspace/ └── .morphir/ └── deps/ ├── morphir/ │ └── sdk/ │ └── v3.0.0-abc123/ # tag + short SHA │ ├── morphir.toml │ ├── src/ │ └── .git/ # Shallow clone └── acme/ └── shared/ └── main-def456/ # branch + short SHA └── ... ``` ### Global Cache ``` ~/.cache/morphir/ └── deps/ ├── github.com/ │ ├── finos/ │ │ └── morphir-sdk/ │ │ ├── v3.0.0/ │ │ └── v3.1.0/ │ └── acme/ │ └── shared/ │ └── main-def456/ └── gitlab.com/ └── ... ``` ### Configuring Cache Behavior **Per-dependency cache location:** ```toml [dependencies] # Use global cache for this dependency "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0", cache = "global" } # Explicit local cache (default) "acme/experimental" = { git = "https://github.com/acme/experimental.git", branch = "main", cache = "local" } ``` **Workspace-wide cache default:** ```toml # morphir.toml [workspace] members = ["packages/*"] [cache] # Use global cache for all repository dependencies by default dependencies = "global" # Or keep default local caching # dependencies = "local" ``` **User-level default:** ```toml # ~/.config/morphir/config.toml [cache] # Set global as default for all workspaces dependencies = "global" ``` ### Cache Management ```bash # Show cache status (local and global) morphir deps cache status # Clean unused cached dependencies (local) morphir deps cache clean # Clean global cache morphir deps cache clean --global # Clean all cached dependencies morphir deps cache clean --all # Verify cache integrity morphir deps cache verify ``` ## Best Practices 1. **Commit Lock Files**: Always commit `morphir.lock` for reproducible builds 2. **Use Tags for Releases**: Prefer git tags over branches for stability 3. **Workspace Inheritance**: Share common dependencies at workspace level with `[workspace.dependencies]` 4. **Path for Development**: Use path dependencies for local cross-project development 5. **Global Cache for CI**: Consider global caching in CI environments to speed up builds across jobs 6. **Pin Branches**: When using branch dependencies, run `morphir deps resolve` regularly to update the lock file 7. **Review Updates**: Use `morphir deps update --dry-run` to preview changes before applying ## Migration Guide ### From Direct Git Clones If you previously cloned dependencies manually: ```bash # Before: manual clone git clone https://github.com/finos/morphir-sdk.git deps/morphir-sdk # After: declare in morphir.toml [dependencies] "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } ``` ### Preparing for Registry When the Morphir registry becomes available, migration will be straightforward: ```toml # Current: repository dependency "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } # Future: registry dependency "morphir/sdk" = "3.0.0" ``` The lock file format will remain compatible, so existing builds will continue to work. ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Projects](./projects.md)** - Project management within a workspace - **[Packages](./packages.md)** - Package format and publishing ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Extension Host Interface Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/02-extension-host-interface # Extension Host Interface **Status:** Draft **Version:** 0.1.0 ## Overview The `ExtensionHost` trait defines the common interface that all protocol-specific hosts must implement. This abstraction allows the Extension Manager to work with any host type uniformly. ## Trait Definition ```rust use async_trait::async_trait; use serde_json::Value; #[async_trait] pub trait ExtensionHost: Send + 'static { /// Returns the protocol identifier (e.g., "jsonrpc2", "grpc", "stdio") fn protocol(&self) -> &str; /// Initialize the host (called once when host is created) async fn initialize(&mut self) -> Result<(), ExtensionError>; /// Load an extension from the given configuration async fn load_extension( &mut self, config: ExtensionConfig, ) -> Result; /// Call a method on a loaded extension async fn call( &mut self, extension_id: ExtensionId, method: &str, params: Value, ) -> Result; /// Unload an extension and clean up resources async fn unload_extension( &mut self, extension_id: ExtensionId, ) -> Result<(), ExtensionError>; /// Query capabilities of a loaded extension async fn capabilities( &self, extension_id: ExtensionId, ) -> Result, ExtensionError>; /// Check if extension is healthy (optional, default returns true) async fn health_check( &self, extension_id: ExtensionId, ) -> Result { Ok(HealthStatus::Healthy) } } ``` ## Common Types ### ExtensionId Unique identifier for a loaded extension: ```rust #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] pub struct ExtensionId(pub u64); ``` ### ExtensionConfig Configuration for loading an extension: ```rust #[derive(Clone, Debug, Deserialize, Serialize)] pub struct ExtensionConfig { /// Human-readable name pub name: String, /// Source location/configuration pub source: ExtensionSource, /// Permissions granted to extension pub permissions: Permissions, /// Extension-specific configuration pub config: Value, /// Restart strategy on failure #[serde(default)] pub restart: RestartStrategy, } #[derive(Clone, Debug, Deserialize, Serialize)] #[serde(tag = "type", rename_all = "lowercase")] pub enum ExtensionSource { Http { url: String }, Grpc { endpoint: String }, Process { command: String, args: Vec, #[serde(default)] env: HashMap, }, Wasm { path: PathBuf }, WasmComponent { path: PathBuf }, } ``` ### Capability Description of what an extension can do: ```rust #[derive(Clone, Debug, Serialize, Deserialize)] pub struct Capability { /// Method name pub name: String, /// Human-readable description pub description: String, /// JSON schema for parameters (optional) #[serde(skip_serializing_if = "Option::is_none")] pub params_schema: Option, /// JSON schema for return value (optional) #[serde(skip_serializing_if = "Option::is_none")] pub return_schema: Option, } ``` ### Permissions Permissions granted to extension: ```rust #[derive(Clone, Debug, Default, Deserialize, Serialize)] pub struct Permissions { /// Allow network access #[serde(default)] pub network: bool, /// Allowed filesystem paths #[serde(default)] pub filesystem: Vec, /// Maximum memory (WASM only) #[serde(skip_serializing_if = "Option::is_none")] pub max_memory: Option, // e.g., "100MB" /// Maximum execution time per call #[serde(skip_serializing_if = "Option::is_none")] pub max_execution_time: Option, } ``` ### HealthStatus Health status of an extension: ```rust #[derive(Clone, Debug, Serialize, Deserialize)] pub enum HealthStatus { Healthy, Degraded { reason: String }, Unhealthy { reason: String }, } ``` ### ExtensionError Common error type for all host operations: ```rust #[derive(Debug, thiserror::Error)] pub enum ExtensionError { #[error("Extension not found: {0}")] NotFound(ExtensionId), #[error("Invalid extension source")] InvalidSource, #[error("Initialization failed: {0}")] InitializationFailed(String), #[error("Method not found: {0}")] MethodNotFound(String), #[error("Extension error: {0}")] ExtensionError(String), #[error("Protocol error: {0}")] ProtocolError(String), #[error("Timeout")] Timeout, #[error("IO error: {0}")] IoError(#[from] std::io::Error), #[error("Serialization error: {0}")] SerializationError(#[from] serde_json::Error), } ``` ## Lifecycle Methods ### initialize() Called once when the host actor is created. Use for: - Setting up global state - Initializing protocol clients - Loading configuration **Example:** ```rust async fn initialize(&mut self) -> Result<(), ExtensionError> { tracing::info!("Initializing {} host", self.protocol()); // Setup protocol-specific state Ok(()) } ``` ### load_extension() Loads a new extension. Must: 1. Validate configuration 2. Spawn/connect to extension 3. Send initialization message 4. Query capabilities 5. Return unique ExtensionId **Sequence:** ``` Host receives config ↓ Validate source matches protocol ↓ Spawn/connect to extension ↓ Send "initialize" with config.config ↓ Wait for success response ↓ Send "capabilities" query ↓ Store extension metadata ↓ Return ExtensionId ``` ### call() Invokes a method on an extension. Must: 1. Validate extension exists 2. Serialize parameters 3. Send request with timeout 4. Deserialize response 5. Return result or error **Error Handling:** - If extension not found → `ExtensionError::NotFound` - If method unknown → `ExtensionError::MethodNotFound` - If timeout → `ExtensionError::Timeout` - If extension returns error → `ExtensionError::ExtensionError` ### unload_extension() Cleanly shuts down an extension. Must: 1. Send shutdown notification (if protocol supports) 2. Wait briefly for cleanup 3. Force terminate if needed 4. Remove from tracking **Best Effort:** Should not fail even if extension already dead. ### capabilities() Returns cached capabilities. Should be fast (no RPC). ## Implementation Guidelines ### Thread Safety All methods receive `&mut self`, so: - **No internal locking needed** (actor guarantees sequential access) - **Can mutate state freely** - **Blocking operations must use spawn_blocking** ### Timeouts All operations with external communication should use timeouts: ```rust tokio::time::timeout( Duration::from_secs(30), some_async_operation() ).await?? ``` ### Error Propagation Use `?` operator and convert to `ExtensionError`: ```rust let response = client.call(request).await .map_err(|e| ExtensionError::ProtocolError(e.to_string()))?; ``` ### Logging Use structured logging with `tracing`: ```rust #[instrument(skip(self), fields(extension_id = %extension_id))] async fn call(&mut self, extension_id: ExtensionId, ...) { debug!("Calling method", method = %method); // ... } ``` ### Metrics Emit metrics for all operations: ```rust let start = Instant::now(); let result = self.call_internal(id, method, params).await; let duration = start.elapsed(); metrics::histogram!("morphir.extension.call.duration_ms") .record(duration.as_millis() as f64); metrics::counter!("morphir.extension.call.count") .increment(1); ``` ## Testing Host implementations should include: 1. **Unit tests**: Test individual methods 2. **Integration tests**: Test with real/mock extensions 3. **Failure tests**: Test error handling 4. **Concurrency tests**: Multiple calls in parallel 5. **Timeout tests**: Verify timeout behavior **Example:** ```rust #[tokio::test] async fn test_load_and_call() { let mut host = StdioExtensionHost::new(); host.initialize().await.unwrap(); let config = ExtensionConfig { name: "test".into(), source: ExtensionSource::Process { command: "python3".into(), args: vec!["./test_extension.py".into()], env: HashMap::new(), }, permissions: Permissions::default(), config: json!({}), restart: RestartStrategy::Never, }; let id = host.load_extension(config).await.unwrap(); let result = host.call( id, "echo", json!({"message": "hello"}), ).await.unwrap(); assert_eq!(result, json!({"message": "hello"})); } ``` ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## File Watching Source: https://finos.github.io/morphir-rust/contributors/design/daemon/watching # File Watching This document defines the file watching system for Morphir workspaces, enabling automatic recompilation on source changes. ## Overview File watching enables: - **Incremental builds**: Recompile only changed files - **IDE integration**: Real-time error feedback - **Development workflow**: Automatic rebuild on save - **Hot reload**: Update running applications (where supported) ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Workspace Daemon │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Watcher │───►│ Debounce │───►│ Compiler │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ ▲ │ │ │ │ FS events │ IR │ │ │ ▼ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ File System │ │ Notifier │ │ │ └─────────────┘ └─────────────┘ │ │ │ │ └──────────────────────────────────────────────┼─────────┘ │ ▼ Clients (IDE, CLI) ``` ## Types ### WatchEventType ```gleam /// Type of file system event pub type WatchEventType { /// File or directory created Created /// File content modified Modified /// File or directory deleted Deleted /// File or directory renamed Renamed } ``` ### WatchEvent ```gleam /// A file system change event pub type WatchEvent { WatchEvent( /// Type of event event_type: WatchEventType, /// Affected path (relative to workspace root) path: String, /// New path (for rename events only) new_path: Option(String), /// Project this file belongs to (if determinable) project: Option(PackagePath), /// Timestamp of event timestamp: DateTime, ) } ``` ### WatchState ```gleam /// Current state of the file watcher pub type WatchState { /// Watcher is not running Stopped /// Watcher is initializing Starting /// Watcher is active Running /// Watcher encountered an error Error(message: String) } ``` ## Operations ### Start Watching Begins watching the workspace for file changes. #### Behavior 1. Initialize file system watcher 2. Register watch paths for all projects 3. Set up debouncing (default: 100ms) 4. Begin emitting events #### Watch Paths By default, watches: - `*/src/**/*.elm` - Elm source files - `*/src/**/*.morphir` - Morphir DSL files - `*/morphir.toml` - Project configuration - `morphir.toml` - Workspace configuration #### WIT Interface ```wit /// Start watching workspace for changes start-watching: func() -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/watch", "params": { "enabled": true } } ``` **Response:** ```json { "result": { "state": "running", "watchedPaths": [ "packages/core/src", "packages/domain/src", "packages/api/src" ] } } ``` #### CLI ```bash morphir workspace watch morphir build --watch ``` ### Stop Watching Stops file system watching. #### WIT Interface ```wit /// Stop watching workspace stop-watching: func() -> result<_, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/watch", "params": { "enabled": false } } ``` ### Poll Events Retrieves pending watch events (for polling-based clients). #### WIT Interface ```wit /// Poll for watch events (non-blocking) poll-events: func() -> list; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/pollEvents", "params": {} } ``` **Response:** ```json { "result": [ { "eventType": "modified", "path": "packages/domain/src/User.elm", "project": "my-org/domain", "timestamp": "2026-01-16T12:34:56Z" } ] } ``` ## Notifications ### workspace/onFileChanged Push notification sent when files change (for streaming clients). ```json { "method": "workspace/onFileChanged", "params": { "events": [ { "eventType": "modified", "path": "packages/domain/src/User.elm", "project": "my-org/domain", "timestamp": "2026-01-16T12:34:56Z" }, { "eventType": "created", "path": "packages/domain/src/Order.elm", "project": "my-org/domain", "timestamp": "2026-01-16T12:34:56Z" } ] } } ``` ### workspace/onProjectStateChanged Push notification when a project's state changes due to file events. ```json { "method": "workspace/onProjectStateChanged", "params": { "project": "my-org/domain", "previousState": "ready", "currentState": "stale", "reason": "Source files modified" } } ``` ### workspace/onBuildComplete Push notification when automatic rebuild completes. ```json { "method": "workspace/onBuildComplete", "params": { "project": "my-org/domain", "success": true, "diagnostics": [], "duration": 1234 } } ``` ## Debouncing File events are debounced to avoid excessive recompilation: ``` Events: ─●─●●──●───●●●────────────────── Debounce: ─────────────────●────────────── └── Trigger rebuild |<── 100ms ──>| ``` ### Configuration ```toml # morphir.toml [watch] debounce-ms = 100 # Debounce interval ignore-patterns = [ # Patterns to ignore "**/node_modules/**", "**/.git/**", "**/*.bak" ] auto-rebuild = true # Automatically rebuild on changes ``` ## Event Processing ### File Change Flow ``` 1. File saved │ 2. FS event received │ 3. Debounce (collect more events) │ 4. Determine affected project(s) │ 5. Mark project(s) as 'stale' │ 6. Emit 'onProjectStateChanged' │ 7. If auto-rebuild enabled: │ ├── Recompile affected project(s) │ └── Emit 'onBuildComplete' │ 8. Emit 'onFileChanged' (batched events) ``` ### Affected Project Detection ```gleam /// Determine which project a file belongs to fn find_project_for_path( workspace: WorkspaceInfo, path: String, ) -> Option(PackagePath) { workspace.projects |> list.find(fn(p) { string.starts_with(path, p.path) }) |> option.map(fn(p) { p.name }) } ``` ## Watch Strategies ### Recursive Watch Watch entire source directories recursively (default): ``` packages/domain/src/ ├── Domain/ │ ├── User.elm ← watched │ └── Order.elm ← watched └── Utils.elm ← watched ``` ### Glob-Based Watch Watch specific patterns: ```toml [watch] patterns = [ "**/*.elm", "**/*.morphir", "!**/*_test.elm" # Exclude tests ] ``` ### Selective Watch Watch only specific projects: ```toml [watch] projects = ["my-org/core", "my-org/domain"] # Only these ``` ## Error Handling | Error | Cause | Recovery | | ------------------ | ----------------------- | ------------------- | | `WatchError` | FS watcher failed | Restart watcher | | `TooManyFiles` | Watch limit exceeded | Use ignore patterns | | `PermissionDenied` | Cannot access directory | Check permissions | | `PathNotFound` | Watched path deleted | Re-scan workspace | ## Platform Considerations ### Linux (inotify) - Default limit: ~8192 watches - Increase with: `fs.inotify.max_user_watches` ### macOS (FSEvents) - No practical limit - Slightly higher latency ### Windows (ReadDirectoryChangesW) - Works per-directory - May miss rapid changes ## Best Practices 1. **Ignore Generated Files**: Don't watch `.morphir-dist/`, `node_modules/` 2. **Reasonable Debounce**: 100-300ms balances responsiveness and efficiency 3. **Batch Events**: Process multiple changes together when possible 4. **Graceful Degradation**: Fall back to polling if native watching fails 5. **Resource Limits**: Monitor memory/CPU usage of watcher ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Build](./build.md)** - Build orchestration and diagnostics - **[Projects](./projects.md)** - Project management within a workspace ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents --- ## JSON-RPC Extension Host Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/03-jsonrpc-host # JSON-RPC Extension Host **Status:** Draft **Version:** 0.1.0 ## Overview The JSON-RPC Extension Host manages extensions that communicate via JSON-RPC 2.0 over HTTP. This protocol is ideal for web services, microservices, and extensions running as standalone HTTP servers. ## Protocol ### Transport - **Transport**: HTTP/HTTPS - **Format**: JSON-RPC 2.0 specification - **Library**: `jsonrpsee` (Rust client) ### Message Format #### Request (Host → Extension) ```json { "jsonrpc": "2.0", "id": 1, "method": "morphir.transform", "params": { "ir": {...} } } ``` #### Response (Extension → Host) Success: ```json { "jsonrpc": "2.0", "id": 1, "result": { "transformed": true } } ``` Error: ```json { "jsonrpc": "2.0", "id": 1, "error": { "code": -32000, "message": "Invalid IR structure" } } ``` ## Implementation ### State ```rust pub struct JsonRpcExtensionHost { extensions: HashMap, next_id: u64, } struct JsonRpcExtension { name: String, client: HttpClient, capabilities: Vec, } ``` ### Loading Process ```rust async fn load_extension(&mut self, config: ExtensionConfig) -> Result { let ExtensionSource::Http { url } = config.source else { return Err(ExtensionError::InvalidSource); }; // Create HTTP client let client = HttpClientBuilder::default() .build(url)?; // Initialize extension let _init: Value = client .request("initialize", rpc_params![config.config]) .await?; // Query capabilities let capabilities: Vec = client .request("capabilities", rpc_params![]) .await?; let id = ExtensionId(self.next_id); self.next_id += 1; self.extensions.insert(id, JsonRpcExtension { name: config.name, client, capabilities, }); Ok(id) } ``` ## Extension Implementation (TypeScript Example) ```typescript import { createServer } from "jayson"; const server = createServer({ initialize: (params, callback) => { console.log("Initialized with config:", params); callback(null, { status: "ready" }); }, capabilities: (params, callback) => { callback(null, [ { name: "transform", description: "Transform Morphir IR to TypeScript", }, ]); }, transform: (params, callback) => { try { const { ir } = params; // Transform IR to TypeScript const output = generateTypeScript(ir); callback(null, { output }); } catch (error) { callback(error); } }, }); server.http().listen(3000); ``` ## Configuration Example ```toml [[extensions]] name = "typescript-generator" enabled = true protocol = "jsonrpc" [extensions.source] type = "http" url = "http://localhost:3000" [extensions.permissions] network = true filesystem = [] ``` ## Performance Characteristics - **Latency**: 10-50ms per call (network overhead) - **Throughput**: 100-500 calls/sec per extension - **Startup**: Instant (service already running) ## Best For ✅ Existing web services ✅ Language ecosystems with good HTTP support ✅ Distributed deployments ✅ Services shared across multiple Morphir instances ❌ High-frequency local calls ❌ Offline/air-gapped environments ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Build Operations Source: https://finos.github.io/morphir-rust/contributors/design/daemon/build # Build Operations This document defines build orchestration, cleaning, and diagnostic operations for Morphir workspaces. ## Overview Build operations coordinate compilation across multiple projects: - **Dependency-ordered builds**: Compile projects in correct order - **Parallel compilation**: Build independent projects concurrently - **Incremental builds**: Only rebuild what changed - **Diagnostic aggregation**: Unified error reporting across workspace ## Build Pipeline ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Resolve │───►│ Order │───►│ Compile │───►│ Report │ │ Dependencies│ │ Projects │ │ (Parallel)│ │ Diagnostics │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ ``` ## Types ### BuildResult ```gleam /// Result of a workspace build pub type BuildResult { BuildResult( /// Overall success/failure success: Bool, /// Per-project results projects: List(ProjectBuildResult), /// Total build duration (milliseconds) duration_ms: Int, ) } /// Result for a single project pub type ProjectBuildResult { ProjectBuildResult( /// Project name name: PackagePath, /// Build status status: BuildStatus, /// Compiled distribution (if successful) distribution: Option(Distribution), /// Compilation diagnostics diagnostics: List(Diagnostic), /// Build duration (milliseconds) duration_ms: Int, ) } /// Build status for a project pub type BuildStatus { /// Build succeeded with no issues Ok /// Build succeeded with warnings Partial /// Build failed Failed /// Build was skipped (up to date) Skipped } ``` ### Diagnostic ```gleam /// Compilation diagnostic pub type Diagnostic { Diagnostic( /// Severity level severity: Severity, /// Error/warning code code: String, /// Human-readable message message: String, /// Source location location: Option(SourceLocation), /// Suggested fixes hints: List(String), ) } pub type Severity { Error Warning Info Hint } pub type SourceLocation { SourceLocation( /// File path (relative to workspace) file: String, /// Start line (1-indexed) start_line: Int, /// Start column (1-indexed) start_col: Int, /// End line end_line: Int, /// End column end_col: Int, ) } ``` ## Operations ### Build All Builds all projects in the workspace. #### Behavior 1. Resolve all dependencies 2. Compute build order (topological sort) 3. Build projects in parallel where possible 4. Aggregate diagnostics 5. Return combined result #### Build Order Projects are built in dependency order: ``` Level 0 (no deps): core Level 1 (→ core): domain, utils Level 2 (→ domain): api, cli ``` Projects at the same level can be built in parallel. #### WIT Interface ```wit /// Build all projects in workspace build-all: func() -> result>, workspace-error>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/buildAll", "params": {} } ``` **Response:** ```json { "result": { "success": true, "projects": [ { "name": "my-org/core", "status": "ok", "distribution": { "..." }, "diagnostics": [], "durationMs": 523 }, { "name": "my-org/domain", "status": "partial", "distribution": { "..." }, "diagnostics": [ { "severity": "warning", "code": "W001", "message": "Unused import: List.Extra", "location": { "file": "packages/domain/src/User.elm", "startLine": 5, "startCol": 1, "endLine": 5, "endCol": 25 } } ], "durationMs": 1247 } ], "durationMs": 1823 } } ``` #### CLI ```bash morphir build # Build all morphir build --parallel 4 # Limit parallelism morphir build --project my-org/api # Build single project ``` ### Clean Removes build artifacts and caches. #### Behavior 1. If project specified: clean that project only 2. If no project: clean entire workspace 3. Remove `.morphir-dist/` directories 4. Optionally remove dependency cache #### WIT Interface ```wit /// Clean build artifacts clean: func( /// Specific project, or all if none project: option, ) -> result<_, workspace-error>; ``` #### JSON-RPC **Request (single project):** ```json { "method": "workspace/clean", "params": { "project": "my-org/domain" } } ``` **Request (entire workspace):** ```json { "method": "workspace/clean", "params": {} } ``` **Request (include dependency cache):** ```json { "method": "workspace/clean", "params": { "includeDeps": true } } ``` #### CLI ```bash morphir clean # Clean all morphir clean my-org/domain # Clean one project morphir clean --deps # Also clean dependency cache ``` ### Get Diagnostics Returns all current diagnostics across the workspace. #### WIT Interface ```wit /// Get workspace-wide diagnostics get-diagnostics: func() -> list>>; ``` #### JSON-RPC **Request:** ```json { "method": "workspace/getDiagnostics", "params": {} } ``` **Response:** ```json { "result": { "my-org/core": [], "my-org/domain": [ { "severity": "error", "code": "E001", "message": "Type mismatch: expected Int, got String", "location": { "file": "packages/domain/src/Order.elm", "startLine": 42, "startCol": 15, "endLine": 42, "endCol": 28 }, "hints": ["Try using String.toInt to convert"] } ], "my-org/api": [ { "severity": "warning", "code": "W002", "message": "Function 'oldHelper' is deprecated", "location": { "..." } } ] } } ``` ## Incremental Builds ### Change Detection Changes are detected via: 1. **File modification time**: Compare against last build 2. **Content hash**: SHA-256 of source files 3. **Dependency changes**: Rebuild if dependency was rebuilt ### Build Cache ``` .morphir/ └── cache/ ├── my-org/ │ └── domain/ │ ├── manifest.json # Build metadata │ ├── source-hash # Hash of all sources │ └── ir-cache/ # Cached IR fragments ``` ### Manifest Format ```json { "version": "1.0.0", "lastBuild": "2026-01-16T12:00:00Z", "sourceHash": "sha256:abc123...", "files": { "src/User.elm": { "hash": "sha256:def456...", "lastModified": "2026-01-16T11:30:00Z" } }, "dependencies": { "my-org/core": "sha256:ghi789..." } } ``` ## Parallel Execution ### Strategy ``` Sequential (dependencies): core ──► domain ──► api │ Parallel (independent): core ──►─┬► domain ──► api └► utils ───►─┘ ``` ### Configuration ```toml # morphir.toml [build] parallel = true # Enable parallel builds max-workers = 4 # Maximum parallel compilations fail-fast = false # Continue on errors (or stop immediately) ``` ## Error Recovery ### Partial Builds When a project fails, dependent projects can still attempt to build using the last successful IR: ``` core (ok) ──► domain (FAILED) ──► api (uses cached domain IR) ``` ### Diagnostic-Only Mode Build without generating artifacts (fast validation): ```bash morphir build --check-only ``` ## Build Events ### workspace/onBuildStarted ```json { "method": "workspace/onBuildStarted", "params": { "projects": ["my-org/core", "my-org/domain"], "incremental": true } } ``` ### workspace/onBuildProgress ```json { "method": "workspace/onBuildProgress", "params": { "project": "my-org/domain", "phase": "compiling", "progress": 0.45, "currentFile": "src/Domain/User.elm" } } ``` ### workspace/onBuildComplete ```json { "method": "workspace/onBuildComplete", "params": { "success": true, "projects": ["my-org/core", "my-org/domain"], "durationMs": 2341, "diagnosticCount": { "errors": 0, "warnings": 3 } } } ``` ## Streaming Builds Large projects benefit from streaming compilation where results are produced incrementally rather than in one shot. This enables: - **Early feedback**: Errors appear as soon as they're discovered - **Progressive output**: Generated artifacts stream as modules complete - **Memory efficiency**: Don't hold entire project in memory - **Interruptibility**: Cancel long builds without losing partial progress ### Streaming Model ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Source │────►│ Compile │────►│ Stream │ │ Files │ │ Module │ │ Results │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ ▼ ▼ │ ┌─────────────┐ ┌─────────────┐ └───────────►│ Compile │─────►│ Stream │ │ Module │ │ Results │ └─────────────┘ └─────────────┘ ``` ### Module-Level Streaming Compilation streams results at the module level: ```json { "jsonrpc": "2.0", "method": "build/moduleCompiled", "params": { "project": "my-org/domain", "module": ["Domain", "User"], "status": "ok", "ir": { "...module IR..." }, "diagnostics": [] } } ``` ### Streaming Build Request **Request:** ```json { "jsonrpc": "2.0", "id": "build-001", "method": "workspace/buildStreaming", "params": { "projects": ["my-org/domain"], "streaming": { "granularity": "module", "includeIR": true, "includeDiagnostics": true } } } ``` **Stream of Notifications:** ```json { "method": "build/started", "params": { "project": "my-org/domain", "modules": 12 } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "Types"], "status": "ok", "ir": {...} } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "User"], "status": "ok", "ir": {...} } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "Order"], "status": "partial", "diagnostics": [...] } } ... { "method": "build/completed", "params": { "success": true, "modulesCompiled": 12 } } ``` **Final Response:** ```json { "jsonrpc": "2.0", "id": "build-001", "result": { "success": true, "modulesCompiled": 12, "durationMs": 3421 } } ``` ### CLI Streaming Output ```bash morphir build --stream ``` **Output:** ``` Building my-org/domain (12 modules) ✓ Domain.Types [42ms] ✓ Domain.User [38ms] ⚠ Domain.Order [51ms] (2 warnings) ✓ Domain.Product [29ms] ... ✓ Domain.Api [67ms] Build complete: 12 modules in 3.4s (2 warnings) ``` ### Incremental Module Compilation For watch mode and IDE integration, individual modules can be recompiled: **Request:** ```json { "jsonrpc": "2.0", "id": "compile-001", "method": "compile/module", "params": { "project": "my-org/domain", "module": ["Domain", "User"], "source": "module Domain.User exposing (..)\n\nimport Domain.Types...", "existingIR": { "...previous module IR for merge..." } } } ``` ### Dependency-Aware Streaming When a module changes, dependent modules are recompiled in order: ``` User.elm changed └─► Recompile Domain.User └─► Recompile Domain.Api (depends on User) └─► Recompile Domain.Service (depends on Api) ``` Each recompilation streams its result immediately: ```json { "method": "build/moduleCompiled", "params": { "module": ["Domain", "User"], "trigger": "source-change" } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "Api"], "trigger": "dependency-change" } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "Service"], "trigger": "dependency-change" } } ``` ## Code Generation Code generation transforms compiled IR into target language code using backend extensions. ### Target Selection The `--target` flag selects which backend to use for code generation: ```bash # Generate Spark/Scala code morphir codegen --target spark # Generate TypeScript code morphir codegen --target typescript # Generate multiple targets morphir codegen --target spark --target typescript # List available targets morphir codegen --list-targets ``` **Built-in Targets:** | Target | Flag | Output | Notes | | ----------- | ---------------------- | --------------------------- | -------------------------- | | Spark | `--target spark` | Scala (Spark DataFrame API) | Default for data pipelines | | Scala | `--target scala` | Pure Scala | General-purpose | | TypeScript | `--target typescript` | TypeScript | Web/Node.js | | JSON Schema | `--target json-schema` | JSON Schema | Type definitions only | **Extension Targets:** WASM-based backend extensions register additional targets: ```toml # morphir.toml [extensions] codegen-flink = { path = "./extensions/flink-codegen.wasm" } # Registers --target flink ``` ```bash # Use extension-provided target morphir codegen --target flink ``` ### Target Configuration Targets can be configured in `morphir.toml`: ```toml [codegen.spark] spark_version = "3.5" scala_version = "2.13" output_dir = "src/main/scala" [codegen.typescript] module_system = "esm" output_dir = "src/generated" ``` Or via CLI flags: ```bash morphir codegen --target spark --option spark_version=3.5 --output src/main/scala ``` ### Automatic Target Association Projects can configure default targets that run automatically on build: ```toml # morphir.toml [project] name = "my-org/domain" # Default targets for this project [codegen] targets = ["spark", "typescript"] # Target-specific configuration [codegen.spark] output_dir = "src/main/scala" [codegen.typescript] output_dir = "src/generated/ts" ``` With this configuration, `morphir build` automatically generates code for all configured targets: ```bash # Builds IR and generates code for spark and typescript morphir build # Skip codegen morphir build --no-codegen # Override targets morphir build --codegen-targets spark ``` ### Module-Level Target Association Associate specific modules with specific targets: ```toml # morphir.toml [codegen] # Default targets for all modules targets = ["spark"] # Override for specific modules [codegen.modules."Domain.Api"] targets = ["typescript", "json-schema"] [codegen.modules."Domain.Internal"] targets = [] # No codegen for internal modules ``` ### Pattern-Based Target Association Use glob patterns for target association: ```toml [codegen] targets = ["spark"] # All Api modules get TypeScript [[codegen.rules]] pattern = "**/Api/**" targets = ["typescript"] # Test modules don't get codegen [[codegen.rules]] pattern = "**/Test/**" targets = [] ``` ### Extension-Declared Targets Extensions can declare which targets they provide and their capabilities: ```wit // In extension's WIT interface world spark-codegen { export codegen-target { // Target metadata name: func() -> string; // "spark" description: func() -> string; // "Apache Spark DataFrame API" file-extension: func() -> string; // ".scala" // Capability flags supports-streaming: func() -> bool; supports-incremental: func() -> bool; } export morphir:extension/codegen; } ``` **Extension Registration:** ```toml # morphir.toml [extensions] codegen-spark = { path = "./extensions/spark-codegen.wasm" } # Extension automatically registers its target # Now --target spark is available ``` **Querying Available Targets:** ```json { "jsonrpc": "2.0", "id": "targets-001", "method": "codegen/listTargets", "params": {} } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "targets-001", "result": { "targets": [ { "name": "spark", "description": "Apache Spark DataFrame API", "fileExtension": ".scala", "source": "builtin", "supportsStreaming": true, "supportsIncremental": true }, { "name": "flink", "description": "Apache Flink DataStream API", "fileExtension": ".scala", "source": "extension:codegen-flink", "supportsStreaming": true, "supportsIncremental": false } ] } } ``` ### JSON-RPC Method ```json { "jsonrpc": "2.0", "id": "codegen-001", "method": "codegen/generate", "params": { "project": "my-org/domain", "target": "spark", "options": { "spark_version": "3.5", "output_dir": "src/main/scala" } } } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "codegen-001", "result": { "target": "spark", "filesGenerated": 24, "outputDir": "src/main/scala" } } ``` ## Streaming Code Generation Code generation also supports streaming to avoid generating all output at once. ### Streaming Codegen Request **Request:** ```json { "jsonrpc": "2.0", "id": "codegen-001", "method": "codegen/generateStreaming", "params": { "project": "my-org/domain", "target": "spark", "streaming": { "granularity": "module", "writeImmediately": true } } } ``` **Stream of Notifications:** ```json { "method": "codegen/started", "params": { "target": "spark", "modules": 12 } } { "method": "codegen/moduleGenerated", "params": { "module": ["Domain", "Types"], "files": ["Types.scala"] } } { "method": "codegen/moduleGenerated", "params": { "module": ["Domain", "User"], "files": ["User.scala", "UserCodecs.scala"] } } ... { "method": "codegen/completed", "params": { "filesGenerated": 24 } } ``` ### Incremental Codegen Only regenerate code for changed modules: ```bash morphir codegen --target spark --incremental ``` **Behavior:** 1. Compare module IR hashes against last codegen 2. Regenerate only changed modules 3. Stream generated files as they're produced ### Codegen Manifest Track what was generated for incremental updates: ```json { "target": "spark", "generatedAt": "2026-01-16T12:00:00Z", "modules": { "Domain.User": { "irHash": "sha256:abc123...", "files": [ { "path": "src/main/scala/domain/User.scala", "hash": "sha256:def456..." } ] } } } ``` ### Parallel Codegen Generate code for independent modules in parallel: ```toml # morphir.toml [codegen] parallel = true max-workers = 4 streaming = true # Enable streaming output ``` ## Ad-Hoc Compilation For quick experimentation and integration workloads, Morphir supports compiling code without a full project setup. This is useful for: - **Quick prototyping**: Test ideas without creating a project - **Shell pipelines**: Integrate with Unix-style workflows - **CI validation**: Check snippets in automated tests - **Code generation testing**: Validate codegen output ### Snippet Compilation Compile source code from stdin or inline. The input language must be specified (or inferred from file extension): ```bash # From stdin (language required) echo 'module Example exposing (add) add a b = a + b' | morphir compile --lang elm - # From a single file (language inferred from .elm extension) morphir compile snippet.elm # Explicit language morphir compile --lang elm snippet.elm # Multiple files morphir compile types.elm logic.elm ``` **Supported Languages:** | Language | Flag | Extensions | Notes | | ------------ | -------------------- | ------------------- | ------------------ | | Elm | `--lang elm` | `.elm` | Default frontend | | Morphir DSL | `--lang morphir-dsl` | `.morphir`, `.mdsl` | Native DSL | | (Extensions) | `--lang ` | Per extension | Via WASM frontends | **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "snippet-001", "method": "compile/snippet", "params": { "language": "elm", "source": "module Example exposing (add)\nadd a b = a + b", "options": { "moduleName": "Example", "packageName": "adhoc" } } } ``` ### Expression Evaluation Compile or evaluate standalone expressions: ```bash # Type-check an expression morphir check --expr "\\x -> x + 1" # Evaluate an expression morphir eval "List.map (\\x -> x * 2) [1, 2, 3]" ``` **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "expr-001", "method": "compile/expression", "params": { "expression": "\\x -> x + 1", "context": { "imports": [] } } } ``` ### Ad-Hoc with Dependencies Specify dependencies inline for snippets that need external packages: ```bash morphir compile snippet.elm --with-dep morphir/sdk --with-dep morphir/json ``` **JSON-RPC:** ```json { "jsonrpc": "2.0", "id": "snippet-002", "method": "compile/snippet", "params": { "language": "elm", "source": "module Example exposing (..)\nimport Json.Decode...", "options": { "dependencies": [ { "name": "morphir/sdk" }, { "name": "morphir/json", "version": "1.0.0" } ] } } } ``` ### Pipeline Compilation Chain compilation and codegen in Unix pipelines: ```bash # Compile then generate cat snippet.elm | morphir compile - | morphir codegen --target spark - # Compile multiple, stream codegen morphir compile "src/*.elm" --stream | morphir codegen --target typescript --stream - ``` ### Fragment Compilation For IDE integration, compile a code fragment within an existing module context: ```json { "jsonrpc": "2.0", "id": "frag-001", "method": "compile/fragment", "params": { "language": "elm", "fragment": "\\x -> x + 1", "context": { "modulePath": ["Domain", "User"], "imports": ["Domain.Types"], "localBindings": { "currentUser": "User" } } } } ``` This enables features like: - Hover type information - Autocomplete with context - Inline error checking See [CLI Interaction](./cli-interaction.md#ad-hoc-compilation-mode) for complete CLI documentation. ## Best Practices 1. **Use Incremental Builds**: Avoid `clean` unless necessary 2. **Parallelize**: Enable parallel builds for faster compilation 3. **Fail Fast in CI**: Use `--fail-fast` in CI to stop on first error 4. **Cache Dependencies**: Keep dependency cache for faster rebuilds 5. **Check Before Push**: Run `morphir build --check-only` before committing 6. **Stream Large Builds**: Use `--stream` for projects with many modules 7. **Incremental Codegen**: Use `--incremental` to only regenerate changed modules ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Projects](./projects.md)** - Project management within a workspace - **[Watching](./watching.md)** - File system watching for incremental builds - **[CLI Interaction](./cli-interaction.md)** - CLI-daemon communication ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## gRPC Extension Host Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/04-grpc-host # gRPC Extension Host **Status:** Draft **Version:** 0.1.0 ## Overview The gRPC Extension Host manages extensions that communicate via gRPC. This protocol provides high performance, strong typing via Protocol Buffers, and excellent cross-language support. ## Protocol ### Transport - **Transport**: gRPC over HTTP/2 - **Serialization**: Protocol Buffers - **Library**: `tonic` (Rust) ### Service Definition (Protocol Buffers) ```protobuf syntax = "proto3"; package morphir.extension; service ExtensionService { rpc Initialize(InitializeRequest) returns (InitializeResponse); rpc GetCapabilities(Empty) returns (CapabilitiesResponse); rpc Call(CallRequest) returns (CallResponse); rpc Stream(stream StreamRequest) returns (stream StreamResponse); } message InitializeRequest { string config_json = 1; } message InitializeResponse { string status = 1; } message CapabilitiesResponse { repeated Capability capabilities = 1; } message Capability { string name = 1; string description = 2; optional string params_schema = 3; optional string return_schema = 4; } message CallRequest { string method = 1; bytes params_json = 2; } message CallResponse { oneof result { bytes success_json = 1; string error = 2; } } ``` ## Implementation ### State ```rust use tonic::transport::Channel; use morphir_extension::extension_service_client::ExtensionServiceClient; pub struct GrpcExtensionHost { extensions: HashMap, next_id: u64, } struct GrpcExtension { name: String, client: ExtensionServiceClient, capabilities: Vec, } ``` ### Loading Process ```rust async fn load_extension(&mut self, config: ExtensionConfig) -> Result { let ExtensionSource::Grpc { endpoint } = config.source else { return Err(ExtensionError::InvalidSource); }; // Connect to gRPC service let channel = Channel::from_shared(endpoint)? .connect() .await?; let mut client = ExtensionServiceClient::new(channel); // Initialize client.initialize(InitializeRequest { config_json: serde_json::to_string(&config.config)?, }).await?; // Get capabilities let response = client .get_capabilities(Empty {}) .await? .into_inner(); let id = ExtensionId(self.next_id); self.next_id += 1; self.extensions.insert(id, GrpcExtension { name: config.name, client, capabilities: response.capabilities, }); Ok(id) } ``` ## Extension Implementation (Go Example) ```go package main import ( "context" "encoding/json" "log" "net" pb "morphir/extension" "google.golang.org/grpc" ) type server struct { pb.UnimplementedExtensionServiceServer } func (s *server) Initialize(ctx context.Context, req *pb.InitializeRequest) (*pb.InitializeResponse, error) { log.Printf("Initialized with config: %s", req.ConfigJson) return &pb.InitializeResponse{Status: "ready"}, nil } func (s *server) GetCapabilities(ctx context.Context, _ *pb.Empty) (*pb.CapabilitiesResponse, error) { return &pb.CapabilitiesResponse{ Capabilities: []*pb.Capability{ { Name: "transform", Description: "Transform Morphir IR", }, }, }, nil } func (s *server) Call(ctx context.Context, req *pb.CallRequest) (*pb.CallResponse, error) { var params map[string]interface{} json.Unmarshal(req.ParamsJson, ¶ms) // Process based on method result := map[string]interface{}{ "transformed": true, } resultJson, _ := json.Marshal(result) return &pb.CallResponse{ Result: &pb.CallResponse_SuccessJson{ SuccessJson: resultJson, }, }, nil } func main() { lis, err := net.Listen("tcp", ":50051") if err != nil { log.Fatalf("failed to listen: %v", err) } s := grpc.NewServer() pb.RegisterExtensionServiceServer(s, &server{}) log.Printf("Server listening at %v", lis.Addr()) if err := s.Serve(lis); err != nil { log.Fatalf("failed to serve: %v", err) } } ``` ## Configuration Example ```toml [[extensions]] name = "go-backend" enabled = true protocol = "grpc" [extensions.source] type = "grpc" endpoint = "http://localhost:50051" [extensions.permissions] network = true filesystem = [] ``` ## Performance Characteristics - **Latency**: 1-5ms per call - **Throughput**: 1,000-10,000 calls/sec per extension - **Startup**: Instant (service already running) ## Best For ✅ High-performance scenarios ✅ Strongly-typed contracts ✅ Streaming data ✅ Production services ✅ Cross-language type safety ❌ Simple scripts ❌ Rapid prototyping (protobuf overhead) ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Package Management Source: https://finos.github.io/morphir-rust/contributors/design/daemon/packages # Package Management This document defines the package format, registry backends, and publishing workflow for Morphir packages. ## Overview Morphir packages are distributable archives containing compiled IR and metadata. The package management system supports multiple registry backends, allowing packages to be published to and retrieved from existing infrastructure. | Component | Description | | -------------------- | --------------------------------------------------------- | | **Package Format** | Standardized archive containing IR, metadata, and sources | | **Registry Backend** | Pluggable storage/retrieval layer (npm, Maven, etc.) | | **Pack Command** | Creates distributable package from project | | **Publish Command** | Uploads package to configured registry | ## Package Format A Morphir package is a gzipped tarball (`.morphir.tgz`) containing the compiled distribution and metadata. The package format supports both classic (single-file) and VFS (directory tree) distribution modes. See [Distribution Structure](../ir/distributions.md) for the complete IR format specification. ### Package Contents ``` my-org-core-1.0.0.morphir.tgz ├── morphir.toml # Package metadata ├── .morphir-dist/ # Distribution (VFS mode, recommended) │ ├── format.json # Distribution manifest │ ├── pkg/ │ │ └── my-org/ │ │ └── core/ │ │ ├── module.json │ │ ├── types/ │ │ │ └── *.type.json │ │ └── values/ │ │ └── *.value.json │ └── deps/ # Dependency specifications │ └── ... ├── src/ # Source files (optional, configurable) │ └── ... └── CHANGELOG.md # Changelog (optional) ``` For simpler packages or backwards compatibility, classic mode is also supported: ``` my-org-core-1.0.0.morphir.tgz ├── morphir.toml # Package metadata ├── morphir-ir.json # Single-file distribution (classic mode) ├── src/ # Source files (optional) └── CHANGELOG.md # Changelog (optional) ``` ### Distribution Types Packages contain one of three distribution types: | Type | Description | Use Case | | --------------- | ---------------------------------------- | ------------------------------- | | **Library** | Package with full definitions | Reusable domain packages | | **Specs** | Specifications only (no implementations) | SDK bindings, FFI, native types | | **Application** | Definitions with named entry points | Executables, services, CLIs | ### Package Metadata The `morphir.toml` in a package contains distribution metadata: ```toml [package] name = "my-org/core" version = "1.0.0" description = "Core domain models for my-org" license = "Apache-2.0" repository = "https://github.com/my-org/morphir-packages" [package.authors] "Jane Doe" = "jane@my-org.com" [package.keywords] keywords = ["domain", "finance", "morphir"] # Dependencies required by this package [dependencies] "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } ``` ### Distribution Manifest The `.morphir-dist/format.json` identifies the distribution: ```json { "formatVersion": "4.0.0", "distribution": "Library", "package": "my-org/core", "version": "1.0.0", "created": "2026-01-16T12:00:00Z" } ``` **Format Version**: The `formatVersion` follows semantic versioning and corresponds to the Morphir IR specification version. Tools should check compatibility before processing. ### Classic Mode (Single-File) For backwards compatibility or simpler packages, a single `morphir-ir.json` can be used: ```json { "formatVersion": "4.0.0", "Library": { "package": { "name": "my-org/core", "version": "1.0.0" }, "def": { "modules": { "...": "..." } }, "dependencies": { "morphir/sdk": { "...": "..." } } } } ``` ## Registry Backends Morphir supports pluggable registry backends, allowing packages to be stored in existing package infrastructure. ### Backend Architecture ``` ┌──────────────────────────────────────────────────────────────────────┐ │ Morphir CLI │ ├──────────────────────────────────────────────────────────────────────┤ │ Registry Interface │ │ ┌─────────────┐ ┌───────────────┐ ┌─────────────┐ ┌──────────┐ │ │ │ npm Backend │ │GitHub Releases│ │Maven Backend│ │ (future) │ │ │ └──────┬──────┘ └───────┬───────┘ └──────┬──────┘ └──────────┘ │ └─────────┼─────────────────┼─────────────────┼────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌───────────┐ ┌───────────┐ ┌───────────┐ │npm Registry│ │ GitHub │ │Maven Repo │ │ (npmjs.org)│ │ Releases │ │ (Central) │ └───────────┘ └───────────┘ └───────────┘ ``` ### Backend Comparison | Backend | Package Format | Infrastructure | Best For | | ------------------- | -------------- | ------------------ | ------------------------------------- | | **npm** | tar.gz | Central registry | JavaScript ecosystem, public packages | | **GitHub Releases** | .morphir.tgz | Per-repository | Open source, no extra infrastructure | | **Maven** | JAR | Central repository | JVM ecosystem, enterprise | ### Registry Interface ```gleam /// Registry backend interface pub type RegistryBackend { /// Publish a package to the registry publish: fn(package: PackageArchive, config: RegistryConfig) -> Result(PublishResult, RegistryError) /// Fetch a package from the registry fetch: fn(name: PackagePath, version: SemVer, config: RegistryConfig) -> Result(PackageArchive, RegistryError) /// List available versions of a package versions: fn(name: PackagePath, config: RegistryConfig) -> Result(List(SemVer), RegistryError) /// Search for packages search: fn(query: String, config: RegistryConfig) -> Result(List(PackageInfo), RegistryError) } ``` ### npm Backend The npm backend stores Morphir packages as npm packages, leveraging existing npm infrastructure and tooling. #### Package Mapping | Morphir | npm | | -------------- | ----------------------------- | | `my-org/core` | `@morphir/my-org--core` | | `1.0.0` | `1.0.0` | | `.morphir.tgz` | `.tgz` (standard npm tarball) | #### npm Package Structure ``` package/ ├── package.json # npm metadata ├── morphir.toml # Morphir metadata ├── morphir-ir.json # Compiled IR └── src/ # Sources (if included) ``` **Generated `package.json`:** ```json { "name": "@morphir/my-org--core", "version": "1.0.0", "description": "Core domain models for my-org", "morphir": { "name": "my-org/core", "formatVersion": 4 }, "files": ["morphir.toml", "morphir-ir.json", "src/"], "keywords": ["morphir", "domain", "finance"], "license": "Apache-2.0", "repository": { "type": "git", "url": "https://github.com/my-org/morphir-packages" } } ``` #### Configuration ```toml # morphir.toml [registry] backend = "npm" [registry.npm] # npm registry URL (default: https://registry.npmjs.org) registry = "https://registry.npmjs.org" # Scope for published packages (default: @morphir) scope = "@morphir" # Package name separator (default: --) separator = "--" # Access level for scoped packages access = "public" # or "restricted" ``` #### Authentication npm authentication uses standard npm configuration: ```bash # Login to npm registry npm login --registry=https://registry.npmjs.org # Or use token-based auth npm config set //registry.npmjs.org/:_authToken=${NPM_TOKEN} ``` The Morphir CLI respects `.npmrc` for authentication. ### GitHub Releases Backend The GitHub Releases backend fetches Morphir packages from GitHub release assets. This is ideal for open source projects and organizations that want to distribute packages directly from their repositories without additional infrastructure. Unlike npm or Maven backends which use a central registry, GitHub Releases packages are fetched directly from individual repositories. Each package specifies its source repository. #### Package Discovery Packages are discovered from release assets matching the Morphir package naming convention: | Release Asset | Morphir Package | | ------------------------------- | ------------------- | | `morphir-sdk-3.0.0.morphir.tgz` | `morphir/sdk@3.0.0` | | `my-org-core-1.0.0.morphir.tgz` | `my-org/core@1.0.0` | #### Dependency Declaration Each GitHub dependency specifies its source repository: ```toml [dependencies] # Package from finos/morphir repository "morphir/sdk" = { github = "finos/morphir", tag = "sdk-v3.0.0" } # Package from a different organization's repository "acme/domain" = { github = "acme-corp/morphir-domain", tag = "v1.0.0" } # Package from a monorepo with package-specific tags "my-org/core" = { github = "my-org/morphir-packages", tag = "core-v1.0.0" } "my-org/utils" = { github = "my-org/morphir-packages", tag = "utils-v2.0.0" } # Private repository (requires authentication) "internal/models" = { github = "my-org/internal-models", tag = "v1.5.0" } ``` #### Workspace-Level Configuration Common settings can be configured at the workspace level: ```toml # morphir.toml [registry.github] # GitHub API URL (for GitHub Enterprise) api_url = "https://api.github.com" # default, or https://github.mycompany.com/api/v3 # Asset naming pattern (default shown) asset_pattern = "{name}-{version}.morphir.tgz" # Default organization for unqualified references default_owner = "my-org" ``` #### Workspace Dependencies with GitHub ```toml # workspace/morphir.toml [workspace] members = ["packages/*"] [workspace.dependencies] # Shared GitHub dependencies "morphir/sdk" = { github = "finos/morphir", tag = "sdk-v3.0.0" } "finos/morphir-json" = { github = "finos/morphir-json", tag = "v1.0.0" } ``` ```toml # workspace/packages/domain/morphir.toml [dependencies] "morphir/sdk" = { workspace = true } # Inherits github source from workspace ``` #### Authentication For private repositories, GitHub authentication uses standard methods: ```bash # Via environment variable export GITHUB_TOKEN=ghp_xxxxxxxxxxxx # Or via gh CLI authentication gh auth login # For GitHub Enterprise export GH_ENTERPRISE_TOKEN=ghp_xxxxxxxxxxxx ``` #### Checksums and Bill of Materials For security and reproducibility, GitHub releases should include checksums and optionally a Software Bill of Materials (SBOM). **Checksum file** (`CHECKSUMS.txt`): ``` sha256:abc123def456... my-org-core-1.0.0.morphir.tgz sha256:789xyz012... my-org-utils-1.0.0.morphir.tgz ``` **Morphir manifest** (`morphir-manifest.json`): ```json { "formatVersion": "4.0.0", "repository": "my-org/morphir-packages", "tag": "v1.0.0", "created": "2026-01-16T12:00:00Z", "packages": [ { "name": "my-org/core", "version": "1.0.0", "asset": "my-org-core-1.0.0.morphir.tgz", "checksum": "sha256:abc123def456...", "dependencies": [{ "name": "morphir/sdk", "version": "3.0.0" }] } ] } ``` The CLI verifies checksums when fetching packages: ```toml [registry.github] # Require checksum verification (default: true) verify_checksums = true # Checksum file name pattern checksum_file = "CHECKSUMS.txt" # Optional manifest file manifest_file = "morphir-manifest.json" ``` #### Publishing to GitHub Releases ```bash # Pack the package (generates checksum) morphir pack # Generate checksums file for all packages morphir pack --checksums dist/CHECKSUMS.txt # Generate manifest morphir pack --manifest dist/morphir-manifest.json # Create a GitHub release with package and checksums gh release create v1.0.0 \ dist/my-org-core-1.0.0.morphir.tgz \ dist/CHECKSUMS.txt \ dist/morphir-manifest.json \ --repo my-org/morphir-packages \ --title "my-org/core v1.0.0" \ --notes "Release notes" # Or use morphir publish (generates and uploads checksums automatically) morphir publish --backend github --repository my-org/morphir-packages --tag v1.0.0 ``` #### Comparison with Git Repository Dependencies | Aspect | Git Repository | GitHub Releases | | ---------- | ------------------------ | ------------------------ | | Source | Full repository clone | Release asset only | | Size | Entire repo history | Single package archive | | Speed | Slower (clone) | Faster (direct download) | | Versioning | Any git ref | Release tags only | | Use case | Development, pre-release | Published releases | For development and pre-release packages, use git repository dependencies. For published, stable releases, GitHub Releases provides a more efficient distribution mechanism. ### Maven Backend The Maven backend stores Morphir packages as JAR files in Maven repositories. #### Package Mapping | Morphir | Maven | | -------------- | ------------------------------- | | `my-org/core` | `dev.morphir:my-org-core` | | `1.0.0` | `1.0.0` | | `.morphir.tgz` | `.jar` containing Morphir files | #### JAR Structure ``` my-org-core-1.0.0.jar ├── META-INF/ │ ├── MANIFEST.MF │ └── maven/ │ └── dev.morphir/ │ └── my-org-core/ │ ├── pom.xml │ └── pom.properties ├── morphir.toml ├── morphir-ir.json └── src/ ``` **Generated `pom.xml`:** ```xml 4.0.0 dev.morphir my-org-core 1.0.0 jar my-org/core Core domain models for my-org my-org/core 4 dev.morphir morphir-sdk 3.0.0 ``` #### Configuration ```toml # morphir.toml [registry] backend = "maven" [registry.maven] # Maven repository URL repository = "https://repo1.maven.org/maven2" # For publishing (e.g., Sonatype OSSRH) snapshot_repository = "https://oss.sonatype.org/content/repositories/snapshots" release_repository = "https://oss.sonatype.org/service/local/staging/deploy/maven2" # Group ID prefix (default: dev.morphir) group_id = "dev.morphir" ``` #### Authentication Maven authentication uses standard Maven settings: ```xml ossrh ${env.MAVEN_USERNAME} ${env.MAVEN_PASSWORD} ``` ## CLI Commands ### Pack Creates a distributable package from a project. ```bash # Pack current project morphir pack # Pack specific project in workspace morphir pack --project my-org/core # Pack with specific output location morphir pack --output dist/ # Pack without sources morphir pack --no-sources # Pack for specific backend (affects format) morphir pack --backend npm morphir pack --backend maven ``` #### Behavior 1. Verify project builds successfully 2. Validate package metadata (name, version, license) 3. Compile IR to distribution format 4. Create archive based on backend format 5. Write to output directory #### Output ``` $ morphir pack Building my-org/core... Validating package metadata... Creating package archive... Created: dist/my-org-core-1.0.0.morphir.tgz Size: 45.2 KB IR Format: v4 Includes sources: yes ``` ### Publish Uploads a package to the configured registry. ```bash # Publish current project morphir publish # Publish specific package file morphir publish dist/my-org-core-1.0.0.morphir.tgz # Publish to specific registry morphir publish --registry https://npm.my-org.com # Dry run (validate without uploading) morphir publish --dry-run # Publish with specific tag (npm) morphir publish --tag beta ``` #### Behavior 1. Verify package archive exists (or run pack) 2. Validate credentials for registry 3. Check if version already exists 4. Transform package for backend (if needed) 5. Upload to registry 6. Verify upload success #### Output ``` $ morphir publish Publishing my-org/core@1.0.0 to npm... Authenticating with registry.npmjs.org... Uploading @morphir/my-org--core@1.0.0... Published: https://www.npmjs.com/package/@morphir/my-org--core ``` ### Other Commands ```bash # List published versions morphir registry versions my-org/core # Search for packages morphir registry search "domain model" # Show package info morphir registry info my-org/core@1.0.0 # Unpublish (if supported by registry) morphir registry unpublish my-org/core@1.0.0 --force ``` ## WIT Interface ```wit /// Package archive record package-archive { /// Package name name: package-path, /// Package version version: semver, /// Archive contents (tar.gz bytes) data: list, /// Checksum checksum: string, } /// Registry configuration record registry-config { /// Backend type backend: registry-backend-type, /// Registry URL url: string, /// Additional backend-specific options options: list>, } /// Backend types enum registry-backend-type { npm, github, maven, } /// Pack a project into a distributable package pack: func( project: package-path, options: pack-options, ) -> result; /// Publish a package to a registry publish: func( archive: package-archive, config: registry-config, ) -> result; /// Fetch a package from a registry fetch-package: func( name: package-path, version: semver, config: registry-config, ) -> result; ``` ## JSON-RPC ### Pack **Request:** ```json { "method": "package/pack", "params": { "project": "my-org/core", "options": { "includeSources": true, "outputDir": "dist/" } } } ``` **Response:** ```json { "result": { "path": "dist/my-org-core-1.0.0.morphir.tgz", "name": "my-org/core", "version": "1.0.0", "size": 46284, "checksum": "sha256:abc123..." } } ``` ### Publish **Request:** ```json { "method": "package/publish", "params": { "archive": "dist/my-org-core-1.0.0.morphir.tgz", "registry": { "backend": "npm", "url": "https://registry.npmjs.org" } } } ``` **Response:** ```json { "result": { "name": "my-org/core", "version": "1.0.0", "url": "https://www.npmjs.com/package/@morphir/my-org--core", "backend": "npm" } } ``` ## Configuration Reference ### Project-Level ```toml # morphir.toml [project] name = "my-org/core" version = "1.0.0" [package] # Package metadata description = "Core domain models" license = "Apache-2.0" repository = "https://github.com/my-org/core" readme = "README.md" changelog = "CHANGELOG.md" # What to include in package [package.include] sources = true # Include src/ directory readme = true # Include README changelog = true # Include CHANGELOG # Files to exclude from package [package.exclude] patterns = ["*.test.morphir", "internal/*"] [registry] # Default backend backend = "npm" [registry.npm] scope = "@my-org-morphir" access = "public" ``` ### Workspace-Level ```toml # morphir.toml (workspace root) [workspace] members = ["packages/*"] # Default registry settings for all members [registry] backend = "npm" [registry.npm] scope = "@my-org-morphir" registry = "https://npm.pkg.github.com" ``` ## Dependency Resolution with Registries When registry dependencies become available, the dependency system will integrate with registry backends: ```toml [dependencies] # Future: registry dependency "morphir/sdk" = "3.0.0" # Current: explicit registry "morphir/sdk" = { version = "3.0.0", registry = "npm" } # With specific registry URL "internal/utils" = { version = "1.0.0", registry = { backend = "npm", url = "https://npm.internal.com" } } ``` Resolution will: 1. Query configured registry backend for available versions 2. Download package archive 3. Extract to dependency cache 4. Verify checksum ## Best Practices 1. **Semantic Versioning**: Follow semver for version numbers 2. **Meaningful Descriptions**: Include clear package descriptions 3. **License Compliance**: Always specify license 4. **Minimal Packages**: Exclude test files and internal modules 5. **Changelog**: Maintain a changelog for each version 6. **CI/CD Publishing**: Automate publishing from CI pipelines ## Future Backends Additional backends may be supported in the future: | Backend | Format | Use Case | | -------------------- | ---------------------- | ----------------------------------------------- | | **OCI Registry** | Container image layers | Cloud-native deployments, artifact registries | | **S3/GCS** | Object storage | Private infrastructure, air-gapped environments | | **Artifactory** | Universal | Enterprise artifact management | | **Morphir Registry** | Native format | Dedicated Morphir ecosystem (planned) | ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Dependencies](./dependencies.md)** - Dependency resolution and caching - **[Build](./build.md)** - Build orchestration ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Release Notes Source: https://finos.github.io/morphir-rust/releases # Release Notes All notable changes to Morphir Rust are documented here. This project follows [Semantic Versioning](https://semver.org/). For the full changelog, see [CHANGELOG.md](https://github.com/finos/morphir-rust/blob/main/CHANGELOG.md) on GitHub. --- ## v0.2.0 (January 24, 2026) ### Added - **Core CLI Commands**: Promoted `compile` and `generate` from experimental to stable - `morphir compile` - Compile source code to Morphir IR using language extensions - `morphir generate` - Generate code from Morphir IR using target extensions - **TUI Pager**: Interactive JSON viewer with syntax highlighting and vim-like navigation - Visual mode (`v`, `V`) for selecting text - Yank to clipboard (`y`) with WSL, X11, Wayland, and macOS support - Word motions (`w`, `b`), line jumps (`g`, `G`), and scroll controls - **Expanded Format**: `--expanded` flag for `morphir ir migrate` produces verbose V4 output - Variables: `{"Variable": {"name": "a"}}` instead of `"a"` - References: `{"Reference": {"fqname": "...", "args": [...]}}` instead of array format - **Launcher Script**: Self-updating launcher with version management (`scripts/morphir.sh`) - Supports `.morphir-version` file for per-project version pinning - Auto-downloads correct version on first run - `morphir self upgrade` to fetch latest version - **Dev Mode**: Run morphir from local source for development and testing - Enable via `--dev` flag, `MORPHIR_DEV=1`, `local-dev` in `.morphir-version`, or `dev_mode=true` in `morphir.toml` - `morphir self dev` command to check dev mode status and configuration - Auto-detects source directory from CI environments and common locations - **Gleam Binding**: Roundtrip testing infrastructure for Gleam code - Compile Gleam to IR V4, generate back to Gleam, verify equivalence - Support for todo/panic expressions in parser ### Fixed - **VFS Consistency**: `MemoryVfs::exists()` now returns `true` for directories, matching `OsVfs` behavior - **Compile Path Resolution**: `source_directory` from config is now resolved relative to the config file location, not the current working directory ### Changed - **V4 Compact Format Improvements**: - Reference with args now uses array format: `{"Reference": ["fqname", arg1, ...]}` - Type variables are bare name strings in compact mode: `"a"` - References without args are bare FQName strings: `"morphir/sdk:int#int"` - **V4 Canonical Naming**: `Name` type now uses kebab-case by default (e.g., `my-function`) - **Documentation Site**: Restructured with just-the-docs theme and morphir.finos.org branding ## v0.1.0 (January 23, 2026) ### Added - Initial release of the Morphir Rust CLI toolchain - **IR Versioning**: Support for both Classic and V4 Morphir IR formats - **Remote Source Support**: IR migration can fetch from URLs, GitHub releases, and archives - **Extension System**: Plugin architecture using Extism with JSON-RPC communication - **Morphir Daemon**: Background service for workspace management and IDE integration - **CLI Commands**: - `morphir validate` - Validate Morphir IR models - `morphir generate` - Generate code from Morphir IR - `morphir transform` - Transform Morphir IR - `morphir tool` - Manage Morphir tools (install/list/update/uninstall) - `morphir dist` - Manage Morphir distributions - `morphir extension` - Manage Morphir extensions - `morphir ir migrate` - Migrate IR between versions - `morphir schema` - Generate JSON Schema for Morphir IR - `morphir version` - Print version info (supports `--json` for machine-readable output) - **Multi-platform Binaries**: Pre-built releases for Linux (x86_64, aarch64, musl), macOS (x86_64, aarch64), and Windows (x86_64, aarch64) - **cargo-binstall Support**: Install pre-built binaries via `cargo binstall morphir` - **WASM Bindings**: WebAssembly backend for browser and edge deployments - **Gleam Binding**: Language binding for Gleam frontend/backend --- ## Upgrading To upgrade to the latest version: ```bash # Using the launcher morphir self upgrade # Using mise mise upgrade github:finos/morphir-rust # Using cargo cargo install --git https://github.com/finos/morphir-rust morphir --force ``` ## Reporting Issues Found a bug or have a feature request? Please [open an issue](https://github.com/finos/morphir-rust/issues/new) on GitHub. --- ## Stdio Extension Host Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/05-stdio-host # Stdio Extension Host **Status:** Draft **Version:** 0.1.0 ## Overview The Stdio Extension Host manages extensions that communicate via JSON Lines over stdin/stdout. This is the simplest protocol, ideal for scripts and command-line tools written in any language. ## Protocol ### Transport - **Input**: JSON Lines written to stdin - **Output**: JSON Lines read from stdout - **Errors**: Human-readable text written to stderr (logged by host) ### Message Format #### Request (Host → Extension) ```json {"id": 1, "method": "transform", "params": {"ir": {...}}} ``` Fields: - `id` (number): Unique request ID for matching responses - `method` (string): Method name to invoke - `params` (any): Method parameters #### Response (Extension → Host) Success: ```json { "id": 1, "result": { "transformed": true } } ``` Error: ```json { "id": 1, "error": "Invalid IR structure" } ``` Fields: - `id` (number): Matches request ID - `result` (any): Success result (mutually exclusive with error) - `error` (string): Error message (mutually exclusive with result) #### Notification (Extension → Host, one-way) ```json { "method": "log", "params": { "level": "info", "message": "Processing..." } } ``` No `id` field, no response expected. ## Implementation ### Architecture ``` ┌──────────────────┐ │ StdioExtensionHost│ │ (Kameo Actor) │ └──────────┬─────────┘ │ ├──► Extension 1 (Process) │ ├─ stdin writer task │ ├─ stdout reader task │ └─ stderr reader task │ ├──► Extension 2 (Process) │ └─ ... │ └──► Extension N (Process) ``` ### State ```rust pub struct StdioExtensionHost { extensions: HashMap, next_id: u64, } struct StdioExtension { name: String, process: Child, request_tx: mpsc::Sender<(StdioRequest, Option>)>, pending: HashMap>, next_request_id: u64, capabilities: Vec, } ``` ### Concurrency Model Each extension has three async tasks: 1. **Stdin Writer**: Receives requests from host, writes JSON lines to stdin 2. **Stdout Reader**: Reads JSON lines from stdout, routes responses to pending requests 3. **Stderr Reader**: Reads text from stderr, logs with extension name prefix Communication flow: ``` Host.call() ↓ Create oneshot channel ↓ Send (request, response_tx) to stdin writer ↓ Store response_tx in pending map ↓ Stdin writer serializes and writes ↓ ... extension processes ... ↓ Stdout reader parses response ↓ Lookup response_tx by request id ↓ Send response through oneshot ↓ Host.call() returns ``` ### Loading Process ```rust async fn load_extension(&mut self, config: ExtensionConfig) -> Result { // 1. Extract command and args let ExtensionSource::Process { command, args, env } = config.source else { return Err(ExtensionError::InvalidSource); }; // 2. Spawn process let mut child = Command::new(&command) .args(&args) .envs(&env) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::piped()) .kill_on_drop(true) // Kill on host drop .spawn()?; // 3. Take stdio handles let stdin = child.stdin.take().unwrap(); let stdout = child.stdout.take().unwrap(); let stderr = child.stderr.take().unwrap(); // 4. Create channels let (request_tx, request_rx) = mpsc::channel(32); let (response_tx, mut response_rx) = mpsc::channel(32); // 5. Spawn tasks tokio::spawn(stdin_writer_task(stdin, request_rx)); tokio::spawn(stdout_reader_task(stdout, response_tx)); tokio::spawn(stderr_reader_task(stderr, config.name.clone())); // 6. Send initialize let init_response = self.send_request( &request_tx, 0, "initialize", config.config.clone(), ).await?; // 7. Query capabilities let caps_response = self.send_request( &request_tx, 1, "capabilities", Value::Null, ).await?; let capabilities = serde_json::from_value(caps_response.result.unwrap_or_default())?; // 8. Store and return let id = ExtensionId(self.next_id); self.next_id += 1; self.extensions.insert(id, StdioExtension { name: config.name, process: child, request_tx, pending: HashMap::new(), next_request_id: 2, capabilities, }); Ok(id) } ``` ### Call Process ```rust async fn call( &mut self, extension_id: ExtensionId, method: &str, params: Value, ) -> Result { let ext = self.extensions .get_mut(&extension_id) .ok_or(ExtensionError::NotFound(extension_id))?; let request_id = ext.next_request_id; ext.next_request_id += 1; let request = StdioRequest { id: request_id, method: method.to_string(), params, }; let (response_tx, response_rx) = oneshot::channel(); ext.pending.insert(request_id, response_tx); // Send request ext.request_tx .send((request, None)) .await .map_err(|_| ExtensionError::ProtocolError("Channel closed".into()))?; // Wait for response with timeout let response = tokio::time::timeout( Duration::from_secs(30), response_rx ).await .map_err(|_| ExtensionError::Timeout)? .map_err(|_| ExtensionError::ProtocolError("Response channel closed".into()))?; ext.pending.remove(&request_id); // Check for error if let Some(error) = response.error { return Err(ExtensionError::ExtensionError(error)); } Ok(response.result.unwrap_or(Value::Null)) } ``` ## Extension Implementation Guide ### Minimal Python Extension ```python #!/usr/bin/env python3 import sys import json def handle_initialize(params): return {"status": "ready"} def handle_capabilities(params): return [ {"name": "transform", "description": "Transform IR"}, ] def handle_transform(params): # Process IR return {"result": params} HANDLERS = { "initialize": handle_initialize, "capabilities": handle_capabilities, "transform": handle_transform, } def main(): for line in sys.stdin: try: request = json.loads(line) method = request["method"] params = request.get("params", {}) if method in HANDLERS: result = HANDLERS[method](params) response = {"id": request["id"], "result": result} else: response = {"id": request["id"], "error": f"Unknown method: {method}"} print(json.dumps(response), flush=True) except Exception as e: error_response = {"id": request.get("id", 0), "error": str(e)} print(json.dumps(error_response), flush=True) print(f"Error: {e}", file=sys.stderr, flush=True) if __name__ == "__main__": main() ``` ### Minimal Go Extension ```go package main import ( "bufio" "encoding/json" "fmt" "os" ) type Request struct { ID uint64 `json:"id"` Method string `json:"method"` Params json.RawMessage `json:"params"` } type Response struct { ID uint64 `json:"id"` Result interface{} `json:"result,omitempty"` Error *string `json:"error,omitempty"` } func handleInitialize(params json.RawMessage) (interface{}, error) { return map[string]string{"status": "ready"}, nil } func handleCapabilities(params json.RawMessage) (interface{}, error) { return []map[string]string{ {"name": "transform", "description": "Transform IR"}, }, nil } func handleTransform(params json.RawMessage) (interface{}, error) { var input map[string]interface{} json.Unmarshal(params, &input) return map[string]interface{}{"result": input}, nil } func main() { scanner := bufio.NewScanner(os.Stdin) encoder := json.NewEncoder(os.Stdout) handlers := map[string]func(json.RawMessage) (interface{}, error){ "initialize": handleInitialize, "capabilities": handleCapabilities, "transform": handleTransform, } for scanner.Scan() { var req Request if err := json.Unmarshal(scanner.Bytes(), &req); err != nil { fmt.Fprintf(os.Stderr, "Parse error: %v\n", err) continue } handler, ok := handlers[req.Method] if !ok { errStr := fmt.Sprintf("Unknown method: %s", req.Method) encoder.Encode(Response{ID: req.ID, Error: &errStr}) continue } result, err := handler(req.Params) if err != nil { errStr := err.Error() encoder.Encode(Response{ID: req.ID, Error: &errStr}) } else { encoder.Encode(Response{ID: req.ID, Result: result}) } } } ``` ### Minimal Gleam Extension ```gleam import gleam/io import gleam/json import gleam/dynamic import gleam/result import gleam/string pub type Request { Request(id: Int, method: String, params: json.Json) } pub type Response { Response(id: Int, result: Result(json.Json, String)) } fn handle_initialize(_params: json.Json) -> Result(json.Json, String) { json.object([ #("status", json.string("ready")), ]) |> Ok } fn handle_capabilities(_params: json.Json) -> Result(json.Json, String) { json.array([ json.object([ #("name", json.string("transform")), #("description", json.string("Transform Morphir IR")), ]), ]) |> Ok } fn handle_transform(params: json.Json) -> Result(json.Json, String) { json.object([ #("transformed", json.bool(True)), #("output", params), ]) |> Ok } pub fn main() { read_loop() } fn read_loop() { case io.get_line("") { Ok(line) -> { case handle_line(line) { Ok(response) -> { io.println(json.to_string(response)) read_loop() } Error(err) -> { io.println_error(err) read_loop() } } } Error(_) -> Nil } } fn handle_line(line: String) -> Result(json.Json, String) { use request <- result.try(parse_request(line)) let result = case request.method { "initialize" -> handle_initialize(request.params) "capabilities" -> handle_capabilities(request.params) "transform" -> handle_transform(request.params) _ -> Error("Unknown method: " <> request.method) } encode_response(request.id, result) } ``` ## Configuration Example ```toml [[extensions]] name = "python-validator" enabled = true protocol = "stdio" [extensions.source] type = "process" command = "python3" args = ["./extensions/validator.py"] [extensions.permissions] network = false filesystem = [] [extensions.restart] strategy = "exponential" initial_delay = "1s" max_delay = "30s" max_retries = 3 ``` ## Testing ### Manual Testing ```bash # Test extension directly echo '{"id":1,"method":"initialize","params":{}}' | python3 extension.py echo '{"id":2,"method":"capabilities","params":{}}' | python3 extension.py echo '{"id":3,"method":"transform","params":{"ir":{}}}' | python3 extension.py ``` ### Integration Test ```rust #[tokio::test] async fn test_stdio_extension() { let mut host = StdioExtensionHost::new(); host.initialize().await.unwrap(); let config = ExtensionConfig { name: "test".into(), source: ExtensionSource::Process { command: "python3".into(), args: vec!["./tests/fixtures/echo.py".into()], env: HashMap::new(), }, permissions: Permissions::default(), config: json!({}), restart: RestartStrategy::Never, }; let id = host.load_extension(config).await.unwrap(); let result = host.call(id, "echo", json!({"msg": "hello"})).await.unwrap(); assert_eq!(result, json!({"msg": "hello"})); host.unload_extension(id).await.unwrap(); } ``` ## Performance Characteristics - **Latency**: 5-20ms per call (depends on language startup) - **Throughput**: 50-100 calls/sec per extension - **Memory**: 10-100MB per extension (process overhead) - **Startup**: 100-1000ms (depends on language) ## Best For ✅ Scripts and command-line tools ✅ Rapid prototyping ✅ Legacy tool integration ✅ Any language with stdin/stdout ✅ Debugging (easy to test manually) ❌ High-throughput scenarios ❌ Low-latency requirements ❌ Large data transfers (JSON serialization overhead) ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Configuration System Source: https://finos.github.io/morphir-rust/contributors/design/daemon/configuration # Configuration System This module describes the Morphir configuration system design, covering project configuration, workspace configuration, and the merge/resolution rules. ## Overview Morphir uses a unified configuration file (`morphir.toml`) that supports both single-project and multi-project (workspace) modes. The same file format handles: - **Project mode**: Single project with its own configuration - **Workspace mode**: Multiple projects with shared configuration ## Configuration File All configuration lives in `morphir.toml` (or alternative locations). There is no separate workspace configuration file. ``` my-workspace/ ├── morphir.toml # Workspace-level config (has [workspace] section) ├── .config/ │ └── morphir/ │ └── config.toml # Workspace-level shared config (gitignored secrets, local overrides) ├── packages/ │ ├── core/ │ │ ├── morphir.toml # Project-level config │ │ └── src/ │ └── domain/ │ ├── morphir.toml # Project-level config │ └── src/ ``` ### Mode Detection The presence of specific sections determines the mode: | Section Present | Mode | Behavior | | ---------------- | ------------------------ | -------------------------------------------- | | `[project]` only | Project | Single project configuration | | `[workspace]` | Workspace | Multi-project with member discovery | | Both | Workspace + Root Project | Workspace with the root also being a project | ## Module Reference | Module | Description | | --------------------------------------- | ---------------------------------------------- | | [morphir.toml](./morphir-toml.md) | Configuration file structure and sections | | [Workspace Mode](./workspace-config.md) | Workspace-specific configuration and discovery | | [Tasks](../extensions/tasks.md) | Task definitions and pre/post hooks | | [Merge Rules](./merge-rules.md) | Configuration inheritance and merge behavior | | [Environment](./environment.md) | Environment variables and runtime overrides | ## Key Sections ### Core Configuration ```toml [morphir] version = "^4.0.0" # IR version constraint [project] name = "my-org/my-project" version = "1.0.0" source_directory = "src" exposed_modules = ["Domain.User", "Domain.Order"] # Frontend/language configuration [frontend] language = "elm" # Default: "elm", also "morphir-dsl", or extension-provided ``` ### Frontend Configuration The `[frontend]` section specifies the input language and parser: ```toml [frontend] language = "elm" # Source language # Language-specific options [frontend.elm] elm_version = "0.19" [frontend.morphir-dsl] strict_mode = true ``` **Automatic Language Detection:** When `language` is not specified, Morphir infers it from file extensions: | Extension | Language | | ------------------- | ------------- | | `.elm` | Elm | | `.morphir`, `.mdsl` | Morphir DSL | | (Extension-defined) | Per extension | **Extension-Provided Frontends:** ```toml [extensions] frontend-ocaml = { path = "./extensions/ocaml-frontend.wasm" } [frontend] language = "ocaml" # Now available via extension [frontend.ocaml] ocaml_version = "5.0" ``` **Pattern-Based Language Selection:** For projects with multiple source languages: ```toml [frontend] language = "elm" # Default [[frontend.rules]] pattern = "src/legacy/**/*.morphir" language = "morphir-dsl" [[frontend.rules]] pattern = "src/experimental/**/*.ml" language = "ocaml" ``` ### Workspace Configuration ```toml [workspace] output_dir = ".morphir" members = ["packages/*"] # Glob patterns for project discovery exclude = ["packages/old-*"] default_member = "packages/core" ``` ### Build & Codegen ```toml [ir] format_version = 4 strict_mode = false [codegen] targets = ["typescript", "scala"] output_format = "pretty" ``` ### Tasks Built-in tasks (`build`, `test`, `check`, `codegen`, `pack`, `publish`) work automatically. Users define custom tasks or hooks: ```toml [tasks] # Custom task integration = "./scripts/integration-tests.sh" # CI pipeline using built-in tasks [tasks.ci] description = "Run CI pipeline" depends = ["check", "test", "build"] # Pre/post hooks extend built-in tasks [tasks."post:build"] run = "prettier --write .morphir-dist/" ``` See [Tasks](./tasks.md) for full documentation. ## Configuration Locations Configuration can be placed in multiple locations: | Location | Scope | Typical Use | | ------------------------------- | ----------------- | ------------------------------------------- | | `./morphir.toml` | Project/Workspace | Primary configuration (committed) | | `./.morphir/morphir.toml` | Project/Workspace | Alternative location | | `./.config/morphir/config.toml` | Workspace | Local overrides, secrets (often gitignored) | | `~/.config/morphir/config.toml` | User | User-level defaults and preferences | | `/etc/morphir/config.toml` | System | System-wide defaults | ## Configuration Resolution Configuration is resolved from multiple sources with the following precedence (highest first): 1. **Command-line flags**: `--config-key=value` 2. **Environment variables**: `MORPHIR__SECTION__KEY` 3. **Workspace local config**: `./.config/morphir/config.toml` 4. **Project/Workspace config**: `./morphir.toml` or `./.morphir/morphir.toml` 5. **Parent configs**: Walk up directory tree (for nested projects) 6. **User config**: `~/.config/morphir/config.toml` 7. **System config**: `/etc/morphir/config.toml` The `.config/morphir/config.toml` within a workspace is useful for: - Local developer overrides (gitignored) - Secrets and credentials - Machine-specific paths - CI/CD environment-specific settings See [Merge Rules](./merge-rules.md) for detailed merge semantics. ## Workspace vs Project Config | Aspect | Project Config | Workspace Config | | ------------ | ----------------------------- | -------------------------------- | | Scope | Single package | Multiple packages | | File | `morphir.toml` in project dir | `morphir.toml` at workspace root | | Key section | `[project]` | `[workspace]` | | Dependencies | Per-project | Can be shared | | Build output | Per-project `.morphir-dist/` | Workspace `.morphir/` | ## Design Principles 1. **Single File Format**: One `morphir.toml` format for all modes 2. **Explicit Over Implicit**: Workspace mode requires explicit `[workspace]` section 3. **Inheritance**: Child projects inherit from parent workspace config 4. **Override**: More specific config overrides less specific 5. **Simple Tasks**: Tasks are shell commands, not a DSL; pre/post hooks extend built-in commands 6. **Separation of Concerns**: Committed config vs local overrides (`.config/`) ## Related Documents ### Morphir Rust Design Documents - [morphir.toml Specification](./morphir-toml.md) - Complete configuration file specification - [Merge Rules](./merge-rules.md) - Configuration inheritance and merge behavior - [Workspace Config](./workspace-config.md) - Multi-project workspace configuration - [Morphir Daemon](./README.md) - Daemon overview and architecture ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents --- ## Extism WASM Extension Host Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/06-extism-wasm-host # Extism WASM Extension Host **Status:** Draft **Version:** 0.1.0 ## Overview The Extism WASM Extension Host manages extensions compiled to WebAssembly using the Extism framework. This provides strong sandboxing, portability, and support for many languages. ## Protocol ### Runtime - **Runtime**: Extism (based on Wasmtime) - **Supported Languages**: Rust, Go, JavaScript/TypeScript, Python, C/C++, Zig, AssemblyScript, and more - **Sandboxing**: Full WASM isolation with configurable memory/CPU limits ### Interface Extensions export functions that accept and return JSON strings: ```rust // Extension exports these functions fn initialize(config: String) -> String; fn capabilities() -> String; fn transform(params: String) -> String; // ... other methods ``` ## Implementation ### State ```rust use extism::{Manifest, Plugin, Wasm}; pub struct ExtismExtensionHost { extensions: HashMap, next_id: u64, } struct ExtismExtension { name: String, plugin: Plugin, capabilities: Vec, } ``` ### Loading Process ```rust async fn load_extension(&mut self, config: ExtensionConfig) -> Result { let ExtensionSource::Wasm { path } = config.source else { return Err(ExtensionError::InvalidSource); }; // Load WASM module let manifest = Manifest::new([Wasm::file(&path)]); let mut plugin = Plugin::new(&manifest, [], true)?; // Initialize let config_json = serde_json::to_string(&config.config)?; plugin.call("initialize", config_json)?; // Get capabilities let caps_json = plugin.call("capabilities", "")?; let capabilities: Vec = serde_json::from_str(&caps_json)?; let id = ExtensionId(self.next_id); self.next_id += 1; self.extensions.insert(id, ExtismExtension { name: config.name, plugin, capabilities, }); Ok(id) } async fn call(&mut self, extension_id: ExtensionId, method: &str, params: Value) -> Result { let ext = self.extensions .get_mut(&extension_id) .ok_or(ExtensionError::NotFound(extension_id))?; let params_json = serde_json::to_string(¶ms)?; // Call WASM function (blocking) let result_json = tokio::task::spawn_blocking({ let mut plugin = ext.plugin.clone(); let method = method.to_string(); move || plugin.call(&method, params_json) }).await??; let result: Value = serde_json::from_str(&result_json)?; Ok(result) } ``` ## Extension Implementation (Rust + Extism PDK) ```rust use extism_pdk::*; use serde::{Deserialize, Serialize}; use serde_json::Value; #[derive(Deserialize)] struct TransformParams { ir: Value, } #[derive(Serialize)] struct TransformResult { transformed: bool, output: Value, } #[plugin_fn] pub fn initialize(config: String) -> FnResult { Ok(r#"{"status": "ready"}"#.to_string()) } #[plugin_fn] pub fn capabilities() -> FnResult { Ok(r#"[{"name": "transform", "description": "Transform IR"}]"#.to_string()) } #[plugin_fn] pub fn transform(params_json: String) -> FnResult { let params: TransformParams = serde_json::from_str(¶ms_json)?; // Transform IR let result = TransformResult { transformed: true, output: params.ir, }; Ok(serde_json::to_string(&result)?) } ``` ## Extension Implementation (Go + Extism PDK) ```go package main import ( "encoding/json" "github.com/extism/go-pdk" ) //export initialize func initialize() int32 { result := map[string]string{"status": "ready"} json, _ := json.Marshal(result) mem := pdk.AllocateString(string(json)) pdk.OutputMemory(mem) return 0 } //export capabilities func capabilities() int32 { caps := []map[string]string{ {"name": "transform", "description": "Transform IR"}, } json, _ := json.Marshal(caps) mem := pdk.AllocateString(string(json)) pdk.OutputMemory(mem) return 0 } //export transform func transform() int32 { input := pdk.InputString() var params map[string]interface{} json.Unmarshal([]byte(input), ¶ms) result := map[string]interface{}{ "transformed": true, "output": params["ir"], } output, _ := json.Marshal(result) mem := pdk.AllocateString(string(output)) pdk.OutputMemory(mem) return 0 } func main() {} ``` ## Building Extensions ### Rust ```bash cargo build --target wasm32-unknown-unknown --release ``` ### Go ```bash tinygo build -o extension.wasm -target wasi main.go ``` ## Configuration Example ```toml [[extensions]] name = "wasm-optimizer" enabled = true protocol = "extism" [extensions.source] type = "wasm" path = "./extensions/optimizer.wasm" [extensions.permissions] max_memory = "100MB" max_execution_time = "5s" ``` ## Performance Characteristics - **Latency**: < 1ms per call (after first call) - **Throughput**: 10,000+ calls/sec - **Memory**: 1-10MB per extension - **Startup**: 10-100ms (module compilation) ## Best For ✅ CPU-bound transformations ✅ Untrusted code (strong sandbox) ✅ Portable extensions ✅ High performance ✅ Near-native speed ❌ Heavy I/O operations ❌ Extensions needing network access ❌ Large memory requirements ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## WASM Component Model Extension Host Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/07-wasm-component-host # WASM Component Model Extension Host **Status:** Draft **Version:** 0.1.0 ## Overview The WASM Component Model Extension Host manages extensions using the Component Model specification. This is the next-generation WASM standard with native support for rich types, interfaces, and composition. ## Protocol ### Runtime - **Runtime**: Wasmtime with Component Model support - **Interface Definition**: WIT (WebAssembly Interface Types) - **Supported Languages**: Rust (best support), C, C++, more coming ### Interface Definition (WIT) ```wit // morphir-extension.wit package morphir:extension; interface extension { record capability { name: string, description: string, params-schema: option, return-schema: option, } record init-result { status: string, } initialize: func(config: string) -> init-result; capabilities: func() -> list; transform: func(params: string) -> result; } world morphir-extension { export extension; } ``` ## Implementation ### State ```rust use wasmtime::{ component::{Component, Linker, ResourceTable}, Config, Engine, Store, }; use wasmtime_wasi::WasiView; pub struct WasmComponentHost { engine: Engine, linker: Linker, extensions: HashMap, next_id: u64, } struct WasmComponentExtension { name: String, instance: MorphirExtension, store: Store, } struct ComponentState { table: ResourceTable, wasi: WasiCtx, } ``` ### Loading Process ```rust async fn load_extension(&mut self, config: ExtensionConfig) -> Result { let ExtensionSource::WasmComponent { path } = config.source else { return Err(ExtensionError::InvalidSource); }; // Load component let component = Component::from_file(&self.engine, &path)?; // Create store with WASI let mut store = Store::new( &self.engine, ComponentState { table: ResourceTable::new(), wasi: WasiCtxBuilder::new().build(), }, ); // Instantiate let (instance, _) = MorphirExtension::instantiate_async( &mut store, &component, &self.linker, ).await?; // Initialize instance.call_initialize(&mut store, &config_json).await?; // Get capabilities let caps = instance.call_capabilities(&mut store).await?; let id = ExtensionId(self.next_id); self.next_id += 1; self.extensions.insert(id, WasmComponentExtension { name: config.name, instance, store, }); Ok(id) } ``` ## Extension Implementation (Rust) ```rust // Cargo.toml [dependencies] wit-bindgen = "0.16" // lib.rs wit_bindgen::generate!({ world: "morphir-extension", exports: { "morphir:extension/extension": Extension, } }); struct Extension; impl Guest for Extension { fn initialize(config: String) -> InitResult { InitResult { status: "ready".to_string(), } } fn capabilities() -> Vec { vec![ Capability { name: "transform".to_string(), description: "Transform Morphir IR".to_string(), params_schema: None, return_schema: None, } ] } fn transform(params: String) -> Result { let input: serde_json::Value = serde_json::from_str(¶ms) .map_err(|e| e.to_string())?; // Transform logic let output = serde_json::json!({ "transformed": true, "output": input, }); Ok(serde_json::to_string(&output).unwrap()) } } ``` ## Building ```bash # Build component cargo component build --release # Result: target/wasm32-wasi/release/extension.wasm ``` ## Configuration Example ```toml [[extensions]] name = "component-backend" enabled = true protocol = "component" [extensions.source] type = "wasm-component" path = "./extensions/backend.wasm" [extensions.permissions] max_memory = "100MB" max_execution_time = "10s" ``` ## Performance Characteristics - **Latency**: < 1ms per call - **Throughput**: 10,000+ calls/sec - **Memory**: 1-10MB per extension - **Startup**: 50-200ms ## Best For ✅ Future-proof extensions ✅ Rich type systems ✅ Interface composition ✅ Strong sandboxing ✅ High performance ❌ Limited language support (currently) ❌ Requires recent toolchain ❌ More complex than Extism ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Workspace Mode Configuration Source: https://finos.github.io/morphir-rust/contributors/design/daemon/workspace-config # Workspace Mode Configuration This document describes workspace-specific configuration in `morphir.toml`. ## Overview A workspace is a collection of related Morphir projects managed together. Workspace mode is activated by including a `[workspace]` section in the configuration file. ## Configuration File Locations The configuration file can be placed in any of these equivalent locations: | Location | Notes | | ----------------------------- | ------------------------ | | `morphir.toml` | Root level, most visible | | `.morphir/config.toml` | Hidden directory variant | | `.morphir/morphir.toml` | Hidden directory variant | | `.config/morphir/config.toml` | XDG-style location | All locations use the same format and are functionally equivalent. The CLI searches in this order and uses the first file found. ## Enabling Workspace Mode ```toml # morphir.toml (or any equivalent location) [workspace] members = ["packages/*"] ``` The presence of `[workspace]` section triggers workspace mode, enabling: - Multi-project discovery and management - Shared dependency resolution - Coordinated builds - Workspace-wide configuration inheritance ## Workspace Section ### `[workspace]` ```toml [workspace] # Workspace root directory (empty = directory containing this file) root = "" # Output directory for workspace-level artifacts output_dir = ".morphir" # Glob patterns for discovering member projects members = [ "packages/*", "libs/*", "apps/*" ] # Patterns to exclude from discovery exclude = [ "packages/deprecated-*", "packages/experimental/*" ] # Default member when no project is specified default_member = "packages/core" ``` ### Field Reference | Field | Type | Default | Description | | ---------------- | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------ | | `root` | string | `""` | Workspace root directory. Empty means the directory containing the config file. | | `output_dir` | string | `".morphir"` | Output directory for generated artifacts, relative to workspace root. | | `members` | string[] | `[]` | Glob patterns for discovering workspace member projects. Each matching directory with a `morphir.toml` becomes a member. | | `exclude` | string[] | `[]` | Glob patterns excluded from member discovery. | | `default_member` | string | `""` | Default member path when none is specified in commands. | ## Project Discovery Projects are discovered by scanning directories matching `members` patterns: ``` workspace/ ├── morphir.toml # [workspace] members = ["packages/*", "apps/*"] ├── packages/ │ ├── core/ # ✓ Discovered (matches packages/*) │ │ └── morphir.toml │ ├── domain/ # ✓ Discovered (matches packages/*) │ │ └── morphir.toml │ └── deprecated-old/ # ✗ Excluded (matches deprecated-*) │ └── morphir.toml ├── apps/ │ └── api/ # ✓ Discovered (matches apps/*) │ └── morphir.toml └── tools/ └── generator/ # ✗ Not discovered (no matching pattern) └── morphir.toml ``` ### Discovery Algorithm 1. Resolve `members` globs relative to workspace root 2. Filter out paths matching `exclude` patterns 3. For each remaining directory, check for configuration file 4. Parse project config and register as workspace member ## Workspace + Project Mode A workspace root can also be a project itself: ```toml # morphir.toml - Both workspace and project [workspace] members = ["packages/*"] [project] name = "my-org/workspace-root" version = "1.0.0" source_directory = "src" ``` This is useful when: - The workspace root contains shared code - You want a "meta" project that depends on all members - Monorepo patterns where root is also a package ## Configuration Inheritance Member projects inherit configuration from the workspace: ```toml # workspace/morphir.toml [workspace] members = ["packages/*"] [ir] format_version = 4 strict_mode = true [codegen] targets = ["typescript"] ``` ```toml # workspace/packages/core/morphir.toml [project] name = "my-org/core" version = "1.0.0" # Inherits [ir] and [codegen] from workspace # Can override specific fields: [codegen] targets = ["typescript", "scala"] # Overrides workspace default ``` ### Inheritance Rules | Section | Inheritance Behavior | | ------------- | -------------------------------------- | | `[morphir]` | Merged (project can constrain further) | | `[project]` | Not inherited (project-specific) | | `[workspace]` | Not inherited (workspace-level only) | | `[ir]` | Merged (project overrides workspace) | | `[codegen]` | Merged (project overrides workspace) | | `[cache]` | Merged | | `[logging]` | Merged | | `[tasks]` | Merged (project can add/override) | ## Shared Dependencies Workspace-level dependencies can be defined for sharing across projects. This ensures consistent dependency versions across all workspace members. ```toml # workspace/morphir.toml [workspace] members = ["packages/*"] [workspace.dependencies] # Repository dependencies (recommended) "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } "finos/morphir-json" = { git = "https://github.com/finos/morphir-json.git", tag = "v1.0.0" } # Path dependencies for local packages "my-org/shared" = { path = "packages/shared" } ``` Member projects reference shared dependencies using workspace inheritance: ```toml # workspace/packages/domain/morphir.toml [project] name = "my-org/domain" [dependencies] "morphir/sdk" = { workspace = true } # Inherit from workspace "my-org/shared" = { workspace = true } # Local workspace member "other/lib" = { path = "../other" } # Project-specific path dependency ``` See [Dependency Management](./dependencies.md) for full documentation on dependency sources, resolution, and caching. ## CLI Workspace Awareness The CLI is **workspace-aware** regardless of the current working directory. When running commands from any subdirectory within a workspace, the CLI: 1. **Walks up** the directory tree to find the workspace root (directory with `[workspace]` in config) 2. **Loads** workspace configuration and discovers all member projects 3. **Determines context** based on current directory: - If in a member project directory: operates on that project by default - If in workspace root: operates on workspace or default member - If in non-member subdirectory: operates in workspace context ### Workspace Discovery ``` workspace/ # Workspace root (config has [workspace]) ├── morphir.toml ├── packages/ │ ├── core/ │ │ └── morphir.toml # Running `morphir build` here builds core │ └── domain/ │ ├── morphir.toml │ └── src/ │ └── User/ # Running `morphir build` here still builds domain └── docs/ # Running `morphir workspace list` here works ``` ### Context Resolution ```bash # From workspace root ~/workspace$ morphir build # Builds default_member or prompts ~/workspace$ morphir workspace build # Builds all members # From member project ~/workspace/packages/core$ morphir build # Builds core ~/workspace/packages/core$ morphir workspace build # Builds all members ~/workspace/packages/core$ morphir workspace info # Shows workspace info # From deep within a project ~/workspace/packages/domain/src/User$ morphir build # Builds domain ~/workspace/packages/domain/src/User$ morphir workspace list # Lists all members # From non-member directory within workspace ~/workspace/docs$ morphir workspace list # Works - finds workspace root ~/workspace/docs$ morphir build # Error or uses default_member ``` ### Explicit Workspace/Project Selection ```bash # Override automatic detection morphir build --project my-org/core # Build specific project morphir build --workspace # Force workspace-wide operation morphir build --no-workspace # Force single-project mode (ignore workspace) morphir --workspace-root /path/to/ws build # Explicit workspace root ``` ## Example: Full Workspace Configuration ```toml # morphir.toml [morphir] version = "^4.0.0" [workspace] output_dir = ".morphir" members = ["packages/*", "apps/*"] exclude = ["packages/experimental-*"] default_member = "packages/core" [workspace.dependencies] "morphir/sdk" = { git = "https://github.com/finos/morphir-sdk.git", tag = "v3.0.0" } [ir] format_version = 4 strict_mode = false [codegen] targets = ["typescript"] output_format = "pretty" [cache] enabled = true dir = ".morphir/cache" [logging] level = "info" format = "text" # Built-in tasks (build, test, check, etc.) work automatically # Only define custom tasks or hooks [tasks.ci] description = "Run CI pipeline" depends = ["check", "test", "build", "pack"] run = "echo 'CI passed'" [tasks."post:build"] run = "prettier --write .morphir-dist/" [tasks."post:codegen"] run = "prettier --write generated/" ``` ## Best Practices 1. **Flat Structure**: Prefer flat `packages/*` over deeply nested hierarchies 2. **Explicit Members**: Use explicit patterns rather than catch-all `**/*` 3. **Shared Config**: Put common settings at workspace level 4. **Default Member**: Set `default_member` for common single-project operations 5. **Workspace Commands**: Use `morphir workspace` prefix for explicit workspace operations ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Configuration](./configuration.md)** - Configuration system overview - **[morphir.toml](./morphir-toml.md)** - Complete configuration file specification - **[Merge Rules](./merge-rules.md)** - Configuration inheritance and merge behavior ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents --- ## Extension Manager Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/08-extension-manager # Extension Manager **Status:** Draft **Version:** 0.1.0 ## Overview The Extension Manager is the central coordinator for all extensions in Morphir. It provides a unified API for loading, calling, and managing extensions regardless of their underlying protocol. ## Architecture ``` ┌─────────────────────────────────────────┐ │ Extension Manager │ │ (Kameo Actor) │ │ │ │ ┌────────────────────────────────┐ │ │ │ hosts: HashMap │ │ │ │ "jsonrpc2" → JsonRpcHost │ │ │ │ "grpc" → GrpcHost │ │ │ │ "stdio" → StdioHost │ │ │ │ "extism" → ExtismHost │ │ │ │ "component"→ ComponentHost │ │ │ └────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────┐ │ │ │ extensions: HashMap │ │ │ │ ExtensionId → Metadata │ │ │ └────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────┐ │ │ │ registry: HashMap │ │ │ │ String → ExtensionId │ │ │ │ (name lookup) │ │ │ └────────────────────────────────┘ │ └─────────────────────────────────────────┘ ``` ## State Management ```rust #[derive(Actor)] pub struct ExtensionManager { /// Protocol hosts, keyed by protocol name hosts: HashMap>, /// Metadata for all loaded extensions extensions: HashMap, /// Name-to-ID registry for fast lookup registry: HashMap, /// Event bus for publishing events event_bus: ActorRef, /// Next extension ID next_id: u64, } struct ExtensionMetadata { id: ExtensionId, name: String, protocol: String, capabilities: Vec, config: ExtensionConfig, loaded_at: Instant, call_count: u64, error_count: u64, } ``` ## Messages ### LoadExtension Loads a new extension from configuration. ```rust #[message] pub async fn load_extension( &mut self, config: ExtensionConfig, ) -> Result { // 1. Determine protocol from source let protocol = match &config.source { ExtensionSource::Http { .. } => "jsonrpc2", ExtensionSource::Grpc { .. } => "grpc", ExtensionSource::Process { .. } => "stdio", ExtensionSource::Wasm { .. } => "extism", ExtensionSource::WasmComponent { .. } => "component", }; // 2. Get host let host = self.hosts .get_mut(protocol) .ok_or(ExtensionError::HostNotFound)?; // 3. Load in host let id = host.load(config.clone()).await?; // 4. Store metadata self.extensions.insert(id, ExtensionMetadata { id, name: config.name.clone(), protocol: protocol.to_string(), capabilities: vec![], // Fetched from host config, loaded_at: Instant::now(), call_count: 0, error_count: 0, }); // 5. Register name self.registry.insert(config.name, id); // 6. Publish event self.event_bus.tell(Publish::new("extension.loaded", id)).await?; Ok(id) } ``` ### CallExtension Calls a method on an extension by name. ```rust #[message] pub async fn call_extension( &mut self, name: String, method: String, params: Value, ) -> Result { // 1. Resolve name → ID let id = self.registry .get(&name) .copied() .ok_or_else(|| ExtensionError::NotFound(name))?; // 2. Get metadata let metadata = self.extensions .get_mut(&id) .ok_or(ExtensionError::NotFound(id))?; // 3. Get host let host = self.hosts .get_mut(&metadata.protocol) .ok_or(ExtensionError::HostNotFound)?; // 4. Call extension metadata.call_count += 1; let start = Instant::now(); let result = host.call(id, method, params).await; let duration = start.elapsed(); // 5. Update metrics if result.is_err() { metadata.error_count += 1; } metrics::histogram!("morphir.extension.call.duration_ms") .record(duration.as_millis() as f64); result } ``` ### ListExtensions Lists all loaded extensions. ```rust #[message] pub fn list_extensions(&self) -> Vec { self.extensions .values() .map(|meta| ExtensionInfo { id: meta.id, name: meta.name.clone(), protocol: meta.protocol.clone(), capabilities: meta.capabilities.clone(), call_count: meta.call_count, error_count: meta.error_count, uptime: meta.loaded_at.elapsed(), }) .collect() } ``` ### UnloadExtension Unloads an extension. ```rust #[message] pub async fn unload_extension( &mut self, name: String, ) -> Result<(), ExtensionError> { let id = self.registry .remove(&name) .ok_or_else(|| ExtensionError::NotFound(name))?; let metadata = self.extensions .remove(&id) .ok_or(ExtensionError::NotFound(id))?; let host = self.hosts .get_mut(&metadata.protocol) .ok_or(ExtensionError::HostNotFound)?; host.unload(id).await?; self.event_bus.tell(Publish::new("extension.unloaded", id)).await?; Ok(()) } ``` ## Usage Example ```rust #[tokio::main] async fn main() -> Result<()> { // Create manager let manager = ExtensionManager::new().spawn(); // Register hosts manager.tell(RegisterHost { protocol: "stdio".to_string(), host: StdioExtensionHost::new().spawn(), }).await?; manager.tell(RegisterHost { protocol: "jsonrpc2".to_string(), host: JsonRpcExtensionHost::new().spawn(), }).await?; // Load extension let config = ExtensionConfig { name: "my-extension".to_string(), source: ExtensionSource::Process { command: "python3".to_string(), args: vec!["./extension.py".to_string()], env: HashMap::new(), }, permissions: Permissions::default(), config: json!({}), restart: RestartStrategy::Never, }; let id = manager.ask(LoadExtension(config)).await?; // Call extension let result = manager.ask(CallExtension { name: "my-extension".to_string(), method: "transform".to_string(), params: json!({"ir": {...}}), }).await?; // List extensions let extensions = manager.ask(ListExtensions).await?; for ext in extensions { println!("{}: {} calls", ext.name, ext.call_count); } Ok(()) } ``` ## Configuration Loading The manager can load extensions from a configuration file: ```rust impl ExtensionManager { pub async fn load_from_config(&mut self, path: &Path) -> Result<()> { let config: ExtensionConfig = toml::from_str(&fs::read_to_string(path)?)?; for ext_config in config.extensions { if ext_config.enabled { self.load_extension(ext_config).await?; } } Ok(()) } } ``` ## Supervision The manager implements supervision for host failures: ```rust impl Actor for ExtensionManager { async fn on_link_died( &mut self, actor_ref: WeakActorRef, id: ActorID, reason: ActorStopReason, ) { // A host died - attempt to restart warn!("Host died: {:?}", reason); // Find which host died and restart extensions // Implementation depends on restart strategy } } ``` ## Events The manager publishes these events via the event bus: - `extension.loaded` - Extension successfully loaded - `extension.failed` - Extension failed to load - `extension.unloaded` - Extension unloaded - `extension.call.started` - Method call started - `extension.call.completed` - Method call completed - `extension.call.failed` - Method call failed - `host.registered` - New host registered - `host.failed` - Host actor failed ## Metrics The manager emits these metrics: - `morphir.extension.count` - Number of loaded extensions - `morphir.extension.load.duration_ms` - Time to load extension - `morphir.extension.call.duration_ms` - Time to call method - `morphir.extension.call.count` - Number of calls (by extension, method, status) - `morphir.extension.error.count` - Number of errors (by extension) ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## morphir.toml Specification Source: https://finos.github.io/morphir-rust/contributors/design/daemon/morphir-toml # morphir.toml Specification This document provides the complete specification for the `morphir.toml` configuration file format. ## File Format Morphir uses [TOML v1.0.0](https://toml.io/en/v1.0.0) for configuration files. All configuration is in `morphir.toml` at the project or workspace root. ## Top-Level Sections | Section | Required | Description | | -------------------- | ----------- | ------------------------------------------------ | | `[morphir]` | No | Morphir toolchain settings | | `[project]` | Conditional | Project metadata (required for project mode) | | `[workspace]` | Conditional | Workspace settings (required for workspace mode) | | `[frontend]` | No | Source language configuration | | `[ir]` | No | IR format and generation settings | | `[codegen]` | No | Code generation targets and options | | `[dependencies]` | No | Project dependencies | | `[dev-dependencies]` | No | Development-only dependencies | | `[extensions]` | No | Extension registration | | `[tasks]` | No | Custom tasks and hooks | | `[test]` | No | Test configuration | | `[publish]` | No | Publishing configuration | ## `[morphir]` Section Morphir toolchain version and global settings. ```toml [morphir] # Required Morphir IR version (semver constraint) # Ensures compatibility with toolchain and extensions version = "^4.0.0" # Minimum CLI version required (optional) min_cli_version = "4.0.0" ``` ### Fields | Field | Type | Default | Description | | ----------------- | ------ | -------- | ------------------------------------------------------------------------ | | `version` | string | Required | Semver constraint for IR version (e.g., `"^4.0.0"`, `"~4.1"`, `"4.0.0"`) | | `min_cli_version` | string | None | Minimum CLI version required | ### Version Constraint Syntax | Syntax | Meaning | Example | | ------------------- | ------------------ | ----------------------- | | `"4.0.0"` | Exact version | Only 4.0.0 | | `"^4.0.0"` | Compatible (major) | `>=4.0.0, <5.0.0` | | `"~4.1.0"` | Compatible (minor) | `>=4.1.0, <4.2.0` | | `">=4.0.0"` | Minimum version | 4.0.0 or higher | | `">=4.0.0, <5.0.0"` | Range | Between 4.0.0 and 5.0.0 | ## `[project]` Section Project metadata and structure. Required for project mode. ```toml [project] # Package identifier (organization/name format) name = "my-org/my-project" # Semantic version version = "1.0.0" # Human-readable description description = "A Morphir domain model for order processing" # Package authors authors = ["Jane Doe "] # License identifier (SPDX) license = "Apache-2.0" # Repository URL repository = "https://github.com/my-org/my-project" # Homepage URL homepage = "https://my-org.github.io/my-project" # Source directory (relative to morphir.toml) source_directory = "src" # Modules exposed as public API exposed_modules = ["Domain.User", "Domain.Order", "Domain.Product"] # Output directory for build artifacts output_directory = ".morphir-dist" # Keywords for package discovery keywords = ["domain", "orders", "e-commerce"] ``` ### Fields | Field | Type | Default | Description | | ------------------ | ------ | ----------------- | ---------------------------------------------------- | | `name` | string | Required | Package identifier in `org/name` format | | `version` | string | Required | Semantic version (e.g., `"1.0.0"`, `"2.1.0-beta.1"`) | | `description` | string | `""` | Human-readable description | | `authors` | array | `[]` | List of authors (name and/or email) | | `license` | string | None | SPDX license identifier | | `repository` | string | None | Repository URL | | `homepage` | string | None | Homepage URL | | `source_directory` | string | `"src"` | Source code directory | | `exposed_modules` | array | All modules | Modules in the public API | | `output_directory` | string | `".morphir-dist"` | Build output directory | | `keywords` | array | `[]` | Keywords for discovery | ### Package Name Format Package names follow the format `organization/project-name`: - **Organization**: Lowercase, alphanumeric, hyphens allowed (e.g., `my-org`, `finos`) - **Project**: Lowercase, alphanumeric, hyphens allowed (e.g., `my-project`, `morphir-sdk`) - **Separator**: Forward slash `/` Examples: - `morphir/sdk` - `my-company/order-domain` - `finos/morphir-examples` ## `[workspace]` Section Workspace configuration for multi-project setups. ```toml [workspace] # Glob patterns for discovering member projects members = [ "packages/*", "libs/*", "apps/*" ] # Patterns to exclude from member discovery exclude = [ "packages/deprecated-*", "packages/internal-*" ] # Default member for commands without --project flag default_member = "packages/core" # Shared output directory output_dir = ".morphir" # Parallel build settings parallel = true max_jobs = 4 ``` ### Fields | Field | Type | Default | Description | | ---------------- | ------- | ------------ | ---------------------------------- | | `members` | array | `[]` | Glob patterns for member discovery | | `exclude` | array | `[]` | Patterns to exclude from discovery | | `default_member` | string | None | Default project for commands | | `output_dir` | string | `".morphir"` | Workspace output directory | | `parallel` | bool | `true` | Enable parallel builds | | `max_jobs` | integer | CPU count | Maximum parallel jobs | ### Member Discovery Members are discovered by: 1. Matching directories against `members` patterns 2. Excluding matches against `exclude` patterns 3. Finding `morphir.toml` with `[project]` section in matched directories ## `[frontend]` Section Source language and parser configuration. ```toml [frontend] # Default source language language = "elm" # Language-specific settings [frontend.elm] elm_version = "0.19" optimize = true [frontend.morphir-dsl] strict_mode = true allow_incomplete = false # Pattern-based language selection [[frontend.rules]] pattern = "src/legacy/**/*.morphir" language = "morphir-dsl" [[frontend.rules]] pattern = "src/experimental/**/*.ml" language = "ocaml" ``` ### Fields | Field | Type | Default | Description | | ---------- | ------ | ----------- | ----------------------- | | `language` | string | Auto-detect | Default source language | ### Built-in Languages | Language | Extensions | Description | | ------------- | ------------------- | ---------------- | | `elm` | `.elm` | Elm source files | | `morphir-dsl` | `.morphir`, `.mdsl` | Morphir DSL | ### Language-Specific Options #### `[frontend.elm]` | Field | Type | Default | Description | | ------------- | ------ | -------- | -------------------- | | `elm_version` | string | `"0.19"` | Elm language version | | `optimize` | bool | `false` | Enable optimizations | #### `[frontend.morphir-dsl]` | Field | Type | Default | Description | | ------------------ | ---- | ------- | ---------------------------- | | `strict_mode` | bool | `false` | Strict parsing mode | | `allow_incomplete` | bool | `true` | Allow incomplete definitions | ### Frontend Rules Rules select language based on file path patterns: ```toml [[frontend.rules]] pattern = "src/**/*.elm" # Glob pattern language = "elm" # Language for matching files ``` Rules are evaluated in order; first match wins. ## `[ir]` Section IR format and generation settings. ```toml [ir] # IR format version format_version = 4 # Output mode mode = "vfs" # "classic" or "vfs" # Strict mode (fail on warnings) strict_mode = false # Include source locations in IR include_source_locations = true # Preserve documentation include_docs = true ``` ### Fields | Field | Type | Default | Description | | -------------------------- | ------- | ------- | ------------------------------------------------------------------ | | `format_version` | integer | `4` | IR format version | | `mode` | string | `"vfs"` | Output mode: `"classic"` (single file) or `"vfs"` (directory tree) | | `strict_mode` | bool | `false` | Treat warnings as errors | | `include_source_locations` | bool | `true` | Include source locations in IR | | `include_docs` | bool | `true` | Include documentation in IR | ## `[codegen]` Section Code generation configuration. ```toml [codegen] # Targets to generate targets = ["typescript", "scala", "spark"] # Output format output_format = "pretty" # "pretty" or "compact" # Output directory (relative to project output) output_dir = "generated" # Target-specific configuration [codegen.typescript] module_format = "esm" strict = true [codegen.scala] package_prefix = "com.myorg" scala_version = "2.13" [codegen.spark] spark_version = "3.5" scala_version = "2.13" # Module-specific target overrides [codegen.modules."Domain.Api"] targets = ["typescript"] # Only generate TypeScript for this module [codegen.modules."Domain.Internal"] targets = [] # Skip codegen for internal modules # Pattern-based rules [[codegen.rules]] pattern = "**/Internal/**" targets = [] [[codegen.rules]] pattern = "**/Api/**" targets = ["typescript", "openapi"] ``` ### Fields | Field | Type | Default | Description | | --------------- | ------ | ------------- | ----------------------- | | `targets` | array | `[]` | Code generation targets | | `output_format` | string | `"pretty"` | Output formatting | | `output_dir` | string | `"generated"` | Output directory | ### Built-in Targets | Target | Description | | ------------- | --------------------- | | `typescript` | TypeScript/JavaScript | | `scala` | Scala 2.x/3.x | | `java` | Java | | `spark` | Apache Spark | | `json-schema` | JSON Schema | | `openapi` | OpenAPI specification | ### Target-Specific Options #### `[codegen.typescript]` | Field | Type | Default | Description | | --------------- | ------ | ------- | --------------------------------- | | `module_format` | string | `"esm"` | `"esm"`, `"commonjs"`, or `"umd"` | | `strict` | bool | `true` | Enable strict TypeScript | | `declaration` | bool | `true` | Generate `.d.ts` files | #### `[codegen.scala]` | Field | Type | Default | Description | | ---------------- | ------ | -------- | -------------- | | `package_prefix` | string | None | Package prefix | | `scala_version` | string | `"2.13"` | Scala version | #### `[codegen.spark]` | Field | Type | Default | Description | | --------------- | ------ | -------- | ------------- | | `spark_version` | string | `"3.5"` | Spark version | | `scala_version` | string | `"2.13"` | Scala version | ## `[dependencies]` Section Project dependencies. ```toml [dependencies] # Path dependency (local) "my-org/common" = { path = "../common" } # Version dependency (from registry) "morphir/sdk" = "^3.0.0" # Git dependency "other-org/utils" = { git = "https://github.com/other-org/utils.git", tag = "v1.0.0" } # Detailed specification [dependencies."finos/morphir-examples"] version = "^2.0.0" features = ["extra-types"] ``` ### Dependency Formats #### Version String (Registry) ```toml "morphir/sdk" = "^3.0.0" ``` #### Path (Local) ```toml "my-org/common" = { path = "../common" } ``` #### Git ```toml # By tag "org/pkg" = { git = "https://github.com/org/pkg.git", tag = "v1.0.0" } # By branch "org/pkg" = { git = "https://github.com/org/pkg.git", branch = "main" } # By commit "org/pkg" = { git = "https://github.com/org/pkg.git", rev = "abc123" } ``` #### Detailed ```toml [dependencies."org/pkg"] version = "^2.0.0" features = ["feature1", "feature2"] optional = false ``` ### Dependency Fields | Field | Type | Description | | ---------- | ------ | --------------------------- | | `version` | string | Semver constraint | | `path` | string | Local path | | `git` | string | Git repository URL | | `tag` | string | Git tag | | `branch` | string | Git branch | | `rev` | string | Git commit hash | | `features` | array | Optional features to enable | | `optional` | bool | Optional dependency | ## `[dev-dependencies]` Section Development-only dependencies (same format as `[dependencies]`). ```toml [dev-dependencies] "morphir/test-utils" = "^1.0.0" "my-org/test-fixtures" = { path = "../test-fixtures" } ``` ## `[extensions]` Section Extension registration and configuration. ```toml [extensions] # WASM component (local path) spark-codegen = { path = "./extensions/spark-codegen.wasm" } # WASM component (URL) scala-codegen = { url = "https://extensions.morphir.dev/scala-codegen-1.0.0.wasm" } # Native executable my-analyzer = { command = "./bin/my-analyzer", args = ["--mode", "jsonrpc"] } # Disable auto-discovered extension legacy-ext = { enabled = false } # Extension-specific configuration [extensions.spark-codegen.config] spark_version = "3.5" scala_version = "2.13" ``` ### Extension Fields | Field | Type | Description | | --------- | ------ | -------------------------------- | | `path` | string | Local path to WASM or executable | | `url` | string | URL to download extension | | `command` | string | Executable command | | `args` | array | Command arguments | | `enabled` | bool | Enable/disable extension | | `config` | table | Extension-specific configuration | ## `[tasks]` Section Custom tasks and hooks. ```toml [tasks] # Simple command task lint = "elm-review" # Task with description [tasks.integration] description = "Run integration tests" run = "./scripts/integration-tests.sh" # Task with dependencies [tasks.ci] description = "Full CI pipeline" depends = ["check", "test", "build"] # Task with working directory [tasks.docs] description = "Generate documentation" run = "morphir docs generate" cwd = "./docs" # Pre/post hooks for built-in tasks [tasks."pre:build"] run = "echo 'Starting build...'" [tasks."post:build"] run = "prettier --write .morphir-dist/" [tasks."post:codegen"] run = "./scripts/post-codegen.sh" ``` ### Task Fields | Field | Type | Description | | ------------- | ------ | ----------------------------- | | `description` | string | Human-readable description | | `run` | string | Shell command to execute | | `depends` | array | Task dependencies (run first) | | `cwd` | string | Working directory | | `env` | table | Environment variables | ### Built-in Tasks | Task | Description | | --------- | ---------------------------- | | `build` | Compile project to IR | | `check` | Lint and validate | | `test` | Run tests | | `codegen` | Generate code for targets | | `pack` | Create distributable package | | `publish` | Publish to registry | | `clean` | Remove build artifacts | ### Hooks Hooks extend built-in tasks: | Hook | Timing | | ------------- | ------------------- | | `pre:` | Before task runs | | `post:` | After task succeeds | ## `[test]` Section Test configuration. ```toml [test] # Test directory directory = "tests" # Test patterns include = ["**/*Test.elm", "**/*Spec.elm"] exclude = ["**/helpers/**"] # Test runner settings timeout = 30000 # milliseconds parallel = true ``` ### Fields | Field | Type | Default | Description | | ----------- | ------- | ---------------- | --------------------- | | `directory` | string | `"tests"` | Test directory | | `include` | array | `["**/*Test.*"]` | Include patterns | | `exclude` | array | `[]` | Exclude patterns | | `timeout` | integer | `30000` | Test timeout (ms) | | `parallel` | bool | `true` | Run tests in parallel | ## `[publish]` Section Publishing configuration. ```toml [publish] # Registry URL registry = "https://registry.morphir.dev" # Include patterns include = [ "src/**/*.elm", "morphir.toml", "README.md", "LICENSE" ] # Exclude patterns exclude = [ "**/*.test.elm", "**/Internal/**" ] ``` ### Fields | Field | Type | Default | Description | | ---------- | ------ | ---------------- | ---------------- | | `registry` | string | Default registry | Registry URL | | `include` | array | Standard files | Files to include | | `exclude` | array | `[]` | Files to exclude | ## Complete Example ### Single Project ```toml [morphir] version = "^4.0.0" [project] name = "my-org/order-domain" version = "1.0.0" description = "Order processing domain model" authors = ["Jane Doe "] license = "Apache-2.0" source_directory = "src" exposed_modules = ["Domain.Order", "Domain.Product", "Domain.Customer"] [frontend] language = "elm" [frontend.elm] elm_version = "0.19" [codegen] targets = ["typescript", "scala"] [codegen.typescript] module_format = "esm" [codegen.scala] package_prefix = "com.myorg.orders" [dependencies] "morphir/sdk" = "^3.0.0" "my-org/common-types" = { path = "../common-types" } [dev-dependencies] "morphir/test-utils" = "^1.0.0" [tasks.ci] description = "CI pipeline" depends = ["check", "test", "build", "codegen"] ``` ### Workspace ```toml [morphir] version = "^4.0.0" [workspace] members = ["packages/*", "apps/*"] exclude = ["packages/deprecated-*"] default_member = "packages/core" output_dir = ".morphir" # Shared settings inherited by all projects [frontend] language = "elm" [frontend.elm] elm_version = "0.19" [codegen] targets = ["typescript"] # Shared dependencies [dependencies] "morphir/sdk" = "^3.0.0" [extensions] spark-codegen = { path = "./extensions/spark-codegen.wasm" } [tasks.ci] description = "Workspace CI" depends = ["check", "test", "build"] ``` ## Validation Rules ### Required Fields | Mode | Required Sections/Fields | | --------- | ---------------------------------------------- | | Project | `[project]`, `project.name`, `project.version` | | Workspace | `[workspace]`, `workspace.members` | ### Naming Constraints | Field | Constraint | | ----------------- | ----------------------------------- | | `project.name` | `^[a-z][a-z0-9-]*/[a-z][a-z0-9-]*$` | | `project.version` | Valid semver | | Module names | PascalCase, dot-separated | ### Path Constraints | Field | Constraint | | --------------------- | --------------------------- | | `source_directory` | Must exist, relative path | | `output_directory` | Relative path | | `dependencies.*.path` | Must contain `morphir.toml` | ## Error Messages | Error | Description | | ------ | ----------------------------------- | | `E001` | Missing required field | | `E002` | Invalid package name format | | `E003` | Invalid version format | | `E004` | Source directory not found | | `E005` | Circular dependency detected | | `E006` | Unknown extension | | `E007` | Invalid glob pattern | | `E008` | Duplicate project name in workspace | ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Configuration](./configuration.md)** - Configuration system overview - **[Workspace Config](./workspace-config.md)** - Multi-project workspace configuration - **[Merge Rules](./merge-rules.md)** - Configuration inheritance and merge behavior - **[Environment](./environment.md)** - Environment variables and runtime overrides ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Configuration Merge Rules Source: https://finos.github.io/morphir-rust/contributors/design/daemon/merge-rules # Configuration Merge Rules This document specifies how configuration from multiple sources is merged and resolved. ## Merge Precedence Configuration is resolved from multiple sources in this order (highest precedence first): 1. **Command-line flags**: `--config key=value` 2. **Environment variables**: `MORPHIR__SECTION__KEY` 3. **Workspace local config**: `./.config/morphir/config.toml` 4. **Project config**: `./morphir.toml` in project directory 5. **Workspace config**: `./morphir.toml` at workspace root 6. **Parent configs**: Walk up directory tree 7. **User config**: `~/.config/morphir/config.toml` 8. **System config**: `/etc/morphir/config.toml` Higher precedence sources override lower precedence sources. ## Merge Strategies Different configuration types use different merge strategies: | Type | Strategy | Description | | ------ | ----------------- | ----------------------------------------- | | Scalar | Replace | Higher precedence value replaces lower | | Array | Replace (default) | Higher precedence array replaces entirely | | Array | Append | Arrays are concatenated | | Table | Deep merge | Tables are recursively merged | ### Scalar Values (Replace) Scalar values (strings, numbers, booleans) are replaced entirely: ```toml # User config (~/.config/morphir/config.toml) [codegen] output_format = "compact" # Project config (./morphir.toml) [codegen] output_format = "pretty" # Result: output_format = "pretty" ``` ### Arrays (Replace by Default) Arrays are replaced entirely by default: ```toml # Workspace config [codegen] targets = ["typescript", "scala"] # Project config [codegen] targets = ["spark"] # Result: targets = ["spark"] ``` ### Arrays (Append Mode) To append instead of replace, use the `+` prefix: ```toml # Workspace config [codegen] targets = ["typescript"] # Project config [codegen] "+targets" = ["spark", "scala"] # Result: targets = ["typescript", "spark", "scala"] ``` ### Tables (Deep Merge) Tables are recursively merged: ```toml # Workspace config [codegen.typescript] module_format = "esm" strict = true # Project config [codegen.typescript] declaration = true # Result: # [codegen.typescript] # module_format = "esm" # strict = true # declaration = true ``` ## Section-Specific Rules ### `[project]` Section The `[project]` section is **never inherited**. Each project must define its own: - `name` - `version` - `source_directory` - `exposed_modules` These fields are project-specific and cannot be overridden from workspace or parent configs. ### `[workspace]` Section The `[workspace]` section exists only at workspace root. It is **not inherited** by member projects. ### `[dependencies]` Section Dependencies merge with these rules: 1. **Same package, different versions**: Project version wins 2. **Path vs version**: More specific (project) wins 3. **New dependencies**: Added to merged set ```toml # Workspace config [dependencies] "morphir/sdk" = "^3.0.0" "org/shared" = "^1.0.0" # Project config [dependencies] "morphir/sdk" = "^3.1.0" # Overrides workspace "org/project-specific" = "^2.0.0" # Added # Result: # "morphir/sdk" = "^3.1.0" # "org/shared" = "^1.0.0" # "org/project-specific" = "^2.0.0" ``` ### `[codegen]` Section Code generation settings merge deeply: ```toml # Workspace config [codegen] targets = ["typescript"] output_format = "pretty" [codegen.typescript] module_format = "esm" # Project config [codegen] "+targets" = ["spark"] # Append [codegen.typescript] strict = false # Add to typescript config [codegen.spark] spark_version = "3.5" # New target config # Result: # targets = ["typescript", "spark"] # output_format = "pretty" # [codegen.typescript] # module_format = "esm" # strict = false # [codegen.spark] # spark_version = "3.5" ``` ### `[extensions]` Section Extensions merge with these rules: 1. **Same extension ID**: Project config wins entirely 2. **Disabled extensions**: `enabled = false` prevents loading 3. **New extensions**: Added to merged set ```toml # Workspace config [extensions] spark-codegen = { path = "./extensions/spark-codegen.wasm" } [extensions.spark-codegen.config] spark_version = "3.4" # Project config [extensions.spark-codegen.config] spark_version = "3.5" # Overrides # Result: spark_version = "3.5" ``` ### `[tasks]` Section Tasks merge with these rules: 1. **Same task name**: Project definition wins 2. **Hooks**: All hooks at all levels run (workspace first, then project) 3. **Dependencies**: Resolved from merged task set ```toml # Workspace config [tasks.lint] run = "elm-review" [tasks."pre:build"] run = "echo 'Workspace pre-build'" # Project config [tasks.lint] run = "elm-review --fix" # Overrides workspace [tasks."pre:build"] run = "echo 'Project pre-build'" # Both run! # Result: # - lint runs: "elm-review --fix" # - pre:build runs BOTH (workspace first, then project) ``` ### `[frontend]` Section Frontend settings merge deeply, with rules evaluated in order: ```toml # Workspace config [frontend] language = "elm" [[frontend.rules]] pattern = "**/*.morphir" language = "morphir-dsl" # Project config [frontend] # language inherited from workspace [[frontend.rules]] pattern = "src/legacy/**" language = "elm" # Project rules evaluated first # Result: Project rules checked first, then workspace rules ``` ## Environment Variables Environment variables override file configuration using this naming convention: ``` MORPHIR__
__ ``` ### Naming Rules - Sections and keys are uppercase - Dots become double underscores - Hyphens become single underscores ### Examples | Environment Variable | Configuration Path | | -------------------------------------- | --------------------------- | | `MORPHIR__PROJECT__NAME` | `project.name` | | `MORPHIR__CODEGEN__OUTPUT_FORMAT` | `codegen.output_format` | | `MORPHIR__CODEGEN__TYPESCRIPT__STRICT` | `codegen.typescript.strict` | | `MORPHIR__FRONTEND__LANGUAGE` | `frontend.language` | ### Type Coercion Environment variables are strings. They are coerced to the expected type: | Expected Type | Coercion | | ------------- | ------------------------------------------- | | string | As-is | | integer | Parse as integer | | boolean | `"true"`, `"1"`, `"yes"` → true; else false | | array | JSON array or comma-separated | ```bash # String export MORPHIR__PROJECT__NAME="my-org/my-project" # Integer export MORPHIR__WORKSPACE__MAX_JOBS="4" # Boolean export MORPHIR__IR__STRICT_MODE="true" # Array (JSON) export MORPHIR__CODEGEN__TARGETS='["typescript","scala"]' # Array (comma-separated) export MORPHIR__CODEGEN__TARGETS="typescript,scala" ``` ## Command-Line Overrides Command-line flags have highest precedence: ```bash # Override single value morphir build --config codegen.output_format=compact # Override nested value morphir build --config codegen.typescript.strict=false # Override array (JSON) morphir build --config 'codegen.targets=["spark"]' # Multiple overrides morphir build \ --config codegen.output_format=compact \ --config ir.strict_mode=true ``` ## Merge Algorithm ``` function mergeConfig(sources: ConfigSource[]): Config { result = {} // Process sources from lowest to highest precedence for source in sources.reverse() { for section in source.sections { if section.name == "project" || section.name == "workspace" { // Never inherit project/workspace sections if source.isProjectConfig { result[section.name] = section.value } } else { result[section.name] = mergeSection( result[section.name], section.value, section.name ) } } } return result } function mergeSection(base: Value, overlay: Value, path: string): Value { if overlay is null { return base } if base is null { return overlay } if overlay is Scalar { return overlay // Replace } if overlay is Array { if path.startsWith("+") { return concat(base, overlay) // Append } return overlay // Replace } if overlay is Table { result = copy(base) for key, value in overlay { result[key] = mergeSection(result[key], value, key) } return result } } ``` ## Examples ### Example 1: Workspace with Project Override **Workspace config** (`workspace/morphir.toml`): ```toml [morphir] version = "^4.0.0" [workspace] members = ["packages/*"] [codegen] targets = ["typescript"] output_format = "pretty" [codegen.typescript] module_format = "esm" ``` **Project config** (`workspace/packages/api/morphir.toml`): ```toml [project] name = "my-org/api" version = "1.0.0" [codegen] "+targets" = ["openapi"] # Append [codegen.typescript] strict = true # Add to typescript config ``` **Resolved config for `packages/api`**: ```toml [morphir] version = "^4.0.0" [project] name = "my-org/api" version = "1.0.0" [codegen] targets = ["typescript", "openapi"] output_format = "pretty" [codegen.typescript] module_format = "esm" strict = true ``` ### Example 2: Environment Override **Project config**: ```toml [codegen] targets = ["typescript"] [codegen.typescript] strict = true ``` **Environment**: ```bash export MORPHIR__CODEGEN__TARGETS="spark,scala" export MORPHIR__CODEGEN__TYPESCRIPT__STRICT="false" ``` **Resolved config**: ```toml [codegen] targets = ["spark", "scala"] # From env [codegen.typescript] strict = false # From env ``` ### Example 3: User Defaults **System config** (`/etc/morphir/config.toml`): ```toml [codegen] output_format = "compact" ``` **User config** (`~/.config/morphir/config.toml`): ```toml [codegen] output_format = "pretty" [ir] include_source_locations = true ``` **Project config** (`./morphir.toml`): ```toml [project] name = "my-org/project" version = "1.0.0" [codegen] targets = ["typescript"] ``` **Resolved config**: ```toml [project] name = "my-org/project" version = "1.0.0" [codegen] targets = ["typescript"] output_format = "pretty" # From user config [ir] include_source_locations = true # From user config ``` ## Debugging Configuration ### Show Resolved Config ```bash # Show fully resolved configuration morphir config show # Show specific section morphir config show codegen # Show where values come from morphir config show --sources ``` ### Example Output ``` $ morphir config show --sources [project] name = "my-org/api" # ./morphir.toml version = "1.0.0" # ./morphir.toml [codegen] targets = ["typescript", "openapi"] # merged - "typescript" # ../morphir.toml (workspace) - "openapi" # ./morphir.toml (append) output_format = "pretty" # ../morphir.toml (workspace) [codegen.typescript] module_format = "esm" # ../morphir.toml (workspace) strict = true # ./morphir.toml ``` ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Configuration](./configuration.md)** - Configuration system overview - **[morphir.toml](./morphir-toml.md)** - Complete configuration file specification - **[Workspace Config](./workspace-config.md)** - Multi-project workspace configuration - **[Environment](./environment.md)** - Environment variables and runtime overrides ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents --- ## Security and Isolation Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/09-security-and-isolation # Security and Isolation **Status:** Draft **Version:** 0.1.0 ## Overview The Morphir Extension System implements defense-in-depth security through multiple layers of isolation, capability-based permissions, and resource limits. ## Security Principles 1. **Principle of Least Privilege**: Extensions only get permissions they explicitly request 2. **Defense in Depth**: Multiple security layers (process, sandbox, permissions) 3. **Fail Secure**: Security failures result in denial, not escalation 4. **Explicit Over Implicit**: All capabilities must be declared 5. **Auditability**: All extension actions are logged and traceable ## Isolation Mechanisms ### Process Isolation (Stdio, JSON-RPC, gRPC) Each extension runs in a separate OS process: **Benefits:** - Complete memory isolation - OS-level resource limits - Crash isolation (extension crash doesn't affect Morphir) - Can be run with restricted user permissions **Implementation:** ```rust let child = Command::new(&command) .args(&args) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::piped()) .kill_on_drop(true) // Clean up on host exit .spawn()?; ``` ### WASM Sandboxing (Extism, Component) WASM extensions run in a strict sandbox: **Restrictions:** - No direct filesystem access - No network access - No system calls - Memory limited and isolated - Execution time limited **Capabilities:** - Extensions can only call explicitly provided host functions - All I/O goes through host APIs ```rust let mut plugin = Plugin::new(&manifest, [], true)?; // true = with wasi // Memory limit plugin.set_memory_limits(10_000_000, 100_000_000)?; // min, max bytes // Timeout plugin.set_timeout(Duration::from_secs(5))?; ``` ## Permission System ### Permission Model Extensions declare required permissions in configuration: ```toml [extensions.permissions] network = true filesystem = ["/data/output", "/tmp"] max_memory = "100MB" max_execution_time = "30s" ``` ### Permission Types #### Network Access ```rust pub struct NetworkPermissions { /// Allow any network access pub enabled: bool, /// Allow only specific hosts pub allowed_hosts: Option>, /// Allow only specific ports pub allowed_ports: Option>, } ``` **Enforcement:** - Process extensions: Use OS-level firewall rules - WASM extensions: No network access (sandbox) #### Filesystem Access ```rust pub struct FilesystemPermissions { /// Allowed paths (read and write) pub paths: Vec, /// Read-only paths pub readonly_paths: Vec, } ``` **Enforcement:** - Process extensions: Use OS-level permissions (chroot, AppArmor, SELinux) - WASM extensions: Mount only specified paths via WASI #### Resource Limits ```rust pub struct ResourceLimits { /// Maximum memory usage pub max_memory: Option, /// Maximum CPU time per call pub max_execution_time: Option, /// Maximum number of concurrent calls pub max_concurrent_calls: Option, } ``` ## Threat Model ### Threats Considered 1. **Malicious Extensions** - Trying to access unauthorized resources - Consuming excessive resources (DoS) - Data exfiltration - Privilege escalation 2. **Compromised Extensions** - Vulnerable dependencies - Injected malicious code - Supply chain attacks 3. **Extension Bugs** - Memory corruption - Resource leaks - Crashes affecting availability ### Threats NOT Considered 1. **Host System Compromise**: If the host OS is compromised, extensions cannot be secured 2. **Side-Channel Attacks**: Timing attacks, speculative execution (Spectre/Meltdown) 3. **Physical Access**: Local attacker with physical access 4. **Supply Chain** (Partially): We trust the extension source as declared, but not the extension behavior ## Security by Protocol ### Stdio Extensions **Security Posture:** Medium **Isolation:** Process **Attack Surface:** OS process APIs **Mitigations:** - Run with restricted user (non-root) - Use seccomp/AppArmor/SELinux - Resource limits via cgroups - Network isolation via firewall rules **Example (Linux):** ```rust // Run with restricted user Command::new(&command) .uid(1001) // unprivileged user .gid(1001) ``` ### JSON-RPC / gRPC Extensions **Security Posture:** Low (networked) **Isolation:** Network + Process **Attack Surface:** Network protocols, HTTP/gRPC libraries **Mitigations:** - mTLS for authentication - Rate limiting - Input validation - Network segmentation **Risks:** - Extension can make arbitrary network calls - Shared with other services - Exposed to network attacks ### Extism WASM Extensions **Security Posture:** High **Isolation:** WASM sandbox **Attack Surface:** WASM runtime, host functions **Mitigations:** - Strict memory isolation - No direct system access - Explicit capability granting - Runtime limits **Example:** ```rust // Strict sandbox let mut plugin = Plugin::new(&manifest, [], true)?; // Memory limits plugin.set_memory_limits(1_000_000, 10_000_000)?; // Timeout plugin.set_timeout(Duration::from_secs(5))?; // No host functions = no I/O ``` ### Component Model Extensions **Security Posture:** High **Isolation:** WASM sandbox + WASI **Attack Surface:** WASM runtime, WASI interfaces **Mitigations:** - All Extism mitigations - Capability-based WASI - Fine-grained resource control **Example:** ```rust // Only allow specific directories let mut ctx = WasiCtxBuilder::new() .preopened_dir( Dir::open_ambient_dir("/data/output", ambient_authority())?, "/output", )? .build(); ``` ## Security Best Practices ### For Extension Developers 1. **Minimize Permissions**: Only request what you need 2. **Validate Inputs**: Never trust input from Morphir core 3. **Handle Errors**: Don't leak sensitive info in error messages 4. **Avoid Dependencies**: Fewer dependencies = smaller attack surface 5. **Use WASM When Possible**: Strongest isolation ### For Morphir Core Developers 1. **Validate Extension Outputs**: Don't trust extension results 2. **Rate Limit Calls**: Prevent DoS via excessive calls 3. **Monitor Resource Usage**: Track memory, CPU, errors 4. **Log Everything**: Audit trail for security incidents 5. **Fail Closed**: On error, deny access ### For Operators 1. **Review Extensions**: Audit code before enabling 2. **Use Minimal Permissions**: Grant least privilege 3. **Monitor Logs**: Watch for suspicious behavior 4. **Update Regularly**: Keep runtime dependencies updated 5. **Network Isolation**: Use firewalls, VLANs ## Audit Logging All extension operations are logged: ```rust #[instrument(skip(self))] async fn call_extension( &mut self, name: String, method: String, params: Value, ) -> Result { info!( extension = %name, method = %method, "Extension call started" ); let result = self.call_internal(name, method, params).await; match &result { Ok(_) => info!("Extension call succeeded"), Err(e) => warn!(error = %e, "Extension call failed"), } result } ``` ## Security Checklist Before enabling an extension: - [ ] Extension source is trusted - [ ] Permissions are minimal - [ ] Network access is justified - [ ] Filesystem access is scoped - [ ] Resource limits are set - [ ] Extension has been reviewed - [ ] Monitoring is enabled - [ ] Incident response plan exists ## Future Enhancements 1. **Certificate Pinning**: For JSON-RPC/gRPC extensions 2. **Code Signing**: Verify extension integrity 3. **Runtime Policy Engine**: Dynamic permission adjustment 4. **Security Profiles**: Predefined security levels (low/medium/high) 5. **Attestation**: Remote attestation for WASM modules ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Environment Variables Source: https://finos.github.io/morphir-rust/contributors/design/daemon/environment # Environment Variables This document specifies all environment variables recognized by Morphir. ## Configuration Override Variables Environment variables can override any configuration value using the naming convention: ``` MORPHIR__
__ ``` See [Merge Rules](./merge-rules.md) for details on how environment overrides work. ## Core Environment Variables ### `MORPHIR_HOME` The Morphir home directory for user-level data. | Aspect | Value | | --------------------- | ---------------------------------------------------- | | Default (Linux/macOS) | `$XDG_DATA_HOME/morphir` or `~/.local/share/morphir` | | Default (Windows) | `%LOCALAPPDATA%\morphir` | | Contains | User extensions, cache, global config | ```bash export MORPHIR_HOME="/opt/morphir" ``` ### `MORPHIR_CONFIG_HOME` The Morphir configuration directory. | Aspect | Value | | --------------------- | ------------------------------------------------- | | Default (Linux/macOS) | `$XDG_CONFIG_HOME/morphir` or `~/.config/morphir` | | Default (Windows) | `%APPDATA%\morphir` | | Contains | `config.toml`, credentials | ```bash export MORPHIR_CONFIG_HOME="/etc/morphir" ``` ### `MORPHIR_CACHE_HOME` The Morphir cache directory. | Aspect | Value | | --------------------- | ----------------------------------------------- | | Default (Linux/macOS) | `$XDG_CACHE_HOME/morphir` or `~/.cache/morphir` | | Default (Windows) | `%LOCALAPPDATA%\morphir\cache` | | Contains | Downloaded dependencies, build cache | ```bash export MORPHIR_CACHE_HOME="/var/cache/morphir" ``` ## Daemon Variables ### `MORPHIR_DAEMON_URL` URL of a running Morphir daemon to connect to. | Aspect | Value | | ------- | ---------------------------------------------- | | Default | None (start embedded daemon) | | Format | `http://host:port` or `unix:///path/to/socket` | ```bash # Connect to local daemon export MORPHIR_DAEMON_URL="http://localhost:3000" # Connect to Unix socket export MORPHIR_DAEMON_URL="unix:///tmp/morphir.sock" # Connect to remote daemon export MORPHIR_DAEMON_URL="http://build-server:3000" ``` ### `MORPHIR_DAEMON_AUTO_START` Whether to automatically start a daemon if none is running. | Aspect | Value | | ------- | ------------------------- | | Default | `true` | | Values | `true`, `false`, `1`, `0` | ```bash export MORPHIR_DAEMON_AUTO_START="false" ``` ### `MORPHIR_DAEMON_TIMEOUT` Timeout for daemon connections in milliseconds. | Aspect | Value | | ------- | ---------------------- | | Default | `30000` (30 seconds) | | Format | Integer (milliseconds) | ```bash export MORPHIR_DAEMON_TIMEOUT="60000" ``` ### `MORPHIR_DAEMON_SHUTDOWN_TIMEOUT` How long the daemon waits before shutting down when idle. | Aspect | Value | | ------- | ---------------------- | | Default | `3600000` (1 hour) | | Format | Integer (milliseconds) | ```bash # Keep daemon running for 8 hours export MORPHIR_DAEMON_SHUTDOWN_TIMEOUT="28800000" # Never auto-shutdown export MORPHIR_DAEMON_SHUTDOWN_TIMEOUT="0" ``` ## Build Variables ### `MORPHIR_PARALLEL` Enable parallel builds. | Aspect | Value | | ------- | --------------- | | Default | `true` | | Values | `true`, `false` | ```bash export MORPHIR_PARALLEL="false" ``` ### `MORPHIR_MAX_JOBS` Maximum number of parallel build jobs. | Aspect | Value | | ------- | ------------------- | | Default | Number of CPU cores | | Format | Integer | ```bash export MORPHIR_MAX_JOBS="4" ``` ### `MORPHIR_INCREMENTAL` Enable incremental builds. | Aspect | Value | | ------- | --------------- | | Default | `true` | | Values | `true`, `false` | ```bash export MORPHIR_INCREMENTAL="false" ``` ## Output Variables ### `MORPHIR_COLOR` Control colored output. | Aspect | Value | | ------- | ------------------------- | | Default | `auto` | | Values | `auto`, `always`, `never` | ```bash export MORPHIR_COLOR="never" ``` ### `MORPHIR_LOG_LEVEL` Logging verbosity level. | Aspect | Value | | ------- | ----------------------------------------- | | Default | `info` | | Values | `error`, `warn`, `info`, `debug`, `trace` | ```bash export MORPHIR_LOG_LEVEL="debug" ``` ### `MORPHIR_LOG_FORMAT` Log output format. | Aspect | Value | | ------- | -------------- | | Default | `text` | | Values | `text`, `json` | ```bash export MORPHIR_LOG_FORMAT="json" ``` ### `MORPHIR_QUIET` Suppress non-essential output. | Aspect | Value | | ------- | --------------- | | Default | `false` | | Values | `true`, `false` | ```bash export MORPHIR_QUIET="true" ``` ## Registry Variables ### `MORPHIR_REGISTRY` Default package registry URL. | Aspect | Value | | ------- | ------------------------------ | | Default | `https://registry.morphir.dev` | | Format | URL | ```bash export MORPHIR_REGISTRY="https://internal-registry.company.com" ``` ### `MORPHIR_REGISTRY_TOKEN` Authentication token for the registry. | Aspect | Value | | ------- | -------------- | | Default | None | | Format | String (token) | ```bash export MORPHIR_REGISTRY_TOKEN="ghp_xxxxxxxxxxxx" ``` ### `MORPHIR_REGISTRY_USERNAME` Username for registry authentication. | Aspect | Value | | ------- | ------ | | Default | None | | Format | String | ```bash export MORPHIR_REGISTRY_USERNAME="myuser" ``` ### `MORPHIR_REGISTRY_PASSWORD` Password for registry authentication. | Aspect | Value | | ------- | ------ | | Default | None | | Format | String | ```bash export MORPHIR_REGISTRY_PASSWORD="mypassword" ``` ## Extension Variables ### `MORPHIR_EXTENSIONS_PATH` Additional paths to search for extensions. | Aspect | Value | | ------- | ------------------------------------------------------------- | | Default | None | | Format | Colon-separated paths (Unix) or semicolon-separated (Windows) | ```bash export MORPHIR_EXTENSIONS_PATH="/opt/morphir/extensions:/home/user/my-extensions" ``` ### `MORPHIR_EXTENSION_TIMEOUT` Timeout for extension operations in milliseconds. | Aspect | Value | | ------- | ---------------------- | | Default | `30000` (30 seconds) | | Format | Integer (milliseconds) | ```bash export MORPHIR_EXTENSION_TIMEOUT="60000" ``` ## CI/CD Variables ### `CI` Standard CI environment indicator. | Aspect | Value | | ------- | -------------------------------------------------------- | | Default | None | | Effect | Disables interactive prompts, enables CI-friendly output | ```bash export CI="true" ``` ### `MORPHIR_CI` Morphir-specific CI mode. | Aspect | Value | | ------- | ------------- | | Default | Value of `CI` | | Effect | Same as `CI` | ### `MORPHIR_NO_INTERACTIVE` Disable interactive prompts. | Aspect | Value | | ------- | ---------------------------------- | | Default | `false` (or `true` if `CI` is set) | | Values | `true`, `false` | ```bash export MORPHIR_NO_INTERACTIVE="true" ``` ## Debugging Variables ### `MORPHIR_DEBUG` Enable debug mode. | Aspect | Value | | ------- | ---------------------------------------------- | | Default | `false` | | Effect | Verbose logging, stack traces, debug endpoints | ```bash export MORPHIR_DEBUG="true" ``` ### `MORPHIR_TRACE` Enable trace-level debugging. | Aspect | Value | | ------- | --------------------------------------------- | | Default | `false` | | Effect | Extremely verbose output, performance logging | ```bash export MORPHIR_TRACE="true" ``` ### `MORPHIR_PROFILE` Enable performance profiling. | Aspect | Value | | ------- | ---------------------------------------------- | | Default | `false` | | Effect | Outputs timing information, generates profiles | ```bash export MORPHIR_PROFILE="true" ``` ## Platform-Specific Variables ### Linux/macOS ```bash # XDG Base Directory Specification export XDG_DATA_HOME="$HOME/.local/share" export XDG_CONFIG_HOME="$HOME/.config" export XDG_CACHE_HOME="$HOME/.cache" ``` Morphir respects XDG variables when `MORPHIR_*` equivalents are not set. ### Windows ```cmd REM Standard Windows locations set LOCALAPPDATA=%USERPROFILE%\AppData\Local set APPDATA=%USERPROFILE%\AppData\Roaming ``` ## Configuration Override Examples ### Override `[codegen]` Settings ```bash # Override targets array export MORPHIR__CODEGEN__TARGETS='["typescript","spark"]' # Override output format export MORPHIR__CODEGEN__OUTPUT_FORMAT="compact" # Override nested TypeScript setting export MORPHIR__CODEGEN__TYPESCRIPT__STRICT="false" ``` ### Override `[ir]` Settings ```bash export MORPHIR__IR__FORMAT_VERSION="4" export MORPHIR__IR__STRICT_MODE="true" export MORPHIR__IR__MODE="vfs" ``` ### Override `[frontend]` Settings ```bash export MORPHIR__FRONTEND__LANGUAGE="morphir-dsl" ``` ## Environment File Morphir supports `.env` files in the project root: ```bash # .env file in project root MORPHIR_LOG_LEVEL=debug MORPHIR_PARALLEL=false MORPHIR__CODEGEN__OUTPUT_FORMAT=compact ``` ### `.env` File Loading Order 1. `.env` (always loaded) 2. `.env.local` (local overrides, gitignored) 3. `.env.{environment}` (e.g., `.env.production`) 4. `.env.{environment}.local` (local overrides for environment) The `environment` is determined by `MORPHIR_ENV` or defaults to `development`. ```bash export MORPHIR_ENV="production" ``` ## Variable Reference Table | Variable | Type | Default | Description | | --------------------------------- | ------ | ------------- | -------------------------- | | `MORPHIR_HOME` | path | XDG default | Morphir data directory | | `MORPHIR_CONFIG_HOME` | path | XDG default | Configuration directory | | `MORPHIR_CACHE_HOME` | path | XDG default | Cache directory | | `MORPHIR_DAEMON_URL` | url | None | Daemon connection URL | | `MORPHIR_DAEMON_AUTO_START` | bool | `true` | Auto-start daemon | | `MORPHIR_DAEMON_TIMEOUT` | int | `30000` | Connection timeout (ms) | | `MORPHIR_DAEMON_SHUTDOWN_TIMEOUT` | int | `3600000` | Idle shutdown timeout (ms) | | `MORPHIR_PARALLEL` | bool | `true` | Enable parallel builds | | `MORPHIR_MAX_JOBS` | int | CPU count | Max parallel jobs | | `MORPHIR_INCREMENTAL` | bool | `true` | Enable incremental builds | | `MORPHIR_COLOR` | enum | `auto` | Color output mode | | `MORPHIR_LOG_LEVEL` | enum | `info` | Log verbosity | | `MORPHIR_LOG_FORMAT` | enum | `text` | Log format | | `MORPHIR_QUIET` | bool | `false` | Suppress output | | `MORPHIR_REGISTRY` | url | Default | Registry URL | | `MORPHIR_REGISTRY_TOKEN` | string | None | Registry auth token | | `MORPHIR_EXTENSIONS_PATH` | paths | None | Extension search paths | | `MORPHIR_EXTENSION_TIMEOUT` | int | `30000` | Extension timeout (ms) | | `CI` | bool | None | CI environment | | `MORPHIR_DEBUG` | bool | `false` | Debug mode | | `MORPHIR_TRACE` | bool | `false` | Trace mode | | `MORPHIR_PROFILE` | bool | `false` | Profiling mode | | `MORPHIR_ENV` | string | `development` | Environment name | ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Configuration](./configuration.md)** - Configuration system overview - **[morphir.toml](./morphir-toml.md)** - Complete configuration file specification - **[Merge Rules](./merge-rules.md)** - Configuration inheritance and merge behavior ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents --- ## Protocol Specifications Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/10-protocol-specifications # Protocol Specifications **Status:** Draft **Version:** 0.1.0 ## Overview This document specifies the wire protocols and message formats for all extension host types. These specifications ensure interoperability between Morphir and extensions. ## Common Lifecycle All protocols follow a common lifecycle: 1. **Connection/Spawn**: Host connects to or spawns extension 2. **Initialize**: Host sends initialization message with configuration 3. **Capabilities**: Host queries extension capabilities 4. **Ready**: Extension is ready to process requests 5. **Call**: Zero or more method calls 6. **Shutdown**: Host notifies extension of shutdown 7. **Terminate**: Extension cleans up and exits ## Stdio Protocol (JSON Lines) ### Transport - **Format**: JSON Lines (one JSON object per line) - **Encoding**: UTF-8 - **Delimiter**: Newline character (`\n`) ### Request Format ```json { "id": 123, "method": "transform", "params": { "ir": {...} } } ``` **Fields:** - `id` (integer): Unique request ID, must be echoed in response - `method` (string): Method name to invoke - `params` (any): Method-specific parameters ### Response Format Success: ```json { "id": 123, "result": { "output": "..." } } ``` Error: ```json { "id": 123, "error": "Error message" } ``` **Fields:** - `id` (integer): Matches request ID - `result` (any): Result value (mutually exclusive with `error`) - `error` (string): Error message (mutually exclusive with `result`) ### Notification Format ```json { "method": "log", "params": { "level": "info", "message": "Processing..." } } ``` **Note**: No `id` field, no response expected. ### Required Methods All extensions must implement: #### `initialize` ```json { "id": 0, "method": "initialize", "params": { "config": {...} } } ``` Response: ```json { "id": 0, "result": { "status": "ready" } } ``` #### `capabilities` ```json { "id": 1, "method": "capabilities", "params": {} } ``` Response: ```json { "id": 1, "result": [ { "name": "transform", "description": "Transform Morphir IR" } ] } ``` ## JSON-RPC 2.0 Protocol ### Transport - **Format**: JSON-RPC 2.0 - **Transport**: HTTP POST - **Content-Type**: `application/json` ### Request Format ```json { "jsonrpc": "2.0", "id": 123, "method": "morphir.transform", "params": { "ir": {...} } } ``` ### Response Format Success: ```json { "jsonrpc": "2.0", "id": 123, "result": { "output": "..." } } ``` Error: ```json { "jsonrpc": "2.0", "id": 123, "error": { "code": -32000, "message": "Error message", "data": {...} } } ``` ### Required Methods - `initialize`: Initialize extension - `capabilities`: Return extension capabilities - Custom methods as declared in capabilities ## gRPC Protocol ### Service Definition ```protobuf syntax = "proto3"; package morphir.extension; service ExtensionService { rpc Initialize(InitializeRequest) returns (InitializeResponse); rpc GetCapabilities(Empty) returns (CapabilitiesResponse); rpc Call(CallRequest) returns (CallResponse); } message InitializeRequest { string config_json = 1; } message InitializeResponse { string status = 1; } message Empty {} message CapabilitiesResponse { repeated Capability capabilities = 1; } message Capability { string name = 1; string description = 2; optional string params_schema = 3; optional string return_schema = 4; } message CallRequest { string method = 1; bytes params_json = 2; } message CallResponse { oneof result { bytes success_json = 1; string error = 2; } } ``` ## Extism WASM Protocol ### Exported Functions Extensions must export these functions: ```rust // Initialize extension #[plugin_fn] pub fn initialize(config_json: String) -> FnResult { // Returns: {"status": "ready"} } // Return capabilities #[plugin_fn] pub fn capabilities() -> FnResult { // Returns: [{"name": "...", "description": "..."}] } // Custom methods #[plugin_fn] pub fn transform(params_json: String) -> FnResult { // Returns: result JSON } ``` ### Data Format All data is passed as JSON strings: **Input**: JSON string containing parameters **Output**: JSON string containing result **Error Handling**: Return error via Extism error mechanism: ```rust Err(extism_pdk::Error::msg("error message")) ``` ## WASM Component Protocol ### Interface Definition (WIT) ```wit package morphir:extension; interface extension { record capability { name: string, description: string, params-schema: option, return-schema: option, } record init-result { status: string, } initialize: func(config: string) -> init-result; capabilities: func() -> list; // Custom methods transform: func(params: string) -> result; } world morphir-extension { export extension; } ``` ### Error Handling Methods return `result` where: - `ok(T)`: Success with value - `err(string)`: Error with message ## Capability Schema All protocols use a common capability format: ```json { "name": "transform", "description": "Transform Morphir IR to target language", "params_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "ir": { "type": "object", "description": "Morphir IR" }, "options": { "type": "object", "description": "Generation options" } }, "required": ["ir"] }, "return_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "output": { "type": "string", "description": "Generated code" } }, "required": ["output"] } } ``` **Fields:** - `name` (string, required): Method name - `description` (string, required): Human-readable description - `params_schema` (object, optional): JSON Schema for parameters - `return_schema` (object, optional): JSON Schema for return value ## Standard Methods All extensions should implement these standard methods: ### `initialize` Initialize the extension with configuration. **Parameters:** ```json { "config": { // Extension-specific configuration } } ``` **Returns:** ```json { "status": "ready" } ``` ### `capabilities` Return list of capabilities. **Parameters:** None **Returns:** ```json [ { "name": "method_name", "description": "Method description", "params_schema": {...}, "return_schema": {...} } ] ``` ### `health_check` (Optional) Check extension health. **Parameters:** None **Returns:** ```json { "status": "healthy" | "degraded" | "unhealthy", "message": "Optional status message" } ``` ## Error Codes Standard error codes for all protocols: | Code | Name | Description | | ---------------- | ---------------- | --------------------------------------- | | -32600 | Invalid Request | Invalid JSON or missing required fields | | -32601 | Method Not Found | Method does not exist | | -32602 | Invalid Params | Invalid method parameters | | -32603 | Internal Error | Internal extension error | | -32000 to -32099 | Server Error | Extension-specific errors | ## Versioning Protocol version is declared in initialization: ```json { "protocol_version": "1.0", "extension_version": "2.1.0" } ``` **Compatibility Rules:** - Major version change: Breaking changes - Minor version change: Backward-compatible additions - Patch version change: Bug fixes only ## Testing All protocol implementations should pass these tests: 1. **Initialize Test**: Extension accepts initialize and returns success 2. **Capabilities Test**: Extension returns non-empty capabilities list 3. **Echo Test**: Extension echoes parameters back 4. **Error Test**: Extension returns proper error format 5. **Concurrent Test**: Extension handles multiple concurrent calls 6. **Timeout Test**: Extension respects timeout limits ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Wasm Extension Architecture (Interactive Session) Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/12-wasm-extension-architecture-session # Wasm Extension Architecture (Interactive Session) **Status:** Draft **Version:** 0.3.0 **Last Updated:** 2026-01-29 ## Session framing **Goal:** Update the extension and daemon design to include the hidden Extism runtime, the envelope protocol, TEA runtime semantics, Kameo actors, and support for multiple client types (CLI, Morphir-Live using Dioxus framework, WASM clients with core Wasm and WASM Component Model + WASI). **Outcome:** This document captures the updated architecture and ABI contracts as an interactive design session, including deployment modes for native, daemon, Dioxus-based multi-platform (web/desktop), core Wasm (browser and non-browser), and WASM Component + WASI environments. ## Interactive design session ### Prompt We want a unified extension model that supports design-time and runtime extensions, with a stable engine-agnostic ABI based on envelopes and TEA. Extism should be hidden behind the Morphir runtime. The daemon should host extension instances via Kameo actors. ### Key decisions - **Hidden engine:** Extism is used internally and never appears in extension ABIs or SDKs. - **Single protocol:** All extension calls use an Envelope with `header`, `content_type`, and `content` bytes. - **TEA semantics:** Runtime extensions follow TEA-style init/update/subscriptions with JSON payload conventions. - **Actor isolation:** Each operation runs inside a Kameo actor that owns runtime state and loaded programs. - **Host-agnostic design:** The same ABI works across CLI, daemon, services, and browser hosts. ### Open questions - Should TEA `update` use two envelopes (msg + model) or a merged JSON wrapper by default? - How should subscription envelopes encode timers, filesystem watches, or external event streams? - What is the minimal host-function surface for initial runtime extensions? ## Goals and non-goals ### Goals - **Unified extension model** for Morphir: - Design-time extensions (frontends, backends, transforms, decorators) - Runtime extensions (IR evaluators, effect handlers, domain logic) - **Portable execution** via WebAssembly: - Core Wasm (`wasm32-unknown-unknown`) - WebAssembly Component Model (where available) - **Stable, host-agnostic protocol**: - Envelope: `header + content_type + content` - Works across CLI, daemon, services, browser, Wasmtime, GraalVM, Node, etc. - **Predictable runtime semantics**: - Elm-style (TEA) architecture for runtime extensions - **Isolation and concurrency**: - Kameo actors as the execution boundary for extensions - **Hidden engine**: - Use Extism internally to manage Wasm plugins - Extension authors are **not aware** of Extism ### Non-goals - Forcing extension authors to depend on Extism APIs or SDKs - Locking Morphir into a single Wasm engine forever - Exposing engine-specific details (Extism, Wasmtime, jco, etc.) in extension ABIs ## High-level architecture ```text +---------------------------+ +---------------------------+ +---------------------------+ | CLI Client | | Morphir-Live | | WASM Client | | (native binary) | | (Dioxus: Web/Desktop) | | (Core/Component+WASI) | +------------+--------------+ +-------------+-------------+ +-------------+-------------+ | | | | | | v v v +---------------------------------------------------------------------------------------------+ | Daemon/Host Layer | | (manages actors, I/O, routing, state) | +---------------------------------------------------------------------------------------------+ | | | v v v +---------------------------+ +---------------------------+ +---------------------------+ | Kameo Actor | | Kameo Actor | | Direct Runtime Call | | (extension instance) | | (extension instance) | | (browser, no daemon) | +------------+--------------+ +-------------+-------------+ +-------------+-------------+ | | | v v v +---------------------------------------------------------------------------------------------+ | Morphir Runtime | | (Rust, uses Extism internally for extensions) | | (can be compiled to native or Wasm itself) | +---------------------------------------------------------------------------------------------+ | | | v v v +---------------------------+ +---------------------------+ +---------------------------+ | Extension Program | | Extension Program | | Extension Program | | (Wasm core/component) | | (Wasm core/component) | | (Wasm core/component) | +---------------------------+ +---------------------------+ +---------------------------+ ``` Key points: - **Multiple client types** access the runtime: - **CLI Client**: Native binary for command-line usage - **Morphir-Live**: Multi-platform app from finos/morphir using Dioxus framework (targets web and desktop) - **WASM Client**: Runtime compiled to Wasm, supporting: - Core Wasm (browser or non-browser hosts) - WASM Component Model with WASI (Wasmtime, browser with polyfills, etc.) - The **daemon/host layer** manages **Kameo actors**, one per operation/extension instance. - **Direct runtime calls** are possible in browser environments without daemon (for WASM clients). - Each **actor** (or direct call) hosts: - the **Morphir runtime** (Rust, compiled to native or Wasm) - one or more **extension programs** (Wasm modules) - The runtime uses **Extism internally** to load and call extension programs. - Extension authors only see: - the **envelope protocol** - the **TEA-style ABI** - the **design-time ABI** - the **standard library** (host functions) ## Core concepts ### Host The **host** is any environment that runs the Morphir runtime: - **CLI process**: Native binary for command-line usage - **Daemon process**: Long-running service managing multiple sessions - **Morphir-Live**: Multi-platform application using Dioxus framework (Rust → Web/Desktop, from finos/morphir) - **WASM Client**: Runtime compiled to Wasm, runnable in: - **Core Wasm**: Browser (via WebAssembly API), Wasmtime, wasmer, Node, Deno, Bun - **WASM Component Model + WASI**: Wasmtime, wasmer with WASI support, browser with WASI polyfills - **Long-running service**: Backend services, API servers - **Other embedded runtimes**: GraalVM, custom Wasm hosts Responsibilities: - Start and manage Kameo actors (daemon/service mode) - **OR** Make direct runtime calls (browser/WASM client mode) - Provide host functions (logging, HTTP, filesystem, timers, etc.) - Manage workspace and session state - Route messages between clients and actors (in daemon mode) - Handle persistence and caching (in daemon mode) ### Runtime The **Morphir runtime** is a Rust library/binary that: - Implements the **envelope protocol** - Implements the **TEA runtime** for runtime extensions - Implements the **design-time extension** interface - Uses **Extism internally** to load and call Wasm extension programs - Exposes a clean API to the host/actors (no Extism types leak out) Conceptual structure: ```text +----------------------------------+ | Morphir Runtime | +----------------------------------+ | +----------------------------+ | | | Extism Integration | | (hidden) | +----------------------------+ | | | Program Registry | | | +----------------------------+ | | | TEA Loop Engine | | | +----------------------------+ | | | Envelope Codec | | | +----------------------------+ | | | Host Function Glue | | +----------------------------------+ ``` ### Program (Extension) A **program** is a Wasm module that implements one of: - **Design-time extension interfaces**: - `frontend-compile(input: Envelope) -> Envelope` - `backend-generate(input: Envelope) -> Envelope` - `get-capabilities() -> Envelope` - **Runtime extension interface** (TEA): - `init(flags: Envelope) -> Envelope` - `update(msg: Envelope, model: Envelope) -> Envelope` - `subscriptions(model: Envelope) -> Envelope` Programs may be: - **Core Wasm modules** (wasm32-unknown-unknown): - Simple `(ptr,len)` ABI for passing bytes - Works in any Wasm host (browser, Node, Wasmtime, etc.) - No access to system resources unless provided by host imports - **WASM Component Model** (WIT-based): - Type-safe interfaces defined via WIT (WebAssembly Interface Types) - Can include WASI (WebAssembly System Interface) for system access: - **WASI Preview 2**: Filesystem, sockets, environment, clocks, random - Enables portable system-level extensions - Requires Component Model-capable host (Wasmtime, wasmer, or polyfills) Programs are dynamically loaded by the runtime via Extism (for core Wasm) or Component Model loaders (for WASM Components). ## Envelope protocol All messages between host, runtime, and programs use an **envelope**: ```rust struct Envelope { header: Header, content_type: String, content: Vec, } struct Header { seqnum: u64, session_id: String, kind: Option, } ``` Conceptually: ```text +----------------------------------------+ | Envelope | +----------------------------------------+ | header: { seqnum, session_id, kind? } | | content_type: "application/json" | | content: | +----------------------------------------+ ``` Header definition: ```text +-------------------------------+ | Header | +-------------------------------+ | seqnum: u64 = 0 | | session_id: String = "" | | kind?: String (optional) | +-------------------------------+ ``` Examples: - `application/json` - `application/morphir-ir+json` - `application/morphir-ir+cbor` - `text/typescript` - `application/x-morphir-backend` Benefits: - Encoding-agnostic - Extensible and versionable - Works in core Wasm and Component Model - Browser-friendly - Uniform across design-time and runtime - Carries lightweight metadata in `header` (sequence number, session ID, optional kind) ## TEA-style runtime (Elm-like architecture) The runtime follows a TEA-like model: ```text +---------------------------+ | Program | | (pure functions) | +---------------------------+ ^ | envelopes v +---------------------------+ | Runtime | | (state + effects) | +---------------------------+ ^ | host functions v +---------------------------+ | Host | +---------------------------+ ``` Program interface: - `init(flags: Envelope) -> Envelope` Returns an envelope containing `(model, cmds)`. - `update(msg: Envelope, model: Envelope) -> Envelope` Returns an envelope containing `(model, cmds)`. - `subscriptions(model: Envelope) -> Envelope` Returns an envelope describing subscriptions. The runtime: - Maintains per-program state (`model`) - Interprets `cmds` and `subs` envelopes - Calls host functions (standard library) - Exposes `start`, `send`, `poll` to the host ## Hidden Extism engine ### Design principle - **Extism is an implementation detail.** - Extension authors: - do **not** import Extism functions - do **not** depend on Extism SDKs - only implement the Morphir-defined ABI (envelope + TEA/design-time) - The runtime: - uses Extism to load and call Wasm modules - can be swapped out later for Wasmtime, jco, etc. without breaking extensions ### Envelope codec (host side) ```rust use serde::{Deserialize, Serialize}; #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Header { pub seqnum: u64, pub session_id: String, #[serde(skip_serializing_if = "Option::is_none")] pub kind: Option, } impl Default for Header { fn default() -> Self { Self { seqnum: 0, session_id: String::new(), kind: None, } } } #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Envelope { pub header: Header, pub content_type: String, #[serde(with = "serde_bytes")] pub content: Vec, } impl Envelope { pub fn json(value: &T) -> anyhow::Result { Ok(Self { header: Header::default(), content_type: "application/json".to_string(), content: serde_json::to_vec(value)?, }) } pub fn as_json Deserialize<'de>>(&self) -> anyhow::Result { if self.content_type != "application/json" { anyhow::bail!("expected application/json, got {}", self.content_type); } Ok(serde_json::from_slice(&self.content)?) } } pub fn encode_envelope(env: &Envelope) -> anyhow::Result> { Ok(serde_json::to_vec(env)?) } pub fn decode_envelope(bytes: &[u8]) -> anyhow::Result { Ok(serde_json::from_slice(bytes)?) } ``` Extism only ever sees `Vec`; the runtime sees `Envelope`. ### Extism runtime wrapper ```rust use extism::{Context, Manifest, Plugin}; use crate::envelope::{Envelope, encode_envelope, decode_envelope}; pub struct ExtismRuntime { ctx: Context, plugin: Plugin, } impl ExtismRuntime { pub fn new(wasm_bytes: Vec) -> anyhow::Result { let manifest = Manifest::new([wasm_bytes]); let ctx = Context::new(); let plugin = ctx.new_plugin(manifest, true)?; Ok(Self { ctx, plugin }) } pub fn call_envelope(&mut self, func: &str, input: &Envelope) -> anyhow::Result { let input_bytes = encode_envelope(input)?; let output_bytes = self.plugin.call(func, &input_bytes)?; let env = decode_envelope(&output_bytes)?; Ok(env) } } ``` Everything above this layer is Extism-free. ## Extension ABIs ### Design-time extension ABI Design-time capabilities are split into focused interfaces that still use the same `Envelope` ABI: - **Frontend**: `frontend-compile(input: Envelope) -> Envelope` - **Backend**: `backend-generate(input: Envelope) -> Envelope` - **Common-backend**: `get-capabilities() -> Envelope` ## WIT definitions (current) These are the current Component Model definitions used by `morphir-ext-core`. ### envelope.wit ```wit package morphir:ext@0.1.0; interface envelope { record header { seqnum: u64, session-id: string, kind: option, } record envelope { header: header, content-type: string, content: list, } } ``` ### runtime.wit ```wit package morphir:ext@0.1.0; interface runtime { use envelope.{envelope}; enum log-level { trace, debug, info, warn, error } variant env-value { text(string), text-list(list), boolean(bool), v-u8(u8), v-u16(u16), v-u32(u32), v-u64(u64), v-s8(s8), v-s16(s16), v-s32(s32), v-s64(s64), v-f32(f32), v-f64(f64) } /// Log a message to the host. log: func(level: log-level, msg: string); /// Get an environment variable. get-env-var: func(name: string) -> option; /// Set an environment variable. set-env-var: func(name: string, value: env-value); } world extension { import runtime; export program; } ``` ### program.wit ```wit package morphir:ext@0.1.0; interface program { use envelope.{envelope}; /// Initialize the extension with startup flags. /// Returns the initial model and any initial commands. init: func(init-data: envelope) -> tuple; /// Update the extension state based on a message. /// Returns the new model and any commands. update: func(msg: envelope, model: envelope) -> tuple; /// Get active subscriptions based on the model. subscriptions: func(model: envelope) -> envelope; /// Get capabilities as a JSON envelope. get-capabilities: func() -> envelope; } interface design-time-frontend { use envelope.{envelope}; /// Frontend compilation: source -> IR frontend-compile: func(input: envelope) -> envelope; } interface design-time-backend { use envelope.{envelope}; /// Backend generation: IR -> artifacts backend-generate: func(input: envelope) -> envelope; } interface common-backend { use envelope.{envelope}; /// Backend capability discovery. get-capabilities: func() -> envelope; } ``` ### Runtime extension ABI (TEA) Extension authors implement: ```text init(flags: Envelope) -> Envelope update(msg: Envelope, model: Envelope) -> Envelope subscriptions(model: Envelope) -> Envelope ``` We standardize the **payloads inside envelopes** as JSON objects: - `init`: - Input: `flags` envelope (e.g. config, initial IR) - Output envelope: `application/json` with: ```json { "model": { ... }, "cmds": [ ... ] } ``` - `update`: - Input envelopes: `msg`, `model` - Output envelope: `application/json` with: ```json { "model": { ... }, "cmds": [ ... ] } ``` - `subscriptions`: - Input: `model` envelope - Output envelope: `application/json` with: ```json { "subs": [ ... ] } ``` The runtime wraps these in helper functions: ```rust pub struct MorphirProgram { extism: ExtismRuntime, } impl MorphirProgram { pub fn init(&mut self, flags: Envelope) -> anyhow::Result<(Envelope, Envelope)> { let out = self.extism.call_envelope("init", &flags)?; // parse out.model, out.cmds from out.content // return (model_env, cmds_env) Ok((model_env, cmds_env)) } pub fn update(&mut self, msg: Envelope, model: Envelope) -> anyhow::Result<(Envelope, Envelope)> { // combine msg + model into one envelope or define a convention // e.g. content_type = "application/json", content = { "msg": ..., "model": ... } let input = Envelope::json(&serde_json::json!({ "msg": { "content_type": msg.content_type, "content": base64::encode(msg.content), }, "model": { "content_type": model.content_type, "content": base64::encode(model.content), } }))?; let out = self.extism.call_envelope("update", &input)?; // parse out.model, out.cmds Ok((model_env, cmds_env)) } pub fn subscriptions(&mut self, model: Envelope) -> anyhow::Result { self.extism.call_envelope("subscriptions", &model) } } ``` Extension authors only see: - `init/update/subscriptions` with `Envelope` arguments - JSON payloads inside `Envelope.content` (if they choose) ## Standard library (host functions) We provide a **standard library** to extensions via host functions, implemented in Rust and registered with Extism. Examples: - `log(env: Envelope)` - `get_env_var(name: Envelope) -> Envelope` - `set_env_var(name: Envelope, value: Envelope)` - `random(env: Envelope) -> Envelope` - `http(env: Envelope) -> Envelope` - `workspace(env: Envelope) -> Envelope` (design-time only) - `ir_helpers(env: Envelope) -> Envelope` Conceptually: ```text +---------------------------+ | Standard Library | +---------------------------+ | log (Envelope) | | get_env_var(Envelope) | | set_env_var(Envelope) | | random (Envelope) | | http (Envelope) | | workspace (Envelope) | | ir_helpers (Envelope) | +---------------------------+ ``` These are registered as Extism host functions, but extension authors only see them as imports in their language of choice (Rust, TypeScript, etc.) via a Morphir-specific SDK. ## Kameo actor model Each **operation** (e.g. "run this transform", "compile this IR", "execute this program") is handled by a **Kameo actor**. Actor responsibilities: - Load the Morphir runtime - Load one or more extension programs (via Extism) - Maintain workspace and session state - Execute TEA loops for runtime extensions - Execute `process` calls for design-time extensions - Interpret commands and subscriptions - Call host functions (standard library) - Stream progress and results back to the daemon Actor structure: ```text +----------------------------------+ | Actor | +----------------------------------+ | +----------------------------+ | | | Morphir Runtime | | | | (uses Extism) | | | +----------------------------+ | | | Loaded Extensions | | | +----------------------------+ | | | Workspace / Session | | | +----------------------------+ | | | Async Host Functions | | +----------------------------------+ ``` ## Client to runtime flows ### Daemon-based flow (CLI, Morphir-Live in remote mode) ```text +-----------+ +-----------+ +-----------+ | Client | -----> | Daemon | -----> | Actor | +-----------+ +-----------+ +-----------+ | | | loads runtime | | loads extension(s) | | executes TEA or | | process() | | returns results | ``` Typical flow: 1. Client (CLI, Morphir-Live, etc.) sends a request: "run transform X on workspace Y". 2. Daemon: - creates or reuses an actor for that workspace/session - instructs it to load extension X (if not already loaded) 3. Actor: - loads the extension via Extism - calls `process` or `init/update/subscriptions` - uses host functions as needed - returns results to daemon 4. Daemon streams results back to client. ### Direct runtime flow (WASM client, Morphir-Live standalone) ```text +------------------+ +------------------+ +-------------------+ | WASM Client | -----> | Runtime (Wasm) | -----> | Extension (Wasm) | +------------------+ +------------------+ +-------------------+ (JS/TS, Rust, (Rust compiled (core/component) Python, etc.) to Wasm) ``` Typical flow: 1. Client code (JS/TS, Rust, Python, etc.) calls runtime API with envelope. 2. Runtime (compiled to Wasm): - loads extension if not already loaded - calls extension function with envelope - interprets commands/subscriptions - calls host functions (browser APIs, WASI, custom imports) 3. Runtime returns result envelope to client. 4. Client handles result (update UI, process data, etc.). **Variants**: - **Core Wasm**: Uses `WebAssembly.instantiate` or equivalent host API - **WASM Component + WASI**: Uses WIT bindings, can access filesystem, network, environment via WASI ## Client deployment modes Because Extism is hidden and the envelope protocol is host-agnostic, multiple deployment modes are supported: ### Native/Daemon Mode - **CLI Client**: Native binary communicating with daemon - **Daemon**: Long-running service managing Kameo actors - **Runtime**: Uses Extism internally to load extensions - **Use cases**: Development, CI/CD, backend services ### Morphir-Live Progressive Web App - **Client**: Progressive Web App from finos/morphir (Rust → Wasm) - **Architecture**: Rust runtime compiled to Wasm, running in browser - **Actor support**: Can run Kameo actors in Wasm or make direct calls - **Daemon option**: Can connect to remote daemon or run standalone - **Use cases**: Interactive exploration, live documentation, educational demos ### WASM Client (Portable Wasm) - **Client**: Runtime compiled to Wasm, runnable in various environments - **Direct calls**: No daemon required, runtime and extensions all Wasm - **Variants**: - **Core Wasm (wasm32-unknown-unknown)**: - Browser: Via `WebAssembly.instantiate` - Node/Deno/Bun: Via WebAssembly APIs - Wasmtime/wasmer: Embedded in other applications - **WASM Component Model + WASI**: - Wasmtime: Full WASI support, filesystem, network - Wasmer: WASI support - Browser: Via WASI polyfills or jco - Enables richer host functions (file I/O, sockets, etc.) - **Extism alternative**: Can replace Extism with: - Direct `WebAssembly.instantiate` for core Wasm - jco/wasm-tools for Component Model - Host-native Wasm APIs - **Use cases**: - Embedded widgets in web apps - Static sites with client-side IR processing - Serverless/edge functions - Portable CLI tools (single Wasm binary) - Sandboxed execution environments ### Service/Backend Mode - **Long-running services**: API servers, data processing pipelines - **Actor pools**: Managed Kameo actors for concurrency - **Persistence**: Cached compilation results, persistent workspace state - **Use cases**: Production deployments, scalable processing ### Portable Runtime Mode - **Other hosts**: Wasmtime, GraalVM, Node/Deno/Bun - **Same ABI**: Envelope protocol + TEA/design-time interfaces - **Engine flexibility**: Can swap Extism for host-specific Wasm implementation - **Use cases**: Embedding in existing platforms, custom integrations The key: **extension authors never see the engine** and **the same extension works across all deployment modes**. ## Crate structure The implementation is organized into the following crates: ### `morphir-ext-core` **Status:** ✅ Complete Core protocol definitions shared by all extension implementations: - `Envelope` struct with `Header`, `content_type`, and `content` - JSON codec for envelope serialization/deserialization - Core WASM ABI definitions (pointer-based memory access) - WIT interface definitions (envelope.wit, program.wit, runtime.wit) ### `morphir-ext` **Status:** 🔄 In Progress Main extension runtime and execution infrastructure: - ✅ `ExtensionRuntime` trait (abstraction over WASM engines) - ✅ `ExtensionInstance` (wraps `ExtensionRuntime`, manages state) - ✅ `ExtensionActor` (Kameo actor wrapper for daemon mode) - 🔴 `ExtismRuntime` (Extism implementation of `ExtensionRuntime`) - 🔴 `DirectRuntime` (direct execution without actors) - 🔴 `DaemonClient` (IPC client connecting to daemon actors) - 🔴 Host functions (log, HTTP, workspace, IR helpers) **Key insight:** `morphir-ext` contains both: - The **interface** (`ExtensionRuntime` trait) - Multiple **implementations** (Extism, direct, daemon client) - **Execution modes** (direct vs actor-based) **Client flexibility:** All clients can choose their execution mode: - **DirectRuntime**: Embedded execution (CLI, Morphir-Live, browser/WASM) - No daemon required - Extensions run in-process - Suitable for single-user, local execution - **DaemonClient**: Remote execution via daemon (CLI, Morphir-Live, IDE) - Shared state and caching across sessions - File watching and incremental compilation - Multi-user/multi-project support The choice depends on deployment needs, not client type. ### `morphir-daemon` **Status:** 🔄 In Progress Daemon service for long-running extension hosting: - JSON-RPC server for CLI/IDE integration - File watching and incremental compilation - Extension registry and loading - Actor pool management (using `morphir-ext` actors) - Workspace and session state management ### `morphir-runtime` **Status:** 🔴 Not Started Runtime execution semantics (TEA-style runtime extensions): - IR evaluation engine - Effect handlers and interpreters - Domain logic execution - Uses `morphir-ext` for extension hosting ### `morphir-design` **Status:** 🔴 Not Started Design-time operations (frontends, backends, transforms): - Frontend compilation (source → IR) - Backend generation (IR → target code) - IR transformations and decorators - Uses `morphir-ext` for extension hosting ### `morphir-builtins` **Status:** ✅ Complete (structure), 🔄 In Progress (migrate implementation) Builtin extensions bundled with Morphir: - ✅ `BuiltinExtension` trait (native + optional WASM) - ✅ `BuiltinRegistry` for discovery - 🔄 `migrate` extension (IR v3 ↔ v4 transformation) - 🔴 Future: TypeScript backend, Scala backend, etc. **Dual execution modes:** - **Native**: Direct Rust implementation (always available, best performance) - **WASM**: Compiled to WASM for extension architecture validation **Key insight:** Builtins are **both** usable as Rust libraries AND as WASM extensions, providing flexibility and testing coverage for the extension architecture. ## Steel Thread: Migrate Command **Goal:** Build a working end-to-end slice of functionality using the `migrate` command as a design-time extension (IR transform/backend). ### Why Migrate Command? The `migrate` command transforms Morphir IR from one version to another, making it an ideal steel thread because: - **Design-time operation**: IR → IR transform (backend pattern) - **Self-contained**: No runtime execution needed - **Real-world use case**: Actual production requirement - **Tests the full stack**: Extension loading, envelope protocol, DirectRuntime, WASM execution ### Steel Thread Scope Implement **minimal viable path** through the architecture: ```text CLI ---> DirectRuntime ---> ExtismRuntime ---> migrate.wasm (morphir-ext) (morphir-ext) (extension) | v ExtensionInstance (state management) ``` ### Implementation Steps (Steel Thread) 1. **Minimal ExtismRuntime** (morphir-rust-ext1) - Load WASM module via Extism - Implement `call_envelope()` only - Skip TEA helpers for now - Basic error handling 2. **Minimal DirectRuntime** (morphir-rust-direct1) - Wrap ExtismRuntime - Expose simple API: `execute(func, input) -> output` - No state management initially 3. **Minimal Host Functions** (morphir-rust-std1) - `log()` only for debugging - Skip HTTP, workspace, etc. for now 4. **Migrate Extension** (morphir-builtins) - ✅ Already created in `morphir-builtins` crate - Implements `BuiltinExtension` trait - Native Rust implementation available immediately - Can be compiled to WASM for architecture testing - Takes IR envelope → returns migrated IR envelope 5. **CLI Integration** - `morphir migrate` command can use either: - **Native mode**: Call `MigrateExtension::execute_native()` directly (fast) - **WASM mode**: Load via DirectRuntime + ExtismRuntime (validates architecture) - Start with native mode, add WASM option later ### Success Criteria (Steel Thread) - [ ] `morphir migrate input.json output.json` works end-to-end - [ ] Extension loaded via ExtismRuntime - [ ] Envelope protocol used throughout - [ ] No daemon required (DirectRuntime only) - [ ] Demonstrates extension architecture viability ### After Steel Thread Once the steel thread works, expand: - Add full TEA runtime support - Add DaemonClient for remote execution - Add more host functions - Build more complex extensions ## Implementation roadmap ### Phase 0: Steel Thread (P0 - FIRST!) Build working end-to-end with `migrate` command: 1. Minimal ExtismRuntime (call_envelope only) 2. Minimal DirectRuntime (simple wrapper) 3. Minimal host functions (log only) 4. Migrate extension (IR v3 → v4) 5. CLI integration **Target:** Working `morphir migrate` using extension architecture ### Phase 1: Core Runtime (P1) 1. ✅ **Envelope layer** (morphir-rust-env1) - `Envelope` + JSON codec implemented in `morphir-ext-core` - JSON payload conventions defined 2. 🔄 **Extism integration** (morphir-rust-ext1) - Implement `ExtismRuntime` in `morphir-ext/src/extism_runtime.rs` - Implement `ExtensionRuntime` trait for `ExtismRuntime` - Register basic host functions (log, random) 3. 🔄 **TEA runtime** (morphir-rust-tea1) - Already have TEA helpers in `ExtensionRuntime` trait - Add command interpreter - Add subscription manager - Implement `start`, `send`, `poll` semantics 4. **Design-time runtime** (morphir-rust-dt1) - Implement `process` calls for design-time extensions - Add extension registry in `morphir-daemon` - Support frontend-compile and backend-generate 5. **Standard library** (morphir-rust-std1) - Implement host functions in `morphir-ext/src/host_functions.rs` - HTTP, workspace, IR helpers - Register with Extism - Provide language-specific SDKs for extension authors ### Phase 2: Native Client Support (P1) 6. ✅ **Kameo actors** (morphir-rust-kam1) - `ExtensionActor` implemented in `morphir-ext/src/actor.rs` - Wraps `ExtensionInstance` - Message handlers for init/update/subscriptions 7. **Direct runtime** (NEW) - Implement `DirectRuntime` in `morphir-ext/src/direct.rs` - For browser/WASM mode (no actors, no IPC) - Wraps `ExtensionInstance` directly 8. **Daemon client** (NEW) - Implement `DaemonClient` in `morphir-ext/src/daemon_client.rs` - IPC to actor-based daemon - Implements `ExtensionRuntime` trait 9. **CLI + daemon** - Define CLI commands in `morphir` crate - Implement daemon routing to actors in `morphir-daemon` - Add session management and persistence ### Phase 3: WASM Client Support (P2) 8. **Runtime → Core WASM compilation** - Compile Morphir runtime to core Wasm (wasm32-unknown-unknown). - Test core runtime APIs in multiple environments: - Browser (via WebAssembly API) - Node/Deno/Bun (via WebAssembly APIs) - Wasmtime/wasmer (embedded mode) - Ensure Extism or alternative works in Wasm context. 9. **WASM client interface** - Define language bindings for runtime API: - JavaScript/TypeScript (for browser, Node, Deno, Bun) - Rust (via wasm-bindgen for browser, direct for Wasmtime) - Python (via wasmtime-py or similar) - Implement envelope protocol over language boundaries. - Add environment-appropriate host functions: - Browser: IndexedDB, fetch, localStorage - WASI: Filesystem, sockets, environment variables - Custom: Host-specific imports 10. **Morphir-Live integration (Dioxus framework)** - Create integration layer for finos/morphir's Morphir-Live app (Dioxus-based). - Support desktop mode (native binary with embedded runtime). - Support web mode (Rust → Wasm with direct runtime calls). - Support both standalone mode (embedded runtime) and daemon mode (connect to remote). - Add web-specific features (offline, service worker, caching) for web target. ### Phase 4: Component Model & Advanced Features (P2-P3) 11. **WASM Component Model + WASI support** - Define WIT for envelope + TEA + design-time. - Compile runtime as WASM Component (with WASI preview 2). - Provide wrappers for Component Model hosts: - Wasmtime (native WASI support) - Wasmer (WASI support) - Browser (via WASI polyfills or jco) - Implement WASI-based host functions: - Filesystem operations (wasi:filesystem) - Network sockets (wasi:sockets) - Environment variables (wasi:cli-base) - Test Component Model extensions across all client types. 12. **Cross-client testing** - Create test suite that runs on: - CLI (native) - Daemon (native) - Morphir-Live (Dioxus: web/desktop) - WASM clients (core Wasm: browser, Node, Wasmtime) - WASM Component clients (WASI: Wasmtime, wasmer) - Verify envelope protocol works consistently across all targets. - Benchmark performance across deployment modes: - Native vs core Wasm vs WASM Component - Browser vs Node vs Wasmtime - Standalone vs daemon mode - Test Dioxus web target with both standalone and daemon modes. - Validate WASI host functions work correctly in Wasmtime/wasmer. ## Recommended packaging For most extensions, **one crate with two feature-gated builds** (design-time vs runtime) provides the lowest maintenance burden while keeping the same Envelope ABI. This keeps shared types and helpers in one place, avoids unnecessary dependencies in each build, and produces a focused export surface. Choose a **single module that exports both interfaces** only when you need one Wasm artifact capable of both design-time and runtime behavior at runtime. ## Summary This design gives Morphir: - A **unified extension model** for design-time and runtime. - A **stable, engine-agnostic ABI** based on envelopes and TEA. - A **hidden Extism integration** that simplifies implementation without leaking into extension APIs. - A **Kameo actor-based host** that provides isolation, concurrency, and lifecycle management (for daemon/service mode). - A **standard library** exposed as host functions, not engine details. - **Multiple deployment modes**: - Native CLI + daemon for development and backend services - Morphir-Live (Dioxus) for web and desktop interactive exploration - Core WASM clients for embedded, browser, Node/Deno/Bun, and edge deployments - WASM Component + WASI for portable CLI tools and sandboxed execution - Service/backend mode for production deployments - A path to **Component Model + WASI** and **portable Wasm execution** without breaking existing extensions. - **Write once, run anywhere**: The same extension works across: - Native: CLI, daemon, services - Dioxus: Morphir-Live (web/desktop) - Core Wasm: Browser, Node, Deno, Bun, Wasmtime, wasmer - WASM Component + WASI: Wasmtime, wasmer, edge runtimes ## Related documents - [Actor-Based Extension System Overview](./00-overview.md) - [Extension Host Interface](./02-extension-host-interface.md) - [Extism WASM Host](./06-extism-wasm-host.md) - [Daemon Architecture](../../daemon/README.md) --- ## CLI-Daemon Interaction Source: https://finos.github.io/morphir-rust/contributors/design/daemon/cli-interaction # CLI-Daemon Interaction This document defines how the Morphir CLI communicates with the Morphir Daemon, including connection modes, transport protocols, and lifecycle management. ## Overview The Morphir CLI can operate in multiple modes depending on the use case: | Mode | Description | Use Case | | ----------------- | -------------------------------- | ---------------------------------------- | | **Embedded** | Daemon runs in-process | Simple commands, scripts, CI | | **Local Daemon** | Connects to local daemon process | IDE integration, watch mode, development | | **Remote Daemon** | Connects to remote daemon | Team servers, cloud builds | ## Connection Modes ### Embedded Mode (Default) In embedded mode, the CLI starts an in-process daemon for the duration of the command. This is the simplest mode and requires no external daemon process. ```bash # Embedded mode (default) morphir build morphir test morphir codegen --target spark ``` **Characteristics:** - No persistent state between commands - Full compilation on each invocation - Suitable for CI/CD pipelines and scripts - No daemon process management required ### Local Daemon Mode In local daemon mode, the CLI connects to a running daemon process on the local machine. This enables incremental builds, file watching, and IDE integration. ```bash # Start the daemon morphir daemon start # Commands connect to running daemon morphir build # Uses cached state morphir watch # Streams file changes morphir workspace add ./packages/new-project # Stop the daemon morphir daemon stop ``` **Characteristics:** - Persistent in-memory state - Incremental compilation - File watching support - Shared across CLI invocations and IDE ### Remote Daemon Mode In remote daemon mode, the CLI connects to a daemon running on a remote server. This enables team-shared build servers and cloud-based compilation. ```bash # Connect to remote daemon morphir --daemon https://build.example.com:9742 build # Or via environment variable export MORPHIR_DAEMON_URL=https://build.example.com:9742 morphir build ``` **Characteristics:** - Shared build cache across team ## Ad-Hoc Compilation Mode For quick experimentation and integration workloads, Morphir supports ad-hoc compilation without requiring a full project structure. This enables: - **Quick prototyping**: Try out Morphir without project setup - **Piped workflows**: Integration with shell pipelines - **Single-file compilation**: Compile individual files directly - **Code generation testing**: Generate code from snippets ### Input Language Selection Ad-hoc compilation requires specifying the input language (frontend) since there's no `morphir.toml` to infer it from: ```bash # Explicit language selection morphir compile --lang elm snippet.elm morphir compile --lang morphir-dsl snippet.morphir # Inferred from file extension morphir compile snippet.elm # Infers Elm frontend morphir compile snippet.morphir # Infers Morphir DSL frontend # For stdin, language must be specified echo "module Ex exposing (..)" | morphir compile --lang elm - ``` **Supported Languages:** | Language | Flag | File Extensions | | ----------- | -------------------- | ------------------- | | Elm | `--lang elm` | `.elm` | | Morphir DSL | `--lang morphir-dsl` | `.morphir`, `.mdsl` | | (Extension) | `--lang ` | Per extension | ### Stdin Input (Piping) Pipe Morphir source code directly to the CLI: ```bash # Compile from stdin (language required) echo 'module Example exposing (add) add : Int -> Int -> Int add a b = a + b' | morphir compile --lang elm - # Codegen from stdin cat myfile.elm | morphir codegen --target spark --lang elm - # Pipe through multiple tools morphir compile --lang elm - < snippet.elm | morphir codegen --target typescript - ``` **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "compile-001", "method": "compile/snippet", "params": { "language": "elm", "source": "module Example exposing (add)\n\nadd : Int -> Int -> Int\nadd a b = a + b", "options": { "moduleName": "Example" } } } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "compile-001", "result": { "ir": { "...compiled module IR..." }, "diagnostics": [] } } ``` ### Single File Compilation Compile a single file without a project: ```bash # Compile single file (language inferred from .elm extension) morphir compile src/Example.elm # Explicit language morphir compile --lang elm src/Example.elm # Compile and generate code morphir compile src/Example.elm | morphir codegen --target spark # Compile with inferred module name morphir compile Example.elm # Module name inferred as "Example" from filename ``` ### Multiple Files (No Project) Compile a set of files without a `morphir.toml`: ```bash # Compile multiple files morphir compile src/Types.elm src/Logic.elm src/Api.elm # Using glob patterns morphir compile "src/**/*.elm" # With explicit output morphir compile src/*.elm --output dist/ ``` **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "compile-002", "method": "compile/files", "params": { "language": "elm", "files": [ { "path": "Types.elm", "source": "module Types exposing (..)\n..." }, { "path": "Logic.elm", "source": "module Logic exposing (..)\n..." } ], "options": { "packageName": "adhoc" } } } ``` ### Inline Expressions Evaluate or compile inline expressions (useful for REPL-like workflows): ```bash # Compile an expression morphir eval "List.map (\\x -> x * 2) [1, 2, 3]" # Type-check an expression morphir check --expr "\\x -> x + 1" # Generate code for an expression morphir codegen --target typescript --expr "\\a b -> a + b" ``` **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "eval-001", "method": "compile/expression", "params": { "expression": "List.map (\\x -> x * 2) [1, 2, 3]", "context": { "imports": ["List"] } } } ``` ### Ad-Hoc Codegen Generate code from IR or source without a project. The `--target` flag selects the backend: ```bash # Codegen from compiled IR (stdin) morphir compile snippet.elm | morphir codegen --target spark - # Codegen from source file directly morphir codegen --target typescript src/Example.elm # Codegen with inline source morphir codegen --target spark --source "module Ex exposing (f)\nf x = x + 1" # Multiple targets morphir codegen --target spark --target typescript src/Example.elm ``` **Available Targets:** | Target | Flag | Output Language | | ------------ | ---------------------- | ----------------- | | Spark | `--target spark` | Scala (Spark API) | | Scala | `--target scala` | Pure Scala | | TypeScript | `--target typescript` | TypeScript | | JSON Schema | `--target json-schema` | JSON Schema | | (Extensions) | `--target ` | Via WASM backends | **Target Options:** ```bash # Pass options to target morphir codegen --target spark --option spark_version=3.5 # List available targets morphir codegen --list-targets ``` **Streaming Ad-Hoc Codegen:** ```bash # Stream codegen output as it's generated morphir compile "src/*.elm" | morphir codegen --target spark --stream - ``` ### Output Formats Ad-hoc compilation supports multiple output formats: ```bash # Output IR as JSON (default) morphir compile snippet.elm # Output IR as compact JSON morphir compile snippet.elm --format json-compact # Output just the types morphir compile snippet.elm --format types # Pretty-print the IR morphir compile snippet.elm --format pretty ``` ### Temporary Project Context For ad-hoc compilation that needs dependencies, specify them inline: ```bash # With SDK dependency morphir compile snippet.elm --with-dep morphir/sdk # With custom package name morphir compile snippet.elm --package my-org/experiment # With multiple dependencies morphir compile snippet.elm \ --with-dep morphir/sdk \ --with-dep "acme/utils@1.0.0" ``` **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "compile-003", "method": "compile/snippet", "params": { "language": "elm", "source": "module Example exposing (..)\nimport Json.Decode\n...", "options": { "packageName": "my-org/experiment", "dependencies": [ { "name": "morphir/sdk", "version": "latest" }, { "name": "morphir/json", "version": "1.0.0" } ] } } } ``` ### Use Cases | Use Case | Command | | -------------------- | ------------------------------------------------------------------- | | Quick syntax check | `echo "x = 1" \| morphir check -` | | Prototype a function | `morphir compile snippet.elm` | | Test codegen output | `morphir codegen --target spark snippet.elm` | | Shell pipeline | `cat src.elm \| morphir compile - \| morphir codegen --target ts -` | | CI validation | `morphir compile --check-only "src/**/*.elm"` | | REPL-style eval | `morphir eval "1 + 2"` | ### Limitations Ad-hoc mode has some limitations compared to project mode: | Feature | Ad-Hoc | Project | | ----------------------- | --------------------- | --------- | | Dependency resolution | Manual (`--with-dep`) | Automatic | | Incremental compilation | No | Yes | | Caching | No | Yes | | Multi-module imports | Limited | Full | | Watch mode | No | Yes | | IDE integration | No | Yes | For anything beyond quick prototyping, create a project with `morphir init`. - Centralized dependency resolution - Requires authentication (see [Security](#security)) ## Transport Protocols ### HTTP/JSON-RPC (Primary) The primary transport is JSON-RPC 2.0 over HTTP. This provides a standard, debuggable protocol that works across network boundaries. ``` ┌─────────────┐ HTTP/JSON-RPC ┌─────────────┐ │ │ ─────────────────────────────► │ │ │ Morphir │ │ Morphir │ │ CLI │ ◄───────────────────────────── │ Daemon │ │ │ JSON-RPC Response │ │ └─────────────┘ └─────────────┘ ``` **Default Ports:** - Local daemon: `http://localhost:9741` - Remote daemon: `https://:9742` (TLS required) **Request Format:** ```json { "jsonrpc": "2.0", "id": "req-001", "method": "workspace/build", "params": { "projects": ["my-org/core"] } } ``` **Response Format:** ```json { "jsonrpc": "2.0", "id": "req-001", "result": { "success": true, "diagnostics": [] } } ``` ### Unix Domain Socket (Local Only) For local daemon connections, Unix domain sockets provide lower latency and better security than TCP. ```bash # Socket location $XDG_RUNTIME_DIR/morphir/daemon.sock # Fallback: /tmp/morphir-/daemon.sock ``` The CLI automatically uses the socket when available: ```bash # CLI checks for socket first, falls back to HTTP morphir build ``` ### Stdio (LSP/Embedded) For IDE integration via LSP and embedded mode, the daemon communicates over stdin/stdout: ``` ┌─────────────┐ stdin/stdout ┌─────────────┐ │ │ ─────────────────────────────► │ │ │ IDE │ JSON-RPC │ Morphir │ │ (LSP) │ ◄───────────────────────────── │ Daemon │ │ │ │ (stdio) │ └─────────────┘ └─────────────┘ ``` **Launch Command (LSP):** ```bash morphir daemon --stdio ``` ## Daemon Lifecycle ### Starting the Daemon ```bash # Start daemon in background (default) morphir daemon start # Start with specific options morphir daemon start --port 9741 --workspace /path/to/workspace # Start in foreground (for debugging) morphir daemon start --foreground # Start with verbose logging morphir daemon start --log-level debug ``` **Startup Sequence:** 1. Check for existing daemon (via pidfile/socket) 2. If running, verify health and exit 3. Bind to transport (socket/port) 4. Write pidfile to `$XDG_RUNTIME_DIR/morphir/daemon.pid` 5. Initialize workspace (if specified) 6. Begin accepting connections ### Daemon Status ```bash # Check daemon status morphir daemon status ``` **Output:** ``` Morphir Daemon Status: running PID: 12345 Uptime: 2h 34m Socket: /run/user/1000/morphir/daemon.sock HTTP: http://localhost:9741 Workspace: /home/user/my-workspace (open) Projects: 3 loaded, 0 stale Memory: 124 MB ``` ### Health Check ```bash # Health check (exit code 0 if healthy) morphir daemon health ``` **JSON-RPC Method:** ```json { "jsonrpc": "2.0", "id": "health-001", "method": "daemon/health", "params": {} } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "health-001", "result": { "status": "healthy", "version": "0.4.0", "uptime_seconds": 9240, "workspace": { "root": "/home/user/my-workspace", "state": "open", "projects": 3 } } } ``` ### Stopping the Daemon ```bash # Graceful shutdown morphir daemon stop # Force stop (SIGKILL) morphir daemon stop --force # Restart morphir daemon restart ``` **Shutdown Sequence:** 1. Stop accepting new connections 2. Complete in-flight requests (with timeout) 3. Flush pending writes 4. Close workspace 5. Remove pidfile and socket 6. Exit ### Auto-Start The CLI can automatically start a daemon when needed: ```toml # morphir.toml or ~/.config/morphir/config.toml [daemon] auto_start = true # Start daemon if not running auto_stop = false # Keep daemon running after CLI exits idle_timeout = "30m" # Stop after idle period (0 = never) ``` ## Extension Management The CLI provides commands to discover, inspect, and manage extensions. ### List Extensions ```bash # List all registered extensions morphir extension list # Output: # NAME VERSION TYPE STATUS CAPABILITIES # spark-codegen 1.2.0 codegen ready generate, streaming, incremental # elm-frontend 0.19.1 frontend ready compile, diagnostics # my-extension 0.1.0 - ready (info only) # List with details morphir extension list --verbose # Filter by type morphir extension list --type codegen morphir extension list --type frontend ``` ### Inspect Extension ```bash # Get detailed info about an extension morphir extension info spark-codegen # Output: # spark-codegen v1.2.0 # ═══════════════════════════════════════════════════════ # Type: codegen # Description: Generate Apache Spark DataFrame code from Morphir IR # Author: Morphir Contributors # Homepage: https://github.com/finos/morphir-spark # License: Apache-2.0 # Source: ./extensions/spark-codegen.wasm # # Capabilities: # ✓ codegen/generate # ✓ codegen/generate-streaming # ✓ codegen/generate-incremental # ✓ codegen/generate-module # ✓ codegen/options-schema # # Targets: spark # # Options: # spark_version string "3.5" Spark version to target # scala_version string "2.13" Scala version to target # Output as JSON morphir extension info spark-codegen --json ``` ### Verify Extension Health ```bash # Ping single extension morphir extension ping spark-codegen # spark-codegen: OK (2ms) # Ping all extensions morphir extension ping --all # spark-codegen: OK (2ms) # elm-frontend: OK (1ms) # my-extension: OK (1ms) # Ping with timeout morphir extension ping spark-codegen --timeout 5s ``` ### Extension Installation ```bash # Install from URL morphir extension install https://extensions.morphir.dev/spark-codegen-1.2.0.wasm # Install from local file morphir extension install ./my-extension.wasm # Install from registry (future) morphir extension install morphir/spark-codegen@1.2.0 # Uninstall morphir extension uninstall spark-codegen # Update morphir extension update spark-codegen ``` ### Extension Configuration ```bash # Show extension config morphir extension config spark-codegen # Set extension option morphir extension config spark-codegen --set spark_version=3.5 # Reset to defaults morphir extension config spark-codegen --reset ``` ## CLI Command Mapping ### Command to JSON-RPC Translation | CLI Command | JSON-RPC Method | Notes | | ------------------------------ | --------------------------- | ----------------------------------------------- | | `morphir build` | `workspace/buildAll` | Builds all projects | | `morphir build --stream` | `workspace/buildStreaming` | Streaming build with per-module notifications | | `morphir build ` | `compile/project` | Builds specific project | | `morphir test` | `workspace/test` | Runs tests | | `morphir check` | `workspace/check` | Runs linting/validation | | `morphir codegen` | `codegen/generate` | Generates code for targets | | `morphir codegen --stream` | `codegen/generateStreaming` | Streaming codegen with per-module notifications | | `morphir watch` | `workspace/watch` | Enables file watching | | `morphir workspace init` | `workspace/create` | Creates new workspace | | `morphir workspace add` | `workspace/addProject` | Adds project to workspace | | `morphir clean` | `workspace/clean` | Cleans build artifacts | | `morphir compile -` | `compile/snippet` | Compile from stdin | | `morphir compile ` | `compile/files` | Compile single file (no project) | | `morphir compile ` | `compile/files` | Compile multiple files (no project) | | `morphir eval ` | `compile/expression` | Evaluate inline expression | | `morphir check --expr ` | `compile/expression` | Type-check inline expression | | `morphir extension list` | `extension/list` | List registered extensions | | `morphir extension info ` | `extension/info` | Get extension details | | `morphir extension ping ` | `extension/ping` | Verify extension connectivity | | `morphir extension install` | `extension/install` | Install extension | | `morphir extension uninstall` | `extension/uninstall` | Remove extension | ### Streaming Operations Morphir tasks like `build` and `codegen` support streaming to avoid producing all output in one shot. This is critical for large projects where: - Compilation/generation takes significant time - Results should appear progressively - Early errors should surface immediately - Memory shouldn't hold the entire output #### Streaming Model ``` ┌─────────┐ ┌─────────┐ ┌─────────────────────────┐ │ CLI │─────►│ Request │─────►│ Daemon │ │ │ └─────────┘ │ │ │ │ │ ┌─────────────────┐ │ │ │◄─────────────────────────│ Notification 1 │ │ │ │ │ └─────────────────┘ │ │ │◄─────────────────────────│ Notification 2 │ │ │ │ │ └─────────────────┘ │ │ │◄─────────────────────────│ Notification N │ │ │ │ │ └─────────────────┘ │ │ │◄─────────────────────────│ Final Response │ │ └─────────┘ └─────────────────────────┘ ``` #### Streaming Build ```bash morphir build --stream ``` **Request:** ```json { "jsonrpc": "2.0", "id": "build-001", "method": "workspace/buildStreaming", "params": { "projects": ["my-org/domain"], "streaming": { "granularity": "module", "includeIR": true } } } ``` **Notifications (streamed as modules compile):** ```json { "method": "build/started", "params": { "project": "my-org/domain", "modules": 12 } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "Types"], "status": "ok" } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "User"], "status": "ok" } } { "method": "build/moduleCompiled", "params": { "module": ["Domain", "Order"], "status": "partial", "diagnostics": [...] } } ``` **Final Response:** ```json { "jsonrpc": "2.0", "id": "build-001", "result": { "success": true, "modulesCompiled": 12, "durationMs": 3421 } } ``` #### Streaming Codegen ```bash morphir codegen --target spark --stream ``` **Request:** ```json { "jsonrpc": "2.0", "id": "codegen-001", "method": "codegen/generateStreaming", "params": { "target": "spark", "streaming": { "granularity": "module", "writeImmediately": true } } } ``` **Notifications (streamed as files are generated):** ```json { "method": "codegen/started", "params": { "target": "spark", "modules": 12 } } { "method": "codegen/moduleGenerated", "params": { "module": ["Domain", "Types"], "files": ["Types.scala"] } } { "method": "codegen/moduleGenerated", "params": { "module": ["Domain", "User"], "files": ["User.scala"] } } { "method": "codegen/fileWritten", "params": { "path": "src/main/scala/domain/User.scala" } } ``` #### CLI Streaming Display The CLI renders streaming notifications in real-time: ``` Building my-org/domain (12 modules) ✓ Domain.Types [42ms] ✓ Domain.User [38ms] ⚠ Domain.Order [51ms] (2 warnings) ● Domain.Product [compiling...] ``` #### Cancellation Streaming operations can be cancelled mid-flight: ```json { "jsonrpc": "2.0", "method": "$/cancelRequest", "params": { "id": "build-001" } } ``` The daemon stops processing and returns partial results: ```json { "jsonrpc": "2.0", "id": "build-001", "result": { "cancelled": true, "modulesCompiled": 5, "modulesRemaining": 7 } } ``` ### Progress Notifications For non-streaming operations, progress is reported via LSP-style notifications: ```bash morphir build --progress ``` **Progress Notifications:** ```json { "jsonrpc": "2.0", "method": "$/progress", "params": { "token": "build-001", "value": { "kind": "report", "message": "Compiling my-org/core...", "percentage": 45 } } } ``` ### Diagnostic Streaming Build diagnostics stream as they're discovered (not batched until the end): ```json { "jsonrpc": "2.0", "method": "textDocument/publishDiagnostics", "params": { "uri": "file:///path/to/src/Domain/User.elm", "diagnostics": [ { "range": { "start": { "line": 10, "character": 5 }, "end": { "line": 10, "character": 15 } }, "severity": 1, "message": "Type mismatch: expected Int, got String" } ] } } ``` This allows the CLI to display errors immediately and IDEs to show diagnostics in real-time. ## Connection Management ### Discovery The CLI discovers the daemon in this order: 1. **Explicit URL**: `--daemon ` or `MORPHIR_DAEMON_URL` 2. **Unix Socket**: `$XDG_RUNTIME_DIR/morphir/daemon.sock` 3. **Local HTTP**: `http://localhost:9741` 4. **Embedded**: Start in-process daemon ```bash # Explicit daemon URL morphir --daemon http://localhost:9741 build # Environment variable export MORPHIR_DAEMON_URL=http://localhost:9741 morphir build ``` ### Connection Timeout ```toml # ~/.config/morphir/config.toml [daemon] connect_timeout = "5s" # Time to wait for connection request_timeout = "60s" # Time to wait for response ``` ### Reconnection If the daemon connection is lost during a long-running operation: 1. CLI detects connection failure 2. Attempts reconnection (3 retries with exponential backoff) 3. If daemon is gone, offers to restart 4. Resumes operation if state is recoverable ``` Connection lost. Attempting to reconnect... Retry 1/3: connection refused Retry 2/3: connection refused Retry 3/3: connected Daemon restarted. Rebuilding project state... ``` ## Security ### Local Daemon Local daemon connections are secured by: - Unix socket permissions (owner-only by default) - Pidfile verification - No authentication required for local connections ### Remote Daemon Remote daemon connections require: - TLS encryption (HTTPS) - Authentication token ```bash # Set authentication token export MORPHIR_DAEMON_TOKEN= # Or via config file # ~/.config/morphir/config.toml [daemon] url = "https://build.example.com:9742" token = "" # Or use keyring/credential helper ``` **Token Authentication:** ```json { "jsonrpc": "2.0", "id": "auth-001", "method": "daemon/authenticate", "params": { "token": "" } } ``` ### Capability Restrictions Remote daemons can restrict capabilities: ```json { "jsonrpc": "2.0", "id": "caps-001", "method": "daemon/capabilities", "params": {} } ``` **Response:** ```json { "jsonrpc": "2.0", "id": "caps-001", "result": { "capabilities": { "workspace/create": false, "workspace/build": true, "workspace/watch": false, "codegen/generate": true, "extensions/install": false } } } ``` ## Configuration ### Daemon Configuration ```toml # morphir.toml (project/workspace level) [daemon] # Connection settings port = 9741 socket = true # Enable Unix socket # Behavior auto_start = true idle_timeout = "30m" # Resource limits max_memory = "2GB" max_projects = 50 ``` ### Global Configuration ```toml # ~/.config/morphir/config.toml (user level) [daemon] # Default daemon URL for remote connections url = "https://build.example.com:9742" token_command = "pass show morphir/daemon-token" # Local daemon settings auto_start = true log_level = "info" log_file = "~/.local/state/morphir/daemon.log" ``` ## Error Handling ### Connection Errors | Error | CLI Behavior | | --------------------- | -------------------------------------- | | Daemon not running | Auto-start (if enabled) or prompt user | | Connection refused | Retry with backoff, then fail | | Authentication failed | Prompt for credentials or fail | | Timeout | Retry or fail with message | ### Request Errors JSON-RPC errors are mapped to CLI exit codes: | JSON-RPC Error Code | Exit Code | Meaning | | ------------------- | --------- | ---------------- | | -32700 | 2 | Parse error | | -32600 | 2 | Invalid request | | -32601 | 2 | Method not found | | -32602 | 2 | Invalid params | | -32603 | 1 | Internal error | | -32000 to -32099 | 1 | Server errors | **Error Response:** ```json { "jsonrpc": "2.0", "id": "req-001", "error": { "code": -32603, "message": "Compilation failed", "data": { "diagnostics": [...] } } } ``` ## Logging and Debugging ### Daemon Logs ```bash # View daemon logs morphir daemon logs # Follow logs morphir daemon logs -f # Log location ~/.local/state/morphir/daemon.log ``` ### Debug Mode ```bash # Enable verbose CLI output morphir --verbose build # Enable JSON-RPC tracing morphir --trace-rpc build # Trace output --> {"jsonrpc":"2.0","id":"1","method":"workspace/build","params":{}} <-- {"jsonrpc":"2.0","id":"1","result":{"success":true}} ``` ### Diagnostics Dump ```bash # Dump daemon state for debugging morphir daemon dump > daemon-state.json ``` ## Related ### Morphir Rust Design Documents - **[Morphir Daemon](./README.md)** - Daemon overview and architecture - **[Build Operations](./build.md)** - Build orchestration details - **[File Watching](./watching.md)** - Watch mode implementation ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification --- ## Examples and Recipes Source: https://finos.github.io/morphir-rust/contributors/design/extensions/actor-based/11-examples-and-recipes # Examples and Recipes **Status:** Draft **Version:** 0.1.0 ## Overview This document provides practical examples and common patterns for building Morphir extensions. ## Quick Start Examples ### Minimal Python Extension (Stdio) ```python #!/usr/bin/env python3 import sys import json def main(): for line in sys.stdin: request = json.loads(line) method = request["method"] if method == "initialize": response = {"id": request["id"], "result": {"status": "ready"}} elif method == "capabilities": response = {"id": request["id"], "result": [ {"name": "hello", "description": "Say hello"} ]} elif method == "hello": name = request["params"].get("name", "World") response = {"id": request["id"], "result": f"Hello, {name}!"} else: response = {"id": request["id"], "error": f"Unknown method: {method}"} print(json.dumps(response), flush=True) if __name__ == "__main__": main() ``` **Configuration:** ```toml [[extensions]] name = "hello-python" protocol = "stdio" source = { type = "process", command = "python3", args = ["./hello.py"] } ``` ### Minimal TypeScript Extension (JSON-RPC) ```typescript import { createServer } from "jayson"; const server = createServer({ initialize: (params: any, callback: any) => { callback(null, { status: "ready" }); }, capabilities: (params: any, callback: any) => { callback(null, [{ name: "greet", description: "Greet someone" }]); }, greet: (params: { name: string }, callback: any) => { callback(null, { message: `Hello, ${params.name}!` }); }, }); server.http().listen(3000); console.log("Extension listening on port 3000"); ``` **Configuration:** ```toml [[extensions]] name = "hello-typescript" protocol = "jsonrpc" source = { type = "http", url = "http://localhost:3000" } ``` ### Minimal Rust Extension (Extism WASM) ```rust use extism_pdk::*; use serde::{Deserialize, Serialize}; #[derive(Deserialize)] struct GreetParams { name: String, } #[derive(Serialize)] struct GreetResult { message: String, } #[plugin_fn] pub fn initialize(_config: String) -> FnResult { Ok(r#"{"status": "ready"}"#.to_string()) } #[plugin_fn] pub fn capabilities() -> FnResult { Ok(r#"[{"name": "greet", "description": "Greet someone"}]"#.to_string()) } #[plugin_fn] pub fn greet(params_json: String) -> FnResult { let params: GreetParams = serde_json::from_str(¶ms_json)?; let result = GreetResult { message: format!("Hello, {}!", params.name), }; Ok(serde_json::to_string(&result)?) } ``` **Build:** ```bash cargo build --target wasm32-unknown-unknown --release ``` **Configuration:** ```toml [[extensions]] name = "hello-rust" protocol = "extism" source = { type = "wasm", path = "./target/wasm32-unknown-unknown/release/hello.wasm" } ``` ## Common Patterns ### Pattern: IR Transformer Transform Morphir IR to another format. ```python import sys import json def transform_ir(ir): # Transform logic here return { "transformed": True, "output": ir # Replace with actual transformation } def main(): for line in sys.stdin: request = json.loads(line) method = request["method"] if method == "transform": ir = request["params"]["ir"] result = transform_ir(ir) response = {"id": request["id"], "result": result} # ... other methods print(json.dumps(response), flush=True) if __name__ == "__main__": main() ``` ### Pattern: IR Validator Validate Morphir IR for custom business rules. ```python def validate_ir(ir): errors = [] warnings = [] # Validation logic if not ir.get("modules"): errors.append("No modules found") # Check for naming conventions for module in ir.get("modules", []): if not module["name"].isupper(): warnings.append(f"Module {module['name']} should be uppercase") return { "valid": len(errors) == 0, "errors": errors, "warnings": warnings } def main(): for line in sys.stdin: request = json.loads(line) if request["method"] == "validate": ir = request["params"]["ir"] result = validate_ir(ir) response = {"id": request["id"], "result": result} print(json.dumps(response), flush=True) ``` ### Pattern: Code Generator Generate code from Morphir IR. ```typescript import { createServer } from "jayson"; function generateCode(ir: any, options: any): string { // Code generation logic const lines: string[] = []; for (const module of ir.modules) { lines.push(`export module ${module.name} {`); for (const func of module.functions) { lines.push(` export function ${func.name}() {`); lines.push(` // Generated from Morphir IR`); lines.push(` }`); } lines.push(`}`); } return lines.join("\n"); } const server = createServer({ generate: (params: any, callback: any) => { try { const code = generateCode(params.ir, params.options || {}); callback(null, { code }); } catch (error) { callback(error); } }, }); server.http().listen(3000); ``` ### Pattern: Stateful Extension Maintain state across calls. ```python import sys import json class StatefulExtension: def __init__(self): self.state = {} def set_state(self, params): key = params["key"] value = params["value"] self.state[key] = value return {"success": True} def get_state(self, params): key = params["key"] return {"value": self.state.get(key)} def clear_state(self, params): self.state = {} return {"success": True} def main(): extension = StatefulExtension() handlers = { "set_state": extension.set_state, "get_state": extension.get_state, "clear_state": extension.clear_state, } for line in sys.stdin: request = json.loads(line) method = request["method"] if method in handlers: result = handlers[method](request["params"]) response = {"id": request["id"], "result": result} else: response = {"id": request["id"], "error": f"Unknown method: {method}"} print(json.dumps(response), flush=True) if __name__ == "__main__": main() ``` ### Pattern: Async Processing Handle long-running operations asynchronously. ```typescript import { createServer } from "jayson"; import { EventEmitter } from "events"; const jobs = new Map(); const server = createServer({ // Start async job startJob: (params: any, callback: any) => { const jobId = crypto.randomUUID(); const emitter = new EventEmitter(); jobs.set(jobId, emitter); // Simulate long-running work setTimeout(() => { emitter.emit("complete", { result: "Job completed!" }); }, 5000); callback(null, { jobId, status: "started" }); }, // Check job status getJobStatus: (params: { jobId: string }, callback: any) => { const emitter = jobs.get(params.jobId); if (!emitter) { callback(new Error("Job not found")); return; } callback(null, { status: "running" }); }, // Get job result (blocks until complete) getJobResult: (params: { jobId: string }, callback: any) => { const emitter = jobs.get(params.jobId); if (!emitter) { callback(new Error("Job not found")); return; } emitter.once("complete", (data) => { jobs.delete(params.jobId); callback(null, data); }); }, }); server.http().listen(3000); ``` ## Real-World Examples ### Example: TypeScript Code Generator A complete TypeScript backend generator. ```typescript // extension.ts import { createServer } from "jayson"; import * as ts from "typescript"; interface MorphirType { type: "function" | "record" | "custom"; name: string; fields?: any[]; } class TypeScriptGenerator { generate(ir: any): string { const sourceFile = ts.createSourceFile( "output.ts", "", ts.ScriptTarget.Latest, false, ts.ScriptKind.TS, ); const statements: ts.Statement[] = []; // Generate types for (const type of ir.types) { statements.push(this.generateType(type)); } // Generate functions for (const func of ir.functions) { statements.push(this.generateFunction(func)); } const printer = ts.createPrinter(); return statements .map((s) => printer.printNode(ts.EmitHint.Unspecified, s, sourceFile)) .join("\n\n"); } private generateType(type: MorphirType): ts.Statement { // Type generation logic return ts.factory.createTypeAliasDeclaration( [ts.factory.createModifier(ts.SyntaxKind.ExportKeyword)], type.name, undefined, ts.factory.createTypeLiteralNode([]), ); } private generateFunction(func: any): ts.Statement { // Function generation logic return ts.factory.createFunctionDeclaration( [ts.factory.createModifier(ts.SyntaxKind.ExportKeyword)], undefined, func.name, undefined, [], undefined, ts.factory.createBlock([]), ); } } const generator = new TypeScriptGenerator(); const server = createServer({ generate: (params: any, callback: any) => { try { const code = generator.generate(params.ir); callback(null, { code, language: "typescript" }); } catch (error) { callback(error); } }, }); server.http().listen(3000); ``` ### Example: Custom Linter Lint Morphir IR for organization-specific rules. ```python #!/usr/bin/env python3 import sys import json from typing import List, Dict, Any class MorphirLinter: def __init__(self): self.rules = [ self.check_naming_conventions, self.check_function_complexity, self.check_documentation, ] def lint(self, ir: Dict[str, Any]) -> Dict[str, Any]: errors = [] warnings = [] for rule in self.rules: rule_errors, rule_warnings = rule(ir) errors.extend(rule_errors) warnings.extend(rule_warnings) return { "valid": len(errors) == 0, "errors": errors, "warnings": warnings, "error_count": len(errors), "warning_count": len(warnings) } def check_naming_conventions(self, ir: Dict) -> tuple: errors = [] warnings = [] for module in ir.get("modules", []): # Module names should be PascalCase if not module["name"][0].isupper(): warnings.append({ "rule": "naming-convention", "message": f"Module '{module['name']}' should start with uppercase", "location": module.get("location") }) return errors, warnings def check_function_complexity(self, ir: Dict) -> tuple: errors = [] warnings = [] # Check cyclomatic complexity # (simplified for example) return errors, warnings def check_documentation(self, ir: Dict) -> tuple: errors = [] warnings = [] for module in ir.get("modules", []): if not module.get("documentation"): warnings.append({ "rule": "missing-documentation", "message": f"Module '{module['name']}' lacks documentation", "location": module.get("location") }) return errors, warnings def main(): linter = MorphirLinter() for line in sys.stdin: request = json.loads(line) method = request["method"] if method == "lint": ir = request["params"]["ir"] result = linter.lint(ir) response = {"id": request["id"], "result": result} else: response = {"id": request["id"], "error": f"Unknown method: {method}"} print(json.dumps(response), flush=True) if __name__ == "__main__": main() ``` ## Testing Extensions ### Unit Testing (Python) ```python import unittest import json from io import StringIO from extension import handle_request class TestExtension(unittest.TestCase): def test_initialize(self): request = {"id": 1, "method": "initialize", "params": {}} response = handle_request(request) self.assertEqual(response["result"]["status"], "ready") def test_capabilities(self): request = {"id": 2, "method": "capabilities", "params": {}} response = handle_request(request) self.assertIsInstance(response["result"], list) def test_transform(self): request = { "id": 3, "method": "transform", "params": {"ir": {"type": "Module"}} } response = handle_request(request) self.assertIn("output", response["result"]) if __name__ == '__main__': unittest.main() ``` ### Integration Testing (Bash) ```bash #!/bin/bash # Test extension manually echo '{"id":1,"method":"initialize","params":{}}' | python3 extension.py echo '{"id":2,"method":"capabilities","params":{}}' | python3 extension.py echo '{"id":3,"method":"echo","params":{"message":"test"}}' | python3 extension.py ``` ## Configuration Recipes ### Development Configuration ```toml [[extensions]] name = "dev-generator" enabled = true protocol = "stdio" source = { type = "process", command = "python3", args = ["./generator.py", "--verbose"] } permissions = { filesystem = ["./output", "./temp"] } restart = { strategy = "immediate", max_retries = 0 } ``` ### Production Configuration ```toml [[extensions]] name = "prod-generator" enabled = true protocol = "grpc" source = { type = "grpc", endpoint = "https://generator.internal:50051" } permissions = { network = true, max_execution_time = "30s" } restart = { strategy = "exponential", initial_delay = "1s", max_delay = "60s", max_retries = 5 } ``` ### Sandboxed Configuration (WASM) ```toml [[extensions]] name = "untrusted-transformer" enabled = true protocol = "extism" source = { type = "wasm", path = "./extensions/transformer.wasm" } permissions = { max_memory = "50MB", max_execution_time = "5s" } restart = { strategy = "never" } ``` ## Related ### Morphir Rust Design Documents - **[Morphir Extensions](../README.md)** - Extension system overview - **[WASM Components](../wasm-component.md)** - Component model integration - **[Tasks](../tasks.md)** - Task system definition ### Main Morphir Documentation - [Morphir Documentation](https://morphir.finos.org) - Main Morphir documentation site - [Morphir LLMs.txt](https://morphir.finos.org/llms.txt) - Machine-readable documentation index - [Morphir IR v4 Design](https://morphir.finos.org/docs/design/draft/ir/) - IR v4 design documents - [Morphir IR Specification](https://morphir.finos.org/docs/morphir-ir-specification/) - Complete IR specification ---