BDD Test Scenarios: IR JSON Schema Verification
BDD Test Scenarios: IR JSON Schema Verification
This document defines comprehensive BDD scenarios using Gherkin syntax for the IR JSON Schema Verification feature. These scenarios will be implemented as Reqnroll feature files in tests/Morphir.Core.Tests/Features/.
Related: IR JSON Schema Verification PRD
Feature 1: IR Schema Verification
Feature File: IrSchemaVerification.feature
Feature: IR Schema Verification
As a Morphir developer
I want to validate IR JSON files against schemas
So that I can catch structural errors early
Background:
Given the Morphir CLI is installed
And the schema files v1, v2, and v3 are available
Rule: Valid IR files pass validation
Scenario: Validate a valid v3 IR file
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "morphir ir verify valid-v3.json"
Then the exit code should be 0
And the output should contain "✓ Validation successful"
And the output should contain "Schema: v3 (auto-detected)"
And the output should contain "File: valid-v3.json"
Scenario: Validate a valid v2 IR file
Given a valid Morphir IR v2 JSON file "valid-v2.json"
When I run "morphir ir verify valid-v2.json"
Then the exit code should be 0
And the output should contain "✓ Validation successful"
And the output should contain "Schema: v2 (auto-detected)"
Scenario: Validate a valid v1 IR file
Given a valid Morphir IR v1 JSON file "valid-v1.json"
When I run "morphir ir verify valid-v1.json"
Then the exit code should be 0
And the output should contain "✓ Validation successful"
And the output should contain "Schema: v1 (auto-detected)"
Scenario Outline: Validate various valid IR files across versions
Given a valid Morphir IR <version> JSON file "<filename>"
When I run "morphir ir verify <filename>"
Then the exit code should be 0
And the output should contain "✓ Validation successful"
And the output should contain "Schema: <version> (auto-detected)"
Examples:
| version | filename |
| v1 | library-v1.json |
| v1 | complex-types-v1.json |
| v2 | library-v2.json |
| v2 | complex-types-v2.json |
| v3 | library-v3.json |
| v3 | complex-types-v3.json |
Rule: Invalid IR files fail validation with clear errors
Scenario: Validate an IR file with incorrect tag capitalization
Given an invalid Morphir IR v3 JSON file "invalid-tags.json" with lowercase tags
When I run "morphir ir verify invalid-tags.json"
Then the exit code should be 1
And the output should contain "✗ Validation failed"
And the output should contain "Invalid type tag"
And the output should contain "Expected: \"Public\" or \"Private\""
And the output should contain "Found: \"public\""
Scenario: Validate an IR file with missing required fields
Given an invalid Morphir IR v3 JSON file "missing-fields.json" missing the "name" field
When I run "morphir ir verify missing-fields.json"
Then the exit code should be 1
And the output should contain "✗ Validation failed"
And the output should contain "Missing required field"
And the output should contain "Path: $.package.modules"
And the output should contain "Required property 'name' is missing"
Scenario: Validate an IR file with invalid type structure
Given an invalid Morphir IR v3 JSON file "invalid-structure.json" with malformed type definitions
When I run "morphir ir verify invalid-structure.json"
Then the exit code should be 1
And the output should contain "✗ Validation failed"
And the error count should be greater than 0
Scenario: Validate an IR file with multiple errors
Given an invalid Morphir IR v3 JSON file "multiple-errors.json" with 5 validation errors
When I run "morphir ir verify multiple-errors.json"
Then the exit code should be 1
And the output should contain "5 errors found"
And the output should list all 5 errors with JSON paths
Rule: Schema version can be manually specified
Scenario: Force validation against specific schema version
Given a Morphir IR JSON file "mixed-version.json"
When I run "morphir ir verify --schema-version 2 mixed-version.json"
Then the validation should use schema v2
And the output should contain "Schema: v2 (manual)"
Scenario: Override auto-detection with explicit version
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "morphir ir verify --schema-version 3 valid-v3.json"
Then the exit code should be 0
And the output should contain "Schema: v3 (manual)"
Scenario: Validate v2 file against v3 schema (should fail)
Given a valid Morphir IR v2 JSON file "valid-v2.json"
When I run "morphir ir verify --schema-version 3 valid-v2.json"
Then the exit code should be 1
And the output should contain "✗ Validation failed against schema v3"
Scenario Outline: Validate with explicit version specification
Given a valid Morphir IR <actual-version> JSON file "<filename>"
When I run "morphir ir verify --schema-version <specified-version> <filename>"
Then the exit code should be <exit-code>
And the output should contain "Schema: <specified-version> (manual)"
Examples:
| filename | actual-version | specified-version | exit-code |
| valid-v1.json | v1 | 1 | 0 |
| valid-v2.json | v2 | 2 | 0 |
| valid-v3.json | v3 | 3 | 0 |
| valid-v1.json | v1 | 3 | 1 |
| valid-v2.json | v2 | 1 | 1 |
Rule: Multiple output formats are supported
Scenario: Output validation results as JSON
Given an invalid Morphir IR JSON file "errors.json"
When I run "morphir ir verify --json errors.json"
Then the output should be valid JSON
And the JSON should have field "valid" with value false
And the JSON should have field "errors" as an array
And each error should include "path", "message", "expected", and "found"
Scenario: Output successful validation as JSON
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "morphir ir verify --json valid-v3.json"
Then the output should be valid JSON
And the JSON should have field "valid" with value true
And the JSON should have field "schemaVersion" with value "3"
And the JSON should have field "detectionMethod" with value "auto"
And the JSON should have field "errorCount" with value 0
Scenario: Quiet mode suppresses success messages
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "morphir ir verify --quiet valid-v3.json"
Then the exit code should be 0
And the output should be empty
Scenario: Quiet mode shows only errors
Given an invalid Morphir IR v3 JSON file "invalid-tags.json"
When I run "morphir ir verify --quiet invalid-tags.json"
Then the exit code should be 1
And the output should contain error messages
And the output should not contain "✗ Validation failed"
And the output should not contain headers or decorations
Scenario: Verbose mode shows detailed information
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "morphir ir verify --verbose valid-v3.json"
Then the exit code should be 0
And the output should contain "✓ Validation successful"
And the output should contain "Schema: v3 (auto-detected)"
And the output should contain "File: valid-v3.json"
And the output should contain validation timestamp
And the output should contain schema file path
Rule: Error messages are clear and actionable
Scenario: Error message includes JSON path
Given an invalid Morphir IR v3 JSON file "bad-path.json" with error at "$.modules[0].types.MyType"
When I run "morphir ir verify bad-path.json"
Then the exit code should be 1
And the output should contain "Path: $.modules[0].types.MyType"
Scenario: Error message includes line and column numbers
Given an invalid Morphir IR v3 JSON file "line-col-error.json" with error at line 42, column 12
When I run "morphir ir verify line-col-error.json"
Then the exit code should be 1
And the output should contain "Line: 42, Column: 12"
Scenario: Error message suggests fixes
Given an invalid Morphir IR v3 JSON file "lowercase-tag.json" with lowercase "public" tag
When I run "morphir ir verify lowercase-tag.json"
Then the exit code should be 1
And the output should contain 'Suggestion: Change "public" to "Public"'
Rule: Edge cases and error handling
Scenario: File not found
When I run "morphir ir verify non-existent-file.json"
Then the exit code should be 2
And the output should contain "File not found: non-existent-file.json"
Scenario: Malformed JSON
Given a file "malformed.json" with invalid JSON syntax
When I run "morphir ir verify malformed.json"
Then the exit code should be 2
And the output should contain "Invalid JSON"
And the output should contain the JSON parsing error location
Scenario: Empty file
Given an empty file "empty.json"
When I run "morphir ir verify empty.json"
Then the exit code should be 2
And the output should contain "File is empty"
Scenario: Very large file
Given a valid Morphir IR v3 JSON file "large-10mb.json" of size 10MB
When I run "morphir ir verify large-10mb.json"
Then the validation should complete within 2 seconds
And the exit code should be 0
Scenario: Invalid schema version specified
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "morphir ir verify --schema-version 5 valid-v3.json"
Then the exit code should be 2
And the output should contain "Schema version must be 1, 2, or 3"
Scenario: File with invalid UTF-8 encoding
Given a file "invalid-utf8.json" with invalid UTF-8 bytes
When I run "morphir ir verify invalid-utf8.json"
Then the exit code should be 2
And the output should contain "Invalid file encoding"
Feature 2: Version Detection
Feature File: IrVersionDetection.feature
Feature: IR Version Detection
As a Morphir developer
I want to automatically detect which schema version my IR uses
So that I can validate against the correct schema
Background:
Given the Morphir CLI is installed
And the schema files v1, v2, and v3 are available
Rule: Auto-detection works for files with formatVersion field
Scenario: Detect version from formatVersion field (v3)
Given a Morphir IR JSON file "with-format-v3.json" containing "formatVersion": 3
When I run "morphir ir verify with-format-v3.json"
Then the validation should use schema v3
And the output should contain "Schema: v3 (auto-detected)"
Scenario Outline: Detect version from formatVersion field
Given a Morphir IR JSON file "<filename>" containing "formatVersion": <version>
When I run "morphir ir verify <filename>"
Then the validation should use schema v<version>
And the output should contain "Schema: v<version> (auto-detected)"
Examples:
| filename | version |
| format-v1.json | 1 |
| format-v2.json | 2 |
| format-v3.json | 3 |
Rule: Auto-detection uses tag capitalization when formatVersion is absent
Scenario: Detect v1 from lowercase tags
Given a Morphir IR JSON file "no-format-v1.json" without formatVersion
And the file uses all lowercase tags like "library", "public", "apply"
When I run "morphir ir verify no-format-v1.json"
Then the validation should use schema v1
And the output should contain "Schema: v1 (auto-detected)"
Scenario: Detect v3 from capitalized tags
Given a Morphir IR JSON file "no-format-v3.json" without formatVersion
And the file uses all capitalized tags like "Library", "Public", "Apply"
When I run "morphir ir verify no-format-v3.json"
Then the validation should use schema v3
And the output should contain "Schema: v3 (auto-detected)"
Scenario: Detect v2 from mixed capitalization
Given a Morphir IR JSON file "no-format-v2.json" without formatVersion
And the file uses mixed case tags
When I run "morphir ir verify no-format-v2.json"
Then the validation should use schema v2
And the output should contain "Schema: v2 (auto-detected)"
Rule: Standalone version detection command
Scenario: Detect version with dedicated command
Given a Morphir IR JSON file "detect-me.json" with v3 structure
When I run "morphir ir detect-version detect-me.json"
Then the exit code should be 0
And the output should contain "Detected schema version: v3"
And the output should contain "Confidence: High"
And the output should contain "Rationale:"
Scenario: Version detection shows rationale
Given a Morphir IR JSON file "v3-with-format.json" containing "formatVersion": 3
When I run "morphir ir detect-version v3-with-format.json"
Then the output should contain "Contains formatVersion: 3"
Scenario: Version detection analyzes tag patterns
Given a Morphir IR JSON file "v3-no-format.json" without formatVersion but with capitalized tags
When I run "morphir ir detect-version v3-no-format.json"
Then the output should contain 'All tags are capitalized ("Library", "Public", "Apply")'
Scenario Outline: Detect version with varying confidence levels
Given a Morphir IR JSON file "<filename>" with <indicators>
When I run "morphir ir detect-version <filename>"
Then the output should contain "Confidence: <confidence>"
Examples:
| filename | indicators | confidence |
| clear-v3.json | formatVersion and cap tags | High |
| likely-v1.json | lowercase tags only | Medium |
| ambiguous.json | minimal structure | Low |
Feature 3: Multiple File Support (Phase 2)
Feature File: IrMultiFileVerification.feature
Feature: Multiple File Verification
As a Morphir developer working with multiple IR files
I want to validate several files at once
So that I can efficiently verify my entire project
Background:
Given the Morphir CLI is installed with Phase 2 features
Rule: Multiple files can be validated in one command
Scenario: Validate two valid files
Given valid IR files "file1.json" and "file2.json"
When I run "morphir ir verify file1.json file2.json"
Then the exit code should be 0
And the output should show results for "file1.json"
And the output should show results for "file2.json"
And both files should pass validation
Scenario: Validate mix of valid and invalid files
Given a valid IR file "valid.json"
And an invalid IR file "invalid.json"
When I run "morphir ir verify valid.json invalid.json"
Then the exit code should be 1
And the output should show "valid.json" passed
And the output should show "invalid.json" failed with errors
Scenario: Validate multiple files with summary
Given 10 valid IR files
And 3 invalid IR files
When I run "morphir ir verify *.json"
Then the exit code should be 1
And the output should contain "Summary: 10 passed, 3 failed"
Rule: Stdin support for piped input
Scenario: Validate IR from stdin
Given a valid Morphir IR v3 JSON file "valid-v3.json"
When I run "cat valid-v3.json | morphir ir verify -"
Then the exit code should be 0
And the output should contain "✓ Validation successful"
And the output should contain "Source: stdin"
Scenario: Validate invalid IR from stdin
Given an invalid Morphir IR JSON file "invalid.json"
When I run "cat invalid.json | morphir ir verify -"
Then the exit code should be 1
And the output should contain "✗ Validation failed"
Scenario: Combine file and stdin (stdin represented as -)
Given a valid IR file "file.json"
And valid IR JSON content in stdin
When I run "cat stdin.json | morphir ir verify file.json -"
Then the exit code should be 0
And the output should show results for "file.json"
And the output should show results for "stdin"
Rule: Batch processing is efficient
Scenario: Validate 100 files efficiently
Given 100 valid IR files in "batch/" directory
When I run "morphir ir verify batch/*.json"
Then the validation should complete within 10 seconds
And the exit code should be 0
And the output should contain "Summary: 100 passed, 0 failed"
Scenario: Stop on first error (--fail-fast option)
Given 5 valid IR files and 1 invalid IR file
When I run "morphir ir verify --fail-fast *.json"
Then the validation should stop at the first error
And the exit code should be 1
And not all files should be processed
Feature 4: Directory Validation (Phase 3)
Feature File: IrDirectoryVerification.feature
Feature: Directory Verification
As a Morphir developer with many IR files
I want to validate entire directories
So that I can ensure all my IR files are correct
Background:
Given the Morphir CLI is installed with Phase 3 features
Rule: Directories can be validated recursively
Scenario: Validate all JSON files in directory
Given a directory "ir-files/" with 5 valid IR JSON files
When I run "morphir ir verify --recursive ir-files/"
Then the exit code should be 0
And all 5 files should be validated
And the output should contain "5 files validated, 5 passed"
Scenario: Validate directory with mixed results
Given a directory "mixed/" with 3 valid and 2 invalid IR files
When I run "morphir ir verify --recursive mixed/"
Then the exit code should be 1
And the output should contain "5 files validated, 3 passed, 2 failed"
Scenario: Skip non-JSON files in directory
Given a directory "mixed-types/" with JSON and non-JSON files
When I run "morphir ir verify --recursive mixed-types/"
Then only JSON files should be validated
And the output should list which files were skipped
Scenario: Validate nested directory structure
Given a nested directory structure:
"""
project/
├── src/
│ ├── module1/
│ │ └── ir.json
│ └── module2/
│ └── ir.json
└── tests/
└── fixtures/
└── ir.json
"""
When I run "morphir ir verify --recursive project/"
Then all 3 IR files should be validated
And the output should show the relative paths of all files
Rule: Directory validation supports filtering
Scenario: Validate only specific file patterns
Given a directory with various JSON files
When I run "morphir ir verify --recursive --pattern 'morphir-*.json' dir/"
Then only files matching "morphir-*.json" should be validated
Scenario: Exclude specific directories
Given a directory structure with "node_modules/" and "src/"
When I run "morphir ir verify --recursive --exclude 'node_modules' ."
Then files in "node_modules/" should be skipped
And files in "src/" should be validated
Feature 5: Error Reporting Quality
Feature File: IrValidationErrorReporting.feature
Feature: Validation Error Reporting
As a Morphir developer fixing validation errors
I want detailed, actionable error messages
So that I can quickly identify and fix issues
Background:
Given the Morphir CLI is installed
Rule: Errors include precise location information
Scenario: Error with JSON path
Given an IR file "error.json" with invalid value at "$.modules[0].types.MyType.accessControlled[0]"
When I run "morphir ir verify error.json"
Then the output should contain the exact JSON path
And the path should be formatted as "$.modules[0].types.MyType.accessControlled[0]"
Scenario: Error with line and column numbers
Given an IR file "error.json" with syntax error at line 42, column 12
When I run "morphir ir verify error.json"
Then the output should contain "Line: 42, Column: 12"
Scenario: Error shows context snippet
Given an IR file with error at line 42
When I run "morphir ir verify --verbose error.json"
Then the output should include a code snippet around line 42
And the error line should be highlighted
Rule: Errors explain what was expected vs found
Scenario: Type mismatch error
Given an IR file with string where number is expected
When I run "morphir ir verify error.json"
Then the output should contain "Expected: number"
And the output should contain 'Found: "some string"'
Scenario: Enum value error
Given an IR file with invalid access control tag
When I run "morphir ir verify error.json"
Then the output should contain 'Expected: One of ["Public", "Private"]'
And the output should contain 'Found: "public"'
Scenario: Array length constraint error
Given an IR file with array that violates length constraints
When I run "morphir ir verify error.json"
Then the output should contain "Expected: Array with 2 elements"
And the output should contain "Found: Array with 3 elements"
Rule: Errors provide helpful suggestions
Scenario: Suggest capitalization fix
Given an IR file with lowercase tag in v3 IR
When I run "morphir ir verify error.json"
Then the output should contain 'Suggestion: Change "public" to "Public"'
Scenario: Suggest adding missing field
Given an IR file missing required "name" field
When I run "morphir ir verify error.json"
Then the output should contain 'Suggestion: Add required field "name"'
Scenario: Suggest similar field names for typos
Given an IR file with "nmae" instead of "name"
When I run "morphir ir verify error.json"
Then the output should contain 'Did you mean "name"?'
Rule: Multiple errors are clearly enumerated
Scenario: List multiple errors with numbering
Given an IR file with 3 validation errors
When I run "morphir ir verify error.json"
Then the output should contain "Error 1:"
And the output should contain "Error 2:"
And the output should contain "Error 3:"
And the output should contain "3 errors found"
Scenario: Group errors by category
Given an IR file with type errors and missing field errors
When I run "morphir ir verify error.json"
Then errors should be grouped by type
And the output should show "Type Errors (2)" and "Missing Fields (3)"
Scenario: Limit error display with --max-errors option
Given an IR file with 50 validation errors
When I run "morphir ir verify --max-errors 10 error.json"
Then only the first 10 errors should be displayed
And the output should contain "... and 40 more errors"
Rule: Error output is machine-readable in JSON mode
Scenario: JSON error format includes all details
Given an IR file with validation errors
When I run "morphir ir verify --json error.json"
Then the JSON output should include:
| field | description |
| valid | false |
| errors | Array of error objects |
| errors[].path | JSON path to error |
| errors[].line | Line number |
| errors[].column | Column number |
| errors[].message| Human-readable error message |
| errors[].code | Machine-readable error code |
Scenario: Error codes are consistent and documented
Given an IR file with a missing required field
When I run "morphir ir verify --json error.json"
Then the error should have code "MISSING_REQUIRED_FIELD"
And the error code should be documented
Feature 6: Performance and Scalability
Feature File: IrValidationPerformance.feature
Feature: Validation Performance
As a developer integrating validation in CI/CD
I want fast validation even for large files
So that builds remain efficient
Background:
Given the Morphir CLI is installed
Rule: Validation meets performance targets
Scenario Outline: Validate files of varying sizes
Given a valid Morphir IR v3 JSON file of size <size>
When I run "morphir ir verify <filename>"
Then the validation should complete within <max-time>
And the exit code should be 0
Examples:
| size | filename | max-time |
| 10KB | small.json | 100ms |
| 100KB | medium.json | 100ms |
| 1MB | large.json | 500ms |
| 10MB | very-large.json | 2000ms |
Scenario: Schema caching improves performance
Given 10 valid IR files
When I run "morphir ir verify file1.json ... file10.json"
Then schemas should only be loaded once
And subsequent validations should be faster
Scenario: Memory usage remains bounded
Given a 50MB IR file
When I run "morphir ir verify huge.json"
Then memory usage should not exceed 500MB
And validation should complete successfully
Rule: Validation supports progress reporting
Scenario: Show progress for multiple files
Given 100 IR files to validate
When I run "morphir ir verify --progress *.json"
Then the output should show a progress indicator
And the progress should update as files are validated
Scenario: Show progress for large single file
Given a 10MB IR file
When I run "morphir ir verify --progress large.json"
Then the output should show validation progress
Implementation Notes
Step Definition Organization
Step definitions should be organized in the following files within tests/Morphir.Core.Tests/StepDefinitions/:
IrVerificationSteps.cs: Common steps for file setup, CLI execution, output assertionsIrSchemaSteps.cs: Steps specific to schema validationIrVersionDetectionSteps.cs: Steps for version detection scenariosIrFileManagementSteps.cs: Steps for file and directory operations
Test Data Strategy
Test IR JSON files should be stored in tests/Morphir.Core.Tests/TestData/IrFiles/:
TestData/
└── IrFiles/
├── v1/
│ ├── valid/
│ │ ├── library-v1.json
│ │ └── complex-types-v1.json
│ └── invalid/
│ ├── invalid-tags-v1.json
│ └── missing-fields-v1.json
├── v2/
│ ├── valid/
│ └── invalid/
└── v3/
├── valid/
└── invalid/
Continuous Integration
These BDD scenarios should run:
- On every pull request
- On merge to main branch
- As part of the release validation process
Target: All scenarios pass with >95% reliability.
Last Updated: 2025-12-13
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.