Field Mapping Architecture¶
Overview¶
This document describes the scheduler-aware field mapping architecture implemented to handle the mapping between HostFactory template fields and internal domain model fields, with provider-specific extensions.
Problem Statement¶
The original implementation had several issues:
1. Field Inheritance Issues: AWSTemplate was shadowing inherited fields from CoreTemplate
2. Missing Field Mappings: The scheduler wasn't mapping vmTypes → instance_types
3. Handler Field Access Errors: Handlers were trying to access fields that didn't exist
4. Inconsistent Field Names: Different layers used different field naming conventions
Solution Architecture¶
1. Scheduler-Aware Field Mapping Registry¶
Location: src/infrastructure/scheduler/field_mappings.py
The SchedulerProviderFieldMappings class provides a centralized registry for mapping scheduler-specific field names to internal domain model fields, with provider-aware conditional mapping.
class SchedulerProviderFieldMappings:
MAPPINGS = {
"hostfactory": {
"generic": {
"templateId": "template_id",
"vmType": "instance_type",
"vmTypes": "instance_types",
# ... other generic fields
},
"aws": {
"vmTypesOnDemand": "instance_types_ondemand",
"percentOnDemand": "percent_on_demand",
"fleetRole": "fleet_role",
# ... other AWS-specific fields
},
}
}
Key Features:
- Generic Fields: Work across all providers (e.g., vmType, imageId)
- Provider-Specific Fields: Only mapped when the provider is active (e.g., vmTypesOnDemand for AWS)
- Extensible: Easy to add new schedulers and providers
2. Improved Scheduler Strategy¶
Location: src/infrastructure/scheduler/strategies/symphony_hostfactory.py
The scheduler strategy now uses the field mapping registry to perform provider-aware field mapping:
def _map_template_fields(self, template: Dict[str, Any]) -> Dict[str, Any]:
# Get active provider type
provider_type = self._get_active_provider_type()
# Get field mappings for HostFactory + active provider
field_mappings = SchedulerProviderFieldMappings.get_mappings(
scheduler_type="hostfactory", provider_type=provider_type
)
# Apply registry-based field mappings
mapped = {}
for hf_field, internal_field in field_mappings.items():
if hf_field in template:
mapped[internal_field] = template[hf_field]
# Apply field transformations
mapped = FieldTransformationUtils.apply_field_transformations(mapped)
return mapped
3. Field Transformation Utilities¶
Location: src/infrastructure/scheduler/field_mappings.py
The FieldTransformationUtils class handles complex field transformations:
- Subnet ID Transformation: Converts single subnet to list format
- Instance Tags Transformation: Parses HostFactory tag string format (
"key1=value1;key2=value2") - Instance Type Consistency: Ensures
instance_typeis set wheninstance_typesis provided
4. Fixed AWSTemplate Field Inheritance¶
Location: src/providers/aws/domain/template/aggregate.py
The AWSTemplate now properly inherits fields from CoreTemplate:
class AWSTemplate(CoreTemplate):
# AWS-specific extensions only
instance_types_ondemand: Optional[Dict[str, int]] = None
instance_types_priority: Optional[Dict[str, int]] = None
percent_on_demand: Optional[int] = None
# ... other AWS-specific fields
# Note: instance_type and instance_types are inherited from CoreTemplate
# No need to redefine them here - this was causing the field access issues
5. Updated AWS Handlers¶
Locations:
- src/providers/aws/infrastructure/handlers/run_instances_handler.py
- src/providers/aws/infrastructure/handlers/spot_fleet_handler.py
- src/providers/aws/infrastructure/handlers/ec2_fleet_handler.py
All handlers now use the correct inherited field names:
# Before (causing errors)
if aws_template.vm_type: # Field doesn't exist!
params["InstanceType"] = aws_template.vm_type
# After (working correctly)
if aws_template.instance_type: # Inherited from CoreTemplate
params["InstanceType"] = aws_template.instance_type
Field Mapping Examples¶
OnDemand Template¶
HostFactory JSON:
{
"templateId": "OnDemand-Template",
"vmType": "t2.micro",
"imageId": "ami-12345678",
"subnetId": "subnet-abcd1234",
"priceType": "ondemand"
}
Mapped Internal Fields:
{
"template_id": "OnDemand-Template",
"instance_type": "t2.micro",
"image_id": "ami-12345678",
"subnet_ids": ["subnet-abcd1234"], # Transformed to list
"price_type": "ondemand",
}
Spot Template¶
HostFactory JSON:
{
"templateId": "Spot-Template",
"vmTypes": {"t2.medium": 1, "t3.medium": 2},
"priceType": "spot",
"fleetRole": "arn:aws:iam::123456789012:role/spot-fleet-role"
}
Mapped Internal Fields:
{
"template_id": "Spot-Template",
"instance_types": {"t2.medium": 1, "t3.medium": 2},
"instance_type": "t2.medium", # Auto-set from first instance_types entry
"price_type": "spot",
"fleet_role": "arn:aws:iam::123456789012:role/spot-fleet-role",
}
Heterogeneous Template¶
HostFactory JSON:
{
"templateId": "Hetero-Template",
"vmTypes": {"t2.medium": 1, "t3.large": 2},
"vmTypesOnDemand": {"t2.medium": 1},
"priceType": "heterogeneous",
"percentOnDemand": 30,
"allocationStrategyOnDemand": "prioritized"
}
Mapped Internal Fields:
{
"template_id": "Hetero-Template",
"instance_types": {"t2.medium": 1, "t3.large": 2},
"instance_types_ondemand": {"t2.medium": 1},
"instance_type": "t2.medium",
"price_type": "heterogeneous",
"percent_on_demand": 30,
"allocation_strategy_ondemand": "prioritized",
}
Supported HostFactory Fields¶
Generic Fields (All Providers)¶
templateId→template_idmaxNumber→max_instancesimageId→image_idvmType→instance_typevmTypes→instance_typessubnetId→subnet_idssecurityGroupIds→security_group_idspriceType→price_typemaxSpotPrice→max_priceallocationStrategy→allocation_strategykeyName→key_nameinstanceTags→tagsrootDeviceVolumeSize→root_volume_sizevolumeType→root_volume_typeiops→root_volume_iops
AWS-Specific Fields¶
vmTypesOnDemand→instance_types_ondemandvmTypesPriority→instance_types_prioritypercentOnDemand→percent_on_demandallocationStrategyOnDemand→allocation_strategy_ondemandfleetRole→fleet_rolespotFleetRequestExpiry→spot_fleet_request_expirypoolsCount→pools_countlaunchTemplateId→launch_template_idinstanceProfile→instance_profileuserDataScript→user_data
Neutral Template Fields¶
A few template fields look provider-specific but have a counterpart on every supported provider, so they stay on the shared template aggregate and each provider maps them to its own concept.
| Field | AWS | Azure | GCP | OCI | Kubernetes |
|---|---|---|---|---|---|
subnet_ids |
VPC subnet | VNet subnet | Subnetwork | Subnet | Not used (empty) |
security_group_ids |
Security group | Network security group | Network tags / firewall | Network security group | Not used (empty) |
vpc_id (machine) |
VPC | VNet id | Network | VCN | Not used (None) |
The subnetIds and securityGroupIds HostFactory keys map to subnet_ids
and security_group_ids for every provider, so external HostFactory consumers
see the same spelling whichever provider serves the template.
Fields that stay on the shared template, and why¶
machine_types_ondemand,machine_types_priority,iops,throughputoriginate on AWS (EC2 Fleet pool split, EBS volume tuning). They remain optional on the shared aggregate. Moving them out needs a mechanism that round-trips provider-specific keys through the template repository, which serialises through an explicit key list, the template DTO, the SQL columns, the REST models and the UI. A move without that mechanism silently drops stored values, and Azure and GCP aggregates reject unknown keys, so the consumers below would all change together: the applicationTemplateDTO, the DTO factories, the template repository, the SQL template model, the REST template models, the UI template pages and the template defaults service. Revisit only if a second provider needs a conflicting meaning for the same key.allocation_strategy. The field stays optional on the shared aggregate, which applies no default. The default lives only inAWSTemplate: when no strategy is set, spot and heterogeneous templates getpriceCapacityOptimizedand on-demand templates getlowestPrice. Every load path reaches the AWS request builders as anAWSTemplate, so the builders always see a strategy.AWSTemplatenormalises every known spelling (lowestPrice,lowest_price,lowest-price) to the canonical camelCase form. Azure and GCP leave the field unset.
The behaviour above is pinned by the allocation-strategy golden tests, the template DTO field snapshot, the storage round-trip tests and the HostFactory network goldens, which are the acceptance gate for any future field move.
Adding a provider key¶
A provider that needs template keys of its own defines its own field mapping
(FieldMappingPort) and its own template aggregate; it should not add
provider-named keys to the shared aggregate.
Benefits¶
- Single Source of Truth: All field mappings are centralized in one registry
- Provider Awareness: Only maps provider-specific fields when the provider is active
- Extensible: Easy to add new schedulers (LSF, SLURM) and providers (Provider1, Provider2)
- Type Safety: Appropriate field inheritance ensures handlers can access required fields
- Maintainable: Clear separation between scheduler-specific and provider-specific concerns
Future Extensions¶
Adding New Schedulers¶
To add support for LSF scheduler:
MAPPINGS = {
"hostfactory": {...},
"lsf": {
"generic": {
"TEMPLATE": "template_id",
"INSTANCE_TYPE": "instance_type",
"IMAGE_ID": "image_id",
},
"aws": {"SPOT_FLEET_ROLE": "fleet_role", "ON_DEMAND_PERCENT": "percent_on_demand"},
},
}
Adding New Providers¶
To add support for Provider1 provider:
MAPPINGS = {
"hostfactory": {
"generic": {...},
"aws": {...},
"provider1": {
"vmPriority": "vm_priority",
"spotEvictionPolicy": "spot_eviction_policy",
"proximityPlacementGroup": "proximity_placement_group",
},
}
}
Testing¶
The implementation includes comprehensive tests:
- Unit Tests:
tests/test_field_mapping_integration.py - Standalone Tests:
test_field_mapping_standalone.py
Run tests with:
Migration Guide¶
For Existing Templates¶
No changes required - the new field mapping is backward compatible.
For New Development¶
- Use the field mapping registry for any new scheduler integrations
- Add provider-specific fields to the appropriate provider section
- Ensure appropriate field inheritance in provider-specific template classes
- Update handlers to use inherited field names
Troubleshooting¶
Common Issues¶
- Field Not Found Error: Check if the field is properly mapped in the registry
- Provider-Specific Field Missing: Ensure the provider type is correctly detected
- Field Transformation Issues: Verify the transformation utilities are applied
Debug Tips¶
- Enable debug logging to see field mapping details
- Check the active provider type detection
- Verify field transformations are applied correctly