Moto structural limits (what moto cannot test)¶
The AWS provider is tested at two levels:
- moto (mocked) —
tests/providers/aws/mocked/andtests/providers/aws/unit/. Fast, deterministic, run in CI on every change. - live (onaws) —
tests/providers/aws/live/. Slow (~45 min/run), run against a real AWS account. See System Tests (onaws).
Moto is a simulator, not AWS. Several behaviours that ORB depends on in production either behave differently under moto or are absent entirely. Any scenario that turns on one of the limits below is live-only: a moto test for it would either be impossible to write or would pass against behaviour that AWS does not actually exhibit, giving false confidence.
This page enumerates the five structural limits found while pairing every live test with its moto counterpart. For each: what real AWS does, what moto does, and where the behaviour is (or must be) covered.
L1 — SpotFleet never fulfils¶
AWS: After request_spot_fleet, the fleet transitions to active and
asynchronously launches instances until FulfilledCapacity reaches
TargetCapacity. describe_spot_fleet_instances then returns the active
instances.
Moto: The request is recorded and reports active, but moto does not launch
backing instances to fulfil the target. FulfilledCapacity does not converge and
describe_spot_fleet_instances stays empty.
Consequence: The "wait for SpotFleet to fulfil, then read active instances" polling loop and any weighted-capacity fulfilment arithmetic cannot be exercised end-to-end under moto.
Coverage: Live — tests/providers/aws/live/test_onaws.py (SpotFleet
scenarios). The weighted-capacity cancellation arithmetic that this fulfilment
gap otherwise hides is regression-covered at unit level in
tests/providers/aws/unit/infrastructure/handlers/test_fleet_release_weighted_capacity.py.
L2 — No cancelled_running state¶
AWS: Cancelling a SpotFleet without terminating instances moves the request
to cancelled_running — cancelled, but instances still alive — before it later
settles to cancelled_terminated. ORB distinguishes these to decide whether a
follow-up terminate is required.
Moto: Moto has no cancelled_running transition. A cancel goes straight to a
terminal cancelled state, so the intermediate window ORB reacts to never
appears.
Consequence: Logic branching on cancelled_running versus
cancelled_terminated cannot be reached under moto.
Coverage: Live — tests/providers/aws/live/test_multi_spot_fleet_termination.py.
L3 — Synchronous describe_instances (no eventual consistency)¶
AWS: State changes are eventually consistent. Immediately after a terminate,
describe_instances can still report running, and a fleet just emptied can
still report stale instances for a short window. ORB polls to tolerate this.
Moto: State changes are applied synchronously and are visible on the very next call. There is no stale-read window.
Consequence: Bugs that only surface under eventual consistency — e.g. ORB
stamping a return request complete while AWS still reports the instance
running, or cancelling a fleet based on a stale empty describe — are invisible
under moto because the reads are always immediately correct.
Coverage: Live — tests/providers/aws/live/test_sdk_onaws.py and
test_cleanup_e2e_onaws.py. At moto level the post-return terminal state is
asserted directly (tests/providers/aws/mocked/test_sdk_onmoto.py,
test_return_flow.py::test_instances_terminated_in_aws_after_return) so at least
the final state — not the consistency window — is guarded.
L4 — No fleet modifying transition¶
AWS: Changing an EC2 Fleet's TotalTargetCapacity (scale-down on partial
return) moves the fleet through a modifying state before returning to active.
ORB tolerates modifying as a non-terminal state while polling.
Moto: Capacity changes are applied instantly; the fleet stays active (or
jumps to a terminal deleted state). The modifying transition never occurs.
Consequence: The modifying branch of fleet-state handling cannot be reached
under moto, and partial-return scale-down cannot be observed mid-flight.
Coverage: Live — tests/providers/aws/live/test_multi_ec2_fleet_termination.py.
The partial-return arithmetic itself (new target capacity computation) is unit
covered in test_fleet_release_weighted_capacity.py and
tests/providers/aws/mocked/test_partial_return.py.
L5 — Instant ASG-instance state transitions¶
AWS: Scaling an Auto Scaling Group down (or deleting it) terminates instances
asynchronously; they pass through shutting-down before terminated, and the ASG
briefly reports instances in Terminating lifecycle state.
Moto: ASG capacity changes and deletions are reflected instantly — instances
disappear or flip to terminated with no intermediate lifecycle state.
Consequence: Any logic that observes the Terminating / shutting-down
window during ASG teardown cannot be exercised under moto.
Coverage: Live — tests/providers/aws/live/test_multi_asg_termination.py.
Live/moto pairing¶
Every live test file under tests/providers/aws/live/ is paired here with the
moto (or unit) tests that cover the same behaviour, or with the reason no moto
counterpart exists. Paths in the moto column are relative to
tests/providers/aws/. When you add a live test file, add a row; the check in
tests/unit/test_moto_live_pairing_matrix.py fails until you do.
| Live test file | Moto counterpart | Not covered under moto |
|---|---|---|
test_onaws.py |
mocked/test_hf_contract.py, mocked/test_partial_return.py, mocked/test_cli_onmoto.py |
SpotFleet fulfilment (L1); partial-return scale-down mid-flight (L4) |
test_rest_api_onaws.py |
mocked/test_rest_api_onmoto.py, mocked/test_rest_real_container_flows.py |
SpotFleet fulfilment (L1); post-return consistency window (L3) |
test_sdk_onaws.py |
mocked/test_sdk_onmoto.py |
Post-return consistency window (L3) |
test_cleanup_e2e_onaws.py |
mocked/test_cleanup_e2e.py |
Stale reads after fleet or ASG deletion (L3) |
test_multi_asg_termination.py |
mocked/test_multi_resource_onmoto.py |
ASG Terminating / shutting-down window (L5) |
test_multi_ec2_fleet_termination.py |
mocked/test_cleanup_e2e.py, mocked/test_partial_return.py |
Fleet modifying transition (L4); moto tests cover one fleet per return, not two |
test_multi_spot_fleet_termination.py |
mocked/test_cleanup_e2e.py, mocked/test_partial_return.py |
Fulfilment (L1) and cancelled_running (L2); moto tests cover one fleet per return, not two |
test_multi_resource_termination.py |
mocked/test_multi_resource_onmoto.py |
SpotFleet and EC2 Fleet members of a mixed batch (L1, L4); moto batches cover RunInstances and ASG |
test_capacity_helpers_regex.py |
none | Not a live test: it unit-tests the live suite's own helpers and runs without AWS |
Summary¶
| Limit | AWS behaviour | Moto behaviour | Live coverage |
|---|---|---|---|
| L1 | SpotFleet fulfils asynchronously | Never fulfils | test_onaws.py (SpotFleet) |
| L2 | cancelled_running before terminal |
No such state | test_multi_spot_fleet_termination.py |
| L3 | Eventually consistent reads | Synchronous reads | test_sdk_onaws.py, test_cleanup_e2e_onaws.py |
| L4 | Fleet modifying on capacity change |
Instant, no modifying |
test_multi_ec2_fleet_termination.py |
| L5 | Async ASG-instance teardown | Instant teardown | test_multi_asg_termination.py |
When adding a moto test, check this list first: if the scenario depends on any of L1–L5, it belongs in the live suite (or a unit test that isolates the pure logic), not in a moto test that would pass against unrealistic simulator behaviour.