Reliability claims are easy to make while the network is healthy. A production MQTT system proves them when a packet is duplicated, a certificate expires, a consumer slows down, or thousands of devices reconnect together.
The drills below convert common MQTT failure modes into repeatable tests. For each drill, record the expected behavior, injected fault, evidence, recovery time, lost or duplicated work, and the person who would own the same failure in production.
Drill 1: duplicate a QoS 1 message
Use QoS 0 for frequent telemetry where the next reading replaces the previous one and occasional loss is acceptable. Use QoS 1 when every event should arrive and consumers can deduplicate. Reserve QoS 2 for the small set of workflows whose delivery semantics justify its additional state and network exchange.
QoS describes delivery between one client and the broker. It does not make a database write, external API call, or device action exactly once. Publish the same business event twice with a stable eventId and verify that the consumer produces one business result. See the MQTT QoS lab for packet flows and a reproducible experiment.
Drill 2: connect two clients with the same ID
A client ID identifies one active MQTT session. Reusing it across devices causes connections to evict each other. Generate a stable, unique ID per device or service instance and never treat a display name as the protocol identity.
In a staging environment, deliberately connect a second client with the same ID. Confirm that the disconnection is visible, the affected device can be identified, and the client does not enter an unbounded reconnect loop.
Drill 3: publish and subscribe outside the policy
Use stable identifiers and separate telemetry, state, events, commands, and command results. A practical hierarchy might be:
prod/{tenant}/{site}/{deviceId}/{channel}
Attempt to publish as another device, subscribe to a sibling tenant, and widen an allowed filter with + or #. Every attempt should fail and create evidence that identifies the principal and requested topic without leaking credentials. The topic design checklist covers the permission model behind these negative tests.
Drill 4: expire or revoke a credential
Production MQTT connections should use TLS with certificate validation. Plain TCP on port 1883 is appropriate only inside a deliberately isolated environment.
Give each identity only the publish and subscribe filters it needs. A sensor usually publishes its telemetry and subscribes to its own command topic; it should not subscribe to an entire tenant or publish as another device. Rotate secrets, rate-limit failed authentication, and remove credentials when hardware is retired.
Revoke one pilot device and verify that only that identity loses access. Then present an expired or untrusted server certificate to the client and confirm that it fails closed instead of silently disabling verification. Use the MQTT security guide to review TLS, authentication, authorization, and credential storage together.
Drill 5: fill a queue with work that will become stale
Set a maximum payload size and validate schema at the ingestion boundary. Include a schema version, observed timestamp, stable device identifier, and unit information where relevant.
Disconnect a persistent client, enqueue time-sensitive commands, and reconnect it after their business deadline. Expired work should not execute. The test should also prove what happens at the queue limit and how an operator distinguishes discarded stale work from unexpected message loss.
Drill 6: reconnect a cohort at once
Use exponential backoff with jitter instead of reconnecting every client on a fixed interval. A coordinated reconnect storm can become the outage.
Choose session expiry deliberately. Persistent sessions help intermittently connected clients receive queued QoS messages, but they also consume broker resources and can deliver obsolete work. Disconnect a representative cohort, restore the network at once, and measure connection success, authentication latency, subscription restoration, queue age, and downstream lag until recovery completes.
Drill 7: replay stale retained state
Retained messages are useful for the latest configuration or state snapshot. They are not an event history. Subscribe a new consumer after intentionally leaving an obsolete retained value and verify that version or freshness checks prevent it from becoming a command or current fact. Publish an empty retained payload when state should be cleared, and restrict who can overwrite retained state.
Drill 8: slow or stop the downstream consumer
Broker uptime alone cannot explain whether the product works. Track:
- successful and failed connections;
- reconnect rate and connection duration;
- publish and delivery rate by bounded topic group;
- authorization denials;
- payload size and processing latency;
- queue depth, expiry, and consumer lag;
- device heartbeat freshness.
Use low-cardinality dimensions such as environment, tenant tier, region, and device class. Raw client IDs or unbounded topic strings can make metrics expensive and hard to use. During the drill, verify that queue age and end-to-end latency alert before memory or storage is exhausted.
Measure recovery, not only survival
Load tests should include slow subscribers, duplicate delivery, packet loss, expired certificates, invalid credentials, broker restarts, downstream outages, and a large cohort reconnecting at once.
Define the expected behavior before running each test. “The system recovered” is not enough: measure recovery time, lost or duplicated work, maximum queue age, and whether operators received a useful alert before customers noticed.
Keep a short production contract
For each topic family, record its owner, allowed publishers and subscribers, QoS, retained behavior, payload schema, maximum size, expiry, and consumer idempotency strategy. Keep the contract close to firmware and backend code and review it when either side changes.
Failure-drill exit checklist
Before connecting real devices or customer data, confirm that the team has evidence for each item:
- every client has a unique identity and revocable credential;
- TLS certificate validation is enabled outside isolated development;
- publish and subscribe permissions have negative authorization tests;
- topic, payload, QoS, retain, and expiry contracts are documented;
- consumers tolerate duplicate delivery and reject stale commands;
- reconnect backoff, queue bounds, and broker capacity have been load-tested;
- dashboards and alerts cover clients, the broker, and downstream consumers;
- backup, recovery, upgrade, and incident ownership are assigned.
If the last two items would require building a new platform function, compare the ongoing responsibility in the managed versus self-hosted MQTT guide. Teams planning large fleets should also use the enterprise MQTT scaling guide to test connection, subscription, and reconnect limits before launch.
You can validate basic client behavior with synthetic data against the testing-only free public MQTT broker, follow the full MQTT tutorial, or compare managed MQTT plans when the workload needs isolation and production controls.