A stable fleet can become a reconnect storm in seconds when a carrier recovers, a region fails over, or a certificate rollout restarts devices together. The resulting load is not normal publish traffic: every client may repeat TLS negotiation, authentication, session restoration, subscriptions, queued delivery, and application catch-up.
Capacity planning therefore needs a recovery target. If 60,000 devices must return within 10 minutes, the system needs to sustain at least 100 successful recoveries per second before accounting for retries, failed credentials, and uneven network timing.
Model recovery work, not idle connections
Capture at least these inputs for each device class:
- concurrent and peak connected clients;
- connects and disconnects per second;
- messages per second in each direction;
- average and 95th-percentile payload size;
- QoS distribution and retained-message count;
- subscriptions per client and wildcard breadth;
- expected offline duration and queued-message policy;
- daily and event-driven burst patterns.
Start with a simple floor:
required successful recoveries per second = disconnected clients / recovery window in seconds
Then add the work behind each recovery: TLS handshakes, identity lookup, restored subscriptions, queued QoS messages, retained delivery, and the device's first telemetry burst. A reconnect after a regional network outage often creates more authentication and subscription work than normal publishing.
Replay real client behavior in staging. A test that holds idle sockets open does not represent certificate handshakes, subscription restoration, queued delivery, or downstream processing.
Choose a recoverable cohort boundary
Separate broker runtimes when fleets have different:
- regulatory or data-residency requirements;
- availability targets;
- deployment schedules;
- traffic profiles;
- operational owners;
- customer isolation commitments.
An environment, region, major tenant group, or product line can be a useful boundary. Avoid creating a broker per tiny customer without an operational reason; excessive fragmentation increases policy, monitoring, and upgrade work.
RunMQTT uses isolated broker instances with scoped templates and device identities, so a policy or capacity decision can remain attached to the workload it governs.
Keep subscriptions and topics partitionable
A scalable topic tree exposes the dimensions used for ownership and routing:
prod/{region}/{tenant}/{site}/{deviceId}/{channel}
Do not put fast-changing labels into the hierarchy. Use stable IDs, and keep high-cardinality payload fields out of topics. Broad subscriptions such as prod/# become expensive operational and security dependencies; route analytics through intentional consumers and bounded prefixes.
The MQTT topic design guide explains wildcard and permission tradeoffs.
Keep authentication off the critical bottleneck
Manual provisioning does not survive enterprise scale. Integrate device manufacturing or enrollment with a workflow that:
- creates the device record;
- assigns a reviewed policy template;
- delivers credentials through a protected channel;
- records activation;
- rotates or revokes credentials;
- decommissions the identity with the hardware.
Use one identity per device whenever possible. If gateways represent downstream equipment, document that trust boundary and limit the gateway to its assigned site or child-device prefix.
Make client backoff part of capacity
Every client needs exponential backoff with jitter. Roll credentials and firmware in cohorts so an error does not force the whole fleet to reconnect at once.
Capacity tests should include:
- a cold start of the expected cohort;
- a regional outage followed by network recovery;
- expired or rejected credentials;
- restoration of persistent sessions and subscriptions;
- delivery of bounded offline queues;
- slow or unavailable downstream consumers.
Set message expiry and queue limits so recovery work remains useful. Dropping expired telemetry can be safer than allowing a backlog to delay current state.
Treat regional failover as a reconnect event
Multi-region design needs a documented business objective. Decide whether the system requires disaster recovery, active-active client distribution, data replication, or all three.
Devices need a deterministic way to discover or select a healthy endpoint. Replicate only the state needed after failover, and define how retained messages, commands, and duplicate events behave across regions. Test recovery with real DNS, certificates, client backoff, and downstream dependencies.
Measure the recovery budget end to end
Monitor broker health alongside client and consumer behavior:
- successful connection rate and authentication latency;
- connection churn and session restoration;
- publish-to-consumer latency;
- authorization denials;
- queue depth, expiry, and dropped work;
- throughput and payload-size distribution;
- device heartbeat freshness;
- downstream lag and error rate.
Set a recovery objective around the product outcome, such as “95% of affected devices resume current telemetry within 10 minutes, with no command older than 60 seconds executed.” Broker availability alone can stay green while authentication is saturated or an analytics consumer is hours behind.
Test a timeline, not a peak number
Write the load test as a timeline: stable traffic, abrupt disconnect, outage interval, network restoration, session recovery, and return to the steady-state latency budget. Record the maximum successful connection rate, failed attempts, P95 authentication latency, queue age, downstream lag, and total recovery time.
Treat firmware, certificate, policy, and endpoint changes as releases because each can create the same reconnect pattern. Use canary cohorts, explicit rollback criteria, and an owner watching the recovery telemetry.
Start with the local MQTT fleet capacity and cost calculator to make connection, message-unit, fan-out, and reconnect assumptions explicit. Before choosing the operating model, compare managed and self-hosted MQTT brokers. When isolation, device identity, and predictable fleet operations are the priority, review RunMQTT pricing.