← Projects

Physical system · October 25, 2026

Murmur: A Delay-Tolerant Swarm Robotics Testbed

A physical testbed of eight ESP32 ground robots, overhead AprilTag ground truth and firmware-injected delay and loss, designed to test whether simulated coordination-collapse results survive contact with real hardware.

1. Problem statement

Five Arboria papers map where decentralized coordination breaks under communication delay. The main results are:

  • the collapse is primitive-independent;
  • its onset scales as dc α≈2.8d_c\,\alpha \approx 2.8;
  • it is governed by staleness, not throughput;
  • cheap peer-state prediction pushes it outward.

Every paper ends with the same caveat: a simulation-based algorithmic result, not physical-device validation.

Murmur is designed to remove that caveat. It is a small physical swarm in which delay and loss are the controlled variables, ground truth comes from outside the robots, and the coordination code is the same code path the simulator scores.

The question it answers is: does the delay cliff measured in Leviathan appear at the same normalized delay on real robots, with real radios, real actuation lag and real sensing noise?

2. Systems approach

Systems engineering V-modelConcept of operationswho, where, what success meansRequirementsmeasurable, verifiable "shall"sArchitecturesubsystems, interfaces, tradesDetailed designschematics, models, budgetsValidationdoes it answer the question?System verificationevery requirement testedIntegration testinterfaces under loadUnit testparts against specImplementationbuild the thingvalidatesverifiesverifies■ Design studies cover the left leg
The design is developed down the left leg of the V. Each requirement below names the test on the right leg that will verify it.

3. Concept of operations

Murmur arena concept of operations, top-down2.4 m2.4 mvirtual fence (halt) · 0.15 m insetcomm range r (firmware-limited)swarm centroidConcept of operations1. Controller loads manifest: task, primitive,gain α, delay d, loss p, seed.2. Robots placed by a scatter routine;camera verifies start poses.3. Robots coordinate over ESP-NOW only,on injected stale peer state.4. Camera logs ground truth at 30 Hz;robots log every packet's fate.5. Run ends at T or on fence/heartbeattrip; Q computed from ground truth.Green robot: the one whose neighbourhood(dashed circle) is shown.
Top-down concept of operations on a 2.4 m × 2.4 m arena.

A run is fully described by a manifest: task (rendezvous, consensus, formation-hold), primitive (gossip-consensus, flocking, CRDT-intent), gain α\alpha, delay dd, loss probability pp, predictor, and a seed. Maneuver.Map already produces exactly this manifest and seed tree for simulated runs, so a physical run and its simulated twin share one identifier.

4. Requirements

IDRequirementRationaleVerification
SR-1Operate N ≥ 8 robots at once for ≥ 30 min per runSmallest fleet where neighbourhood structure is non-trivialTest
SR-2Ground-truth pose at ≥ 30 Hz, ≤ 5 mm position and ≤ 2° heading errorQ must be scored independently of the robots' own beliefsTest against a surveyed grid
SR-3Injected one-way delay programmable 0–40 control steps (0–4 s), accurate to ±1 stepCovers the full simulated cliff for α from 0.07 to 0.28Test
SR-4Injected loss independent per link, p ∈ [0, 0.99]Reproduces the delivery-gated loss sweepAnalysis + χ² test on logs
SR-5Native link latency (no injection) p99 ≤ 20 msMust stay below one control step, or it contaminates the delay axisTest
SR-6Fleet clock synchronisation ≤ 2 msPacket ages must be attributableTest
SR-7Every run reproducible from its manifest: seed, firmware hash, configSame standard as the simulation papersInspection
SR-8Policy parity: firmware primitive matches Gossamer on recorded inputsOtherwise sim-vs-real compares two algorithmsGolden-vector test in CI
SR-9Robot halts within 200 ms of heartbeat loss or fence breachSafetyTest
SR-10Per-robot bill of materials ≤ $60Fleet must be cheap enough to scale to 16Analysis

5. Architecture

Murmur testbed system architectureARENA INFRASTRUCTUREOverhead cameraOV9281 global shutter · 120 fpsGround-truth serverAprilTag 36h11 · poses @ 30 HzSync + safety beaconESP32 · 1 Hz clock sync · heartbeatRun controllerManeuver.Map · manifest + seed treeFLEET · 8 ROBOTSESP-NOW radio2.4 GHz broadcast · fixed channelDelay / loss injectorper-link Bernoulli gate + FIFOCoordination policyGossamer primitive, ported to CMotion layerunicycle · 200 Hz wheel PIDRun log → SDevery packet: seq, age, fatepeer broadcastData lakeParquet + manifestAnalysisQ(d), d_c, master curveground-truth poses
Subsystems and interfaces. Ground truth never passes through the robots, so the robots cannot grade their own work.

The key interface decision is where delay and loss are injected. Murmur injects them on the receive side: each receiver flips its own loss coin and holds packets in its own FIFO. That makes loss independent per link, which SR-4 requires. It also mirrors Leviathan's delivered-edge model exactly: the simulator decides per link whether a message arrives, then gates coordination on the links that did.

Murmur message path with receive-side delay and loss injectionSENDER jState x_j[k]pose + velocityStampseq, step kESP-NOW txbroadcastnative air path, ≤ 20 ms p99 (SR-5)RECEIVER iESP-NOW rxLoss gatedrop w.p. pDelay FIFOrelease at k + dPeer tablelatest + agePredictornone · CV · KalmanController u_i→ wheel speeds
Message path. Green blocks are the experimental controls; everything else is the system under test.

6. Theory and models

6.1 Robot kinematics

Each robot is a differential-drive unicycle with wheel radius rr and track width LL:

x˙=vcos⁡θ,y˙=vsin⁡θ,θ˙=ω,v=r2(ωR+ωL),ω=rL(ωR−ωL).\dot x = v\cos\theta,\qquad \dot y = v\sin\theta,\qquad \dot\theta = \omega,\qquad v = \tfrac{r}{2}(\omega_R + \omega_L),\quad \omega = \tfrac{r}{L}(\omega_R - \omega_L).

Coordination primitives emit a velocity ui∈R2u_i \in \mathbb{R}^2 for a point-mass agent, as they do in simulation. A unicycle cannot follow that directly, so Murmur controls a point a distance ℓ\ell ahead of the axle, p=(x+ℓcos⁡θ,  y+ℓsin⁡θ)p = (x + \ell\cos\theta,\; y + \ell\sin\theta). That point's dynamics are fully actuated:

p˙=R(θ)[100ℓ][vω]⟹[vω]=[1001/ℓ]R(θ)⊤ui.\dot p = R(\theta)\begin{bmatrix}1 & 0\\ 0 & \ell\end{bmatrix}\begin{bmatrix}v\\ \omega\end{bmatrix} \quad\Longrightarrow\quad \begin{bmatrix}v\\ \omega\end{bmatrix} = \begin{bmatrix}1 & 0\\ 0 & 1/\ell\end{bmatrix} R(\theta)^{\top} u_i .

This feedback linearization makes the robot behave like the simulator's point mass, up to actuator limits. It is what lets the same primitive run unchanged on both platforms.

6.2 Coordination under delay

For gossip-consensus rendezvous, robot ii at control step kk acts on the newest delivered state of each neighbour:

ui[k]=α∑j∈Ni[k](p^j[k]−pi[k]),p^j[k]=pj[k−d]+d Δt p˙j[k−d].u_i[k] = \alpha \sum_{j \in \mathcal{N}_i[k]} \big(\hat p_j[k] - p_i[k]\big), \qquad \hat p_j[k] = p_j[k - d] + d\,\Delta t\,\dot p_j[k-d].

The prediction term is optional and is used only in the predictor experiments (constant-velocity shown here; Kalman is the alternative). The robot's own state is always fresh. Only its view of its peers is stale.

Analytic anchor. In the fully delayed scalar form x[k+1]=x[k]−q x[k−d]x[k+1] = x[k] - q\,x[k-d], which applies per Laplacian eigenvalue with q=αλq = \alpha\lambda, the system is asymptotically stable if and only if

0<q<2sin⁡ ⁣(π2(2d+1))  →  d→∞    q d→π2.0 < q < 2\sin\!\left(\frac{\pi}{2(2d+1)}\right) \;\xrightarrow{\;d\to\infty\;}\; q\,d \to \frac{\pi}{2}.

This is the Levin–May condition. It predicts the inverse scaling dc∝1/αd_c \propto 1/\alpha that the simulation measured as dc α≈2.8d_c\,\alpha \approx 2.8. The constant differs because QQ measures task quality rather than stability, and because the robot's own term is undelayed. The scaling law is what Murmur tests. The constant is what it calibrates.

6.3 Mapping simulation steps to physical time

QuantitySimulationMurmur
Control step Δt\Delta t1 step100 ms
Gain α0.07–0.28same, per step
Predicted onset dc≈2.8/αd_c \approx 2.8/\alpha10–40 steps1.0–4.0 s
Robot speed capnormalized0.2 m/s

Delays are specified in steps, so the comparison is dimensionless. The arena only has to be large enough that a robot cannot cross it within the longest delay. At 0.2 m/s × 4 s = 0.8 m, the 2.4 m arena satisfies that.

7. Detailed design

Murmur robot board schematicMURMUR ROBOT · MAIN BOARD · REV A (DESIGN)U1 · MCU + radioESP32-S3-WROOM-13V33V3GNDGNDIO8SDAIO9SCLIO1VBAT_SNSIO2LEDIO4AIN1IO5AIN2IO6BIN1IO7BIN2IO15ENC_LIO16ENC_RIO10-13SD_SPIU2 · Motor driverTI DRV8833AIN1AIN1AIN2AIN2BIN1BIN1BIN2BIN2VMVBATAOUTBOUTMMN20 gearmotors + magnetic encodersU3 · 3V3 railAP63203 buckVINVBATGNDGNDVOUT3V3U4 · IMUBosch BMI270SDASDASCLSCLVDD3V3U5 · Power monitorTI INA219SDASDASCLSCLIN+BATTIN−VBATPower: 2S 500 mAh LiPo (BATT) → 2 A polyfuse + reverse-polarity FET → VBAT. microSD on SD_SPI logs every packet.
Robot main board. All slow peripherals share one I²C bus; the SD card logs every packet with its sequence number, age and fate (delivered, dropped, superseded).
Murmur robot isometric render2.4 GHz antenna (ESP-NOW)AprilTag 36h11 · ground-truth fiducialBattery + electronics deckN20 drive wheels · rear caster~100 mm footprint · ~$55 BOM
Robot render. The tag sits on the highest surface so a single overhead camera always sees it.

Bill of materials (per robot)

ItemPartEst. cost
MCU + radioESP32-S3-WROOM-1$5
Motor driverTI DRV8833$2
Drive2 × N20 gearmotor with magnetic encoder$14
IMUBosch BMI270$4
Power monitorTI INA219$2
RegulatorAP63203 buck + protection$3
Battery2S 500 mAh LiPo$9
StoragemicroSD socket + card$5
MechanicalPCB, printed chassis, wheels, caster, tag$11
Total≈ $55 (meets SR-10)

Arena infrastructure adds roughly $250: an OV9281 global-shutter camera, a Raspberry Pi 5 ground-truth server, a ceiling frame and a matte floor.

8. Budgets

8.1 Native latency budget (SR-5)

StageBudget
State sample + packet build1.2 ms
ESP-NOW airtime, 32 B payload≈ 0.5 ms
Channel contention (8 robots @ 10 Hz)0–5 ms
Receive callback + queue1 ms
Wait for next 100 Hz inner-loop tick≤ 10 ms
Worst case≈ 18 ms ≤ 20 ms

Channel load is 8×10 Hz×0.5 ms≈4%8 \times 10\,\text{Hz} \times 0.5\,\text{ms} \approx 4\% of airtime, far below the point where contention would add tail latency.

8.2 Power budget (SR-1)

LoadAverage
Two drive motors, cruising≈ 1.0 W
ESP32-S3, radio always on≈ 0.37 W
IMU, power monitor, SD, LED≈ 0.1 W
Total≈ 1.5 W

A 2S 500 mAh pack holds about 3.7 Wh, which gives ≈ 2.4 h of runtime: five times the 30-minute requirement.

8.3 Ground-truth accuracy (SR-2)

An OV9281 (1280 × 800) mounted 2.5 m up with a 2.6 m field of view gives ≈ 2 mm per pixel. AprilTag corner refinement is sub-pixel, so ≤ 5 mm position error is realistic. It still has to be shown against a surveyed grid before any run counts (verification test VT-2).

9. Trade studies

Radio. Scored 1–5, weighted:

OptionLatency (0.30)Cost (0.25)Integration (0.20)Determinism (0.15)Range control (0.10)Score
ESP-NOW555334.50
802.15.4 (Thread/Zigbee)433433.45
BLE advertising353233.35
UWB (DW3000)422543.25

ESP-NOW wins on latency and cost. Its weak determinism is acceptable because SR-5 bounds native latency below one step, and every packet's real age is logged anyway.

Ground truth:

OptionAccuracy (0.35)Cost (0.30)Rate (0.20)Setup (0.15)Score
Overhead AprilTag45444.30
Wheel odometry only15553.60 (fails SR-2: drift)
Optical mocap51533.50
UWB anchors23332.65

10. Failure modes and effects

Severity (S), occurrence (O) and detection (D) are scored 1–10. RPN is the risk priority number, S × O × D.

Failure modeEffectSODRPNMitigation
Wi-Fi interference spikes native delayContaminates the delay axis854160Fixed quiet channel, pre-run spectrum check, analyse on logged age, not commanded delay
Battery sag changes motor responseDynamics drift during a run56390Encoder speed loop, abort below 7.0 V
Firmware primitive diverges from GossamerSim-vs-real compares different algorithms93381Golden-vector parity test in CI (SR-8)
Robot leaves camera viewLost ground truth74256Virtual fence 0.15 m inside the edge triggers a halt
Clock driftMis-attributed packet ages633541 Hz sync beacon, reject a run if skew exceeds 2 ms
Tag occlusion or misdetectionGaps in ground truth44232Tag on top surface, odometry-aided smoothing, flag gaps

11. Verification and validation

TestVerifiesPass criterion
VT-1 Fleet enduranceSR-18 robots, 30 min, no resets
VT-2 Surveyed-grid pose checkSR-2≤ 5 mm / ≤ 2° at 95th percentile
VT-3 Loopback delay sweepSR-3, SR-5Measured age = commanded ±1 step; native p99 ≤ 20 ms
VT-4 Loss statisticsSR-4χ² test fails to reject independence at 5%
VT-5 Golden vectorsSR-8Bit-identical outputs on 10⁴ recorded inputs
VT-6 Safety tripsSR-9Halt ≤ 200 ms in 50/50 trials

Validation experiments, each with a falsifiable hypothesis:

  • E1, calibration. At zero delay, rendezvous reaches Q≥0.95Q \ge 0.95. This mirrors the papers' Vicsek sanity check: the substrate is trusted before it is used.
  • E2, H1: the cliff transfers. The physical dcd_c lies within ±25% of the simulated dcd_c at the same α. The run budget is 11 delays × 3 primitives × 5 seeds × 3 min ≈ 8 h of robot time.
  • E3, H2: the scaling law transfers. dc αd_c\,\alpha stays constant within ±20% across a 4× range of α.
  • E4, H3: staleness, not throughput. At fixed dd, QQ varies by less than 0.05 across p∈[0,0.9]p \in [0, 0.9].
  • E5, prediction transfers. The constant-velocity predictor recovers at least half the lost QQ at d=2dcd = 2 d_c.
Murmur hypothesis sketch: coordination quality against delay times gain00.510123456normalized delay d · αcoordination quality Qsimulated d_c·α ≈ 2.8H1 acceptance band (±25%)Sketch of the hypothesis, not measured data.
What H1 looks like if it holds: the physical curve's midpoint lands inside the green band. If it lands outside, the sim-to-real gap itself becomes the result.

12. Risks and roadmap

The main technical risk is actuation lag, since a real robot does not respond instantly as the simulated point mass does. It adds an effective delay the simulator does not model. Mitigation: measure the step response in phase 1 and add it to Leviathan's agent model, rather than hiding it.

PhaseDeliverableExit criterion
0This design studyRequirements baselined
1One robot: motion layer + loggingStep response characterised
2Arena + ground truthVT-2 passes
3Fleet of 8 + injectorVT-1, VT-3, VT-4, VT-5, VT-6 pass
4Experiments E1–E5Paper: physical validation of the delay cliff