Top Five Bugs from Five Years of Robot Software

After five years writing robot software at Dex, the bugs I remember best are not the hard algorithm problems. They are the small ones hiding in the plumbing: a connection here, a sleep call there, one stale number in a struct.

These are my top five. None of them shows up in a quick unit test. They show up after hours of running, on real hardware, under load.

Each section follows the same shape: what we saw, what was actually going on, the fix, and the lesson. The code is rewritten as small examples, not copied from the codebase.

1. Redis: “Too many open files”

What we saw. Processes started failing to connect to Redis with Too many open files (errno 24, EMFILE).

At first, a service would simply stop responding in the middle of a run. As we narrowed it down, a pattern showed up: the error came back every time, but only after certain programs had been running for a while.

What was going on. Every component opened its own Redis connection. Each driver plugin, each subscriber, and each Python controller and listener created a fresh client. Some helpers opened a new client on every call.

Each connection is a socket, and each socket is a file descriptor. The default soft limit on Linux is 1,024 per process. Add enough plugins, robots and reconnects, and one process runs out.

The tempting fix is to raise the limit with ulimit -n or Redis maxclients. That only moves the cliff further away.

The fix. Share one client per process, and batch commands into it. Most Redis clients buffer commands locally and send them together:

client.set("robot/state", state);
client.publish("robot/state", state);
client.commit();  // nothing leaves the process until here

Then the buffer bit us. Batching brings its own failure: commands that go into the buffer and never get sent.

A command added after the last commit() waits until something else commits. If nothing does, it never goes out.
A read with a timeout stops waiting but leaves the request queued. In a 1 kHz loop against a slow server, one more stranded request piles up every millisecond.
Async writes while the server is slow or reconnecting grow the outgoing buffer without any bound.

The lesson. Treat connections like memory: one owner, created once. Every buffer needs someone who flushes it and a limit on how big it can grow.

2. The timer that overslept

What we saw. A loop configured for 1 kHz actually ran at roughly 910–950 Hz. Everything downstream that assumed the configured rate slowly fell behind.

The symptoms were easy to miss. Sometimes it showed up as a sporadic jitter in the arm’s motion; other times as odd noise in what otherwise looked like a smooth run.

A timer bug was not on our list of suspects. We looked for high-latency code, misbehaving firmware, or an algorithm that put jittery waypoints into the motion path.

What was going on. The loop timer looked like this:

void Timer::Sleep() {
  auto now = clock::now();
  if (now < next_) std::this_thread::sleep_for(next_ - now);
  next_ = clock::now() + period_;  // bug: measured from when we woke up
}

sleep_for never wakes exactly on time. On Linux a normal thread can wake up to 50 µs late by default, and later under load. Because the next deadline is measured from the wake-up time, each tick’s lateness is added to the period and never paid back.

At 1 kHz, waking 50–100 µs late makes every tick 1.05–1.1 ms long. That is 910–950 Hz instead of 1,000.

The Python version had the same bug, with a twist. Someone had already written the correct line, next_tick += period, and the line right below it overwrote it.

The fix. Advance the deadline from the previous deadline, and sleep until an absolute time:

next_ += period_;
std::this_thread::sleep_until(next_);

If a tick runs over, decide on purpose what happens: skip to the next future deadline, or run the late ticks back to back. Don’t let drift decide for you.

A related trap: scheduling on the wall clock (system_clock) instead of a monotonic one. When NTP steps the clock backward, a timer waiting for a wall-clock time oversleeps by the size of the step.

The lesson. Periodic loops should track deadlines, not durations. Measure the rate you actually get in production, not just the one you configured.

3. Garbage collection stalls

What we saw. Mid-motion, the robot would occasionally hesitate and then jerk forward.

This was a bigger jerk than the jitters from #2, and we were sure something in the code stack was causing it. This time we profiled everything and went looking for gaps in the stack.

What was going on. Our Python motion API streamed a new target to the controller on every tick. To keep GC pauses out of that loop, it switched the collector off for the length of a motion and ran a full collection at the end:

gc.disable()
run_motion_threads()
gc.collect()
gc.enable()

That moved the pause instead of removing it:

Reference cycles pile up during a long motion: message objects, closures, partials, exception tracebacks. The final gc.collect() walks all of them at once.
A full collection holds the GIL, so every Python thread stops. That includes the safety listeners watching for protective-stop signals.
gc.disable() affects the whole process. With two robots moving, the first to finish turned GC back on while the other was still mid-motion. Some motion paths never turned it off at all.

The stall hurt because each target was computed from elapsed time, time.time() - start. After a 200 ms pause, the next target is 200 ms further along the trajectory. The controller sees a step in its setpoint, and the robot jerks.

The fix.

Call gc.freeze() after startup so long-lived objects are not rescanned. Tune thresholds instead of switching GC off.
Break reference cycles in the hot path so reference counting frees objects right away.
Limit how far the setpoint can move in one tick, so a late tick can’t become a step.
Keep hard-timing work out of the Python process where you can.

The lesson. Turning GC off doesn’t remove the pause. It only picks when the pause happens. And in Python, any thread that holds the GIL can delay your safety code.

4. Half of every torque command arrived one tick late

What we saw. Nothing obvious. No visible misbehavior pointed us here; we found it by reading the integrator code while checking another part of the stack.

What was going on. Each tick, the controller computed joint torques. It then integrated a robot model one step forward to get the position, velocity and acceleration commands sent to the robot.

The integrator was Heun’s method, a second-order Runge-Kutta scheme. Take an Euler step, compute the acceleration at the predicted state, then average the two accelerations. The problem was where the first acceleration came from:

State s = model.state();  // s.ddq was computed LAST tick, with LAST tick's torque
s.fq = new_torque;

State pred;
pred.q  = s.q  + s.dq  * dt;
pred.dq = s.dq + s.ddq * dt;  // predictor uses the old acceleration
auto a_pred = ForwardDynamics(model, pred, new_torque);

next.dq = s.dq + 0.5 * (s.ddq + a_pred) * dt;  // half old torque, half new

The velocity update was really this, where τ is the torque and the tilde marks the predicted state:

\[\dot q_{n+1} = \dot q_n + \frac{\Delta t}{2}\Big(a(q_n, \dot q_n, \tau_{n-1}) + a(\tilde q_{n+1}, \tilde{\dot q}_{n+1}, \tau_n)\Big)\]

Half of each new torque command took effect one tick late. A one-tick delay inside a stiff feedback loop is phase lag, and with high gains phase lag turns into chatter or a growing oscillation.

The fix. Compute the first stage from the current state and the current torque:

auto a0 = ForwardDynamics(model, s, new_torque);
pred.dq = s.dq + a0 * dt;

It costs one extra forward-dynamics call per tick.

The lesson. A cached value in a state struct always has an age, even when the struct doesn’t store one. Before using it, ask which inputs it was computed from.

5. Values nobody set: `ddq` and `axis_des`

Two bugs from the same family: a value the code trusted that nobody had actually set.

What we saw. The robot jumped the moment a controller switched on, and a tool-orientation controller sometimes aimed at a direction nobody had asked for.

In simulation the arm went wild: it snapped into some arbitrary configuration or spun fast. On the real robot it would not run at all.

Stale ddq. State updates were partial. An empty field meant “keep the old value”:

if (update.q.size()   > 0) state.q   = update.q;
if (update.dq.size()  > 0) state.dq  = update.dq;
if (update.ddq.size() > 0) state.ddq = update.ddq;

When a controller switched on, we synced the model to the sensors: copy the model’s state, overwrite q and dq with measurements, write it back. The acceleration ddq kept whatever value it had when the last controller stopped. That could be minutes earlier, in a different pose.

The integrator from #4 then used that acceleration in its very first step. Two bugs combined into one visible jump.

Uninitialized axis_des. A controller that keeps the tool pointing along an axis stored its target as a member:

Eigen::Vector3d axis_des_;  // Eigen fixed-size types are not zero-initialized

It was only set by a reset or by a new input. Any path that stepped the controller before either one read whatever happened to be in that memory.

A new input was also easy to reject by accident. The check required the axis to have length 1 within machine epsilon, about 2e-16. A unit vector that has been printed to a string and parsed back usually fails that test. The input was dropped with an error, and the controller kept its old axis.

The reset had its own problem. It read the robot model from the input thread while the control thread was updating that model. A read in the middle of an update can produce an orientation that never existed.

The fix.

Initialize every member where it is declared. NaN is a good default for values that must be set, because misuse then fails loudly.
When syncing a model to sensors, set every field explicitly. Zero the acceleration.
Check norms with a tolerance that matches your data, such as 1e-6, then normalize.
Read a shared model only from the thread that owns it, or copy its state under a lock.

The lesson. “Not set” needs its own representation: NaN, std::optional, or an explicit flag. Never “whatever was there before”.

What the five have in common

None of these bugs lived in the headline algorithm. They lived in the plumbing around it: connections, clocks, the garbage collector, and a few cached numbers.

The rules I took away:

Give every shared resource one owner. That goes for connections, buffers, and the GIL.
Schedule by deadlines, not durations. Then measure the rate you really get.
Know when every cached value was computed, and with which inputs.
Make “not set” loud. A NaN that crashes a test beats a leftover value that moves a robot.

If you are starting on a robot software team, be rigorous about every detail. When a problem looks like it could go deep, put a person or a team on it, or at least ask an expert. Don’t hand-wave it away. If there is no time to do it properly now, mark it as tech debt and come back to pay it off.