The Six-Hour UART Framing Bug

This one was for the drone's telemetry link: a simple length-prefixed frame format over UART, [START][LEN][PAYLOAD...][CHECKSUM]. It worked perfectly on the bench, one frame every 100ms, for two straight days. Then I turned on the motors, which meant real vibration, real electrical noise, and a much higher chance of a byte getting dropped or corrupted mid-frame — and the parser started returning garbage within about thirty seconds of flight.

The setup

The receive side was a small state machine: WAIT_START → READ_LEN → READ_PAYLOAD → READ_CHECKSUM → DISPATCH. Each state consumed exactly the bytes it needed and moved to the next. On paper this handles a corrupted byte just fine — a bad checksum gets rejected, and the state machine goes back to WAIT_START to look for the next frame.

case READ_CHECKSUM:
    if (compute_checksum(buf, len) == byte) {
        dispatch_frame(buf, len);
    }
    state = WAIT_START; /* look for the next frame no matter what */
    break;

That part was correct. The bug was upstream of it.

What went wrong

If a byte was dropped during READ_PAYLOAD — not corrupted, actually missing, which a noisy UART line does — the state machine would keep reading bytes into the payload buffer waiting for a length that would now never arrive, because every subsequent byte was really the start of the next frame, being consumed as if it were the tail end of the current one. The parser wasn't rejecting bad frames. It was silently absorbing the start of every good frame that arrived while it was still confused about the last one, and it had no way to notice that it had been sitting in READ_PAYLOAD for way longer than a frame should ever take.

There was no timeout anywhere in the state machine. It assumed that once it started reading a frame, a frame was actually coming. Under bench conditions with a clean line, that assumption never got tested. Under flight conditions, it failed constantly, and every failure took out not just the corrupted frame but every frame after it until the byte alignment happened to drift back into sync by chance — which could take seconds.

The fix

Two changes, both small:

#define FRAME_TIMEOUT_MS 20

case READ_PAYLOAD:
    if (elapsed_ms(frame_start) > FRAME_TIMEOUT_MS) {
        state = WAIT_START;   /* give up on this frame, resync */
        break;
    }
    buf[idx++] = byte;
    if (idx == len) state = READ_CHECKSUM;
    break;

A frame that takes longer than FRAME_TIMEOUT_MS to arrive is abandoned, and the parser goes back to hunting for the next START byte instead of continuing to trust a length field it read from what might be garbage. The second change was adding a distinctive START byte value that's vanishingly unlikely to appear inside a valid payload, so re-sync after a timeout is fast rather than best-effort.