The handoff that never happened

There is a particular kind of bug that only shows up in systems that have to hand work between two parties. Not a crash. Not a stack trace. Just a quiet disagreement about who was holding the plate.

I found one in my own trading daemon, and it cost me most of a day.

The symptom

The logs said a position opened and then closed 55 seconds later, classified as a "natural close." Natural close is the boring outcome — the position ran its course, nothing dramatic. Except it kept happening. Hundreds of times. Each one paying fees on the way in and the way out.

Fee bleed with no explanation is the worst kind of loss, because nothing is technically failing. Every individual line in the log is plausible.

What was actually happening

The entry orders were maker-only. They sat on the book waiting to be filled. The tier that managed them had a reconciliation pass that asked, in effect, do I still have a position here? — and when the answer was no, it recorded a natural close and moved on.

The problem is that "no" is the correct answer for an order that hasn't filled yet. The system was closing positions that had never opened. It was reconciling against reality before reality had finished happening.

The tell was a pending entry that stayed in the tracking table for seven hours after its supposed 55-second lifecycle. Nothing had cleaned it up, because nothing believed it existed.

Why I didn't see it sooner

Because the classification was reassuring. If those closes had been labelled unknown or orphaned, I'd have looked at them in week one. Instead they were labelled with the name of the healthy path, and I skipped over them for months.

This is the thing I keep re-learning: a default value is a claim about the world. Defaulting the close reason to "natural" wasn't a neutral choice. It was the system asserting that everything was fine, in the absence of any evidence either way.

The kitchen version

Every cook knows this failure. You call a dish away. The grill station hears you. The sauté station doesn't. Nobody says anything, because from grill's point of view the ticket is moving — they're working, the plate is coming together, all the signals look right. The dish dies in the window waiting for a component that was never started.

Kitchens solve this with a callback. You don't just call the order, you make the station repeat it. The handoff isn't complete when the message is sent; it's complete when it's acknowledged.

My scalp tier already did this. It had three layers of defense: check whether the entry actually filled, hold a grace period before reconciling, and check the pending table before declaring anything closed. The swing tier had none of them. Same codebase, same author, six months apart, and I'd simply never gone back to port the lesson across.

What I changed

The fix wasn't clever. I ported the three guards from one tier to the other and made the default close reason unclassified instead of natural. That second change is the one that matters long-term — it means the next version of this bug announces itself instead of hiding inside the happy path.

The part worth keeping

Distributed systems people call this a two-phase commit problem, and there's a deep literature on it. But you don't need the literature to catch it. You need one habit:

When two parts of a system hand work to each other, write down who is holding it, and make the receiving side confirm. Not the sending side. The receiving side.

Everything else is bookkeeping.