The customer tapped Pay. The page sat. The radio dropped. They tapped again. Your processor, which got a 504 from your callback URL, retried at one minute and at five. Two debits posted against one sale.
Decide, before the next tap, what that second message is. If you treat it as a new payment, you invented a charge. If you treat it as the same payment id, you have a control. Retries can debit twice. Idempotence here is deduplication on a stable id.
Why these three models
The decision is what a second tap, or a webhook retry, is allowed to do to the account. Three features fire. A posted debit does not reverse when you restore the button. The pay control is a rule that maps messages to money, and the retry policy is the part people actually play. The only thing the customer and your retry loop can see, in the moment, is a timeout, which looks like “not paid.”
Hysteresis-memory gives a cycle answer: the path back is not the path out. Mechanism design gives an equilibrium answer: the outcome is assigned by the rule, not by the warning on the spinner. Monitoring-noise gives a complex answer: a noisy proxy punishes the already-debited. Cycle, equilibrium, complex. Their mistakes do not line up.
Behaviour is folded: a reasonable person taps twice when a spinner looks like failure. That is already the monitoring-noise claim. Governance, who may write a debit, is already the mechanism. LOOP is folded too: the customer will be back if you refund cleanly, but the lever is a second signal, not a patience talk. PULSE sits inside the first card. The debit is a stock. The button is not.
1. The debit does not walk back with the button
Hysteresis is system memory. After a threshold, restoring the input does not restore the state.
A posted debit is that threshold. The input people reach for is the screen: stop the spinner, grey the button, print “please do not click twice.” None of that moves the stock. The money has already left the wallet. A receipt SMS, a till row, and a webhook that will fire again have already moved. Putting the button back does not put the account back. You need the original payment id to recognise the second copy, or a new correction event.
Stopford’s recovery path is the condition that breaks the trap, and only if you kept the inputs. Fix the bug. Rewind the log to before the bug was introduced. Replay from the original stream of requests. New events go downstream and correct the corrupted ones.1 Event sourcing records every state change, “much like a version control system. As the saying goes, ‘Accountants don’t use erasers.'”1
A refund that silently deletes the extra credit is an eraser. The week looks clean. Next month you cannot count the stairwell. Append the correction against the same payment id. A slip nobody has paid against can still be voided. A payment that already hit the customer’s phone cannot. Without the id, hysteresis stands.
Hysteresis-memory, the cycle lens
- Assumes: after a debit posts, restoring the button does not restore the account.
- Fits because: the extra charge is a one-way stock change; the spinner is not the stock.
- Breaks when: you kept the original payment id and can treat the second copy as the same fact, or append a dated correction.
- Evidence: grade B+. Shape established; payment transfer analogical. Stopford supplies the Breaks-when.
- Counteracts: treating refresh or “try again” as a restore.
- May reinforce: refusing a same-minute void nobody has paid against.
2. The retry rule is the incentive
The pay button is a mechanism. Hurwicz’s definition is enough to work with: participants send messages, and a pre-specified rule assigns an outcome to every collection of received messages.2 The messages here are taps, timeouts, webhook retries, and provider callbacks. The outcome is a debit, a pending row, or a no-op. Whatever rule you actually run is the mechanism. The sentence on the spinner is not.
“Please do not click twice” assigns the control to the person with the worst radio. They are trapped. A different patient in the same chair, same spinner, same dead air, taps again. Lecturing them is a negotiation with someone who has no other move. The lever is the rule, not the plea.
Two rules are in use. Rule A mints a new collection on every tap or retry. The incentive is to keep tapping, because each tap is the only way to turn “not paid” into “paid.” Rule B issues a payment id when the customer agrees to pay, before the network answers, and treats every later message with that id as the same fact. People still tap twice. The second tap no longer prints a second debit. Incentive compatibility here is whether one debit follows from the messages they will actually send.
Stopford draws the same sequence. The interface calls a payment service. That service calls an external provider, fails before it returns, and the interface retries. “Retries can result in duplicate processing, and this can cause very real problems. Taking a payment twice from someone’s account will lead to an incorrect balance.”3 Most systems dodge this because they push to a store that deduplicates on a primary key. Those processes are naturally idempotent, “so long as each one has a unique ID.” Deduplication at the end of the use case is enough. The idea is older than any broker: the end-to-end principle.3 Then the Latin drops away. “Idempotence, in this context, is just deduplication.”4
The control is not a warning on the button. The control is a stable id, born before the tap, checked when the money would move.
HTTP already knew Pay is a non-idempotent method. A client should not automatically retry one unless it can tell the semantics are actually idempotent, or tell that the original request was never applied.5 Your id is that means. Stripe keeps the first response for a given idempotency key so a connection error can be repeated without creating a second object.6 Their write-up is blunt. Networks are unreliable. Calling a charge twice by accident double-charges the customer.7
Mechanism design, the equilibrium lens
- Assumes: a pre-specified rule maps messages to a debit, and people and machines play the rule they face.
- Fits because: the pay control plus its retry policy is that rule.
- Breaks when: the id is minted after the first debit, or two genuine purchases are forced to share one id.
- Evidence: grade A. Incentive compatibility is definitional; the retry mapping is observable.
- Counteracts: asking the customer not to click twice.
- May reinforce: trusting any id the provider can also issue twice.
3. A timeout is not a proof of zero debit
The customer cannot see the debit. They can see a spinner, a dead page, or a USSD session that died under a tin roof. Your webhook cannot see it either: a 504, a closed socket, silence. Both treat the missing receipt as “not paid.” Both then create the second debit.
That is noisy monitoring. Green and Porter’s firms cannot observe the demand shock directly. They observe the market price. When it dips below a trigger they agreed on, they revert to a punishment spell, even though the dip may have been weather rather than cheating.8 Punishments happen on the path, against the innocent, because the signal cannot separate the two. A dropped MoMo callback is the unobserved shock. The spinner is the price. The second tap, and the webhook retry, are the reversion.
A second signal turns the punishment off. The customer’s phone often already has it: the MoMo SMS, the card alert. Your side can query the provider with the payment id before minting a new collection. If the first debit posted, the query says so. If it did not, a new attempt can go out under the same id. Do not infer “not paid” from the absence of your own callback. Absence is the noisy proxy. It is how honest people get charged twice, and how honest webhooks become the second debit.
Variety is short here. The world arrives as paid, unpaid, pending, paid-but-callback-lost, webhook-retry, two genuine purchases. The usual controller has a plea on the button and a human refund three days later. Responses are fewer than states. Two repairs, no third. Attenuate incoming variety: one id per agreed payment. Amplify yours: a query-by-id, a “seen again” line, a same-id refund. A second person on the till changes neither count.
Monitoring-noise, the complex lens
- Assumes: the observer sees a noisy proxy, not the action, and treats the proxy as the action.
- Fits because: a dropped connection looks like “not paid” to the customer and the webhook.
- Breaks when: a second signal (provider query, MoMo SMS) is checked before a new debit.
- Evidence: grade A. Noisy monitoring produces on-path punishment of the innocent.
- Counteracts: reading a spinner or a 504 as proof of zero debit.
- May reinforce: holding every receipt for a second signal, which some customers also read as failure.
GEER: what you can change on the checkout without a new stack
Net exposure sits on the second debit, not on the spinner. Ruin here is a week of extras: a processor that freezes you, a bank that files you as noisy, a thread that tells three other patients you take money twice. Cap the change so it cannot mint a second collection while you are “improving the flow.”
Cheapest and most reversible first.
- Mint the payment id when the customer agrees to pay, before the network answers. Put it in the link, the narration, the till row. An afternoon on the sheet you already have.
- Dedup at the end of the use case. The second tap, the second callback, and the cashier who types the reference twice are noise in the middle. The books look at the id once. Stopford’s end-to-end condition is the same one Saltzer, Reed and Clark stated for any function that can only be finished at the application: the communication system cannot complete it for you.3, 9
- Store the provider’s reference as a child of your id, not as a new sale. Providers may see two attempts. You may not see two sales.
- Query the provider by id before you mint a new collection. That is the second signal. A morning on most MoMo and card APIs.
- Append a correction. Do not erase. A true double gets a refund line against the same id. Next month you can count the stairwell.
- Do not buy broker-style exactly-once as the MoMo fix. Stopford is explicit. Kafka transactions “work only in situations where both the input and the output go through Kafka.” Call an external service over HTTP, update a database, or write to anything other than that broker, and “transactional guarantees won’t apply and calls can be duplicated.”10 A MoMo callback is HTTP. A bank is a bank. Exactly-once does not travel with you across either hop. A sales page that lists exactly-once is describing hops inside its own log. The tool may help you store the id. It cannot turn a mobile-money hop into a closed broker.
RADAR: what to have live before the next dropped Saturday
Do now, by T+3, one afternoon, effect visible on the next tap. Issue a payment id before the button is live. Add a seen column. The first credit with that id writes the amount and the time. Every later credit with that id writes “seen again” and does not add to collections. Reversible: you can still refund by hand. Dominant across every story about whose fault the double was, because the column picks the fact, not a villain.
Hedge, by T+14, premium is one extra status call, cover live before the next campaign or clinic day. Query the provider by id before a new collection, and write a same-id refund playbook with a same-day owner. The cover has to be live before the next bad-radio Saturday, or the hedge is a note. If doubles were never the problem you have spent a few queries, and that is the whole downside.
Defer and trigger, size declared when the trigger is set. Rebuilding checkout onto a broker, or buying a processor because the brochure says exactly-once, is expensive and sticky. Pre-commit the observable: two same-id doubles in one week after the seen column and the query are live, or one provider-side double under two of their references that your id could not collapse. When that fires you may change rails. The id has to exist first, or you cannot see whether the broker would have helped.
CHAIN: checkouts that treat a retry as a new sale
Name the shape before you explain it. This is a one-way step in a stock, then a lagged correction if anyone complains. It is not oscillation. The refund does not restore Tuesday. It posts a Wednesday. Two clocks can also appear, till versus SMS, but the harm here is the extra debit, not the disagreement window.
Ask the break questions before you borrow anyone else’s week. Has a rule changed, has an actor entered or left, has a measurement become a target? The last one fires when someone is paid on “collections today” and a retry looks like volume. Last year’s “we always refund extras on Friday” then describes a target, not a book. The seen column still holds. The incentive on the till does not.
The class that matches on structure is any write that crosses an unreliable hop (MoMo callback, card HTTP, webhook, USSD) with retries and no stable id at the start of the use case. The directional base rate does not need a percentage. Stopford drew the failure as a figure because it is ordinary. Stripe shipped keys because connection errors are ordinary. Shops that treat every incoming credit as a new fact produce extra debits on bad radio. Shops that name the payment first produce a “seen again” line.
Take off the lookalikes. Two genuine purchases are two ids. Do not refund those. A cell overwritten after the fact is a memory problem. A till and a MoMo SMS that disagree for ninety seconds, then agree, is a lag problem. Credit a repair only if the second copy stopped creating a second debit.
One flag. If the provider posts twice under two references you cannot join, or a regulator requires their reference as your only key, the matrix has moved. Your id remains the right local control. It is no longer sufficient. Change rails, or require their second reference as a child and stop minting until both are in.
What this read cannot collapse
These three models can tell you that a posted debit will not walk back with the button, that the retry rule is the incentive, and that a timeout is a noisy proxy for “not paid.” None of them can tell you whether the provider will still answer a status query on that id at 18:40 on a Saturday, or whether the person who has always refunded extras in the chat will stop because you wrote a column. You agreed to be surprised by both.
There is a second limit, and it cuts against the whole argument. This applies when the second message is a copy of one agreement to pay. Two people paying two invoices, or one person buying a second item, must mint a second id. Applying this piece there would collapse two sales into one.
One property no member models: once every tap has an id, people start treating informal cash as a defect because it has no row. The chat was absorbing awkward cases, a balance paid later, a guardian paying from a different wallet. That absorption does not appear in the seen column. You will notice it only when it stops.
The one action that survives the ignorance: before Friday, mint a payment id on the next ten collections before the button is live. On one of them, tap Pay twice on a dropped connection, or simulate the webhook retry. Accept only one receipt. If two receipts print, the column is decoration. If your only control is “please do not click twice,” you do not have a control.
Who owns the payment id this week
The person who needs this is usually the one about to add a spinner, or the one about to buy a processor because Saturday’s thread said “charged twice.” The cheapest first test costs an afternoon and no new vendor: name the payment before the tap, refuse a second debit of that name, query before you mint. The customer in the stairwell will keep tapping. Make that tap idle for the books.
Sources and notes
- Ben Stopford, Designing Event-Driven Systems: Concepts and Patterns for Streaming Services with Apache Kafka, O’Reilly, 2018, chapter 7, printed p. 58. Recovery: fix the bug, rewind the log to before the bug was introduced, replay from the stream of order requests; new events correct the corrupted ones. Event Sourcing records every state change, “much like a version control system. As the saying goes, ‘Accountants don’t use erasers.'” Printed pages follow the book’s own pagination (printed = PDF page minus 15). Author landing for the free O’Reilly edition: benstopford.com.
- The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2007, awarded to Leonid Hurwicz, Eric S. Maskin and Roger B. Myerson for the foundations of mechanism design theory. The prize committee’s popular background states that Hurwicz defined a mechanism as a game in which the participants send messages to each other and/or to a “message center,” and where a pre-specified rule assigns an outcome to every collection of received messages. nobelprize.org.
- Stopford, ibid., chapter 12, printed p. 112. “Retries can result in duplicate processing, and this can cause very real problems. Taking a payment twice from someone’s account will lead to an incorrect balance.” Many systems are naturally idempotent because a database deduplicates on the primary key; this applies to payments “so long as each one has a unique ID.” Deduplication at the end of each use case is the end-to-end principle, “dating back to the early days of TCP.”
- Stopford, ibid., chapter 12, printed p. 115. “Idempotence, in this context, is just deduplication.” Each producer and each message carry identifiers the broker uses to discard a batch already in the log.
- R. Fielding, M. Nottingham and J. Reschke (eds.), HTTP Semantics, RFC 9110, June 2022, section 9.2.2. A request method is idempotent if the intended effect on the server of multiple identical requests is the same as the effect of a single such request. A client should not automatically retry a non-idempotent method unless it has some means to know the request semantics are actually idempotent, or some means to detect that the original request was never applied. rfc-editor.org/rfc/rfc9110.html.
- Stripe, Idempotent requests, API reference. “The API supports idempotency for safely retrying requests without accidentally performing the same operation twice.” After a connection error, the same idempotency key can be repeated without creating a second object or performing the update twice. Stripe saves the status code and body of the first request for that key. docs.stripe.com/api/idempotent_requests.
- Brandur Leach, Designing robust and predictable APIs with idempotency, Stripe, 22 February 2017. “Networks are unreliable.” A charge endpoint called twice by accident “would lead to the customer being double-charged.” The client sends a unique id with the payload so the server can recognise a retry of the same operation. stripe.com/blog/idempotency.
- Edward J. Green and Robert H. Porter, Noncooperative Collusion under Imperfect Price Information, Social Science Working Paper 367, California Institute of Technology, revised December 1982; published in Econometrica 52 (1984), pages 87 to 100. Each period’s demand shock is one the firms “cannot observe directly.” Unobserved demand fluctuations produce unstable performance: firms act monopolistically while prices stay high and revert to Cournot behaviour for a fixed spell when the market price dips below an agreed trigger. Open working paper (text layer): authors.library.caltech.edu. Human-readable record: authors.library.caltech.edu/records/7nsw7-ab254. The cartel application is not transferred. The geometry is: an unobserved shock, a noisy public proxy, an on-path punishment of the innocent.
- J. H. Saltzer, D. P. Reed and D. D. Clark, End-to-end arguments in system design, ACM Transactions on Computer Systems 2(4), November 1984, pages 277 to 288. “The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the end points of the communication system.” Duplicate-message suppression is listed among the functions for which a low-level mechanism is at most a performance enhancement. Open text: web.mit.edu/Saltzer.
- Stopford, ibid., chapter 12, printed p. 119, under the heading “What Can’t Transactions Do?” Transactions “work only in situations where both the input and the output go through Kafka.” Calling an external service via HTTP, updating a database, writing to stdout, or anything other than writing to and from the Kafka broker means “transactional guarantees won’t apply and calls can be duplicated.” “So, much like using a transactional database, transactions work only when you are using Kafka.”
A note on what is deliberately absent. This piece does not tell you which cell in last quarter’s sheet may be overwritten. That is a memory question. It does not tell you whether two live screens of one stream may disagree for a minute. That is a lag question. The only decision here is whether a second tap, or a second webhook, is allowed to print a second debit.
Joshua Agonya Pi’Rwot, Founder.