Payments have been failing for ninety minutes. Your engineers have three theories, your provider says they are investigating, and four customers have asked what is going on.
You are waiting until you can give them a proper answer. A proper answer is two days away, and the decision about what this costs you is being made in the next two hours.
Why these three models
The decision is what to tell customers during an incident you do not yet understand, and what to do while you do not understand it. The features that fire are a system whose internals you cannot inspect in the moment, a failure that reveals a dependency structure nobody had mapped, and a standing that is being spent and replenished in real time.
Three lenses. The black-box probe produces a random answer about how to act without a diagnosis. Exposure topology produces a complex answer about what else is at risk right now. Reputation decay produces a cycle answer about why the communication is the durable part. Most incident advice is about the fix. All three of these are about the ninety minutes before anyone knows what the fix is.
1. You cannot model it, so probe it
The first instinct in an incident is to understand it. That instinct is correct in a post-mortem and wrong during the event, because the information required to understand it is exactly what you do not have.
The alternative is to treat the system as a black box: stop trying to explain the mechanism and instead act on what you can observe from outside, changing one input at a time and reading what comes back.1 Which region is failing. Which payment method. Whether it is all customers or one integration. Whether it started at a deploy or at an hour.
Those probes do not require a theory and they narrow the problem faster than a discussion does. They also produce, within twenty minutes, the only thing your customers actually need from you: a description of scope. Not a cause, a scope. Which of them is affected, and which is not.
You do not need to know why to tell them who, and who is the part they are waiting for.
The probes are worth ordering, because the first one you run decides how fast the scope narrows. Start with the split that removes the most customers from the affected set. Usually that is by channel or by integration: if mobile money is failing while card payments clear, you have halved the problem and produced a sentence you can send. Time-based splits, meaning what changed at the moment it started, are better for the fix and worse for the message, because they tell you about your systems and nothing about who is affected.
Run one probe from outside your own network as well. A founder in Kampala watching an aggregator fail cannot tell from inside the office whether the fault is the aggregator, the telco or the office connection. A phone on a different network, running one transaction, settles that in two minutes, and it is the step that gets skipped because everybody is already reading logs.
The discipline that makes this work is to separate the diagnosis stream from the communication stream, and to give them to different people. One person runs probes and narrows scope. Another reports scope on a fixed cadence. When the same person does both, communication stops the moment the technical work gets interesting, which is always.
The cadence matters more than the content. Every thirty minutes, on the half hour, even when nothing has changed, and especially then. A message that says we still do not know, here is what we have ruled out, next update at the hour is far better received than silence followed by an excellent explanation, because it tells the customer they can stop checking.
2. Find out what else is behind the same door
The second lens runs in parallel with the first and is the one nobody assigns during an incident.
The thing that failed is a node, and the useful question during the failure is what else routes through it. If your payment provider is down, what else depends on that provider: refunds, reconciliation, the daily settlement file, the fraud check, your own reporting. If a database is unavailable, which other services quietly read from it.
Structures with concentrated routing tolerate ordinary faults well and fail broadly when the concentrated node goes, and from inside the organisation the concentration is usually invisible until it is demonstrated.2 An incident is the one moment when it is visible for free, and the list you write during it is the most valuable artefact the event will produce.
Practically: while the incident is live, have someone write down every consequence as it emerges, including the ones that resolve themselves. That list is your dependency map, drawn from evidence rather than from architecture diagrams, and it will contain two or three items nobody knew about.
Keep that list somewhere that survives the incident, which sounds trivial and is why most of these lists are lost. A shared document with a timestamp against each line beats a chat thread, because the thread gets scrolled past by the third hour and nobody opens it afterwards. One person, one document, one line per consequence with the time it appeared.
Two categories on that list need escalating while the incident is still running rather than at the review. Anything that touched money and will not correct itself: a double charge, a settlement file that will run with a gap, a refund that failed silently. And anything that touched data belonging to somebody else, because in several jurisdictions the notification clock starts when you became aware rather than when you finished investigating. Both are cheap to raise at hour two. Both become expensive the moment they surface after you have published an account that did not mention them.
It also changes what you say. A team that knows reconciliation is affected can tell customers before they discover it, which is a different conversation from the one where they find out on Thursday that Tuesday’s totals are wrong.
3. The next 48 hours are what people remember
The third lens explains why this is worth the founder’s attention at all rather than being delegated as an engineering matter.
Standing with customers is a flow rather than a balance. It is drawn down by events, topped up by what people observe you doing, and it decays on its own when neither happens, which means it is replenished by recent conduct rather than by an accumulated record.3 An incident is a large draw. The communication during and after it is the top-up, and the two land within days of each other.
What tops it up is specific and cheap. Frequency during the event. A written account afterwards with what happened, what you changed, and what would have caught it earlier. Proactively contacting the affected customers rather than waiting for complaints. Naming the ones who were not affected, so they do not assume they were.
What does not top it up: a discount offered instead of an explanation, an apology without a mechanism, or a post-mortem that describes the failure as unprecedented, which readers correctly hear as we did not consider it.
The discount question arrives in every one of these, so settle it in advance. Money offered instead of an account reads as payment for silence. Money offered after the account, unrequested, and calculated from something the customer can check, such as the hours the service was unavailable against what they pay for it, reads as a term being honoured. The order is the entire difference and it costs you the same either way.
The asymmetry is worth stating plainly, because it is what justifies interrupting a founder’s day for a communication task. An outage is an operational event with a bounded cost. How it was handled is a story that gets told at renewal, in reference calls, and to the next customer who asks whether you are reliable. The outage is over by Thursday and the account of it is not.
The timing is the part founders get wrong. The credit for good handling accrues while people are still paying attention, which is a window of about two days. A thorough report published in week three is a document nobody reads, and the same content sent in eighteen hours changes how the event is remembered.4
So name the person who writes that account before the incident rather than after. It is the task that slips, because by Thursday the system is working, the pressure has gone, and the window closes without anybody deciding to let it. Put it on one named person with a deadline stated in hours from resolution, and take a shorter document delivered on time over a better one delivered in week three.
What the three say together
- Stop diagnosing in public. Probe to establish scope, and report scope rather than cause.
- Split the two jobs: one person narrows the problem, another communicates on a fixed cadence.
- Update every thirty minutes even when nothing has changed, and say when the next update comes.
- Assign someone to write down every consequence as it appears. That is your real dependency map.
- Publish the account within two days, with what changed and what would have caught it earlier.
Where they disagree
The probe argument and the reputation argument conflict about how much to say early.
Probing says communicate constantly and provisionally, because scope is knowable long before cause and a customer’s main need is to know whether they are affected. Reputation says that early statements become the record, that a scope you announce and then widen twice reads as either incompetence or concealment, and that each correction costs more than the silence would have.
The resolution is in what you commit to rather than in how often you speak. State only what you have observed, and never what you expect. Payments to this provider are failing for customers in this region is an observation and remains true. We believe this is limited to one provider is a forecast, and when it is wrong you have spent standing on a guess you were not required to make. Frequent updates that contain only observations can be issued every thirty minutes for a day without a single correction, which is the arrangement both models accept.
What none of them contain
None of the three handles the incident that is somebody else’s. When your provider is down, you can report scope and cadence and you cannot fix it or promise a time. The models assume you control the system, and much of the operational risk for African founders sits with providers, banks and networks where you have no visibility and no leverage, only a support ticket.
None of them prices the internal cost of the cadence. Thirty-minute updates during a long incident consume someone senior for the whole event, and if that person is also the best debugger you have made the outage longer to make the communication better. That trade is real and it argues for naming the communicator in advance rather than discovering the conflict at hour two.
None of them addresses the second incident. A single outage handled well can strengthen a relationship. Three in a quarter, each handled well, is a pattern, and communication quality stops compensating at a point nobody announces. The models here describe one event, and customers are counting.
And one property the ensemble will not produce: some customers will be reassured by a rapid, calm cadence and others will read frequent updates as evidence of a serious problem. The same behaviour produces both reactions, and which one you get depends on the relationship before the incident rather than on anything you do during it.
The one action that survives the ignorance: today, before anything breaks, write down who communicates during an incident and who debugs, and make sure they are two different people. That single decision, made in advance, is most of the difference between an outage customers forget and one they cite at renewal.
Who has to move
The founder, and entirely before the event, because during it nobody has the attention to design a process. The instinct in the moment is to concentrate everyone on the fix and communicate when there is something to say, which optimises the technical outcome and spends the relationship. The cheapest first test costs fifteen minutes: name the incident communicator, write the four-line update template, and agree the cadence. It sits unused until the day it saves you an account.
Sources and notes
- Stafford Beer, Cybernetics and Management, English Universities Press, 1959. The black box, and the argument that a system whose internals cannot be inspected is nevertheless controllable by manipulating inputs and observing outputs, run throughout this volume, which uses the term extensively. Used in section 1 for the claim that a live incident should be probed rather than modelled, and that scope is establishable long before mechanism.
- Albert-László Barabási, Network Science, Cambridge University Press. Robustness and fragility in networks with concentrated structure, where the removal of ordinary nodes has little effect and the removal of a hub has a large one, is developed in the chapter on network robustness. Used in section 2 for the claim that a failure exposes routing that was invisible beforehand, and for the recommendation to record consequences during the event rather than reconstructing them afterwards.
- Martin W. Cripps, George J. Mailath and Larry Samuelson, Imperfect Monitoring and Impermanent Reputations, Econometrica 72(2), 2004, pages 407 to 432, developed at book length in George J. Mailath and Larry Samuelson, Repeated Games and Reputations: Long-Run Relationships, Oxford University Press, 2006. Under imperfect monitoring a reputation is temporary rather than permanent and is replenished by recent conduct. Used in section 3 for treating incident communication as the top-up that lands within days of the draw.
- Sanjit S. Dhami, The Foundations of Behavioral Economic Analysis, Oxford University Press. Limited attention, and the finding that the weight placed on information depends heavily on whether it arrives while the recipient is attending to the topic, sit in the part on bounded rationality. Used in section 3 for the claim that the same account is worth far more at eighteen hours than at three weeks. Direction only; no magnitude is claimed.
A note on a number this article does not give. There is no correct update interval. Thirty minutes suits an incident measured in hours and is wrong for one measured in days. What transfers is that the interval should be stated in the first message and then honoured exactly, because the commitment to a next update is most of what the message is doing.
Joshua Agonya Pi’Rwot, Founder.