Home / Algorithmic Risk / Disaster Recovery
ResilienceAdvanced

Disaster Recovery

Disaster recovery is the pre-planned set of procedures for restoring safe control of a trading operation after a major failure, so that positions are reconciled, exposure is contained and trading resumes only once the cause is understood and the state is known.

Quick answer: Disaster recovery is the pre-planned set of procedures for restoring safe control of a trading operation after a major failure, so that positions are reconciled, exposure is contained and trading resumes only once the cause is understood and the state is known.

In simple words

Disaster recovery is your plan for what to do after something has already gone badly wrong, a crash, an outage, a runaway algo, a broker going down mid-position. It is not about preventing the disaster; it is about limiting the damage and getting back to a safe, known state without making things worse. The heart of it is knowing your true position and containing risk first, then resuming only when you understand what happened. A trader without a recovery plan improvises during the worst possible moment, which is exactly when improvisation fails.

Purpose

Disaster recovery exists because prevention is never perfect, so a professional operation prepares in advance for how it will regain safe control after a serious failure, rather than improvising under stress when judgement is worst.

Professional explanation

Recovery is the last line when prevention fails

Kill switches, redundancy, monitoring and limits reduce the chance and size of failures, but none of them guarantees prevention, so there must be a plan for the moment a serious failure happens anyway. Disaster recovery is that plan: a pre-written, rehearsed procedure for regaining control, containing exposure and returning to normal operation safely. Its defining feature is that it is prepared in advance and in calm, precisely because a major failure arrives with adrenaline, uncertainty and time pressure, the conditions under which improvised decisions are worst. The existence of a written, tested plan is what separates a contained incident from a compounding one.

First priority: contain exposure, not restore the system

The instinctive response to a system going down is to rush to bring it back, but that is usually wrong. The first priority in recovery is to contain risk: know what positions are open and ensure they are protected or flattened, using whatever channel still works, the broker terminal, a phone call to the dealing desk, a broker-side square-off. Restoring the failed algorithm can wait; an unmanaged open position cannot. A recovery plan therefore orders the steps deliberately, secure the book first, diagnose second, restore third, because rushing to restart a system while a naked position drifts is how a technical incident becomes a financial one.

Reconciliation: establishing ground truth

After any major failure the single most important fact is the true state of positions and orders, and it is often uncertain, some orders may have filled, others cancelled, others sent but unacknowledged. Recovery hinges on reconciliation: comparing every source of truth, the broker's records, the exchange, the strategy's own logs, to establish exactly what is held before any new action is taken. Acting on a stale or assumed position after a crash is one of the most dangerous things a trader can do, because it can double a position, leave a real one unhedged, or exit something that was already closed. No restart or new trade should occur until the book is reconciled and known with certainty.

Safe resumption and the temptation to rush back

Deciding when and how to resume is where discipline is tested. Resumption should be gated on three conditions: the cause of the failure is understood, positions are reconciled and safe, and the infrastructure is verified healthy. Restarting a strategy while the cause is still unknown risks walking straight back into the same failure, and resuming with reduced size or in a monitoring-only mode first is often prudent. The strong temptation after an outage is to rush back to recover lost opportunity or lost money, which is precisely the tilt that turns one incident into two. A good plan makes resumption a deliberate, checklist-gated decision, not an emotional reflex.

Preparation, backups and rehearsal

A recovery plan is only real if the materials and muscle memory exist before they are needed. That means backups of strategy configuration and state, documented recovery procedures accessible even when the main system is down, out-of-band contact details for the broker, and alternative access to positions and orders. Just as important is rehearsal: periodically running through the recovery procedure, ideally against a simulated failure, so the steps are familiar and the gaps, an out-of-date phone number, a backup that does not restore, a procedure nobody can find, are discovered in a drill rather than a real crisis. An untested recovery plan is a document, not a capability, and the difference only becomes visible when it is too late to fix.

With vs without a disaster recovery plan

AspectWith a planWithout a plan
First actionContain exposure via any working channelRush to restart the failed system
Position knowledgeReconciled against broker before actingAssumed, often wrong, acted on blindly
ResumptionGated on cause understood and state knownImmediate, into the same unresolved fault
Emotional stateFollowing a rehearsed checklistImprovising under stress and time pressure

Practical example

Illustrative example (Indian market)

A strategy on Rs 5,00,000 holding 2 Nifty lots suffers a total failure at 2 pm: the VPS reboots and the broker API session drops simultaneously. Following its recovery plan, the trader does not scramble to restart the algo. First they open the broker's web terminal on a separate device and confirm the true position, 2 lots long, with the protective stops still resting at the exchange, so exposure is already contained. Next they diagnose the reboot cause and verify the API session before touching the strategy. Only once positions are reconciled, protection confirmed and infrastructure checked healthy do they resume, initially in monitoring-only mode. The disaster was identical to one that could have wiped out an unprepared trader; the plan turned it into a contained, orderly recovery with no forced or panicked trades.

For an Indian retail algo, the practical recovery kit is: the broker's dealing-desk phone number, web and mobile terminal access independent of the API, knowledge of the broker's auto square-off policy and timing, and resting stop-loss orders at the exchange so a position stays protected through any client-side outage. Because broker and NSE issues cluster around the volatile open and expiry, the plan should assume failures happen at the worst moments, not quiet ones.

Limitations

  • A plan cannot prevent the disaster, only limit the damage and speed the return to safe control
  • An unrehearsed plan often fails on contact, an out-of-date number, a backup that will not restore
  • Recovery depends on some channel still working; a simultaneous broker and personal outage limits options
  • Reconciliation takes time, during which a fast-moving unhedged position can still lose
  • The plan assumes the discipline to follow it, and the temptation to rush back defeats many good plans

Common mistakes

  • Rushing to restart the system before containing the open position
  • Acting on an assumed position after a crash instead of reconciling against the broker and exchange
  • Resuming trading before the cause of the failure is understood, walking into it again
  • Having no out-of-band way to reach the broker or view positions when the main system is down
  • Never rehearsing the plan, so gaps surface only during a real crisis
  • Letting the urge to recover lost money drive an immediate, oversized return to trading

Professional usage

Institutional trading operations treat disaster recovery as a documented, rehearsed capability, not a hope. They order recovery deliberately, contain exposure first through any working channel, reconcile positions against broker and exchange to establish ground truth, then gate resumption on the cause being understood and the infrastructure verified. They keep configuration and state backups, out-of-band broker contacts and independent position access, and they drill the procedure against simulated failures so gaps surface in rehearsal. Resumption is often staged, monitoring-only or reduced size first, and never driven by the impulse to win back what was lost.

Key takeaways

  • Disaster recovery is the pre-planned, rehearsed process for regaining safe control after a serious failure
  • Contain exposure first, diagnose second, restore third; a drifting naked position cannot wait for a restart
  • Reconcile the true position against broker and exchange before any new action; never act on an assumed state
  • Gate resumption on the cause being understood and the state known, and rehearse the plan before you need it

Frequently asked questions

What is disaster recovery in trading?
Disaster recovery is the pre-planned set of procedures for restoring safe control of a trading operation after a major failure, such as a crash, outage or runaway algo. It is not about preventing the disaster but about containing the damage, reconciling positions and resuming only once the cause is understood and the state is known.
What is the first priority after a system failure?
Containing exposure, not restoring the system. The first step is to know what positions are open and ensure they are protected or flattened using any channel that still works, because an unmanaged open position cannot wait while a naked position drifting is how a technical incident becomes a financial one.
Why is reconciliation central to disaster recovery?
Because after a major failure the true state of positions and orders is often uncertain: some may have filled, others cancelled, others sent but unacknowledged. Reconciling every source of truth, broker, exchange and strategy logs, establishes exactly what is held before any new action, since acting on an assumed position can double it or leave it unhedged.
When should I resume trading after a disaster?
Only when three conditions hold: the cause of the failure is understood, positions are reconciled and safe, and the infrastructure is verified healthy. Resuming before the cause is known risks walking straight back into the same failure, and staging the return in monitoring-only or reduced-size mode first is often prudent.
How is disaster recovery different from a kill switch?
A kill switch stops the strategy when a limit or malfunction occurs; disaster recovery is what happens afterward to regain safe control. The kill switch is an immediate halt, while recovery is the ordered process of containing exposure, reconciling positions, diagnosing the cause and resuming safely. One acts in the moment, the other governs the aftermath.
Why should a recovery plan be written in advance?
Because a major failure arrives with adrenaline, uncertainty and time pressure, the conditions under which improvised decisions are worst. A pre-written, rehearsed plan removes the need to invent a procedure during the crisis, which is what separates a contained incident from a compounding one.
What should be in a disaster recovery kit for a retail algo?
The broker's dealing-desk phone number, web and mobile terminal access independent of the API, knowledge of the broker's auto square-off policy and timing, backups of strategy configuration and state, and resting stop-loss orders at the exchange so positions stay protected through a client-side outage.
Why is rushing to restart the system a mistake?
Because it prioritises the wrong thing. Restarting the algorithm while an unmanaged position drifts leaves real risk uncontrolled, and restarting before the cause is diagnosed risks re-entering the same failure. The correct order is contain exposure first, diagnose second, restore third.
Why must a recovery plan be rehearsed?
Because an untested plan is a document, not a capability. Rehearsing against a simulated failure reveals gaps, an out-of-date phone number, a backup that will not restore, a procedure nobody can find, in a drill rather than a real crisis, where discovering them is too late to fix.
What is the danger of trying to recover lost money after a disaster?
It is the tilt that turns one incident into two. The urge to win back an outage's lost opportunity or money drives an immediate, often oversized, return to trading before the state is safe and the cause understood. A good plan makes resumption a deliberate, checklist-gated decision rather than an emotional reflex.
Can disaster recovery prevent losses?
No, it can only limit them. Prevention is the job of kill switches, redundancy, monitoring and limits; recovery is the last line for when those fail anyway. It reduces the damage of a failure and speeds the return to safe control, but the disaster itself has already happened by the time recovery begins.
How does disaster recovery relate to system and API failure?
System and API failure are causes of disaster; recovery is the response. Whether the trigger is your own hardware dying or the broker connection breaking, the recovery process is the same: contain exposure through any working channel, reconcile the true position, diagnose, and resume only when safe and understood.
What if the broker is down at the same time as my system?
That is the hardest case, and it limits options because recovery depends on some channel still working. The mitigations are resting protective stops at the exchange that survive both outages, a broker-side auto square-off, and multiple independent access routes, so that even a broad outage leaves some form of protection in place.

Voice search & related questions

Natural-language questions people ask about Disaster Recovery.

What is disaster recovery in trading?
It is your plan for what to do after something has already gone badly wrong, so you limit the damage, get back to a safe known state, and do not make things worse.
What do I do first when my system crashes mid-trade?
Contain your risk first, not restart the system. Find your true position on another device and make sure it is protected before you worry about bringing the algo back.
Should I restart my algo straight after an outage?
No. First know your real position, then understand why it failed, then check the infrastructure. Restarting into an unknown cause can walk you right back into the same failure.
Why is knowing my true position so important after a crash?
Because if you act on a guess, you might double a position, leave one unhedged, or exit something already closed. Reconcile against the broker before doing anything.
Do I really need to practise a recovery plan?
Yes. An untested plan usually fails when it matters, an old phone number, a backup that will not restore. Rehearse it so the gaps show up in a drill, not a crisis.
How is disaster recovery different from a kill switch?
A kill switch is the instant halt when something breaks. Disaster recovery is everything you do afterward, containing risk, checking your true position, and resuming safely.
What is in a basic recovery kit for a retail algo?
Your broker's phone number, terminal access on another device, knowledge of the auto square-off policy, config backups, and stops resting at the exchange that survive an outage.

Sources & references

    Last reviewed 12 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Monitoring
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Risk-management techniques reduce but never remove risk, and trading derivatives involves substantial risk of loss. See our Risk Disclosure and SEBI Disclaimer.