How Does dbtech Handle Downtime When Both the EHR and the Backup System Experience Issues Simultaneously?
This question addresses one of the most anxiety-inducing scenarios in healthcare IT: the double failure. The EHR is offline, the downtime workstation is also having problems, and the clinical team has neither primary nor backup access to patient information. Understanding how likely this scenario is, what dbtech’s architecture does to prevent it, and what the response protocol looks like when problems do arise gives healthcare organizations a realistic picture of their worst-case exposure and how to manage it.
Why Simultaneous Failure Is Less Likely Than It Appears
The reason simultaneous failure of both the EHR and the dbtech downtime system is less common than it might seem is that the two systems are deliberately architected to have minimal shared dependencies. The EHR typically fails due to:
- Software bugs introduced by an upgrade or patch
- Server hardware failures in the EHR’s hosting environment
- Network connectivity issues affecting the EHR’s application servers
- Vendor-side cloud infrastructure problems for cloud-hosted EHR platforms
- Ransomware or other cybersecurity events that encrypt or disable EHR systems
dbtech’s on-premise architecture stores patient data locally on the downtime workstations or on a local server within the facility, connected to the EHR through an HL7 interface that runs on the organization’s local network. The causes of EHR failure listed above do not generally affect the locally stored downtime data or the workstation hardware. A software bug in the EHR application does not corrupt the patient data cached on dbtech workstations. A vendor-side cloud outage does not reach the local hardware where dbtech data is stored.
The scenarios where both systems might experience issues simultaneously are narrower than they appear:
- A severe ransomware attack that encrypts network-accessible systems broadly, which might reach both the EHR and any dbtech workstations connected to the compromised network
- A physical disaster such as a fire or flood that damages the infrastructure supporting both systems
- A facility-wide power failure that takes down all electronic systems simultaneously, including both the EHR and the downtime workstations
Each of these scenarios has specific mitigations that are worth building into the preparedness program.
How the Architecture Reduces Simultaneous Failure Risk
Several specific dbtech architectural choices reduce the probability of simultaneous failure:
Patient data is stored locally rather than in a cloud environment shared with the EHR. When the EHR’s cloud infrastructure fails, the locally stored downtime data is unaffected. When the EHR’s application layer has a software problem, the downtime data store is not running the same software and is not subject to the same failure mode.
The HL7 interface that feeds patient data from the EHR to the downtime workstations runs on the local network rather than through the EHR application layer. An EHR application failure that prevents clinical users from accessing the system does not necessarily prevent the HL7 feed from continuing to deliver patient data to the downtime workstations, depending on the specific nature of the failure.
For ransomware scenarios specifically, the guidance in our post on the difference between EHR downtime and a ransomware attack is directly relevant. When a ransomware attack is detected, the containment response may involve isolating network segments. Downtime workstations that are on isolated network segments may lose their HL7 feed but retain the patient data that was cached before isolation. Whether to continue using isolated downtime workstations during a ransomware containment is a clinical and security decision that the organization’s cyber incident response plan should address in advance.
What to Do When a Downtime Workstation Has a Problem During an Outage
When a specific downtime workstation experiences a problem during an actual EHR outage, the response protocol should address:
First, determine whether the problem is isolated to a single workstation or affects all workstations. A single workstation with a hardware or software problem is a manageable issue. All workstations simultaneously experiencing problems suggests a network or infrastructure issue that may have a common cause.
For a single workstation problem, the immediate response is directing clinical staff to the nearest functional downtime workstation in an adjacent department or area. The department-level procedure should specify which neighboring department’s workstations can serve as a backup when the primary workstation is unavailable.
For a network-affecting problem, the IT team should assess whether the local network infrastructure is functioning and whether the issue is related to the same event that caused the EHR outage. If the network is functional, the dbtech Downtime Dashboard provides visibility into which workstations are active and which are experiencing issues, allowing IT to prioritize their response to the workstations serving the highest-acuity clinical areas.
The Last Resort: Printed Downtime Binders
The scenario where both the EHR and all downtime workstations are simultaneously unavailable is the reason that most downtime preparedness experts recommend maintaining a printed downtime binder in each clinical area as a last-resort backup. This binder, described in our post on what happens to downtime workstation data after a full facility power outage, contains a printed patient census updated at each shift start, basic medication reference information for current inpatients, and critical patient safety information for the highest-acuity patients on the unit.
The printed binder is not a substitute for the dbtech downtime system. It is a fallback for the specific scenario where no electronic access to patient information is available from any source. Updated at the start of each shift from the live EHR, it provides a snapshot of patient information that is at most 12 hours old, which is a significantly better starting point than no information at all.
To discuss how dbtech’s architecture reduces your organization’s exposure to simultaneous system failure and what your specific network configuration means for your fallback protocol, contact our team or request a demo.