Ransomware Hit at 6:00 AM. Production Was Running Again by 8:00.

The attackers found and destroyed every backup they could reach. The five that mattered were racked a few feet away — powered off, dark, and invisible to everything on the network.

The setup

A growing manufacturer running seven locations across five states — ERP, file shares, and the line-of-business software every site depended on, all on a centralized server environment. No enterprise security team. No enterprise security budget. One IT director responsible for the entire footprint.

That constraint drove the design. The backup strategy could not depend on anyone being awake, present, or paying attention at the moment of an attack. It had to run automatically, and it had to assume the production network would eventually be compromised. Not might be. Would be.

The architecture: five units, one per weekday

Five rack-mounted NAS units on a Monday-through-Friday rotation, one unit per weekday. Each powered itself on five minutes before its scheduled job, captured the day’s data, and powered itself back off five minutes after the job finished. Veeam ran the jobs against production. Nobody had to remember to do anything, and nobody had to be in the building.

Each unit was seeded with its own full backup and took incrementals against that chain thereafter — five self-sufficient chains, one per device, so any single unit could carry a restore on its own and losing one never broke the other four. Because the weekly job was incremental, it ran about an hour and forty-five minutes. Add five minutes on either side and each NAS was awake for roughly two hours a week — and asleep for the other 166.

The defining property: at any given moment, four of the five units were powered off. A machine that is off has no network interface, no listening service, and no session to hijack. It cannot be discovered by a scanning tool, reached by lateral movement, encrypted, or deleted. And there is no single credential that unlocks all five, because four of them are not answering anything at all.

Two details make that hold up under scrutiny. The schedule lived on each NAS itself — not on a network-managed power strip an attacker could reach and override — and Wake-on-LAN was disabled on every unit. The cable stayed in the switch port, but there was no path across the network to bring a sleeping unit up. The only thing that could wake one was its own internal clock.

The second wall: backups that didn’t trust the domain

Physical isolation was the outer wall. Credentials were the second one — and independent of the first.

Backing up the production servers required domain credentials. That part is unavoidable: Hyper-V will not let Veeam connect to the virtual machines without authenticating against the domain. The five rotation units were a different story. Each used its own unique, complex local account created directly on the device, with no relationship to the domain at all.

That distinction is the whole ballgame. Ransomware escalates privileges inside the domain, then uses those now-trusted credentials to move laterally — which is precisely how it finds and destroys connected backups. Anything that trusts a compromised domain is exposed. The rotation units trusted nothing: full Domain Admin compromise would have granted zero access to them, and cracking one unit’s credential would not have yielded the other four.

Those local credentials lived in exactly one place — the Veeam console, behind its own complex password, which was never breached. Even in the hypothetical where a rotation unit had been connected at the wrong moment, an attacker would have had to get into Veeam first just to learn that unit’s credential.

The canary: treating failure as the alert

The rotation carried a deliberate tripwire. If a unit’s data had been tampered with, its scheduled job would fail to run as expected — and that failure was itself the alarm.

Most backup systems are built to alert on success: green check, job complete, nobody looks. This design used failure-to-run as an early-warning signal, surfacing a problem while clean current data could still be captured.

The morning it happened

Those connected units authenticated against the domain. Once the attacker escalated inside it, the backup targets trusted the identity that had already been taken. Reachable, and authenticated by a trusted-but-owned account — so they did exactly what modern ransomware is built to do: destroy the reachable backups first, eliminate the recovery option, and maximize the pressure to pay.

What the attackers never saw were the five rotating units. The attack landed over a weekend, with the business closed and all five sitting dark in the rack. Friday’s complete backup was on a machine that had powered itself off Friday evening and had no reason to wake until Monday.

Recovery

Recovery ran from Friday’s clean unit — roughly two hours from the 6:00 AM call to restored production.

Data loss was zero. Not “minimal.” Zero. The business was closed Saturday and Sunday, so Friday’s end-of-week backup was the current state of the business; there was no activity between the last backup and the attack that wasn’t already captured. Had the business been operating over the weekend, the same rotation principle would simply have extended to cover those days.

Why it worked: three independent walls

Any one of these would have helped. All three were present.

The most instructive part of the whole incident is the contrast: the backups that died and the backups that survived ran on the same network, in the same building, under the same IT director. The difference was that the survivors didn’t trust the domain and couldn’t be reached.

This is not a rare scenario

Industry data has since caught up to what this incident demonstrated firsthand. In Sophos’s State of Ransomware 2024 research, 94% of organizations hit by ransomware said the attackers attempted to compromise their backups during the attack — 93% in manufacturing specifically. The logic is simple: a backup reachable from the network is a backup that can be destroyed.

And when attackers do succeed, the victim is nearly twice as likely to pay the ransom. The defense isn’t tighter permissions on a connected system — it’s a copy the attacker cannot touch at all.

Ahead of the product category

What was built here as a self-designed rig, years before 2020, is now a named commercial product category. Vendors today sell automated multi-unit rotation, “disconnected-by-default” appliances that wake only for their backup window, and diode-based one-way write vaults — all variations on the same core idea: keep at least one clean copy unreachable, on a schedule, automatically. The scheduled wake-and-sleep behavior those products are built around is precisely what these five units were doing years earlier. This architecture arrived at the conclusion independently, and then proved it under live fire.

What I’d build today

The physical-isolation principle is timeless. The threat surface has moved. A current design keeps what worked and closes the gaps a 2020 on-premises rig didn’t address:

• Air-gap by default, retained. At least one clean copy always unreachable, on an automated schedule that runs whether anyone remembers it or not.
• Microsoft 365 tenant backup. Exchange Online, SharePoint, Teams, and OneDrive data lives in the cloud and is not protected by an on-prem NAS. It needs its own backup layer.
• An immutable offsite copy. So that fire, flood, or a tornado cannot take primary and backup together.
• Tested restores on a schedule. An untested backup is a hope, not a plan.

The takeaway

Ransomware will try to destroy your backups before it touches anything else. If your backup can be reached over the network, plan on it being destroyed. The businesses that recover in hours instead of weeks — and never write a ransom check — are the ones holding a clean copy the attacker simply could not get to.

The goal isn’t to win the fight. It’s to make the fight irrelevant.

Designing that architecture, and the disaster recovery and business continuity plan around it, is the work I own as a fractional CIO — along with the security posture and risk register that sits above it. For manufacturers and distributors, where a line down is money lost by the hour, it is the difference between a bad morning and a bad quarter.

Source: Sophos, The State of Ransomware 2024 — independent survey of 5,000 IT and cybersecurity leaders across 14 countries. Company identity withheld by design.

← All articles