How I'd simulate a domain controller failure (and time how long it takes to come back)

Having two domain controllers is reassuring, but that reassurance is worth nothing until you switch one off and watch what happens. Can users still sign in? Does DNS still resolve? How long does the one you took down take to come back? If it were up to me, I’d run a small, planned, measured continuity test. Here’s how, with the before-and-after commands, a script to time the outage from a workstation and what I’d write down at the end.

This is a generic proposal: domain, names and IPs are examples. I don’t have Windows Server at home (my Proxmox laptop doesn’t have the RAM for it), so this isn’t the story of one specific test but how I’d plan it.

1. Decide what you want to know before you switch anything off

A test without questions ends in “seems fine”. I’d write down three before starting:

1. Do clients find the other DC? Sign-ins, GPOs, DNS and access to file shares.
2. How long is the outage a user notices? Not the one you think, the one you measure.
3. How long does the failed DC take to come back and catch up on replication?

And a time target to compare against. In part 3 of the Proxmox series I told how a Windows VM took 5 minutes and 31 seconds to serve again after its node rebooted, when the target was 2. The target was wrong, and the only way to find out was to measure.

2. Make sure the domain is healthy

If replication is already broken before the test, you won’t know what the test broke and what was already broken. From a console with domain admin rights:

dcdiag /q                    # only shows errors; empty is a good sign
repadmin /replsummary        # replication failures between DCs
netdom query fsmo            # which DC holds each FSMO role
Get-ADDomainController -Filter * | Select-Object Name, IPv4Address, IsGlobalCatalog

Three things I’d check carefully:

• Both DCs are global catalogs and both run DNS. If the one that goes down was the only global catalog, sign-ins fail even though the other DC is up.
• The DNS servers clients get. The classic mistake: DHCP hands out only one DC’s IP. If that one goes down, computers can’t find the domain even though the other DC is perfectly fine. Every client needs both DNS servers.
• The DCs’ own DNS. The recommended setup is for each DC to point first to the other one and then to itself (127.0.0.1 as secondary, not as the only one).

Also note who holds the PDC emulator: it provides time to the domain and processes account lockouts. If it’s the one you’re taking down, you’ll know what to watch.

3. Choose how to take it down

A clean shutdown isn’t the same as a power cut. From gentlest to harshest:

Scenario	How	What it tests
Isolate it from the network	Disconnect the virtual NIC	That clients switch DCs. Instantly reversible
Stop only the directory	Stop-Service NTDS -Force	That clients don’t stick to a server that answers ping but doesn’t authenticate
Clean shutdown	Stop-Computer	Planned maintenance
Hard stop	Kill the VM	A real failure, and how long it takes to come back

I’d start by isolating it from the network: if something goes wrong, you reconnect it and it’s back as it was within seconds. In Proxmox it’s the Disconnect checkbox on the VM’s network device (Hardware > Network Device). From the shell, copy the net0 line that qm config <id> gives you and add ,link_down=1. If you don’t repeat the MAC, Proxmox makes up a new one:

qm config 101 | grep net0
# net0: virtio=BC:24:11:AA:BB:CC,bridge=vmbr0
qm set 101 --net0 virtio=BC:24:11:AA:BB:CC,bridge=vmbr0,link_down=1   # cut
qm set 101 --net0 virtio=BC:24:11:AA:BB:CC,bridge=vmbr0                # restore

I’d keep the hard stop (qm stop 101) for a second test, once the first one has gone cleanly.

4. Time it from a workstation, not from the server

The VM booting doesn’t mean the domain works. A DC can answer ping minutes before it serves LDAP and Kerberos. That’s why I’d measure from a client computer, with a loop that records what it sees every 5 seconds:

$Domain = 'your-domain.local'
$DCs    = [ordered]@{ dc01 = '10.0.0.11'; dc02 = '10.0.0.12' }
$Csv    = "$env:USERPROFILE\Desktop\dc-test.csv"

function Test-Port($Ip, $Port) {
    $c = New-Object Net.Sockets.TcpClient
    try { $c.ConnectAsync($Ip, $Port).Wait(1000) } catch { $false } finally { $c.Dispose() }
}

while ($true) {
    $row = [ordered]@{ Time = Get-Date -Format s }
    $row.DNS = [bool](Resolve-DnsName $Domain -DnsOnly -ErrorAction SilentlyContinue)
    $line = nltest /dsgetdc:$Domain /force 2>$null | Select-String 'DC: \\\\'
    $row.DC = if ($line) { ($line.Line -split '\\\\')[-1] } else { 'NONE' }
    foreach ($n in $DCs.Keys) {
        $row["$n-LDAP"]     = Test-Port $DCs[$n] 389
        $row["$n-Kerberos"] = Test-Port $DCs[$n] 88
    }
    [pscustomobject]$row | Export-Csv $Csv -Append -NoTypeInformation
    Start-Sleep 5
}

• nltest /dsgetdc ... /force asks which DC the computer would pick right now, skipping the cache. That’s exactly what you want to see change.
• Ports are tested with a TcpClient and a one-second timeout. Test-NetConnection takes too long to give up when a port doesn’t answer and ruins the measurement.
• The IPs are hard-coded on purpose: if DNS fails, you still want to keep measuring.

Start it a few minutes before the cut and stop it (Ctrl+C) once everything is back. To get the timings:

$f = Import-Csv "$env:USERPROFILE\Desktop\dc-test.csv"
$down = $f | Where-Object { $_.'dc01-LDAP' -eq 'False' }
"dc01 without LDAP from $($down[0].Time) to $($down[-1].Time)"
New-TimeSpan ([datetime]$down[0].Time) ([datetime]$down[-1].Time)
$f | Where-Object DC -eq 'NONE' | Measure-Object   # rows with no DC at all

If the last line gives 0, the client was never left without a DC, which is what you’re after.

5. What to test while it’s down

With the loop running, standing at another computer:

• Sign in with a user who has never signed in on that computer. That’s the real test. With a regular user, Windows falls back on cached credentials and lets them in even with no DC, so it proves nothing.
• A new file share. The ones already open keep working with the user’s existing Kerberos ticket (10 hours by default).
• gpupdate /force and gpresult /r, to see policies being read from the remaining DC.
• Changing a password, especially if the failed DC was the PDC emulator.
• Resolving internal names with Resolve-DnsName.

Write down the time of each test and what happens. You’ll match it against the CSV later.

6. Measure the way back too

When you reconnect (or power on) the DC, the test isn’t over. I’d check:

repadmin /showrepl dc01          # latest replications should be successful
repadmin /syncall dc01 /AdeP     # force replication if you're in a hurry
dcdiag /s:dc01 /q

And in Event Viewer, the Directory Service log on the returning DC: replication warnings in the first few minutes are normal, but they shouldn’t keep showing up after a while. The time I’d record is from the cut until repadmin shows no errors, not until the VM answers ping.

FSMO roles stay put

If the failed DC held roles, in a test you don’t touch them. If you know an outage will be long, you transfer them beforehand, with both DCs up:

Move-ADDirectoryServerOperationMasterRole -Identity dc02 -OperationMasterRole PDCEmulator

Seizing them (-Force with the owner down) is for when a DC will never come back. That DC can’t simply be powered on again afterwards: it has to be cleaned out of the directory and reinstalled.

7. Write it down

A one-page report is enough:

Field	Example
Date and scenario	20/10, dc01 isolated from the network
Outage seen by the client	0 s with no DC; 12 s to switch to dc02
dc01 coming back	1 min 40 s until replicating with no errors
What failed	The front desk printer had a static DNS pointing only to dc01
Actions	Fix DNS on static-IP devices; repeat in 6 months

The most useful row is “what failed”. Something always turns up: a device with a static IP and a single DNS server, an application with one DC’s name hard-coded, a service authenticating against an IP. That’s what the test is for.

What not to do

• Restore a DC from a snapshot to “undo” the test. It can cause a USN rollback and leave replication broken without warning. Windows Server 2012 and later protect themselves with VM-GenerationID (Proxmox supports it), but don’t make it your plan.
• Run it on a Monday at nine. Give notice, pick a quiet time and have someone at a workstation.
• Test only with your own user. Cached credentials will tell you everything’s fine.
• Seize FSMO roles during a test.
• Accept a time without measuring it. “It came back quickly” isn’t data.

Summary

1. Write down what you want to know and a time target first.
2. Check the domain’s health and, above all, the DNS servers clients receive.
3. Start by isolating the DC from the network: it’s instantly reversible.
4. Measure from a workstation with a loop and a CSV, not by watching the server console.
5. Test with a user who has never signed in on that computer, time the way back until replication is clean and write down whatever fails.

Two domain controllers are a promise. The test is what turns it into data, and it almost always uncovers something nobody knew.