These scenarios almost always involve a virtual environment, so if you have virtual domain controllers, you need to make sure that backups are performed on physical domain controllers (you can never go wrong when performing backups on FSMOs to boot) and/or make sure that the Windows Backup tool is used to back up the AD database (the SUPPORTED backup method is key). Event Type: Error Event Source: NTDS General Event Category: Service Control Event ID: 2103 Date: 10/3/2005 Time: 4:26:51 PM User: USN2B25VB$ Computer: 2B9A Description: The Active Directory database was restored using an unsupported restore method. Active Directory cannot log on to users as long as this state persists. Therefore, the Net Logon service was stopped. User Action For more information, see the previous event logs. For more information, see support.microsoft.com Help and Support Center. Each domain controller has a server object in the site container stored in the configuration partition. This object has a child object named NTDS Settings with a globally unique identifier (GUID) attribute that is replicated as part of the replication metadata and used by KCC to create the replication topology. Each domain controller has its own copy of the Active Directory database, which is stored in the ntds.dit file, and this unique DB instance on a domain controller is identified by its own GUID identifier called a call ID. The call ID is created when the domain controller is promoted and changes only when the AD database is restored using a supported method or an application partition is added or removed. This is because if an AD database is restored to an earlier point, the USN will also be restored at that time. This means that any changes from the restored USN value to the pre-restored original USN value will be ignored by other domain controllers that retrieve replication from the restored domain controller (because they follow the other USN domain names they replicate with and only retrieve updates when the USN of the destination domain controller exceeds the last saved update value that has the domain controller – more on that later). To avoid this situation, the AD database on the domain controller generates a new call ID and stores the old call ID in an attribute of the server`s NTDS settings object named retiredReplDSASignatures.

In this way, domain controllers treat a new call ID as a new database and ensure that it receives updates from it in the future. You can view the call ID and server GUID of a domain controller by running repadmin /showrepl. Note: If the GUID of the source domain controller and the GUID of the source domain controller are the same, this is an original write operation. If they are different, the change operation is a replicated write operation and the replication system uses the update vector. Domain controllers use all of this information, including the update vector and the high-water mark, as part of “propagation damping” to ensure that replication does not get stuck in a loop. I recommend doing solution step #1. Because typical scenarios that result in a USN restore involve mapping or performing an unsupported recovery, a USN recovery in the environment typically indicates process issues. Often, this is linked to inappropriate collection procedures. Because updates are replicated when the USN on the source domain controller is greater than the destination domain controller for the source domain controller (based on UTDV and HWM), a USN restore scenario on one domain controller prevents AD updates from replicating on that domain controller to another. When a USN runs backwards, it is usually due to a supported restore from a backup. When this process occurs, the call ID changes. Because all replication partners track replication based on domain controller GUID, call ID, and USNs, a supported recovery method keeps the previous call ID as “retired” and effectively ignores it.

The new database call ID and associated USN are used to make AD changes from the domain controller. unless the USN is reset without changing the call ID. This means that AD updates can be performed on this domain controller when a domain controller is in a USN restore state, without ANY of the changes replicated to its replication partners. This is bad news. If a USN restore is identified on Windows 2003 SP1 or later (or earlier with the KB patch 875495), the following actions are performed on that domain controller: Each replicated change contains the following information: Event Type: Error Event Source: NTDS Replication Event Category: Replication Event ID: 2095 Date: 3/10/2005 Time: 16:26:51 User: USN2B25VB$ Computer: 2B9A Description: During an Active Directory replication request, the The domain controller (DC) identifies a remote domain controller that has received replication data from the local domain controller by using previously verified USN trace numbers. Because the remote domain controller assumes that it has a newer Active Directory database than the local domain controller, the remote domain controller does not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners from that local domain controller. If this scenario is not resolved immediately, it will cause inconsistencies in the Active Directory databases of this source domain controller and one or more direct and transitive replication partners. Specifically, the consistency of users, computers, and trusts, their passwords, security groups, security group memberships, and other Active Directory configuration data can vary, affecting the ability to log on, search for objects of interest, and perform other critical operations. To determine whether this misconfiguration occurs, use support.microsoft.com to query this event ID or contact Microsoft Product Support Services. The most likely cause of this situation is the failure to recover Active Directory from the local domain controller. User actions: If this situation occurred due to incorrect or accidental recovery, force the domain controller to be demoted. Remote domain controller: b55ee67f-ed73-4970-b2d4-7dc6f571439f Partition: CN=Configuration,DC=USN, DC=USN loc reported by Remote DC: 24707 USN reported by Local DC: 20485 For more information, see support.microsoft.com Help and Support Center.

Event Type: Warning Event Source: NTDS General Event Category: Replication Event ID: 1113 Date: 10/3/2005 Time: 4:26:51 PM User: USN2B25VB$ Computer: 2B9A Description: Inbound replication has been disabled by the user. For more information, see support.microsoft.com Help and Support Center. Most domain controller information is visible when the Get-ADDomainController PowerShell commandlet is run: in this case, we recommend that you demote the domain controller by running dcpromo /forceremoval and performing metadata cleanup on that domain controller. If the demoted domain controller hosted one of the FSMO roles, it must be assigned to another domain controller. The update sequence number (USN) is a performance counter for Active Directory DB instances that is incremented whenever a single change is committed to the AD database on a domain controller. The USN is unique to each DC and has no correlation with one USN on another DC (and that doesn`t matter, as you`ll see later in this article). Active Directory replication tracks the USN of each domain controller and uses this information to determine when replication is required. Active Directory has two basic types of writes to the AD database: a replicated write (where the change is made to a different domain controller) and an original write (where the change is made to the local domain controller). AD replicates and uses information about changes made to which domain controllers and then replicated. Note that in this case, any changes made on DC02 will NOT be replicated to another domain controller. This means that changes to AD on DC02 exist only on DC02, while other domain controllers may have different versions of the same objects, including passwords and group membership. You see, bad news.

In the next replication cycle, DC01 and DC03 will send DC02 the latest USN update they have for DC02 (USN #31,131). DC02 checks its local USN and finds that the local USN (USN #29,000) is smaller than the USN sent (USN #31.131), so no updates are replicated to DC01 or DC03. When changes are made to Active Directory on DC02, they are not replicated to other controllers ADSecurity.org until the USN for DC02 exceeds the USN for other domain controllers for DC02. This means that 2,131 changes are made to DC02 before other domain controllers receive a single update – and only changes starting with USN 31,132. PS C:> import-module activedirectory; get-addomaincontroller -identity “ADSecurityORGDC01.ADSecurity.org” Microsoft has identified several scenarios that can cause USN restore on a domain controller (for example, unsupported configurations). All contain the same common theme, which is to take and run a previous state of a domain controller (backup or copy). The following events are logged on the domain controller with USN recovery: If you encountered Event ID #2095, understand how a USN restore can negatively impact AD consistency. USN recovery detection is extremely difficult because domain controllers running Windows 2000 or Windows 2003 RTM were not looking for repeated USNs for the same call ID.