TFS Database Backup Failed : There is an error in XML document - but which Doc? - tfs

Question

Our Nightly TFS 2012 backup has just started to fail. It also fails when run directly through TFS Express Administration Console.
Which file is the following error actually referring to? If I can find it then I should be able to fix the "Root element is missing" error :)
[13/08/2014 23:00:00] [Info] Full database backup job
[13/08/2014 23:00:00] [Info] Getting backup lock
[13/08/2014 23:00:05] [Error]
Exception Message: There is an error in XML document (0, 0). (type InvalidOperationException)
Exception Stack Trace: at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at Microsoft.TeamFoundation.Admin.BackupSets.Load(String folder)
at Microsoft.TeamFoundation.Admin.Jobs.FullDatabaseBackupJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime jobQueueTime, String& resultMessage)
Inner Exception Details:
Exception Message: Root element is missing. (type XmlException)
Exception Stack Trace: at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo(String res)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlReader.MoveToContent()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read7_BackupSets()
[13/08/2014 23:00:05] [Info] Full Backups Failed
Thanks.

Answer 1

Dylan answered my original question as to where to find the unspecified xml file that was in error, but in case it helps anyone else...
The Backupsets.xml file was empty. Why this is I do not know...
Attempting to configure backups through TFS Express Administration Console also failed with the same error, so I
Deleted the Backupsets.xml file altogether
Reconfigured Backups using the wizard - Now that it didn't find the xml file at all it created a new one.
Ran a full backup - which was sucessful. Hopefully the scheduled backups will now also work from now on.
NB The newly created Backupsets.xml file (Before the first full backup) :
<?xml version="1.0"?>
<BackupSets xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Version>1</Version>
<BackupSets />
</BackupSets>

Answer 2

Look in the folder where your backups are configured to be placed. there will be an XML file there, can't remember the name, but maybe something like BackupSets.xml

Answer 3

FIX (user workaround):
Rename the file BackupSets.xml in the backup destination folder
Re-run Full Backup*
In TFS Admin Console select Scheduled Backups, then click Take Full Backup Now. Or use command line, PowerShell script, API call as desired.
CAUSE: backupsets.xml in backup destination does not contain valid XML.
Why does this cause failure? Backup wizard opens backupsettings.xml then calls XML deserializer function System.Xml.Serialization.XmlSerializer.Deserialize preparing to add new entry. Invalid XML content including empty/zero byte or text-only content will cause deserialize exception.
PRODUCTS IMPACTED: Repro confirmed in TFS2010 and on 2017-11-25 I had repro with TFS2015 SP3 :-O
Fix is fairly straightforward... once you understand what is going on. -Zephan
MICROSOFT CODE BUGFIX/feature improvement request:
BACKUP Wizard exception handling for backupsets.xml deserialize or parsing exceptions.
If XML deserialization error then close backupsets.xml, rename it to backupsets-YYMMDD-hhmm-corrupt-backup.xml, then jump to backupsets.xml file not found functionality.
SEVERITY: HIGH (data loss)
This is a long-standing problem that can lead to major data loss. I've personally seen over 1 month of data loss due to this issue silently blocking backups and making all earlier restore sets unusable (since parsing BackupSets.xml is VERY finicky I couldn't even hack to restore last successful backup.)