SQL VSS Backup: Unexpected error calling routine XML document is too long

Recently I was troubleshooting an issue with SQL Hardware snapshotting using CommVault software in combination with HPE Primera snap technology. Basically, the idea is to have SQL Hardware snapshots orchestrated by CommVault. The setup is as follows:

1. CommVault subclient containing +200 databases. 2 large DB’s having +- 300 datafiles.
2. SQL physical cluster consisting of 4 servers.
3. HPE Primera storage, SAN-connected.

Issue

When trying to snapshot these 2 large databases, CommVault jobs would go pending and consequently fail after a while.

In Windows Event Viewer, the following is encountered:

1/ A event (EventID 3197) is logged that the the I/O is frozen for the DB:

I/O is frozen on database DBCUST_PROD. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.

2/Followed by the event (EventID 3198) where the I/O is resumed on the same DB.

I/O was resumed on database DBCUST_PROD. No user action is required.

3/ Right after this, 2 VSS errors are encountered as seen below (Event ID 8193).

EventID 8193: Volume Shadow Copy Service error: Unexpected error calling routine XML document is too long.  hr = 0x80070018, The program issued a command but the command length is incorrect.
EventID 8193: Volume Shadow Copy Service error: Unexpected error calling routine IXMLDOMNode::replaceChild.  hr = 0x80070057, The parameter is incorrect.

4/ SQL VSS Writer goes in Failed state with the error:

SqlServerWriter’ Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a} Writer Instance Id: {351a63e0-bfce-4d71-8b9b-c0b09953f3a6} State: [11] Failed Last error: Retryable error

Solution / Workaround

The problem occurs when that list generates a metadata file that’s larger than the size limitation. Default maxsize of XML document used by VSS is 16MB.

To work around this issue, create a new registry key to override the default value of 16 MB for VSS XML document size (you may want to back up the registry before). 

Valid values would be 32/64/128.

RegKey SYSTEM\CurrentControlSet\Services\VSS\Settings “MaxXmlDocumentSizeMB”   (type “DWORD”) 

To create the new registry key, follow these steps:

1. Click Start, click Run, type regedit, and then press ENTER.
2. Locate and then right-click the following registry subkey:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Settings
3. Point to New, and then click DWORD Value.
4. Type MaxXmlDocumentSizeMB, and then press ENTER.
5. Double-click MaxXmlDocumentSizeMB, type 32 in the Value data box, and then click OK.
6. Exit Registry Editor.