Recently I came across an interesting application consistent backup issue of Hyper-V virtual machines in CommVault. The backup jobs would partly run or complete with one or more errors. According to the error log shown in CommVault, I would start by checking the free space on the Hyper-V host and within the virtual machines. However, this was not the case here as both machines had enough free space. Hence I started digging further into CommVault snapshots.
Error Code: [91:155]
Description: The system or provider has insufficient storage space. If possible delete any old or unnecessary persistent shadow copies and try again.
Source: <server name>, Process: vsbkp
Diagnosis
Checking CommVault logs under %Programfiles%\CommVault\Simpana\Log Files couldn’t shed light on the root cause. The next thing I had in mind was to check for any old checkpoints in Hyper-V Manager. An easy way to collect all the available checkpoints on a Hyper-V server is by using the following Powershell command:
Get-VMSnapshot -VMName *
Or even easier, to collect all checkpoint available on all cluster nodes you could use the following command.
Get-VM -ComputerName (Get-ClusterNode -Cluster %clustername%) | Get-VMSnapshot
From my experience with backups, I can say that CommVault sometimes can struggle taking backups of virtual machines that have checkpoints available. You could remove unused checkpoints, however, it might happen in very rare situations.
Digging further into the Windows Logs of the Hyper-v host (Microsoft-Windows-Hyper-V-VMMS/Admin) I noticed a ‘sneaky and innocent’ error referring to a .avhdx file of a snapshot that does no longer exist.
The absolute path ‘C:\ClusterStorage\<CSVname>\<VMName><DiskGUID>.avhdx’ is valid for the “Hard Disk Image pool, but references a file that does not exist.
The missing file could be seen on the Hyper-V Manager, the virtual machine was still referring to inexisting CommVault snapshots in Hyper-V.
Following event ID’s from Windows Event Viewer could be interesting to look for: 10172, 32902
Solution
Restarting Hyper-V Virtual Machine Management (vvms) service will solve the problem. I did it using graphical interface, and a scary warning appeared asking me if I wanted to ‘Turn Off’ the service. Finally, it just halts the management service and does not affect the running VM’s.
The same operation can be accomplished using the following Powershell commands:
stop-service -name vmms start-service -name vmms
Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailing list to get the latest.
One Response
Oh boy, automatic all caps 😉
Great post, ran into the same issue: no checkpoints, but .avhd(x) files were filling up the clustered volumes. Restarting the VM MgmT service caused the merge process to start!