Error Code: [91:155]
Description: The system or provider has insufficient storage space. If possible delete any old or unnecessary persistent shadow copies and try again.
Source: <server name>, Process: vsbkp
Recently I came accross an interesting (application consistent) backup issue of Hyper-V virtual machines in CommVault. The backup jobs would partly run or complete with one or more errors. Following the error log shown in CommVault I would start by checking the free space on the Hyper-V host and within the virtual machines. However this was not the case here as both had enough free space.
Checking CommVault logs under %Programfiles%\CommVault\Simpana\Log Files couldn’t shed light on the root cause. The next thing I had in mind was to check for any old checkpoints in Hyper-V Manager. An easy way to collect all the available checkpoints on a Hyper-V server is by using the following Powershell command:
Get-VMSnapshot -VMName *
Or even easier, to collect all checkpoint available on the all cluster nodes you could use the following command.
Get-VM -ComputerName (Get-ClusterNode -Cluster %clustername%) | Get-VMSnapshot
From my experience with backups, I can say that CommVault sometimes can struggle taking backups of virtual machines that have checkpoints available. You could remove unused check points, however it might happen in very rare situations.
Digging further into the Windows Logs of the Hyper-v host (Microsoft-Windows-Hyper-V-VMMS/Admin) I noticed a ‘sneaky and innocent’ error referring to an .avhdx file of a snapshot that does no longer exist.
The absolute path ‘C:\ClusterStorage\<CSVname>\<VMName><DiskGUID>.avhdx’ is valid for the “Hard Disk Image pool, but references a file that does not exist.
The missing file could be clearly seen on the Hyper-V Manager, the virtual machine was still referring to an inexisting snapshot.
(Apologize for partly obfuscating the images but they contain sensitive and remarkable names).
Following event ID’s from Windows Event Viewer could be interesting to look for: 10172, 32902
Restarting Hyper-V Virtual Machine Management (vvms) service will solve the problem. I did it using graphical interface, and a scary warning appeared asking me if I wanted to ‘Turn Off’ the service. Finally it just halts the management service and does not affect the runing VM’s.
The same operation can be accomplished using the following Powershell commands:
stop-service -name vmms start-service -name vmms
Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailinglist to get the latest.