Month: November 2019

Home / Month: November 2019

Recently I came across an interesting application consistent backup issue of Hyper-V virtual machines in CommVault. The backup jobs would partly run or complete with one or more errors. According to the error log shown in CommVault, I would start by checking the free space on the Hyper-V host and within the virtual machines. However, this was not the case here as both machines had enough free space. Hence I started digging further into CommVault snapshots.

Error Code: [91:155]
Description: The system or provider has insufficient storage space. If possible delete any old or unnecessary persistent shadow copies and try again.
Source: <server name>, Process: vsbkp

Diagnosis

Checking CommVault logs under %Programfiles%\CommVault\Simpana\Log Files couldn’t shed light on the root cause. The next thing I had in mind was to check for any old checkpoints in Hyper-V Manager. An easy way to collect all the available checkpoints on a Hyper-V server is by using the following Powershell command:

Get-VMSnapshot -VMName *

Or even easier, to collect all checkpoint available on all cluster nodes you could use the following command.

Get-VM -ComputerName (Get-ClusterNode -Cluster %clustername%) | Get-VMSnapshot
Powershell command to get available snapshots on all cluster nodes
Powershell command to get available snapshots on all cluster nodes

From my experience with backups, I can say that CommVault sometimes can struggle taking backups of virtual machines that have checkpoints available.  You could remove unused checkpoints, however, it might happen in very rare situations.

Digging further into the Windows Logs of the Hyper-v host (Microsoft-Windows-Hyper-V-VMMS/Admin) I noticed a ‘sneaky and innocent’ error referring to a .avhdx file of a snapshot that does no longer exist.

The absolute path ‘C:\ClusterStorage\<CSVname>\<VMName><DiskGUID>.avhdx’ is valid for the “Hard Disk Image pool, but references a file that does not exist.

Event log showing the missing reference of the file
Event log showing the missing reference of the file

The missing file could be seen on the Hyper-V Manager, the virtual machine was still referring to inexisting CommVault snapshots in Hyper-V.

Hyper-V VM referring to a missing file
Hyper-V VM referring to a missing file

Following event ID’s from Windows Event Viewer could be interesting to look for: 10172, 32902

Solution

Restarting Hyper-V Virtual Machine Management (vvms) service will solve the problem. I did it using graphical interface, and a scary warning appeared asking me if I wanted to ‘Turn Off’ the service. Finally, it just halts the management service and does not affect the running VM’s.

Stop VMM service warning
Stop VMM service warning

The same operation can be accomplished using the following Powershell commands:

stop-service -name vmms
start-service -name vmms

Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailing list to get the latest.

Brocade ISL Trunk configuration

07/11/2019 | SAN | 1 Comment

One of the most interesting parts of administrating FC switches is implementing ISL’s (Inter-Switch Links) between 2 datacenters. In this article, we will cover the steps that need to be taken in order to create a fabric. We assume that the physical link (cabling) has already been set up and that the switch is already configured.

On the demonstration below I’m using Brocade SAN switches G62-series running Fabric OS version 8.0.2e.

  1. We start off by disabling the switch.
    FOS_STORCOM1:admin> switchdisable
  2. Next, we need to configure the port speed of the ports which will be inter-connected.
    FOS_STORCOM1:admin> portcfgspeed -i <port number> -f <port speed>

  3. Brocade SAN switches can be easily configured using the configure command. Once entered  it will lead you through some important configuration steps.
  4. Next, we’ll need to calculate the ISL distance. A rule of thumb will be to multiply the real physical distance with 1.5 to get the ISL distance.
    real_distance_km x 1.5 = ISL_logical_distance

    In my case, I have two switches with a physical distance of 146 km. I will use 220 km as ISL distance.

  5. To activate the port in LS (Long Distance Dynamic) mode enter the following command
    FOS_STORCOM1:admin> portcfglongdistance <port number> LS 1

    A vc_link_init value of 1 uses the ARB fill word (default). A value of 0 uses IDLE. The required value might depend on the link being used. The commands must be repeated for each ISL port.

  6. Optionally, you can enable the QOS on the ISL ports by using the following command:
    FOS_STORCOM1:admin> portcfgqos --enable <port number>
  7. To check and confirm the port parameters use the following command:
    FOS_STORCOM1:admin> portshow <port number>
  8. At this step the port is ready. Enable the switch and the ports using the following commands
    FOS_STORCOM1:admin> switchenable
    FOS_STORCOM1:admin> portcfgpersistantenable <port number>
  9. Log on to the second switch and perform the same operations from Step 1 to Step 7
  10. Your SAN fabric should be ready now. Verify it using the following commands:
    FOS_STORCOM1:admin> fabricshow
    FOS_STORCOM1:admin> trunkshow

The article Essential troubleshooting command lines every Storage Administrator should know offers interesting stuff related to the switch administration.

A complete command line list and other switch administration can be found on the Brocade Fabric OS Administration Guide.

Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailing list to get the latest.