vRealize Automation Virtual Machine is Missing after being Moved or Recovered

Have you ever seen a deployed virtual machine in the vRA portal have its status changed to ‘Missing‘? This can happen for two reasons; The first, obvious reason, is that the source virtual machine has been deleted. Not much you can do about that except re-provision it or restore the vm from backup. The second reason, is because the reference information that vRA holds about the virtual machine is no longer valid (perhaps from an earlier restore).

In this post, I will discuss and provide a solution that will address the latter for a vSphere hosted virtual machine.

When in this state, you will not be able to perform day 2 actions on this virtual machine (apart from destroy).

The following scenarios can explain why vRA has incorrect reference data about the virtual machine:

• The virtual machine has been recovered from backup and a new instance uuid has been created;
• The virtual machine has moved to another vCenter Server, maybe during a DR event. I know that Zerto will create a new virtual machine at the DR site which will have a different instance uuid (SRM’s shadow VMs are most likely similar but I can’t confirm);
• The virtual machine was cloned and the original has been deleted;
• The instance uuid for the virtual machine has been changed, either manually or by some other process.

All code that I have provided or talked about in this post can be downloaded as a vRO package for your consumption here.

How vRA References the vCenter Virtual Machine

The virtual machine entity in vRA, is simply a record in the database, with information about that virtual machine (such as description, vCPUs, memory, etc.) and relationships it has with other objects, such as a reservation. Importantly, the record also contains some fields, which provide references and/or unique identifiers to the virtual machine on the platform it was deployed on. In the case of vSphere, the virtual machine id (moref) and instance uuid is stored.

When a virtual machine has been recovered, or any process which causes these values to differ from what vRA has, will result in the virtual machine being displayed as ‘Missing‘.

There is also an internal custom property called “VirtualMachine.Admin.UUID” that also stores the instance uuid and would need to be updated.

Updating vRA Virtual Machine References

I have created a workflow called ‘Pair vRA VM with vCenter VM‘ that will allow you to select the vRA virtual machine and then select the vCenter virtual machine that it should be paired with. from the vRO inventory.

The solution includes running 2 workflows. The first workflow, ‘Update vRA VM Pointers for VcVm‘, will allow you to select the vRA virtual machine and the vCenter virtual machine that it should be paired with. Once this has completed, the status of the virtual machine in vRA should display the correct status and you will be able to manage it as normal.

When you run the workflow you will be able to select the virtual machines that you want to pair from the vRO inventory.

In some cases you might see a duplicated virtual machine in the vRA inventory. This happens when the data collection process creates a new entity for the virtual machine that it found on vCenter. The reason for this is that the new virtual machine has a different uuid.

When selecting the vRA virtual machine, make sure to select the one that has the ‘isMissing‘ property set to ‘true‘, as this is the one you see in the vRA portal.

As you can see in the screenshot above, I have 2 entities with the same VM name, but only one of these has the ‘isMissing‘ property set to ‘true‘.

When the workflow runs, it is going to process the following high level steps:

• Updates the VMUniqueID, ExternalReferenceId and sets the OwnerExists property to true.
• Updates the ‘VirtualMachine.Admin.UUID’ custom property.
• Checks for duplicate/invalid vRA VM entities (see above) and removes these.
• Runs a data collection if at least one invalid entity has been found (as these can still cause the VM to be displayed as missing).

Below is the log output from the workflow:

[] [I] [Action: getVirtualMachineEntityFromVcacVirtualMachine]: Getting virtual machine entity from vcac virtual machine.
[] [I] [Action: getVirtualMachineEntityFromVcacVirtualMachine]: Found virtual machine entity: SG-BG-Deliv0006
[] [I] [Action: getHostId] Getting hostId from vCAC Entity.
[] [I] [Action: getHostId] Found hostId: 4cad7d75-012e-4304-ac54-c587d2e0ee97
[] [I] [Action: findVcacHostById] Finding vCAC Host with id '4cad7d75-012e-4304-ac54-c587d2e0ee97'
[] [I] [Action: findVcacHostById] Found vCAC host: IaaS host for Default [https://web.vra.sgroot.local/]
[] [I] [Action: getMorefFromVcVirtualMachine] Getting the managed object reference (MORef) id for virtual machine: SG-BG-Deliv0006
[] [I] [Action: getMorefFromVcVirtualMachine] Found MORef id: vm-988
[] [I] [Action: getInstanceUuidFromVcVirtualMachine] Getting the instance uuid for virtual machine: SG-BG-Deliv0006
[] [I] [Action: getInstanceUuidFromVcVirtualMachine] Found instance uuid: 501a1c4b-b98e-8f41-4324-3298f2303bd5
[] [I] [Action: updatePropertyValueForVcacEntity] Attempting to update 'VMUniqueID' on vcac entity 'SG-BG-Deliv0006' to '501a1c4b-b98e-8f41-4324-3298f2303bd5'
[] [I] [Action: updatePropertyValueForVcacEntity] Updating 'VMUniqueID' on vcac entity: SG-BG-Deliv0006
[] [I] [Action: updatePropertyValueForVcacEntity] Successfully updated 'VMUniqueID' on vcac entity 'SG-BG-Deliv0006' to '501a1c4b-b98e-8f41-4324-3298f2303bd5'
[] [I] [Action: updatePropertyValueForVcacEntity] Attempting to update 'ExternalReferenceId' on vcac entity 'SG-BG-Deliv0006' to 'vm-988'
[] [I] [Action: updatePropertyValueForVcacEntity] Updating 'ExternalReferenceId' on vcac entity: SG-BG-Deliv0006
[] [I] [Action: updatePropertyValueForVcacEntity] Successfully updated 'ExternalReferenceId' on vcac entity 'SG-BG-Deliv0006' to 'vm-988'
[] [I] [Action: updatePropertyValueForVcacEntity] Attempting to update 'OwnerExists' on vcac entity 'SG-BG-Deliv0006' to 'true'
[] [I] [Action: updatePropertyValueForVcacEntity] Property 'OwnerExists' already set to 'true', no update required.
[] [I] [Action: updateVmAdminUuidCustomProperty] Updating custom property 'Virtual Machine UUID' with key 'VirtualMachine.Admin.UUID'
[] [I] [Action: addOrUpdateCustomProperty] Adding or updating custom property 'VirtualMachine.Admin.UUID' with value '501a1c4b-b98e-8f41-4324-3298f2303bd5'
[] [I] [Action: addOrUpdateCustomProperty] An existing Custom Property was found and will be updated.
[] [I] [Action: addOrUpdateCustomProperty] Successfully updated 'VirtualMachine.Admin.UUID' to '501a1c4b-b98e-8f41-4324-3298f2303bd5'
[] [I] [Action: updateVmAdminUuidCustomProperty] Successfully updated custom property for: Virtual Machine UUID
[] [I] [Action: getVirtualMachineEntityFromVcacVirtualMachine]: Getting virtual machine entity from vcac virtual machine.
[] [I] [Action: getVirtualMachineEntityFromVcacVirtualMachine]: Found virtual machine entity: SG-BG-Deliv0006
[] [I] [Action: getVirtualMachineNameForVirtualMachineEntity] Retrieving property 'VirtualMachineName' from vm entity.
[] [I] [Action: getVirtualMachineNameForVirtualMachineEntity] Found VirtualMachineName: SG-BG-Deliv0006
[] [I] [Action: getVirtualMachineIDForVirtualMachineEntity] Retrieving property 'VirtualMachineID' from vm entity.
[] [I] [Action: getVirtualMachineIDForVirtualMachineEntity] Found VirtualMachineID: 62751722-b60b-40be-8d6d-e9c6c71a8c56
[] [I] [Action: getHostId] Getting hostId from vCAC Entity.
[] [I] [Action: getHostId] Found hostId: 4cad7d75-012e-4304-ac54-c587d2e0ee97
[] [I] [Action: findVcacHostById] Finding vCAC Host with id '4cad7d75-012e-4304-ac54-c587d2e0ee97'
[] [I] [Action: findVcacHostById] Found vCAC host: IaaS host for Default [https://web.vra.sgroot.local/]
[] [I] [Action: findEntitiesForVirtualMachineByName] Finding entities for virtual machine with the name: SG-BG-Deliv0006
[] [I] [Action: getVcacEntitiesByCustomFilter] Retrieving vCAC Entities for set: VirtualMachines
[] [I] [Action: findEntitiesForVirtualMachineByName] Found 2 entities.
[] [I] [Action: findInvalidEntityIdsFromVmEntities] Getting a list of invalid entities that do not match reference id: 62751722-b60b-40be-8d6d-e9c6c71a8c56
[] [I] [Action: findInvalidEntityIdsFromVmEntities] Found 1 invalid entities.
[] [I] [Action: findInvalidEntityIdsFromVmEntities] The following invalid entity references were found.
c98740b0-3823-409e-952e-fa8a2bca2a57
[] [I] [Action: getVcacHost] Locating vCAC IaaS Host...
[] [I] [Action: getVcacHost] Found vCAC IaaS host: IaaS host for Default [https://web.vra.sgroot.local/]
[] [I] [Action: findVirtualMachineEntityByVirtualMachineId] finding virtual machine entity with entity id: c98740b0-3823-409e-952e-fa8a2bca2a57
[] [I] [Action: findVirtualMachineEntityByVirtualMachineId] Found virtual machine entity: SG-BG-Deliv0006 with entity id: c98740b0-3823-409e-952e-fa8a2bca2a57
[] [I] [Action: deletevCACEntity] Deleting vCAC entity with id: (guid'c98740b0-3823-409e-952e-fa8a2bca2a57')
[] [I] [Action: deletevCACEntity] Successfully deleted vCAC entity with id: (guid'c98740b0-3823-409e-952e-fa8a2bca2a57')
[] [I] [Action: getComputeResourceNameForVirtualMachineEntity] Getting compute resource name for virtual machine entity: SG-BG-Deliv0006
[] [I] [Action: getComputeResourceNameForVirtualMachineEntity] Found compute resource 'SITEA-CLS-CLOUD-01' with entity id '355c617e-f9e0-4726-abcf-81f5748c50db'
[] [I] [Workflow: Pair vRA VM with vCenter VM] Trying to create lock: Attempt 1 of 180
[] [I] [Action: createLock] Creating lock on 'SITEA-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: createLock] Lock created for 'inventory'
[] [I] [Action: getComputeResourceEntityByName] Retrieving vCAC Compute Resource entity with name: SITEA-CLS-CLOUD-01
[] [I] [Action: getVcacEntitiesByCustomFilter] Retrieving vCAC Entities for set: Hosts
[] [I] [Action: getComputeResourceEntityByName] Found vCAC Compute Resource entity: SITEA-CLS-CLOUD-01 with ID: 355c617e-f9e0-4726-abcf-81f5748c50db
[] [I] [Action: getAgentStatusForComputeResourceEntity] Getting Agent status for compute resource 'SITEA-CLS-CLOUD-01'
[] [I] [Action: getAgentStatusForComputeResourceEntity] Agent Status: true
[] [I] [Action: getHostIDFromComputeResourceEntity] Retrieving Host ID for compute resource entity: SITEA-CLS-CLOUD-01
[] [I] [Action: getHostIDFromComputeResourceEntity] Found Host ID for compute resource entity:355c617e-f9e0-4726-abcf-81f5748c50db
[] [I] [Action: getDataCollectionEnabledStatus] Checking if 'inventory' data collection is enabled.
[] [I] [Action: getDataCollectionEnabledStatus] Data collection 'inventory' enabled: true
[] [I] [Action: getDataCollectionRunningStatus] inventory data collection task running: false
[] [I] [Action: getComputeResourceEntityById] Retrieving vCAC Compute Resource entity with id: 355c617e-f9e0-4726-abcf-81f5748c50db
[] [I] [Action: getComputeResourceEntityById] Found vCAC Compute Resource 'SITEA-CLS-CLOUD-01' with ID '355c617e-f9e0-4726-abcf-81f5748c50db'
[] [I] [Action: startDataCollectionOnComputeResource] Starting 'inventory' data collection on compute resource 'SITEA-CLS-CLOUD-01'
[] [I] [Action: startDataCollectionOnComputeResource] The 'inventory' data collection process on compute resource 'SITEA-CLS-CLOUD-01' has started.
[] [I] [Action: getDataCollectionRunningStatus] inventory data collection task running: true
[] [I] [Action: getDataCollectionRunningStatus] inventory data collection task running: true
[] [I] [Action: getDataCollectionRunningStatus] inventory data collection task running: false
[] [I] [Action: getDataCollectionResultStatus] Retrieving the 'inventory' data collection result status.
[] [I] [Action: getDataCollectionResultStatus] Data collection succeeded at: Mon Mar 11 2019 10:30:23 GMT-0000 (UTC)
[] [I] [Action: removeLock] Removing lock on 'SITEA-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: removeLock] Lock removed for 'SITEA-CLS-CLOUD-01'
[] [I] [Workflow: Pair vRA VM with vCenter VM] Successfully paired vRA and vCenter Virtual Machines.

When I refresh the browser I can now see the correct status for the virtual machine and perform day 2 actions as normal.

We can also see that there is now only one virtual machine in the vRO inventory (the invalid entry was removed):

I hope this has been helpful. If you discover bugs with any of my code, require some help or simply need an ad-hoc solution, then please drop me a message via the Drift app.