A major challenge among IT professionals, more specifically, IT professionals who have to wear many hats, is the VM migration process. In an effort to better educate and document our VM migration process, here’s a snippet of a document we created for a recent customer VM migration.
More specifically, this document will cover the procedures used to leverage VMware Cross vCenter Migration (XVM) Utility to migrate VM’s from a production VMware environment to a new Hyper-converged (VxRail in this case) environment.
It includes the download, initialization and configuration of the VXM Utility as well as necessary VMware vCenter network configuration changes for both Universal Printing Company’s Production and VxRail environments to facilitate Virtual Machine migrations.
The XVM utility allows users to easily migrate virtual machines in bulk from a graphical user interface between vCenter servers using the Cross-vCenter vMotion feature.
Reconfiguration of both environments is needed temporary to facilitate the migrations. These changes do not impact Production services and will be “undone” to return both environments to a “default” state when migration functionality is not needed or completed.
This guide can also be used as a basic procedure for any migration that meets the requirements of XVM Utility.
XVM Requirements are as follows:
– vCenter Server 6.0 Update 3 or above (ESXi hosts must also be 6.0u3+
– Java Runtime Environment 1.8-10
– Web Browser
– Please review Cross vCenter vMotion requirements
The methodology being used for this specific Data Migration is as follows:
1) Initialize XVM Utility.
2) Register Source and Target vCenters in the XVM Utility.
3) Load up a ‘migraton host’ in the existing Production VMware environment with VM’s that will be migrated during a single migration session.
4) Modify vMotion network configuration for both the existing Production and VxRail environments in order to create an L3 routable vMotion subnet between the 2 environments.
5) Migrate VM’s from existing Production environment to VxRail.
6) Revert vMotion network configuration changes for both environments back to previous settings
Rinse and repeat the above steps for any N number of migrations sessions needed to move all VM’s from existing Production environment to the VxRail.
Download, Initialize and Register vCenters in the XVM Utility
1) Download XVM Utility to a Windows machine running the appropriate version of Java and also has network access to both the Production vCenter as well as the VxRail vCenter.
As of this document the latest version is 3.1 (filename: xvm-3.1.jar)
3) From a CMD prompt, run the following command to initialize the XVM Utility
C:\<location of downloaded file>\java -jar xvm-3.1.jar
NOTE: for additional, more advanced initialization parameters see the “Instructions” subpage on the Fling site:
4) The default port for the XVM Utility is set to 8443 on initialization. Once the Utility is initialized, open a browser and browse to https://localhost:8443. When the WebUI comes up, click on the ‘Register’ button to register your Source and Target vCenters.
5) On the Register page, enter all the required information for the Source and Target vCenters and click submit (you will need to register the vCenters 1 at a time)
Site Name: Can be any name to identify what vCenter it is.
Hostname: The FQDN or IP address of the vCenter Server.
Username/Password: Username and Password of an account that has Administrator permissions on the vCenter Sever.
Trust Server: If by default the XVM Utility fails to connect, try again with the box selected for ‘Skip certificate verification’
When complete, make sure you see both vCenters listed under ‘Site Information’ and a checkbox in the ‘Connection’ column.
The XVM Utility is now initialized and registered with Source and Target vCenters.
Understanding VMware TCP/IP stacks, how they affect vMotion VMkernels why the temporary changes are being made to Production and VxRail to facilitate migration
For Cross vCenter vMotion to work the vMotion subnet for both the Source and Target VMware environments need to be able to talk to each other. This is possible if both environments use the same subnet or via L3 routing between different subnets.
If the Vmotion subnets are different (such as in this case), additional configuration may be required to make this communication. There are a few different ways to do this, but this only touches on basic concepts and the specific changes for Universal Printing Company.
Universal Printing Company is using VMware’s Default TCP/IP stack in their Production environment. The VMKernels used for various services are on different networks and the Default Gateway for the Default TCP/IP is the gateway of the management subnet. There is no gateway for the vMotion subnet as it is an isolated network that exists only in the UCS environment and not upstream.
You can override the default gateway on a vMotion VMKernel to use the gateway of a vMotion subnet *if* there is a gateway configured for that subnet. However, in many cases/implementations of VMware (such as this one), there may not be a gateway for the vMotion subnet). If there is no gateway for the vMotion subnet, you are left with 2 options:
1) Reconfigure the vMotion subnet and add a gateway. This option is not always the easiest as it requires the VLAN and gateway to be added to all upstream devices all the way back to the core networking to ensure full L3 routing of the vMotion subnet.
2) Pick an existing subnet (VLAN) that is accessible in both the Source and Target VMware infrastructures, has 2 available IP addresses and a gateway.
3) Configure a temporary VMKernel for Vmotion on that subnet and use the override gateway option to set the correct gateway for the subnet.
NOTE: The VxRail environment is using a non-default TCP/IP stack specifically for vMotion. This allows for a gateway to be configured directly on the VMKernal, rather than having to override a Default Gateway. More detail and the procedure for the necessary changes on the VxRail will be covered later in the document.
Reconfiguration of Production Environment to enable L3 routing of vMotion Subnet
Because this is a temporary change just to facilitate Cross vCenter vMotion, we only need to apply changes to 1 ESXi host in the Production Cluster. This will be referred to as the “migration host’ and for Universal Printing Company, this will be host 1.
1) Make sure that any VM’s you want to migrate are moved to the migration host and DRS is disabled or set to Manual on the Cluster so that vSphere does not move any of VM’s for migration off the swing host.
2) Log into the vCenter WebUI, select host 1 and go to Configure -> VMKernel Adapters
3) For both vmk1 and vmk2 (vMotion01 and vMotion02) select the adapter, click on ‘Edit’ and uncheck the box for vMotion. This will ensure that host one will not use either vmk1 or vmk2 for vMotion as this is an isolated subnet with no gateway.
NOTE: At this point, the isolated VLAN for vMotion on Host 1 in the Production Cluster is disabled. This means that VM’s will not be able to migrate to or from Host 1 in the Production Cluster.
4. Create a VMKernal port on a subnet that lives in both environments.
– If not using an existing Portgroup for the VMK, you need to create a new one in VMware Networking and tag it with the appropriate VLAN. In this case a PortGroup was created called ‘VMK-Temp-Vmo’ and tagged with VLAN 192. From the same location in vCenter, Host 1 -> Configure -> Networking -> VMKernel adapters, click on “Add Networking” and create a vmk3 adapter using the ‘VMK-Temp-Vmo’ PortGroup.
A vMotion VMKernel has now been provision no Production Host 1 for a subnet that can communicate with the VxRail.
Reconfiguration of VxRail to enable L3 routing of vMotion Subnet
Unlike the VMware hosts in the Production Environment, the VxRail uses the vMotion TCP/IP stack for the vMotion VMKernel which allows for a gateway to be configured so that vMotion traffic can be routed if needed. However, similar to the Production Environment, the VxRail vMotion network is isolated to the VxRail and does not have a gateway to be able to route upstream.
A similar workaround to the Production Environment will be used to create a VMkernel for Vmotion on the 192 VLAN so that it can talk to the migration host in the Production Cluster
Like the procedure for the Production cluster, this reconfiguration is only taking place on host 1 in the VxRail.
VLAN 192 has already been added to the Trunk on VxRail Host 1
A Distributed PortGroup called ‘Temp Vmotion for Host 1’ has already been created on the VxRail, associated with VMNIC4 and tagged with VLAN 192.
1) On the VxRail, navigate to Host 1 -> Configure -> Networking -> VMKernel adapters and click on ‘Add Networking…”
2) Add a new VMkernel Adapter (vmk5).
3) Select the ‘Temp Vmotion for Host 1’ PortGroup.
Be sure to choose the vMotion TCP/IP Stack.
4) Set the IP address with a gateway.
5. Once vmk5 is created, you may see some Warning or Error alerts reported on the Vcenter server. This is because it has detected a vmk5 on host 1 but does not see a vmk5 on any of the other hosts so it is reporting a mismatch configuration. We can safely ignore this for now and it will clear itself once the temporary changes are reverted.
In the Production environment, we were able to disable the vMotion services on the existing VMKernels, essentially disabling the default vMotion subnet in order to force the host to use the subnet for vMotion.
Because the VxRail uses the vMotion TCP/IP stack, we cannot just disable the vMotion service on the existing VMKernel (vmk4). Instead we have to remove vmk4 from VxRail host 1 entirely. This will force all vMotion traffic to use vmk5 on the subnet.
6. From the same location, Host 1 -> Configure -> Networking -> VMKernel adapters, select vmk4 and record the IP information.
7. Click the 3 dots to the left of vmk4 and select remove.
NOTE: At this point, the VxRail VLAN for vMotion has been removed from VxRail Host 1. This means that VM’s will not be able to migrate to or from Host 1 in the VxRail Cluster.
Test communication between Production Host 1 and VxRail Host 1 across vMotion VMK’s
1) Using Putty open one SSH session to VxRail Host 1 and one SSH session to Production Host 1 (note: you may need to enable the SSH service on the hosts).
2) Use the ‘vmkping’ command to test connectivity using a specific VMKernel port. The syntax will be different between Production Host 1 and VxRail Host 1 because of the TCP/IP stack used.
3) Command for Production Host 1: ‘vmkping -I vmk3 <IP Address>
This sends the ping out the vmk3 interface (that is on the subnet) and attempts to ping the IP assigned to the vmk5 interface on the VxRail that was set to an IP of <Insert IP>.
4) Command for VxRail Host 1: ‘vmkping -S vmotion <IP Address>
This sends the ping out the vmk interface configured on the vMotion TCP/IP stack (vmk5 on the subnet) to the IP assigned to vmk3 on Production Host 1
If communication is good both ways, then a Cross vCenter vMotion should work.
Using the XVM Utility to migrate a VM from Production Host 1 to VxRail Host 1
1) Bring up the XVM Utility WebUI and click on the ‘Migrate’ button.
2) Select the appropriate information for each field. In this example below we are going to ‘Relocate’ (aka migrate) Virtual Machine ADDC02 from the Datacenter Cluster in the Production vCenter to VSAN Datastore on VxRail Host 1. The existing Production network, ‘VLAN10’ maps to the ‘Servers’ network on the VxRail. Meaning, both Networks are for the VLAN 10 subnet, just labeled differently in both environments.
3) Before hitting submit, make sure that there are no CD/DVD’s or USB drives mapped to the VMor the vMotion will fail as the device is not available at the Target Site (VxRail). If CD/DVD is mapped, make sure to disconnect it and set it back to Client Device.
4) To ensure connectivity throughout the VM migration process, start a timed ping to the VM(s) being migrated.
5) Click on submit to start the VM migration process.
Once the migration has started, it can be monitored from the XVM Utility WebUI or Tasks/Events section of wither the Production vCenter or the VxRail.
When the VM migration process completes, the VM(s) will no longer be seen in the Production vCenter and will now be seen in the VxRail vCenter. The migration is complete.
You can rinse and repeat this process for as many Virtual Machines that were loaded up onto the migration host or select multiple VM’s in the ‘Virtual Machine(s)’ field on the ‘Migrate’ page on the XVM Utility to migrate more than one at a time.
When migration activities are complete, you will want to revert the network changes made to both Production Host 1 and VxRail Host 1 in order to restore default functionality for both hosts/environments. See next section.
Revert network changes when Migration activities are complete
Remember, the networking changes made to Production Host 1 and VxRail Host 1 allows for VM’s to migrated between the 2 hosts but prevents vMotion within each Cluster to or from these hosts. You will not be able to move migrated VM’s off of VxRail Host 1 to other VxRail Hosts to balance the load in the Cluster. You also will not be able to move more VM’s onto Production Host 1 for migration to the VxRail.
The network changes made to each Host need to be reverted to a default state to resume normal functionality or to prepare more VM’s for migrate. High level steps for this are listed below but can also be inferred from previous steps in this document used to make the changes.
Revert Changes on Production Host 1
1) From the Production vCenter, navigate to Production Host 1 -> Configure -> Networking -> VMkernel adapters and select vmk3 (VMK-Temp-Vmo). Click on ‘Edit’ and uncheck the box for vMotion to disable the vMotion Service on this vmk.
2) From the same location, Production Host 1 -> Configure -> Networking -> VMkernel adapters, one at a time, select vmk2 and vmk3. Click on ‘Edit’ and check the box for vMotion to re-enable the vMotion service on both vmk’s.
3) If DRS on the Production vCenter Cluster was disabled or set to manual, it can now be re-enabled or set back to Automatic. You can test a vMotion between another host in the Production Cluster to make sure it works as expected.
4) If there are more VM’s to migrate, you can now move them to Production Host 1 in preparation for Migration.
Production Host 1 has now been reverted to its default state.
Revert Changes on VxRail Host 1
1) From the Production vCenter, navigate to Production Host 1 -> Configure -> Networking -> VMkernel adapters and click on the 3 dots to the left of vmk5 (Temp Vmotion for Host 1) and click remove.
2) Re-create vmk4 for the VxRail vMotion network. From the same location click on ‘Add Networking…’.
3) Select Connection Type: ‘VM Kernel Network Adapter’.
4) Select target device: ‘Select an existing network’.
5) Port Properties: Be sure that ‘vMotion’ is selected for the ‘TCP/IP stack’.
6) IPv4 settings: Enter the IP information that was recorded before the vmk was removed.
7) Click on ‘Next’ and then ‘Complete’.
Once vmk5 has been removed and vmk4 has been added back in, any Warning or Errors that vCenter was reporting regarding Network should clear up and go away on their own.
vMotion is now re-enabled between VxRail Host 1 and the other VxRail hosts in the cluster. Migrated VM’s can be vMotioned off VxRail Host 1 and onto other Hosts in the cluster to balance the load and free up Host 1 for more migrations.
All network configuration changes to facilitate Cross Vcenter Vmotions have now been undone. Both Production and VxRail Environments are back to default state. To shutdown the XVM Utility, simply close out the Browser Window and Close the CMD prompt running in the background.
Closing Out
It’s important to remember that this VM migration process was customized for a specific customer use-case and cannot be a step-by-step guide for all VM migrations.
However, this VM Migration process should serve as a guide on what to expect, specifically in the use case of migrating from a production VMware environment to hyper-converged technology.
Let us know if this process worked for you or how we can help.
