Citrix Xenserver VM Stuck in shutdown

I've been troubleshooting XenServer a couple of times today, with problems after cancelling a new VM. What I encountered today is what you can call a Xen hosted VM that is stuck in a “Pending” state, typically a VM that was stuck with a pending Shutdown task.

Here is what you need to do to release it from this state. Also note that in the XenCenter, the VM is no responsive in a panic state and the Shutdown and Reboot buttons are greyed out.

A couple of potential causes for this condition include:

1. a mounted iso from the console that is not avaliable for some reason
2. From Citrix: -> Windows being Windows. At times, when a server executes shutdown, it hangs, spinning it’s wheels. XenServer waits for a confirmation from XenTools that the shutdown is proceeding before it completes the task. If there are problems during the Shutdown, XenServer does not consider the task completed leading to a loss of connectivity to the XenServer tools. If connectivity between XenServer and XenTools is lost during the shutdown process meaning XenServer never receives a proper signal from Xentools about the state of the shutdown, you get that “stuck” symptom and the only thing you can do is force-kill the task and down a Force Reboot.
3. If you cancel a job either creating a new VM or Move/Copy a VM

Here is what you need to do!

From the XenCenter connect to the XenServer console CLI

1. xe task-list to view the Pending tasks
2. xe task-cancel force=true uuid=<UUID> the UUID from the above command

Please note that I also tried to use the xe vm-reboot or xe vm-shutdown commands do nothing but create more pending tasks. If  xe-toolstack-restart and xe task-cancel uuid=<UUID> (without the force=true) had no effect.

Once the Pending tasks affecting the VM had been cancelled, the VM rebooted and worked normally, xe-toolstack-restart after cancelling the tasks was not necessary, but you may need to use it if the server is not responsive.

Please NOTE that if you need to run the xe-toolstack-restart command, be sure to turn off HA before you run the command in CLI!

PS.We take no responsibility for what might or might not happen!