Nutanix Cluster Troubleshooting
In this article you will learn basic Nutanix Cluster Troubleshooting tips and tricks. Learn how to identify which Nutanix Cluster Services is down.
Check back from time to time, as this is a article that is updated on a regular basis.
If you run into issues with the Services in the Nutanix Cluster or the Nutanix CVM it´s good to know what to look for and troubleshoot it. A normal operating Nutanix cluster / CVM services will report that all services are running. Let´s first learn how to check that all is good or not.
The NCC health check cluster_services_down_check verifies if the cluster services are running on all Controller VMs in the cluster. If any cluster service is not running on any of the Controller VMs in the cluster, this check fails.
Running the NCC Check in a Nutanix Cluster
You can run this check as part of the complete NCC Health Checks:
nutanix@cvm$ ncc health_checks run_all
Or you can run this check separately:
nutanix@cvm$ ncc health_checks system_checks cluster_services_down_check
You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.
Please note that this check is scheduled to run every minute, by default.
Here is a list of some issues you might run into performing different operations to the Nutanix Cluster.
Check Nutanix Services in a Nutanix Cluster
Let´s check the status of each Nutanix Service.
Run the following command to confirm if any service is currently down:
nutanix@cvm$ cluster status | grep -v UP
If any services are listed as DOWN, start those services by using the following command:
nutanix@cvm$ cluster start
Here is a list of possible issues one can run into in different scenarios.
Upgrade stuck due to Genesis not able to start services after Cassandra service
Resolution : To resolve the Upgrade stuck due to Genesis not able to start services after Cassandra service – Run following command from any Nutanix CVM in the cluster.
nutanix@cvm$ genesis statusornutanix@cvm$ gs
nutanix@cvm$ allssh 'genesis restart'
Using the following command is a simple way to check to see if any services are crashing on a single node in the cluster.
nutanix@cvm$ watch -d genesis status
Note: if the cluster is large then you may have to execute the command one node at a time, hence it may be a wise decision to correlate with the alert on what CVM so as to check to see which services are reported to be crashing.
Unreachable DNS server can prevent 2 node clusters from starting services after failure
Resolution : To resolve the Unreachable DNS server can prevent 2 node clusters from starting services after failure – check the DNS / Name server entry in CVM configuration file and check connectivity.
Command 1: Check DNS / Name server entry on cluster configuration
nutanix@cvm:~$ zeus_config_printer | grep name_server
Command 2: then check the DNS / Name server entry in all CVMs configuration file.
nutanix@cvm:~$ allssh "cat /etc/resolv.conf"
If DNS entry is not found then add DNS server IP address / host name from Prism as showing following screenshot.
Make sure DNS server is reachable before putting DNS IP address / host name.
Expand CVM Memory
Some times you want to check how much memory is allocated to the Nutanix CVM via CLI. There is also an option in Nutanix Prism to expand memory allocation for the CVM.
Step 1: check the Nutanix CVM Memory allocation that must be at least 24 GB or greater would be fine.
nutanix@cvm$ free -m
Step 2: If Nutanix CVM Memory allocation is less then 24 GB then need to scale-up the memory to at least 24 GB or greater.
Increase / scale-up Nutlanix CVM memory from Prism console
Step 3: Restart Genesis service on all Nutanix CVMs
nutanix@cvm$ allssh genesis restart nutanix@cvm$ allssh genesis stop prism nutanix@cvm$ cluster start
Nutanix CVM / Cluster Services are down
Let’s troubleshoot the Nutanix Cluster / CVM services down issue.
Here are a list of the most Critical Nutanix services:
Resolution: check the Nutanix CVM / Cluster services status and restart them.
Step 1: Check Nutanix CVM / Cluster services status
nutanix@CVM$ ncc health_checks run_all nutanix@CVM$ ncc health_checks system_checks cluster_services_status nutanix@CVM$ ncc health_checks system_checks cvm_services_status nutanix@cvm$ ncc health_checks hypervisor_checks check_services nutanix@cvm$ ncc health_checks system_checks cluster_services_down_check
The NCC health check cluster_services_status verifies if the Controller VM (CVM) services have restarted recently across the cluster.
The following services are checked:
Step 2: Shortlist the down services on all Nutanix CVM
nutanix@pcvm$ cluster status | grep -v UP
Step 3: Start Nutanix CVM / Cluster services
nutanix@pcvm$ cluster start
Note: Above command will not impact your production running VMs.
Optional Step 4: If step 3 command does not start the down services then you can reboot your either Nutlanix Node or Nutanix CVM.
Step 4.1: Reboot Nutanix CVM
nutanix@cvm$ cvm_shutdown -r now
Step 4.1.1: OR Shutdown Nutanix CVM
nutanix@cvm$ cvm_shutdown -P now
Step 4.1.2: Power-on the Shudown Nutanix CVM
SSH to Nuanix AHV host
root# virsh list --all | grep CVM
In output you will see CVM Name, just copy it and run following command to start the Nutanix CVM
root# virsh start <CVM_Name>
Wait for 5 Minutes to boot-up the Nutanix CVM and services.
OR Step 4.2 : You can put your host in maintenance mode and then reboot node
Final Step : Now check Nutanix cluster status and running services.
nutanix@pcvm$ cluster status
Nutanix Gateway not reachable. Http request error
Resolution: Need to restart the Nutanix Console services on the host, which is Prism leader.
Step 1: Find the Nutanix Prism Leader – Verify which cluster node is the Prism leader, that is, the CVM running the Prism container services.
nutanix@cvm$ curl http://0:2019/prism/leader && echo
Output should look similar as following
It means xx.xx.xx.10 CVM is the Prism Leader.
Step 2: SSH to Prism Leader and run the following command to restart Prism service.
nutanix@cvm$ genesis stop prism nutanix@cvm$ cluster start
Note: There is no impact on running production of above commands.
The following command can be used to check out the tasks in the Prism web console that have not been completed.
nutanix@cvm$ ecli task.list include_completed=false
LCM upgrade fails with error “Services not up” on a 2-node cluster
Resolution: Above both services run in LCM framework.
This is known issue. therefore it is recommended to upgrade Nutanix NCC a
Upgrades at Nutanix are always designed to be done without needing any downtime for User VMs and their workloads. The following article is intended to serve as an introduction describing how each type of upgrade works and to share some useful best practices for administrators. You will find similar information in the Acropolis Upgrade Guide (remember to always choose the guide that matches the AOS currently running on your cluster). The link to the article can be found here.
Check upgrade status
Hypervisor upgrade status
Description: Check hypervisor upgrade status from the CLI on any CVM
LCM upgrade status
Detailed logs (on every CVM)
nutanix@cvm$ less ~/data/logs/host_upgrade.out
Note: In order to understand how upgrades work, please refer to the following link to the article which can be found here.
Find cluster error logs
Description: Find ERROR logs for the cluster
nutanix@cvm$ allssh "cat ~/data/logs/<COMPONENT NAME or *>.ERROR"
Example for Stargate
nutanix@cvm$ allssh "cat ~/data/logs/stargate.ERROR"
Find cluster fatal logs
Description: Find FATAL logs for the cluster
nutanix@cvm$ allssh "cat ~/data/logs/<COMPONENT NAME or *>.FATAL"
Example for Stargate
nutanix@cvm$ allssh "cat ~/data/logs/stargate.FATAL"
Similarly, you can also run the following script to list the fatals across all the nodes in the cluster
nutanix@cvm$ for i in `svmips`; do echo "CVM: $i"; ssh $i "ls -ltr /home/nutanix/data/logs/*.FATAL"; done
Here is a list of KBs that might come in handy when doing Nutanix Cluster Troubleshooting
The following article provides commands for entering and exiting maintenance mode for CVM (Controller VM) and hypervisor. The link can be found here.
How to Shutdown Nutanix CVM and Nutanix AHV Host
How to Fix a Nutanix CVM being Stuck in Maintenance Mode
|cvm_services_status verifies if a service has crashed and generated a core dump in the last 15 minutes.||KB 2472|
|cluster_services_status verifies if the Controller VM (CVM) services have restarted recently across the cluster.||KB 3378|
|check_ntp verifies the NTP configuration of the CVMs (Controller VMs) and hypervisor hosts, and also checks if there are any time drifts on the cluster.||KB 4519|