Openstack vm instance stuck in spawning state

Question

Ive installed openstack 2.3.1 (Mitaka) using conjure-up on a single machine(Ubuntu 16.04 Sever). My lxd container uses zfs as a storage backend which is 4TB in size, so there is enough space.

I could launch a test instance by using the image which came along with the installation

Flavor Name m1.small

Flavor ID ee64beb2-e13b-454a-b085-3916947c96d1

RAM 2GB

VCPUs 1 VCPU

Disk 20GB

The size of this image is 124.94 MB

Disk Format is RAW

The network specs for this image are

subtestnet1 190.168.1.0/24 (gave some random range)

This is a private network, so when launching the instance the assigned IP for the is 190.168.1.4

I shutdown this instance , since I just wanted to test If i can launch.

Now, I try to launch my custom vdi image(Ubuntu 16.04 with some software) which is 12GB in size with the below specs

Disk Format RAW

subtestnet1 190.168.1.0/24 (gave some random range)

Flavor Name m1.medium

Flavor ID dbda6460-bdf0-4387-9e05-0ebf7a26f45a

RAM 4GB

VCPUs 2 VCPU

Disk 40GB

But it just hangs in spawning state (its been almost 30 minutes)

Ive checked the logs in nova-compute node but it didnt show any error log. I checked in nova-cloud-controller/0 and there were some error logs in the files below

nova-api-os-compute.log

2017-11-27 03:18:43.276 23931 ERROR nova.api.openstack.extensions [req-3095878e-c8d3-4ba6-9c44-ad7dabde21fe a048e14509c84c0f9205819fb29a2501 003ad069faba48709198af746c92b5fa - default default] Unexpected exception in API method: MessagingTimeout: Timed out waiting for a reply to message ID 9fdb48d40a1046b2ad7b63b2703df928

nova-conductor.log 2017-11-27 04:29:30.123 23928 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.

NoValidHost: No valid host was found. There are not enough hosts available. 2017-11-27 04:29:30.177 23928 WARNING nova.scheduler.utils [req-dc65dc2a-0bf0-4376-b8ba-88fd788ad233 a048e14509c84c0f9205819fb29a2501 003ad069faba48709198af746c92b5fa - default default] [instance: 4a7c771d-65a7-4369-97c0-d432630c24d7] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

I've restarted all the nova related services on the compute and controller nodes but didn't help.

Has anyone faced such an issue?

Any help to debug further will be appreciated. If you need more info, do let me know.

Thanks.

Answer 1

Does the compute node has enough RAM/VCPUs/Disks required by the VM. "No valid host found" error generally indicates lack of one of these three resources.

Answer 2

Try to enable debug mode to get detailed logs.

Set debug = True in these files:

• /etc/nova/nova.conf
• /etc/nova/cinder.conf
• /etc/glance/glance-registry.conf

... and restart the reconfigured services. Now you should be able to find the source of the problem.

For example you can check the filters log in nova-scheduler.log, look at lines above this one :

.. INFO nova.filters [req-..] Filter DiskFilter returned 0 hosts

Answer 3

This error is a bit generic and can come because of many reasons, I had the same error and took the following steps to resolve it:

You can check the following Openstack Summit video for an indepth troubleshooting on this matter:

OpenStack Summit

Error:

fault | {"message": "No valid host was found. There are not enough hosts available.}

Unable to create VM; details below:

+--------------------------------------+-----------------+----------------------------------+--------+------------+-------------+----------+
| ID                                   | Name            | Tenant ID                        | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+----------------------------------+--------+------------+-------------+----------+
| f40c379c-b322-403e-b8b8-7fddeb949cdf | Test            | 2367198c9f704e52a171dd1e84b41e37 | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+-----------------+----------------------------------+--------+------------+-------------+----------+

Troubleshooting:

• Check the VM Power state, id and status using: nova list

• Check the VM details using: nova show <VM id from the above command>

• In the above command check if the compute name is present, In my case it was present.

• Check if the compute node name is present in: nova host-list

• Went to the compute node:

  ubuntu@compute:/var/log/nova$ cat nova-compute.log
  2018-11-25 22:06:49.095 43926 ERROR nova.virt.libvirt.driver Error defining a domain with XML: <domain type="kvm">
  TRACE nova.compute.manager [instance: f40c379c-b322-403e-b8b8-7fddeb949cdf] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
  TRACE nova.compute.manager [instance: f40c379c-b322-403e-b8b8-7fddeb949cdf] libvirtError: internal error: no supported architecture for os type 'hvm'
  

Resolution:

• Checked nova-compute config which had virt_type set to kvm (if required use root to access this directory):

  $ cd /etc/nova
  $ vim nova-compute.conf
  

  KVM was not present:

  $ kvm-ok 
  INFO: /dev/kvm does not exist
  HINT:   sudo modprobe kvm_intel
  INFO: For more detailed results, you should run this as root
  HINT:   sudo /usr/sbin/kvm-ok
  ubuntu@compute:/var/log/nova$ 
  

• Changed the virt_type to qemu in nova-compute.conf (virt_type=qemu) and restarted the services to resolve the issue:

  $ sudo service nova-compute restart