Hi all!

I'm having a trouble now trying to allocate a VM con Openstack. I'm using STX 2.0 AIO-SX with DPDK so when i'm trying to allocate a VM with a flavor that have hw:mem_page_size=large metadata the system gives me error 500 No valid host was found. There are not enough hosts available.

File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances instance_uuids, return_alternates=True) File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances return_alternates=return_alternates) File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations instance_uuids, return_objects, return_alternates) File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call retry=self.retry) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send retry=retry) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send call_monitor_timeout, retry=retry) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send raise result 


Logs from Nova Scheduler

2019-09-23 17:37:56.537 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filter NUMATopologyFilter returned 0 hosts
2019-09-23 17:37:56.537 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filter NUMATopologyFilter returned 0 hosts
2019-09-23 17:37:56.538 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filtering removed all hosts for the request with instance ID '66175096-37a4-4a2c-b23a-e7a4a6f10ca1'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 0)']
2019-09-23 17:37:56.538 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filtering removed all hosts for the request with instance ID '66175096-37a4-4a2c-b23a-e7a4a6f10ca1'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 0)']

Logs from Nova Conductor

2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return_alternates=return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=self.retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise result
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     claimed_instance_uuids)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise exception.NoValidHost(reason=reason)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
    return func(*args, **kwargs)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
    claimed_instance_uuids)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return_alternates=return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=self.retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise result
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     claimed_instance_uuids)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise exception.NoValidHost(reason=reason)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.627 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
    return func(*args, **kwargs)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
    claimed_instance_uuids)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.627 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
    return func(*args, **kwargs)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
    claimed_instance_uuids)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.630 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] [instance: 66175096-37a4-4a2c-b23a-e7a4a6f10ca1] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.630 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] [instance: 66175096-37a4-4a2c-b23a-e7a4a6f10ca1] Setting instance to ERROR state.

How can i change Nova log mode to DEBUG?

Anything more i can check?

Regards,
Mariano