Specifying "hw:mem_page_size=large" means that it will only use hugepages, not 4K pages.   On the compute node do you have enough CPU and free hugepages on the same NUMA node to satisfy the request?  The 'NUMATopologyFilter: (start: 1, end: 0)' part of the log says that this filter ruled out your only compute node.

What does "virsh capabilities|grep -C5 pages" show on the compute node?

What does /etc/nova/nova.conf show inside the nova-compute container?

It's possible to enable debug mode for nova.conf but it's a little tricky to get the syntax right.  Basically you'd be using the "system helm-override-update" command.  Can you give the output of "system helm-override-show" for the nova chart in the openstack application?

Chris

On 9/24/2019 6:16 AM, Mariano Ucha wrote:
Hi all!

I'm having a trouble now trying to allocate a VM con Openstack. I'm using STX 2.0 AIO-SX with DPDK so when i'm trying to allocate a VM with a flavor that have hw:mem_page_size=large metadata the system gives me error 500 No valid host was found. There are not enough hosts available.

File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances instance_uuids, return_alternates=True) File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances return_alternates=return_alternates) File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations instance_uuids, return_objects, return_alternates) File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call retry=self.retry) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send retry=retry) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send call_monitor_timeout, retry=retry) File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send raise result 


Logs from Nova Scheduler

2019-09-23 17:37:56.537 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filter NUMATopologyFilter returned 0 hosts
2019-09-23 17:37:56.537 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filter NUMATopologyFilter returned 0 hosts
2019-09-23 17:37:56.538 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filtering removed all hosts for the request with instance ID '66175096-37a4-4a2c-b23a-e7a4a6f10ca1'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 0)']
2019-09-23 17:37:56.538 1 INFO nova.filters [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Filtering removed all hosts for the request with instance ID '66175096-37a4-4a2c-b23a-e7a4a6f10ca1'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 0)']

Logs from Nova Conductor

2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return_alternates=return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=self.retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise result
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     claimed_instance_uuids)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise exception.NoValidHost(reason=reason)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
    return func(*args, **kwargs)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
    claimed_instance_uuids)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_alternates=True)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return_alternates=return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     instance_uuids, return_objects, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=self.retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     call_monitor_timeout, retry=retry)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise result
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager Traceback (most recent call last):
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     return func(*args, **kwargs)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     allocation_request_version, return_alternates)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     claimed_instance_uuids)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager   File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager     raise exception.NoValidHost(reason=reason)
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.542 1 ERROR nova.conductor.manager
2019-09-23 17:37:56.627 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
    return func(*args, **kwargs)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
    claimed_instance_uuids)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.627 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
    return func(*args, **kwargs)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/manager.py", line 168, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
    allocation_request_version, return_alternates)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule
    claimed_instance_uuids)

  File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.
: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.630 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] [instance: 66175096-37a4-4a2c-b23a-e7a4a6f10ca1] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2019-09-23 17:37:56.630 1 WARNING nova.scheduler.utils [req-f57095ac-6539-4efb-a6a5-f27aebb0a27c edbcf8808d8a4f48a39b8eebb3951292 8ae61d9905d94b8aa66193b20a3ab973 - default default] [instance: 66175096-37a4-4a2c-b23a-e7a4a6f10ca1] Setting instance to ERROR state.

How can i change Nova log mode to DEBUG?

Anything more i can check?

Regards,
Mariano



_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss