Thanks Austin.
Seems the issue is connectivity to rabbitmq though nothing altered manually.
2021-06-18T04:37:58.675414124Z
stdout F 2021-06-18 04:37:58.672 1 INFO nova.scheduler.manager
[req-25ec038c-661d-41fa-ac07-57a2eeb2fd6c
d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d -
default default] Got no allocation candidates from the Placement API.
This could be due to insufficient resources or a temporary occurrence as
compute nodes start up.
2021-06-18T04:37:58.67589708Z stdout F
2021-06-18 04:37:58.672 1 INFO nova.scheduler.manager
[req-25ec038c-661d-41fa-ac07-57a2eeb2fd6c
d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d -
default default] Got no allocation candidates from the Placement API.
This could be due to insufficient resources or a temporary occurrence as
compute nodes start up.
2021-06-18T04:37:59.291144415Z
stdout F 2021-06-18 04:37:59.289 1 WARNING nova.scheduler.utils
[req-25ec038c-661d-41fa-ac07-57a2eeb2fd6c
d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d -
default default] [instance: 3dbd2298-acf5-4123-a6e5-608113b0dbb7]
Setting instance to ERROR state.:
nova.exception_Remote.NoValidHost_Remote: No valid host was found.
2021-06-18T04:37:59.533353963Z
stdout F 2021-06-18 04:37:59.532 1 ERROR
oslo.messaging._drivers.impl_rabbit
[req-25ec038c-661d-41fa-ac07-57a2eeb2fd6c
d9f7048c1cd947cfa8ecef128a6cee89 e8813293073545f99658adbec2f80c1d -
default default] Connection failed: failed to resolve broker hostname
(retrying in 0 seconds): OSError: failed to resolve broker hostname
controller-0:/var/log/pods$ sudo rabbitmqctl cluster_status
Password:
Cluster status of node rabbit@localhost ...
[{nodes,[{disc,[rabbit@localhost]}]},
{running_nodes,[rabbit@localhost]},
{cluster_name,<<"rabbit@controller-0">>},
{partitions,[]},
{alarms,[{rabbit@localhost,[]}]}]