Hi all,

I create some VMs on openstack for stx3.0. And I cannot reboot/shutdown VMs from either dashboard or cli.

This is log from nova-api-proxy pod:

2020-04-26 07:55:24,632.632 6 INFO nova_api_proxy.apps.acceptor [-] reboot

2020-04-26 07:55:24,633.633 6 INFO nova_api_proxy.apps.acceptor [-] type : SOFT

2020-04-26 07:55:24,633.633 6 INFO nova_api_proxy.apps.acceptor [-] POST request issued by user (admin) tenant (admin) remote address (172.16.2.145) "POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/125f4586-17a5-4358-9110-04c518cc9a92/action"

2020-04-26 07:55:24,633.633 6 INFO nova_api_proxy.apps.acceptor [-] Forward to NFV "POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/125f4586-17a5-4358-9110-04c518cc9a92/action", action: (reboot), val:({u'type': u'SOFT'})

2020-04-26 07:55:24,636.636 6 INFO nova_api_proxy.apps.proxy [-] POST response body: ['<html>\r\n<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The plain HTTP request was sent to HTTPS port</center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n']

2020-04-26 07:57:22,882.882 6 INFO nova_api_proxy.apps.acceptor [-] reboot

2020-04-26 07:57:22,883.883 6 INFO nova_api_proxy.apps.acceptor [-] type : SOFT

2020-04-26 07:57:22,883.883 6 INFO nova_api_proxy.apps.acceptor [-] POST request issued by user (admin) tenant (admin) remote address (172.16.2.145) "POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/8498e70d-8fb6-44f9-974a-14bfa9ef6288/action"

2020-04-26 07:57:22,883.883 6 INFO nova_api_proxy.apps.acceptor [-] Forward to NFV "POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/8498e70d-8fb6-44f9-974a-14bfa9ef6288/action", action: (reboot), val:({u'type': u'SOFT'})

2020-04-26 07:57:22,885.885 6 INFO nova_api_proxy.apps.proxy [-] POST response body: ['<html>\r\n<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The plain HTTP request was sent to HTTPS port</center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n']

2020-04-26 08:38:23,794.794 6 INFO nova_api_proxy.apps.acceptor [-] reboot

2020-04-26 08:38:23,799.799 6 INFO nova_api_proxy.apps.acceptor [-] type : SOFT

2020-04-26 08:38:23,799.799 6 INFO nova_api_proxy.apps.acceptor [-] POST request issued by user (admin) tenant (admin) remote address (172.16.2.145) "POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/8498e70d-8fb6-44f9-974a-14bfa9ef6288/action"

2020-04-26 08:38:23,801.801 6 INFO nova_api_proxy.apps.acceptor [-] Forward to NFV "POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/8498e70d-8fb6-44f9-974a-14bfa9ef6288/action", action: (reboot), val:({u'type': u'SOFT'})

2020-04-26 08:38:23,832.832 6 INFO nova_api_proxy.apps.proxy [-] POST response body: ['<html>\r\n<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The plain HTTP request was sent to HTTPS port</center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n']

And when I try to stop VM from cli("openstack server stop xxx"), this error occurs:

Starting new HTTP connection (1): nova-api-proxy.openstack.svc.cluster.local

http://nova-api-proxy.openstack.svc.cluster.local:8774 "GET /v2.1/d27de62f56b84376b5de5e50069dfa86/servers/e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a HTTP/1.1" 200 1688

RESP: [200] Connection: keep-alive Content-Length: 1688 Content-Type: application/json Date: Sun, 26 Apr 2020 09:15:11 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-66b83d55-3013-4b01-9b0b-ace29cf45813 X-Openstack-Nova-Api-Version: 2.1 X-Openstack-Request-Id: req-66b83d55-3013-4b01-9b0b-ace29cf45813

RESP BODY: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"public-net0": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:4a:4c:63", "version": 4, "addr": "192.168.101.42", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://nova-api.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a", "rel": "self"}, {"href": "http://nova-api.openstack.svc.cluster.local:8774/d27de62f56b84376b5de5e50069dfa86/servers/e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a", "rel": "bookmark"}], "image": "", "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-0000000e", "OS-SRV-USG:launched_at": "2020-04-26T09:06:51.000000", "flavor": {"id": "88d9d1c3-9d3f-400f-9833-bc9a961581a1", "links": [{"href": "http://nova-api.openstack.svc.cluster.local:8774/d27de62f56b84376b5de5e50069dfa86/flavors/88d9d1c3-9d3f-400f-9833-bc9a961581a1", "rel": "bookmark"}]}, "id": "e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a", "security_groups": [{"name": "default"}], "user_id": "9781ba055b664b88bc0d81efa8385a0a", "OS-DCF:diskConfig": "AUTO", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ACTIVE", "updated": "2020-04-26T09:06:51Z", "hostId": "ed5b6f56f20a881943d21c2c5c5b438bfc474ad968c2129652789499", "OS-EXT-SRV-ATTR:host": "controller-0", "OS-SRV-USG:terminated_at": null, "key_name": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": "controller-0", "name": "test", "created": "2020-04-26T09:06:08Z", "tenant_id": "d27de62f56b84376b5de5e50069dfa86", "os-extended-volumes:volumes_attached": [{"id": "0e6068c3-2b43-4786-8caf-3e508d746421"}], "metadata": {}}}

GET call to compute for http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a used request id req-66b83d55-3013-4b01-9b0b-ace29cf45813

REQ: curl -g -i -X POST http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/d27de62f56b84376b5de5e50069dfa86/servers/e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a/action -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}2924d2898529a443db522ba7e087d34d55c1a1816a618c6fcd2b15650466fc38" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"os-stop": null}'

http://nova-api-proxy.openstack.svc.cluster.local:8774 "POST /v2.1/d27de62f56b84376b5de5e50069dfa86/servers/e1ef4f18-6fc3-40ca-ae4f-fa7f263f2f2a/action HTTP/1.1" 400 248

RESP: [400] Connection: keep-alive Content-Length: 248 Content-Type: text/html Date: Sun, 26 Apr 2020 09:15:11 GMT Server: nginx

RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.

Bad request (HTTP 400)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand

result = cmd.run(parsed_args)

File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run

return super(Command, self).run(parsed_args)

File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run

return_code = self.take_action(parsed_args) or 0

File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 2347, in take_action

server,

File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 175, in stop

return self.manager.stop(self)

File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1049, in stop

resp, body = self._action_return_resp_and_body('os-stop', server, None)

File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 2023, in _action_return_resp_and_body

return self.api.client.post(url, body=body)

File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 381, in post

return self.request(url, 'POST', **kwargs)

File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 78, in request

raise exceptions.from_response(resp, body, url, method)

BadRequest: Bad request (HTTP 400)

clean_up StopServer: Bad request (HTTP 400)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run

ret_val = super(OpenStackShell, self).run(argv)

File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run

result = self.run_subcommand(remainder)

File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand

ret_value = super(OpenStackShell, self).run_subcommand(argv)