Hi Marc, I think you are right in that this is a neutron bug. It was probably introduced with [1]: And was backported to the Train release with [2]. This would explain why this hasn't been seen until recently. And the reason we don't see this in master or stx-4.0 is [3]: I have filed a bug with the Neutron folks here: https://bugs.launchpad.net/neutron/+bug/1888920 Cheers, Steve [1] https://opendev.org/openstack/neutron/commit/c5d8fd6329e405d1d584f4fedd7d0ed... [2] https://opendev.org/openstack/neutron/commit/764ff2a70a53100a301a07832c4d9a3... [3] https://opendev.org/openstack/openstack-helm/commit/e83f50633d29ba39f808c11e... ________________________________________ From: Marc Ferland <ferlandm@amotus.ca> Sent: Friday, July 24, 2020 8:39 AM To: Webster, Steven Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Problem applying stx-openstack-1.0-19-centos-stable-latest There you go. controller-0:~$ cat /etc/build.info<http://build.info> ### ### StarlingX ### Release 19.12 ### OS="centos" SW_VERSION="19.12" BUILD_TARGET="Host Installer" BUILD_TYPE="Formal" BUILD_ID="r/stx.3.0" JOB="STX_BUILD_3.0" BUILD_BY="starlingx.build@cengn.ca<mailto:starlingx.build@cengn.ca>" BUILD_NUMBER="21" BUILD_HOST="starlingx_mirror" BUILD_DATE="2019-12-13 02:30:00 +0000" [sysadmin@controller-0 ~(keystone_admin)]$ system show +----------------------+--------------------------------------+ | Property | Value | +----------------------+--------------------------------------+ | contact | None | | created_at | 2020-07-22T14:42:28.979711+00:00 | | description | None | | https_enabled | False | | location | None | | name | eafea042-28ed-4182-b206-3ce0bda1da44 | | region_name | RegionOne | | sdn_enabled | False | | security_feature | spectre_meltdown_v1 | | service_project_name | services | | software_version | 19.12 | | system_mode | duplex | | system_type | All-in-one | | timezone | UTC | | updated_at | 2020-07-22T14:43:48.392565+00:00 | | uuid | dec3eebe-8afc-4c3a-a40d-334dbf53669c | | vswitch_type | none | +----------------------+--------------------------------------+ On Fri, Jul 24, 2020 at 1:08 AM Webster, Steven <Steven.Webster@windriver.com<mailto:Steven.Webster@windriver.com>> wrote: Hi Marc, Can you provide your /etc/build.info<http://build.info> as well as a 'system show' command? Cheers, Steve ________________________________________ From: Marc Ferland <ferlandm@amotus.ca<mailto:ferlandm@amotus.ca>> Sent: Thursday, July 23, 2020 11:46 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Problem applying stx-openstack-1.0-19-centos-stable-latest Hi, This is related to: http://lists.starlingx.io/pipermail/starlingx-discuss/2020-July/009242.html When applying the stx-openstack charts I saw an error in the neutron-ovs-agent-controller-0-937646f6-t8cnc_openstack_neutron-ovs-agent-init log file: {"log":"+ timeout 3m neutron-sanity-check --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --ovsdb_native --nokeepalived_ipv6_support\n","stream":"stderr","time":"2020-07-21T16:42:56.518790392Z"} {"log":"2020-07-21 16:42:57.321 683 INFO neutron.common.config [-] Logging enabled!\u001b[00m\n","stream":"stderr","time":"2020-07-21T16:42:57.3220551Z"} {"log":"2020-07-21 16:42:57.322 683 INFO neutron.common.config [-] /var/lib/openstack/bin/neutron-sanity-check version 15.1.1.dev39\u001b[00m\n","stream":"stderr","time":"2020-07-21T16:42:57.322196968Z"} {"log":"2020-07-21 16:42:57.497 683 CRITICAL neutron [-] Unhandled error: NoSuchOptError: no such option vf_management in group [DEFAULT]\n","stream":"stderr","time":"2020-07-21T16:42:57.498852473Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron Traceback (most recent call last):\n","stream":"stderr","time":"2020-07-21T16:42:57.498867171Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron File \"/var/lib/openstack/bin/neutron-sanity-check\", line 8, in \u003cmodule\u003e\n","stream":"stderr","time":"2020-07-21T16:42:57.498871069Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron sys.exit(main())\n","stream":"stderr","time":"2020-07-21T16:42:57.498887171Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron File \"/var/lib/openstack/lib/python2.7/site-packages/neutron/cmd/sanity_check.py\", line 404, in main\n","stream":"stderr","time":"2020-07-21T16:42:57.498889813Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron enable_tests_from_config()\n","stream":"stderr","time":"2020-07-21T16:42:57.498892855Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron File \"/var/lib/openstack/lib/python2.7/site-packages/neutron/cmd/sanity_check.py\", line 347, in enable_tests_from_config\n","stream":"stderr","time":"2020-07-21T16:42:57.498895879Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron cfg.CONF.set_default('vf_management', True)\n","stream":"stderr","time":"2020-07-21T16:42:57.498899118Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_config/cfg.py\", line 2051, in __inner\n","stream":"stderr","time":"2020-07-21T16:42:57.498901668Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron result = f(self, *args, **kwargs)\n","stream":"stderr","time":"2020-07-21T16:42:57.498905003Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_config/cfg.py\", line 2455, in set_default\n","stream":"stderr","time":"2020-07-21T16:42:57.498907669Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron opt_info = self._get_opt_info(name, group)\n","stream":"stderr","time":"2020-07-21T16:42:57.498910612Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_config/cfg.py\", line 2845, in _get_opt_info\n","stream":"stderr","time":"2020-07-21T16:42:57.498913487Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron raise NoSuchOptError(opt_name, group)\n","stream":"stderr","time":"2020-07-21T16:42:57.498916583Z"} {"log":"2020-07-21 16:42:57.497 683 ERROR neutron NoSuchOptError: no such option vf_management in group [DEFAULT]\n","stream":"stderr","time":"2020-07-21T16:42:57.498919996Z"} Basically, vf_management isn't a recognized option anymore. I think this might be a bug in OpenStack itself. Poking around, I manually modified /var/lib/openstack/lib/python2.7/site-packages/neutron/cmd/sanity_check.py like so: --- sanity_check.py.orig 2020-07-23 10:29:00.551745591 -0400 +++ sanity_check.py 2020-07-23 11:28:39.935302535 -0400 @@ -357,7 +357,6 @@ run all necessary tests, just by passing in the appropriate configs. """ - cfg.CONF.set_default('vf_management', True) cfg.CONF.set_default('arp_header_match', True) cfg.CONF.set_default('icmpv6_header_match', True) if 'vxlan' in cfg.CONF.AGENT.tunnel_types: The charts then all applied correctly and my OpenStack deployment is now up and running. I was able to log into the Horizon GUI, etc. Am I the only one seeing this? Regards, Marc