[Starlingx-discuss] Openstack reapply failing
Hi all, My earlier email was never replied to so I will try to re-word it: I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such. The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found" Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know! Best wishes, Austin Gillmann
Hi Austin, Could you check the pdb resources in openstack namespace when reapplying the opentack app? `$ Kubectl -n openstack get pdb` I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again. Mingyuan -----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Openstack reapply failing Hi all, My earlier email was never replied to so I will try to re-word it: I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such. The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found" Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know! Best wishes, Austin Gillmann _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Mingyuan, I checked the PDB list and neutron-server was indeed listed so I deleted it, however it seems kubernetes is playing whack-a-mole with me; so far I had to delete secrets, serviceaccounts, and RBAC roles as it seems like helm does not have control on *any* neutron related items Best regards, Austin On Thu, Feb 13, 2020 at 6:42 AM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Hi Austin,
Could you check the pdb resources in openstack namespace when reapplying the opentack app? `$ Kubectl -n openstack get pdb` I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again.
Mingyuan
-----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Openstack reapply failing
Hi all,
My earlier email was never replied to so I will try to re-word it:
I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such.
The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found"
Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know!
Best wishes, Austin Gillmann
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi again! Seems I found a workaround to my issue, by removing the application that seems to clean up everything well enough to apply successfully again! Albeit destructive it seems to have worked well. Best regards, Austin On Thu, Feb 13, 2020 at 8:28 AM Austin Gillmann <ji@sibyl.li> wrote:
Hi Mingyuan,
I checked the PDB list and neutron-server was indeed listed so I deleted it, however it seems kubernetes is playing whack-a-mole with me; so far I had to delete secrets, serviceaccounts, and RBAC roles as it seems like helm does not have control on *any* neutron related items
Best regards, Austin
On Thu, Feb 13, 2020 at 6:42 AM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Hi Austin,
Could you check the pdb resources in openstack namespace when reapplying
the opentack app? `$ Kubectl -n openstack get pdb`
I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again.
Mingyuan
-----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Openstack reapply failing
Hi all,
My earlier email was never replied to so I will try to re-word it:
I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such.
The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found"
Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know!
Best wishes, Austin Gillmann
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Yes, you can always remove the stx-openstack application if you do not need to keep the data within mariadb database. Mingyuan From: Austin Gillmann <ji@sibyl.li> Sent: Friday, February 14, 2020 8:36 To: Qi, Mingyuan <mingyuan.qi@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Openstack reapply failing Hi again! Seems I found a workaround to my issue, by removing the application that seems to clean up everything well enough to apply successfully again! Albeit destructive it seems to have worked well. Best regards, Austin On Thu, Feb 13, 2020 at 8:28 AM Austin Gillmann <ji@sibyl.li<mailto:ji@sibyl.li>> wrote: Hi Mingyuan, I checked the PDB list and neutron-server was indeed listed so I deleted it, however it seems kubernetes is playing whack-a-mole with me; so far I had to delete secrets, serviceaccounts, and RBAC roles as it seems like helm does not have control on *any* neutron related items Best regards, Austin On Thu, Feb 13, 2020 at 6:42 AM Qi, Mingyuan <mingyuan.qi@intel.com<mailto:mingyuan.qi@intel.com>> wrote:
Hi Austin,
Could you check the pdb resources in openstack namespace when reapplying the opentack app? `$ Kubectl -n openstack get pdb` I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again.
Mingyuan
-----Original Message----- From: Austin Gillmann <ji@sibyl.li<mailto:ji@sibyl.li>> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Openstack reapply failing
Hi all,
My earlier email was never replied to so I will try to re-word it:
I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such.
The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found"
Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know!
Best wishes, Austin Gillmann
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hello again Mingyuan and team, While this workaround still works fine for me, the issue of applying still persists and can get annoying having to wipe network and disk images to make a helm edit. As said before it seems like it's not just PDB that is out of helms control, but everything including secrets, service accounts, and RBAC. I am curious if running SAS disks on my OSD has anything to do with this... If there is a possible fix or an associated bug I can track please let me know! Thank you! Austin Gillmann On Thu, Feb 13, 2020 at 9:51 PM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Yes, you can always remove the stx-openstack application if you do not need to keep the data within mariadb database.
Mingyuan
From: Austin Gillmann <ji@sibyl.li> Sent: Friday, February 14, 2020 8:36 To: Qi, Mingyuan <mingyuan.qi@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Openstack reapply failing
Hi again!
Seems I found a workaround to my issue, by removing the application that seems to clean up everything well enough to apply successfully again! Albeit destructive it seems to have worked well.
Best regards,
Austin
On Thu, Feb 13, 2020 at 8:28 AM Austin Gillmann <ji@sibyl.li> wrote:
Hi Mingyuan,
I checked the PDB list and neutron-server was indeed listed so I deleted it, however it seems kubernetes is playing whack-a-mole with me; so far I had to delete secrets, serviceaccounts, and RBAC roles as it seems like helm does not have control on *any* neutron related items
Best regards, Austin
On Thu, Feb 13, 2020 at 6:42 AM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Hi Austin,
Could you check the pdb resources in openstack namespace when reapplying the opentack app? `$ Kubectl -n openstack get pdb` I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again.
Mingyuan
-----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Openstack reapply failing
Hi all,
My earlier email was never replied to so I will try to re-word it:
I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such.
The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found"
Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know!
Best wishes, Austin Gillmann
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Furthermore, I get the following traceback when attempting to apply or delete the application after letting the cluster sit: ``` sysinv 2020-03-15 02:43:38.049 1094205 INFO sysinv.conductor.kube_app [-] Application stx-openstack (1.0-19-centos-stable-latest) apply started. sysinv 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app [-] File does not exists: /tmp/tmpz3U7_z: ConfigException: File does not exists: /tmp/tmpz3U7_z 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1947, in perform_app_apply 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app self._create_local_registry_secrets(app.name) 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1002, in _create_local_registry_secrets 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app self._kube.kube_get_secret(DOCKER_REGISTRY_SECRET, ns)): 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 192, in kube_get_secret 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app c = self._get_kubernetesclient_core() 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 78, in _get_kubernetesclient_core 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app self._load_kube_config() 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 63, in _load_kube_config 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app config.load_kube_config('/etc/kubernetes/admin.conf') 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 531, in load_kube_config 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app loader.load_and_set(config) 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 413, in load_and_set 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app self._load_cluster_info() 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 392, in _load_cluster_info 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app file_base_path=self._config_base_path).as_file() 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 116, in as_file 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app raise ConfigException("File does not exists: %s" % self._file) 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app ConfigException: File does not exists: /tmp/tmpz3U7_z 2020-03-15 02:43:38.832 1094205 ERROR sysinv.conductor.kube_app ``` On Sat, Mar 14, 2020 at 8:12 PM Austin Gillmann <ji@sibyl.li> wrote:
Hello again Mingyuan and team,
While this workaround still works fine for me, the issue of applying still persists and can get annoying having to wipe network and disk images to make a helm edit. As said before it seems like it's not just PDB that is out of helms control, but everything including secrets, service accounts, and RBAC. I am curious if running SAS disks on my OSD has anything to do with this... If there is a possible fix or an associated bug I can track please let me know!
Thank you! Austin Gillmann
On Thu, Feb 13, 2020 at 9:51 PM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Yes, you can always remove the stx-openstack application if you do not need to keep the data within mariadb database.
Mingyuan
From: Austin Gillmann <ji@sibyl.li> Sent: Friday, February 14, 2020 8:36 To: Qi, Mingyuan <mingyuan.qi@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Openstack reapply failing
Hi again!
Seems I found a workaround to my issue, by removing the application that seems to clean up everything well enough to apply successfully again! Albeit destructive it seems to have worked well.
Best regards,
Austin
On Thu, Feb 13, 2020 at 8:28 AM Austin Gillmann <ji@sibyl.li> wrote:
Hi Mingyuan,
I checked the PDB list and neutron-server was indeed listed so I deleted it, however it seems kubernetes is playing whack-a-mole with me; so far I had to delete secrets, serviceaccounts, and RBAC roles as it seems like helm does not have control on *any* neutron related items
Best regards, Austin
On Thu, Feb 13, 2020 at 6:42 AM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Hi Austin,
Could you check the pdb resources in openstack namespace when reapplying the opentack app? `$ Kubectl -n openstack get pdb` I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again.
Mingyuan
-----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Openstack reapply failing
Hi all,
My earlier email was never replied to so I will try to re-word it:
I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such.
The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found"
Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know!
Best wishes, Austin Gillmann
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
You can file a new bug or search for similar issues on https://bugs.launchpad.net/starlingx Mingyuan -----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, March 15, 2020 9:13 To: Qi, Mingyuan <mingyuan.qi@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Openstack reapply failing Hello again Mingyuan and team, While this workaround still works fine for me, the issue of applying still persists and can get annoying having to wipe network and disk images to make a helm edit. As said before it seems like it's not just PDB that is out of helms control, but everything including secrets, service accounts, and RBAC. I am curious if running SAS disks on my OSD has anything to do with this... If there is a possible fix or an associated bug I can track please let me know! Thank you! Austin Gillmann On Thu, Feb 13, 2020 at 9:51 PM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Yes, you can always remove the stx-openstack application if you do not need to keep the data within mariadb database.
Mingyuan
From: Austin Gillmann <ji@sibyl.li> Sent: Friday, February 14, 2020 8:36 To: Qi, Mingyuan <mingyuan.qi@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Openstack reapply failing
Hi again!
Seems I found a workaround to my issue, by removing the application that seems to clean up everything well enough to apply successfully again! Albeit destructive it seems to have worked well.
Best regards,
Austin
On Thu, Feb 13, 2020 at 8:28 AM Austin Gillmann <ji@sibyl.li> wrote:
Hi Mingyuan,
I checked the PDB list and neutron-server was indeed listed so I deleted it, however it seems kubernetes is playing whack-a-mole with me; so far I had to delete secrets, serviceaccounts, and RBAC roles as it seems like helm does not have control on *any* neutron related items
Best regards, Austin
On Thu, Feb 13, 2020 at 6:42 AM Qi, Mingyuan <mingyuan.qi@intel.com> wrote:
Hi Austin,
Could you check the pdb resources in openstack namespace when reapplying the opentack app? `$ Kubectl -n openstack get pdb` I suspect that neutron-server pdb is out of helm chart's control for some reason. And you could try to delete it only and re-apply the app again.
Mingyuan
-----Original Message----- From: Austin Gillmann <ji@sibyl.li> Sent: Sunday, February 9, 2020 2:00 To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Openstack reapply failing
Hi all,
My earlier email was never replied to so I will try to re-word it:
I recently deployed StarlingX 3.0 under a dedicated storage standard cluster (the storage servers running on dell r740xd's with HDD's and compute/controllers running on HP bl460c G8 blades with 2 SSD's) the stx-openstack application applies successfully on the first try, but will abort upon reapply for a config change or such.
The errors armada puts out is details = "release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists" and "getting deployed release "osh-openstack-neutron": release: "osh-openstack-neutron" not found"
Everything else in logs look normal to me, but let me know if you need any files attached! If anyone has any idea how to get past this please let me know!
Best wishes, Austin Gillmann
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (2)
-
Austin Gillmann
-
Qi, Mingyuan