[Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z
Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/202105...) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949 - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/202105... Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
Hi All: Some Investigation found on LP #1928949: 1) Not sure why controller-1 build info is master info. controller-0 is r/stx.5.0 , "20210519T042950Z" . Is it possible ? @Miller, Frank<mailto:Frank.Miller@windriver.com> , @Little, Scott<mailto:Scott.Little@windriver.com> @Panech, Davlet<mailto:Davlet.Panech@windriver.com>: could this happen possibly ? or setup issue ? 2) This issue is because rabbitmq pod running on controller-1 fail. + exec rabbitmq-server Failed to create thread: Resource temporarily unavailable (11) Once changes, --pod-max-pids 750 to --pod-max-pids 2000 in /etc/sysconfig/kubelet on controller-1 . reapply openstack the rabbitmq pod running on controller-1 successfully . This change is related w/ [1] merged on 0518 maybe need increase some value for pid limited , the question is what value should be appropriate. [1] https://review.opendev.org/c/starlingx/stx-puppet/+/791267/3/ Thanks. BR Austin Sun. From: Dimofte, Alexandru <alexandru.dimofte@intel.com> Sent: Wednesday, May 19, 2021 10:52 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/202105...) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949 - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/202105... Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
If each controller is running a different load, then this would be a lab setup issue. @Dimofte, Alexandru<mailto:alexandru.dimofte@intel.com> can you please investigate this? We will wait for the resolution of this issue before declaring the 5.0 release as ready. Thanks, Ghada From: Sun, Austin <austin.sun@intel.com> Sent: Wednesday, May 19, 2021 10:14 PM To: Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io; Miller, Frank <Frank.Miller@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: Re: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z [Please note: This e-mail is from an EXTERNAL e-mail address] Hi All: Some Investigation found on LP #1928949: 1) Not sure why controller-1 build info is master info. controller-0 is r/stx.5.0 , "20210519T042950Z" . Is it possible ? @Miller, Frank<mailto:Frank.Miller@windriver.com> , @Little, Scott<mailto:Scott.Little@windriver.com> @Panech, Davlet<mailto:Davlet.Panech@windriver.com>: could this happen possibly ? or setup issue ? 2) This issue is because rabbitmq pod running on controller-1 fail. + exec rabbitmq-server Failed to create thread: Resource temporarily unavailable (11) Once changes, --pod-max-pids 750 to --pod-max-pids 2000 in /etc/sysconfig/kubelet on controller-1 . reapply openstack the rabbitmq pod running on controller-1 successfully . This change is related w/ [1] merged on 0518 maybe need increase some value for pid limited , the question is what value should be appropriate. [1] https://review.opendev.org/c/starlingx/stx-puppet/+/791267/3/<https://urldefense.com/v3/__https:/review.opendev.org/c/starlingx/stx-puppet/*/791267/3/__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLtcRIzyc$> Thanks. BR Austin Sun. From: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>> Sent: Wednesday, May 19, 2021 10:52 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLri9xlkK$>) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949<https://urldefense.com/v3/__https:/bugs.launchpad.net/starlingx/*bug/1928949__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLoaBS07J$> - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLryQ0UxK$> Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
For #1 I suggest you check if controller-1 is booting from disk instead of installing from controller-1 - might be a BIOS boot order issue? For #2, is the rabbitmq pod creating 750 processes? That's surprising but as we just use the rabbitmq software from the OpenStack project I suspect Austin you do not know. In order to determine a more appropriate number someone needs to see how many pids are used by the rabbitmq process. Can you check the pod Austin and determine this number? Then normal practice would be to double this number at a minimum. Also the commit you reference is not in the r/stx.5.0 branch so why is this showing up in r/stx.5.0 sanity? Is this a result of controller-1 not running the right load? Was the rabbitmq pod being launched on controller-1? Frank From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: Thursday, May 20, 2021 2:54 PM To: Sun, Austin <austin.sun@intel.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io; Miller, Frank <Frank.Miller@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: RE: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z If each controller is running a different load, then this would be a lab setup issue. @Dimofte, Alexandru<mailto:alexandru.dimofte@intel.com> can you please investigate this? We will wait for the resolution of this issue before declaring the 5.0 release as ready. Thanks, Ghada From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Wednesday, May 19, 2021 10:14 PM To: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z [Please note: This e-mail is from an EXTERNAL e-mail address] Hi All: Some Investigation found on LP #1928949: 1) Not sure why controller-1 build info is master info. controller-0 is r/stx.5.0 , "20210519T042950Z" . Is it possible ? @Miller, Frank<mailto:Frank.Miller@windriver.com> , @Little, Scott<mailto:Scott.Little@windriver.com> @Panech, Davlet<mailto:Davlet.Panech@windriver.com>: could this happen possibly ? or setup issue ? 2) This issue is because rabbitmq pod running on controller-1 fail. + exec rabbitmq-server Failed to create thread: Resource temporarily unavailable (11) Once changes, --pod-max-pids 750 to --pod-max-pids 2000 in /etc/sysconfig/kubelet on controller-1 . reapply openstack the rabbitmq pod running on controller-1 successfully . This change is related w/ [1] merged on 0518 maybe need increase some value for pid limited , the question is what value should be appropriate. [1] https://review.opendev.org/c/starlingx/stx-puppet/+/791267/3/<https://urldefense.com/v3/__https:/review.opendev.org/c/starlingx/stx-puppet/*/791267/3/__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLtcRIzyc$> Thanks. BR Austin Sun. From: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>> Sent: Wednesday, May 19, 2021 10:52 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLri9xlkK$>) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949<https://urldefense.com/v3/__https:/bugs.launchpad.net/starlingx/*bug/1928949__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLoaBS07J$> - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLryQ0UxK$> Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
Dan and I had a discussion on issue #2 with rabbitmq and we think the best solution is to set the default --pod-max-pids to the maximum number. Then a user can decide to tune this number to something smaller based on the app(s) they need. Dan will make this change on Friday. Frank From: Miller, Frank <Frank.Miller@windriver.com> Sent: Thursday, May 20, 2021 3:03 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Sun, Austin <austin.sun@intel.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: Re: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z For #1 I suggest you check if controller-1 is booting from disk instead of installing from controller-1 - might be a BIOS boot order issue? For #2, is the rabbitmq pod creating 750 processes? That's surprising but as we just use the rabbitmq software from the OpenStack project I suspect Austin you do not know. In order to determine a more appropriate number someone needs to see how many pids are used by the rabbitmq process. Can you check the pod Austin and determine this number? Then normal practice would be to double this number at a minimum. Also the commit you reference is not in the r/stx.5.0 branch so why is this showing up in r/stx.5.0 sanity? Is this a result of controller-1 not running the right load? Was the rabbitmq pod being launched on controller-1? Frank From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: Thursday, May 20, 2021 2:54 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: RE: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z If each controller is running a different load, then this would be a lab setup issue. @Dimofte, Alexandru<mailto:alexandru.dimofte@intel.com> can you please investigate this? We will wait for the resolution of this issue before declaring the 5.0 release as ready. Thanks, Ghada From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Wednesday, May 19, 2021 10:14 PM To: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z [Please note: This e-mail is from an EXTERNAL e-mail address] Hi All: Some Investigation found on LP #1928949: 1) Not sure why controller-1 build info is master info. controller-0 is r/stx.5.0 , "20210519T042950Z" . Is it possible ? @Miller, Frank<mailto:Frank.Miller@windriver.com> , @Little, Scott<mailto:Scott.Little@windriver.com> @Panech, Davlet<mailto:Davlet.Panech@windriver.com>: could this happen possibly ? or setup issue ? 2) This issue is because rabbitmq pod running on controller-1 fail. + exec rabbitmq-server Failed to create thread: Resource temporarily unavailable (11) Once changes, --pod-max-pids 750 to --pod-max-pids 2000 in /etc/sysconfig/kubelet on controller-1 . reapply openstack the rabbitmq pod running on controller-1 successfully . This change is related w/ [1] merged on 0518 maybe need increase some value for pid limited , the question is what value should be appropriate. [1] https://review.opendev.org/c/starlingx/stx-puppet/+/791267/3/<https://urldefense.com/v3/__https:/review.opendev.org/c/starlingx/stx-puppet/*/791267/3/__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLtcRIzyc$> Thanks. BR Austin Sun. From: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>> Sent: Wednesday, May 19, 2021 10:52 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLri9xlkK$>) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949<https://urldefense.com/v3/__https:/bugs.launchpad.net/starlingx/*bug/1928949__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLoaBS07J$> - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLryQ0UxK$> Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
Hi Frank, Thank you for the hint. Your assumption was right! There was a wrong bios boot order. BR, Alex From: Miller, Frank <Frank.Miller@windriver.com> Sent: Thursday, May 20, 2021 10:03 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Sun, Austin <austin.sun@intel.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: RE: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z For #1 I suggest you check if controller-1 is booting from disk instead of installing from controller-1 - might be a BIOS boot order issue? For #2, is the rabbitmq pod creating 750 processes? That's surprising but as we just use the rabbitmq software from the OpenStack project I suspect Austin you do not know. In order to determine a more appropriate number someone needs to see how many pids are used by the rabbitmq process. Can you check the pod Austin and determine this number? Then normal practice would be to double this number at a minimum. Also the commit you reference is not in the r/stx.5.0 branch so why is this showing up in r/stx.5.0 sanity? Is this a result of controller-1 not running the right load? Was the rabbitmq pod being launched on controller-1? Frank From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: Thursday, May 20, 2021 2:54 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: RE: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z If each controller is running a different load, then this would be a lab setup issue. @Dimofte, Alexandru<mailto:alexandru.dimofte@intel.com> can you please investigate this? We will wait for the resolution of this issue before declaring the 5.0 release as ready. Thanks, Ghada From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Wednesday, May 19, 2021 10:14 PM To: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z [Please note: This e-mail is from an EXTERNAL e-mail address] Hi All: Some Investigation found on LP #1928949: 1) Not sure why controller-1 build info is master info. controller-0 is r/stx.5.0 , "20210519T042950Z" . Is it possible ? @Miller, Frank<mailto:Frank.Miller@windriver.com> , @Little, Scott<mailto:Scott.Little@windriver.com> @Panech, Davlet<mailto:Davlet.Panech@windriver.com>: could this happen possibly ? or setup issue ? 2) This issue is because rabbitmq pod running on controller-1 fail. + exec rabbitmq-server Failed to create thread: Resource temporarily unavailable (11) Once changes, --pod-max-pids 750 to --pod-max-pids 2000 in /etc/sysconfig/kubelet on controller-1 . reapply openstack the rabbitmq pod running on controller-1 successfully . This change is related w/ [1] merged on 0518 maybe need increase some value for pid limited , the question is what value should be appropriate. [1] https://review.opendev.org/c/starlingx/stx-puppet/+/791267/3/<https://urldefense.com/v3/__https:/review.opendev.org/c/starlingx/stx-puppet/*/791267/3/__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLtcRIzyc$> Thanks. BR Austin Sun. From: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>> Sent: Wednesday, May 19, 2021 10:52 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLri9xlkK$>) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949<https://urldefense.com/v3/__https:/bugs.launchpad.net/starlingx/*bug/1928949__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLoaBS07J$> - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLryQ0UxK$> Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
Hello Ghada, We investigated this issue, we changed the boot order on controller-1(for standard_ext on setup2 which was the affected configuration). Now we have the right load on both controllers, the sanity report will come later for RC5.0. Master image is still affected by https://bugs.launchpad.net/starlingx/+bug/1928949 Kind regards, Alex From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: Thursday, May 20, 2021 9:54 PM To: Sun, Austin <austin.sun@intel.com>; Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io; Miller, Frank <Frank.Miller@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: RE: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z If each controller is running a different load, then this would be a lab setup issue. @Dimofte, Alexandru<mailto:alexandru.dimofte@intel.com> can you please investigate this? We will wait for the resolution of this issue before declaring the 5.0 release as ready. Thanks, Ghada From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Wednesday, May 19, 2021 10:14 PM To: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z [Please note: This e-mail is from an EXTERNAL e-mail address] Hi All: Some Investigation found on LP #1928949: 1) Not sure why controller-1 build info is master info. controller-0 is r/stx.5.0 , "20210519T042950Z" . Is it possible ? @Miller, Frank<mailto:Frank.Miller@windriver.com> , @Little, Scott<mailto:Scott.Little@windriver.com> @Panech, Davlet<mailto:Davlet.Panech@windriver.com>: could this happen possibly ? or setup issue ? 2) This issue is because rabbitmq pod running on controller-1 fail. + exec rabbitmq-server Failed to create thread: Resource temporarily unavailable (11) Once changes, --pod-max-pids 750 to --pod-max-pids 2000 in /etc/sysconfig/kubelet on controller-1 . reapply openstack the rabbitmq pod running on controller-1 successfully . This change is related w/ [1] merged on 0518 maybe need increase some value for pid limited , the question is what value should be appropriate. [1] https://review.opendev.org/c/starlingx/stx-puppet/+/791267/3/<https://urldefense.com/v3/__https:/review.opendev.org/c/starlingx/stx-puppet/*/791267/3/__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLtcRIzyc$> Thanks. BR Austin Sun. From: Dimofte, Alexandru <alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com>> Sent: Wednesday, May 19, 2021 10:52 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity RC 5.0 Test LAYERED build ISO 20210519T010439Z Sanity Test from 2021-May-19 (http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/iso/__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLri9xlkK$>) Status: RED This might be related to: https://bugs.launchpad.net/starlingx/+bug/1928949<https://urldefense.com/v3/__https:/bugs.launchpad.net/starlingx/*bug/1928949__;Kw!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLoaBS07J$> - stx-openstack apply-failed because of osh-openstack-rabbitmq install issue Helm-Chart used: http://mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz<https://urldefense.com/v3/__http:/mirror.starlingx.cengn.ca/mirror/starlingx/rc/5.0/centos/flock/20210519T010439Z/outputs/helm-charts/helm-charts-stx-openstack-centos-stable-versioned.tgz__;!!AjveYdw8EvQ!O0bB7OtiCCPZIDipG-T_lalpmSeoNwF_PwryeAYLmGdp0heiK3ZEgweneLoTLryQ0UxK$> Regards, Alexandru Dimofte [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
participants (4)
-
Dimofte, Alexandru
-
Khalil, Ghada
-
Miller, Frank
-
Sun, Austin