Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause? brucej From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, November 12, 2019 7:26 AM To: Sun, Austin <austin.sun@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io; Cabrales, Ada <ada.cabrales@intel.com>; Qian, Bin <Bin.Qian@windriver.com> Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Adding Bin explicitly to the email thread From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, November 12, 2019 4:02 AM To: Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi All: About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage , this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439 after revert this commit , AIO on virtual environment can be deployed successfully @Cabrales, Ada<mailto:ada.cabrales@intel.com> and Maria : would you like to verify this by EB which reverts 691714/692439 ? EB is available on http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too. @Qianbin : would you double check these two issues caused by 691714/692439 gerrit ? Thanks. BR Austin Sun. From: Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>> Sent: Tuesday, November 12, 2019 1:17 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity Test - ISO 20191111 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20191111T000000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 61 TCs ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.h... BUG platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533 regards Maria G

While the lab is still being installed, I check the collected logs. From system inventory's point of view, all nodes provisioned successfully. The impact of my commits ends after controller-0 being provisioned (unlocked successfully the 1s time). As Bob brought up in the comment, the port 44134 was taken by other process (as client port). it could be the rage setting of ephemeral port incorrect. I check a couple of our existing lab, I got: cat /proc/sys/net/ipv4/ip_local_port_range 49216 61000 which excludes target port 44134 from being randomly assigned as client port. I think it might be a good start that someone have access to the labs that failed the sanity to check the same settings? Thanks, Bin ________________________________ From: Qian, Bin [Bin.Qian@windriver.com] Sent: Tuesday, November 12, 2019 8:26 AM To: Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 The 2 commits are inter related, taking one out will fail. I am trying to reproduce the issue. Not sure what exactly causes the issue yet. The commits work no difference on virtual or baremetal. Bin ________________________________ From: Jones, Bruce E [bruce.e.jones@intel.com] Sent: Tuesday, November 12, 2019 7:54 AM To: Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada; Qian, Bin Cc: Eslimi, Dariush Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111 Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause? brucej From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, November 12, 2019 7:26 AM To: Sun, Austin <austin.sun@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io; Cabrales, Ada <ada.cabrales@intel.com>; Qian, Bin <Bin.Qian@windriver.com> Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Adding Bin explicitly to the email thread From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, November 12, 2019 4:02 AM To: Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi All: About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage , this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439 after revert this commit , AIO on virtual environment can be deployed successfully @Cabrales, Ada<mailto:ada.cabrales@intel.com> and Maria : would you like to verify this by EB which reverts 691714/692439 ? EB is available on http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too. @Qianbin : would you double check these two issues caused by 691714/692439 gerrit ? Thanks. BR Austin Sun. From: Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>> Sent: Tuesday, November 12, 2019 1:17 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity Test - ISO 20191111 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20191111T000000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers – Bare Metal Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] =========================================== Sanity Test is executed in a Containers – Virtual Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.h... BUG platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533 regards Maria G

Thanks Cristopher, I found the same on latest build as well. I further look into the sys config, I found that recently the config-files/initscripts-config/files/sysctl.conf was altered, and the following was removed. # Limit local port range net.ipv4.ip_local_port_range = 49216 61000 net.ipv4.tcp_tw_reuse = 1 The file is deployed to /etc/sysctl.conf, the setting is the range of local port. Probably some changes that messed up? Thanks, Bin ________________________________ From: Lemus Contreras, Cristopher J [cristopher.j.lemus.contreras@intel.com] Sent: Tuesday, November 12, 2019 9:56 AM To: Qian, Bin; Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi Bin, I checked on the configs that failed with today’s build (BUILD_ID="20191112T000000Z"), and I got this output on both: [sysadmin@controller-0 ~(keystone_admin)]$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 60999 Which I think, causes the conflict with port 44134. With this build, the two configs that failed are Standard with Dedicated Storage (2+2+2), virtual and baremetal environments. Please, let me know if you need further details. Thanks & Regards Cristopher Lemus From: "Qian, Bin" <Bin.Qian@windriver.com> Date: Tuesday, November 12, 2019 at 11:38 AM To: "Jones, Bruce E" <bruce.e.jones@intel.com>, "Khalil, Ghada" <Ghada.Khalil@windriver.com>, "Sun, Austin" <austin.sun@intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales@intel.com> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 While the lab is still being installed, I check the collected logs. From system inventory's point of view, all nodes provisioned successfully. The impact of my commits ends after controller-0 being provisioned (unlocked successfully the 1s time). As Bob brought up in the comment, the port 44134 was taken by other process (as client port). it could be the rage setting of ephemeral port incorrect. I check a couple of our existing lab, I got: cat /proc/sys/net/ipv4/ip_local_port_range 49216 61000 which excludes target port 44134 from being randomly assigned as client port. I think it might be a good start that someone have access to the labs that failed the sanity to check the same settings? Thanks, Bin ________________________________ From: Qian, Bin [Bin.Qian@windriver.com] Sent: Tuesday, November 12, 2019 8:26 AM To: Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 The 2 commits are inter related, taking one out will fail. I am trying to reproduce the issue. Not sure what exactly causes the issue yet. The commits work no difference on virtual or baremetal. Bin ________________________________ From: Jones, Bruce E [bruce.e.jones@intel.com] Sent: Tuesday, November 12, 2019 7:54 AM To: Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada; Qian, Bin Cc: Eslimi, Dariush Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111 Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause? brucej From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, November 12, 2019 7:26 AM To: Sun, Austin <austin.sun@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io; Cabrales, Ada <ada.cabrales@intel.com>; Qian, Bin <Bin.Qian@windriver.com> Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Adding Bin explicitly to the email thread From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, November 12, 2019 4:02 AM To: Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi All: About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage , this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439 after revert this commit , AIO on virtual environment can be deployed successfully @Cabrales, Ada<mailto:ada.cabrales@intel.com> and Maria : would you like to verify this by EB which reverts 691714/692439 ? EB is available on http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too. @Qianbin : would you double check these two issues caused by 691714/692439 gerrit ? Thanks. BR Austin Sun. From: Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>> Sent: Tuesday, November 12, 2019 1:17 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity Test - ISO 20191111 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20191111T000000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers – Bare Metal Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] =========================================== Sanity Test is executed in a Containers – Virtual Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.h... BUG platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533 regards Maria G

Hi Austin, An update about the EB that you sent us (http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs), in summary: * A full sanity was executed for all configs once. All TC passed. * An Standard with external storage was installed 3 times (because this is where the issue appeared more times), tiller pod was running fine. * Port range is also like this: controller-0:~$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 60999 If I understood, we had good luck the three times and tiller port was not in use. Please, let me know if you need additional details. Thanks & Regards Cristopher Lemus From: "Church, Robert" <Robert.Church@windriver.com> Date: Tuesday, November 12, 2019 at 12:15 PM To: "Qian, Bin" <Bin.Qian@windriver.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Khalil, Ghada" <Ghada.Khalil@windriver.com>, "Sun, Austin" <austin.sun@intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales@intel.com> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 I think this is commit: https://opendev.org/starlingx/config-files/commit/fbc09b8db8a14fbf24976a0f7d... Bob From: "Qian, Bin" <Bin.Qian@windriver.com> Date: Tuesday, November 12, 2019 at 12:12 PM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Khalil, Ghada" <Ghada.Khalil@windriver.com>, "Sun, Austin" <austin.sun@intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales@intel.com> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Thanks Cristopher, I found the same on latest build as well. I further look into the sys config, I found that recently the config-files/initscripts-config/files/sysctl.conf was altered, and the following was removed. # Limit local port range net.ipv4.ip_local_port_range = 49216 61000 net.ipv4.tcp_tw_reuse = 1 The file is deployed to /etc/sysctl.conf, the setting is the range of local port. Probably some changes that messed up? Thanks, Bin ________________________________ From: Lemus Contreras, Cristopher J [cristopher.j.lemus.contreras@intel.com] Sent: Tuesday, November 12, 2019 9:56 AM To: Qian, Bin; Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi Bin, I checked on the configs that failed with today’s build (BUILD_ID="20191112T000000Z"), and I got this output on both: [sysadmin@controller-0 ~(keystone_admin)]$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 60999 Which I think, causes the conflict with port 44134. With this build, the two configs that failed are Standard with Dedicated Storage (2+2+2), virtual and baremetal environments. Please, let me know if you need further details. Thanks & Regards Cristopher Lemus From: "Qian, Bin" <Bin.Qian@windriver.com> Date: Tuesday, November 12, 2019 at 11:38 AM To: "Jones, Bruce E" <bruce.e.jones@intel.com>, "Khalil, Ghada" <Ghada.Khalil@windriver.com>, "Sun, Austin" <austin.sun@intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales@intel.com> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 While the lab is still being installed, I check the collected logs. From system inventory's point of view, all nodes provisioned successfully. The impact of my commits ends after controller-0 being provisioned (unlocked successfully the 1s time). As Bob brought up in the comment, the port 44134 was taken by other process (as client port). it could be the rage setting of ephemeral port incorrect. I check a couple of our existing lab, I got: cat /proc/sys/net/ipv4/ip_local_port_range 49216 61000 which excludes target port 44134 from being randomly assigned as client port. I think it might be a good start that someone have access to the labs that failed the sanity to check the same settings? Thanks, Bin ________________________________ From: Qian, Bin [Bin.Qian@windriver.com] Sent: Tuesday, November 12, 2019 8:26 AM To: Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 The 2 commits are inter related, taking one out will fail. I am trying to reproduce the issue. Not sure what exactly causes the issue yet. The commits work no difference on virtual or baremetal. Bin ________________________________ From: Jones, Bruce E [bruce.e.jones@intel.com] Sent: Tuesday, November 12, 2019 7:54 AM To: Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada; Qian, Bin Cc: Eslimi, Dariush Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111 Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause? brucej From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, November 12, 2019 7:26 AM To: Sun, Austin <austin.sun@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io; Cabrales, Ada <ada.cabrales@intel.com>; Qian, Bin <Bin.Qian@windriver.com> Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Adding Bin explicitly to the email thread From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, November 12, 2019 4:02 AM To: Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi All: About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage , this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439 after revert this commit , AIO on virtual environment can be deployed successfully @Cabrales, Ada<mailto:ada.cabrales@intel.com> and Maria : would you like to verify this by EB which reverts 691714/692439 ? EB is available on http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too. @Qianbin : would you double check these two issues caused by 691714/692439 gerrit ? Thanks. BR Austin Sun. From: Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>> Sent: Tuesday, November 12, 2019 1:17 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity Test - ISO 20191111 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20191111T000000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers – Bare Metal Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] =========================================== Sanity Test is executed in a Containers – Virtual Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.h... BUG platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533 regards Maria G

Fix for this up for review: https://review.opendev.org/#/c/694355/. Regards, Bob From: "Sun, Austin" <austin.sun@intel.com> Date: Wednesday, November 13, 2019 at 2:54 AM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, Robert Church <Robert.Church@windriver.com>, "Qian, Bin" <Bin.Qian@windriver.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Khalil, Ghada" <Ghada.Khalil@windriver.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales@intel.com> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com> Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111 Thanks a lot Cristopher. The commit fbc09b8db8a14fbf24976a0f7d8924af7a330f85 was merged two months ago. LP# 1851533 was reported since 11/05, and never reported before 11/05. I cannot understand this part. Thanks. BR Austin Sun. From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, November 13, 2019 10:37 AM To: Church, Robert <Robert.Church@windriver.com>; Qian, Bin <Bin.Qian@windriver.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Sun, Austin <austin.sun@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io; Cabrales, Ada <ada.cabrales@intel.com> Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi Austin, An update about the EB that you sent us (http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs), in summary: * A full sanity was executed for all configs once. All TC passed. * An Standard with external storage was installed 3 times (because this is where the issue appeared more times), tiller pod was running fine. * Port range is also like this: controller-0:~$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 60999 If I understood, we had good luck the three times and tiller port was not in use. Please, let me know if you need additional details. Thanks & Regards Cristopher Lemus From: "Church, Robert" <Robert.Church@windriver.com<mailto:Robert.Church@windriver.com>> Date: Tuesday, November 12, 2019 at 12:15 PM To: "Qian, Bin" <Bin.Qian@windriver.com<mailto:Bin.Qian@windriver.com>>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Khalil, Ghada" <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Sun, Austin" <austin.sun@intel.com<mailto:austin.sun@intel.com>>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com<mailto:Dariush.Eslimi@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 I think this is commit: https://opendev.org/starlingx/config-files/commit/fbc09b8db8a14fbf24976a0f7d... Bob From: "Qian, Bin" <Bin.Qian@windriver.com<mailto:Bin.Qian@windriver.com>> Date: Tuesday, November 12, 2019 at 12:12 PM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Khalil, Ghada" <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Sun, Austin" <austin.sun@intel.com<mailto:austin.sun@intel.com>>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com<mailto:Dariush.Eslimi@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Thanks Cristopher, I found the same on latest build as well. I further look into the sys config, I found that recently the config-files/initscripts-config/files/sysctl.conf was altered, and the following was removed. # Limit local port range net.ipv4.ip_local_port_range = 49216 61000 net.ipv4.tcp_tw_reuse = 1 The file is deployed to /etc/sysctl.conf, the setting is the range of local port. Probably some changes that messed up? Thanks, Bin ________________________________ From: Lemus Contreras, Cristopher J [cristopher.j.lemus.contreras@intel.com] Sent: Tuesday, November 12, 2019 9:56 AM To: Qian, Bin; Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi Bin, I checked on the configs that failed with today’s build (BUILD_ID="20191112T000000Z"), and I got this output on both: [sysadmin@controller-0 ~(keystone_admin)]$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 60999 Which I think, causes the conflict with port 44134. With this build, the two configs that failed are Standard with Dedicated Storage (2+2+2), virtual and baremetal environments. Please, let me know if you need further details. Thanks & Regards Cristopher Lemus From: "Qian, Bin" <Bin.Qian@windriver.com<mailto:Bin.Qian@windriver.com>> Date: Tuesday, November 12, 2019 at 11:38 AM To: "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Khalil, Ghada" <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Sun, Austin" <austin.sun@intel.com<mailto:austin.sun@intel.com>>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: "Eslimi, Dariush" <Dariush.Eslimi@windriver.com<mailto:Dariush.Eslimi@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 While the lab is still being installed, I check the collected logs. From system inventory's point of view, all nodes provisioned successfully. The impact of my commits ends after controller-0 being provisioned (unlocked successfully the 1s time). As Bob brought up in the comment, the port 44134 was taken by other process (as client port). it could be the rage setting of ephemeral port incorrect. I check a couple of our existing lab, I got: cat /proc/sys/net/ipv4/ip_local_port_range 49216 61000 which excludes target port 44134 from being randomly assigned as client port. I think it might be a good start that someone have access to the labs that failed the sanity to check the same settings? Thanks, Bin ________________________________ From: Qian, Bin [Bin.Qian@windriver.com] Sent: Tuesday, November 12, 2019 8:26 AM To: Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Cc: Eslimi, Dariush Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 The 2 commits are inter related, taking one out will fail. I am trying to reproduce the issue. Not sure what exactly causes the issue yet. The commits work no difference on virtual or baremetal. Bin ________________________________ From: Jones, Bruce E [bruce.e.jones@intel.com] Sent: Tuesday, November 12, 2019 7:54 AM To: Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada; Qian, Bin Cc: Eslimi, Dariush Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111 Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause? brucej From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, November 12, 2019 7:26 AM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Qian, Bin <Bin.Qian@windriver.com<mailto:Bin.Qian@windriver.com>> Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com<mailto:Dariush.Eslimi@windriver.com>> Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Adding Bin explicitly to the email thread From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, November 12, 2019 4:02 AM To: Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Cabrales, Ada Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111 Hi All: About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage , this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439 after revert this commit , AIO on virtual environment can be deployed successfully @Cabrales, Ada<mailto:ada.cabrales@intel.com> and Maria : would you like to verify this by EB which reverts 691714/692439 ? EB is available on http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too. @Qianbin : would you double check these two issues caused by 691714/692439 gerrit ? Thanks. BR Austin Sun. From: Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com<mailto:maria.g.perez.ibarra@intel.com>> Sent: Tuesday, November 12, 2019 1:17 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Sanity Test - ISO 20191111 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20191111T000000Z/outputs/iso/>) Status: RED =========================================== Sanity Test is executed in a Containers – Bare Metal Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] =========================================== Sanity Test is executed in a Containers – Virtual Environment AIO – Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO – Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.h... BUG platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533 regards Maria G