[Starlingx-discuss] Sanity Test - ISO 20191111

Church, Robert Robert.Church at windriver.com
Tue Nov 12 18:15:54 UTC 2019


I think this is commit: https://opendev.org/starlingx/config-files/commit/fbc09b8db8a14fbf24976a0f7d8924af7a330f85

Bob

From: "Qian, Bin" <Bin.Qian at windriver.com>
Date: Tuesday, November 12, 2019 at 12:12 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>, "Jones, Bruce E" <bruce.e.jones at intel.com>, "Khalil, Ghada" <Ghada.Khalil at windriver.com>, "Sun, Austin" <austin.sun at intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com>, "starlingx-discuss at lists.starlingx.io" <starlingx-discuss at lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales at intel.com>
Cc: "Eslimi, Dariush" <Dariush.Eslimi at windriver.com>
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

Thanks Cristopher,

I found the same on latest build as well.

I further look into the sys config, I found that recently the config-files/initscripts-config/files/sysctl.conf was altered, and the following was removed.
# Limit local port range
net.ipv4.ip_local_port_range = 49216 61000
net.ipv4.tcp_tw_reuse = 1

The file is deployed to /etc/sysctl.conf, the setting is the range of local port.

Probably some changes that messed up?

Thanks,

Bin

________________________________
From: Lemus Contreras, Cristopher J [cristopher.j.lemus.contreras at intel.com]
Sent: Tuesday, November 12, 2019 9:56 AM
To: Qian, Bin; Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io; Cabrales, Ada
Cc: Eslimi, Dariush
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111
Hi Bin,

I checked on the configs that failed with today’s build (BUILD_ID="20191112T000000Z"), and I got this output on both:

[sysadmin at controller-0 ~(keystone_admin)]$ cat /proc/sys/net/ipv4/ip_local_port_range
32768    60999

Which I think, causes the conflict with port 44134.

With this build, the two configs that failed are Standard with Dedicated Storage (2+2+2), virtual and baremetal environments.

Please, let me know if you need further details.

Thanks & Regards

Cristopher Lemus

From: "Qian, Bin" <Bin.Qian at windriver.com>
Date: Tuesday, November 12, 2019 at 11:38 AM
To: "Jones, Bruce E" <bruce.e.jones at intel.com>, "Khalil, Ghada" <Ghada.Khalil at windriver.com>, "Sun, Austin" <austin.sun at intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com>, "starlingx-discuss at lists.starlingx.io" <starlingx-discuss at lists.starlingx.io>, "Cabrales, Ada" <ada.cabrales at intel.com>
Cc: "Eslimi, Dariush" <Dariush.Eslimi at windriver.com>
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

While the lab is still being installed, I check the collected logs. From system inventory's point of view, all nodes provisioned successfully. The impact of my commits ends after controller-0 being provisioned (unlocked successfully the 1s time).
As Bob brought up in the comment, the port 44134 was taken by other process (as client port). it could be the rage setting of ephemeral port incorrect. I check a couple of our existing lab, I got:
cat /proc/sys/net/ipv4/ip_local_port_range
49216   61000
which excludes target port 44134 from being randomly assigned as client port.

I think it might be a good start that someone have access to the labs that failed the sanity to check the same settings?

Thanks,

Bin

________________________________
From: Qian, Bin [Bin.Qian at windriver.com]
Sent: Tuesday, November 12, 2019 8:26 AM
To: Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io; Cabrales, Ada
Cc: Eslimi, Dariush
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111
The 2 commits are inter related, taking one out will fail. I am trying to reproduce the issue. Not sure what exactly causes the issue yet. The commits work no difference on virtual or baremetal.

Bin
________________________________
From: Jones, Bruce E [bruce.e.jones at intel.com]
Sent: Tuesday, November 12, 2019 7:54 AM
To: Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io; Cabrales, Ada; Qian, Bin
Cc: Eslimi, Dariush
Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111
Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause?

      brucej

From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Tuesday, November 12, 2019 7:26 AM
To: Sun, Austin <austin.sun at intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra at intel.com>; starlingx-discuss at lists.starlingx.io; Cabrales, Ada <ada.cabrales at intel.com>; Qian, Bin <Bin.Qian at windriver.com>
Cc: Eslimi, Dariush <Dariush.Eslimi at windriver.com>
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

Adding Bin explicitly to the email thread

From: Sun, Austin [mailto:austin.sun at intel.com]
Sent: Tuesday, November 12, 2019 4:02 AM
To: Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>; Cabrales, Ada
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

Hi All:
     About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage ,
this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439
after revert this commit , AIO on virtual environment can be deployed successfully
@Cabrales, Ada<mailto:ada.cabrales at intel.com> and Maria :  would you like to verify this by EB which reverts 691714/692439 ?
 EB is available on  http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs
 This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too.

@Qianbin :  would you double check these two issues caused by 691714/692439 gerrit ?

Thanks.
BR
Austin Sun.

From: Perez Ibarra, Maria G <maria.g.perez.ibarra at intel.com<mailto:maria.g.perez.ibarra at intel.com>>
Sent: Tuesday, November 12, 2019 1:17 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Sanity Test - ISO 20191111


Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20191111T000000Z/outputs/iso/>)


Status: RED



===========================================



Sanity Test is executed in a Containers – Bare Metal Environment


AIO – Simplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [FAIL]

Sanity OpenStack  49 TCs [BLOCKED]

Sanity Platform   07 TCs [BLOCKED]



TOTAL: [ 61 TCs ]


AIO – Duplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]



TOTAL: [ 64 TCs ]



Standard - Local Storage (2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]



TOTAL: [ 65 TCs ]



Standard - Dedicated Storage (2+2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   09 TCs [PASS]



TOTAL: [ 66 TCs ]


===========================================



Sanity Test is executed in a Containers – Virtual Environment


AIO – Simplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]



TOTAL: [ 61 TCs ]


AIO – Duplex



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]



TOTAL: [ 64 TCs ]



Standard - Local Storage (2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]



TOTAL: [ 65 TCs ]



Standard - Dedicated Storage (2+2+2)



Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   09 TCs [PASS]



TOTAL: [ 66 TCs ]

List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.html

BUG
platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533


regards
Maria G


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20191112/ac9fb113/attachment-0001.html>


More information about the Starlingx-discuss mailing list