While the lab is still being installed, I check the collected logs. >From system inventory's point of view, all nodes provisioned successfully. The impact of my commits ends after controller-0 being provisioned (unlocked successfully the 1s time).
As Bob brought up in the comment, the port 44134 was taken by other process (as client port). it could be the rage setting of ephemeral port incorrect. I check a couple of our existing lab, I got:
cat /proc/sys/net/ipv4/ip_local_port_range
49216   61000
which excludes target port 44134 from being randomly assigned as client port.

I think it might be a good start that someone have access to the labs that failed the sanity to check the same settings?

Thanks,

Bin


From: Qian, Bin [Bin.Qian@windriver.com]
Sent: Tuesday, November 12, 2019 8:26 AM
To: Jones, Bruce E; Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada
Cc: Eslimi, Dariush
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

The 2 commits are inter related, taking one out will fail. I am trying to reproduce the issue. Not sure what exactly causes the issue yet. The commits work no difference on virtual or baremetal. 

Bin

From: Jones, Bruce E [bruce.e.jones@intel.com]
Sent: Tuesday, November 12, 2019 7:54 AM
To: Khalil, Ghada; Sun, Austin; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada; Qian, Bin
Cc: Eslimi, Dariush
Subject: RE: [Starlingx-discuss] Sanity Test - ISO 20191111

Is it possible to bisect the two suspicious commits, or otherwise figure out which parts of the commits are the root cause?

 

      brucej

 

From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
Sent: Tuesday, November 12, 2019 7:26 AM
To: Sun, Austin <austin.sun@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io; Cabrales, Ada <ada.cabrales@intel.com>; Qian, Bin <Bin.Qian@windriver.com>
Cc: Eslimi, Dariush <Dariush.Eslimi@windriver.com>
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

 

Adding Bin explicitly to the email thread

 

From: Sun, Austin [mailto:austin.sun@intel.com]
Sent: Tuesday, November 12, 2019 4:02 AM
To: Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io; Cabrales, Ada
Subject: Re: [Starlingx-discuss] Sanity Test - ISO 20191111

 

Hi All:

     About this https://bugs.launchpad.net/starlingx/+bug/1851533, after triage ,

this issue should be caused by https://review.opendev.org/#/c/691714/ and https://review.opendev.org/#/c/692439

after revert this commit , AIO on virtual environment can be deployed successfully

@Cabrales, Ada and Maria :  would you like to verify this by EB which reverts 691714/692439 ?

 EB is available on  http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/314/outputs

 This change 691714 will cause https://bugs.launchpad.net/starlingx/+bug/1852155 issue too.

 

@Qianbin :  would you double check these two issues caused by 691714/692439 gerrit ?

 

Thanks.

BR

Austin Sun.

 

From: Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>
Sent: Tuesday, November 12, 2019 1:17 AM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] Sanity Test - ISO 20191111

 

Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-November-11 (link)

 

Status: RED

 

===========================================

 

Sanity Test is executed in a Containers – Bare Metal Environment

 

AIO – Simplex

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [FAIL]

Sanity OpenStack  49 TCs [BLOCKED]

Sanity Platform   07 TCs [BLOCKED]

 

TOTAL: [ 61 TCs ]

 

AIO – Duplex

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]

 

TOTAL: [ 64 TCs ]

 

Standard - Local Storage (2+2)

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]

 

TOTAL: [ 65 TCs ]

 

Standard - Dedicated Storage (2+2+2)

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   09 TCs [PASS]

 

TOTAL: [ 66 TCs ]

 

===========================================

 

Sanity Test is executed in a Containers – Virtual Environment

 

AIO – Simplex

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]

 

TOTAL: [ 61 TCs ]

 

AIO – Duplex

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]

 

TOTAL: [ 64 TCs ]

 

Standard - Local Storage (2+2)

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]

 

TOTAL: [ 65 TCs ]

 

Standard - Dedicated Storage (2+2+2)

 

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   09 TCs [PASS]

 

TOTAL: [ 66 TCs ]

 

List of docker images : http://lists.starlingx.io/pipermail/starlingx-discuss/2019-November/006894.html

 

BUG

platform-integ-apps apply failed - tiller on Crash https://bugs.launchpad.net/starlingx/+bug/1851533

 

 

regards

Maria G