Nicolae please go ahead with a new sanity run. Frank -----Original Message----- From: Penney, Don <Don.Penney@windriver.com> Sent: Tuesday, May 12, 2020 2:09 PM To: Jascanu, Nicolae <nicolae.jascanu@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Request freeze on commits The CENGN flock build was skipping the stx-openstack helm charts due to a script issue. You can now find the charts here: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/latest... -----Original Message----- From: Jascanu, Nicolae [mailto:nicolae.jascanu@intel.com] Sent: Tuesday, May 12, 2020 1:04 PM To: Miller, Frank; Saul Wold; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Request freeze on commits @Saul Wold - I confirm that links. When the sanity started that helm-chart was the latest available Regards, Nicolae Jascanu, Ph.D. TSD Engineering Manager Internet Of Things Group Galati, Romania -----Original Message----- From: Miller, Frank <Frank.Miller@windriver.com> Sent: Tuesday, May 12, 2020 19:49 To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Request freeze on commits Saul: This isn't a sanity process issue but a build issue. The layered builds are not re-building the openstack helm charts for each build. The current build should have them since this build will build an ISO as well as docker images and helm charts. But Don is looking into why the openstack helm charts aren't being generated for every build since Scott is out this week. Frank -----Original Message----- From: Saul Wold <sgw@linux.intel.com> Sent: Tuesday, May 12, 2020 12:37 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Request freeze on commits On 5/12/20 9:35 AM, Jascanu, Nicolae wrote:
The LAYERED build 20200511T230211Z used the helm chart: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/contai ners/20200505T204039Z/outputs/helm-charts/helm-charts-stx-openstack-ce ntos-stable-versioned.tgz
Please reconfirm the link! It shows 20200505T204039Z, with is from May 5th, not the May 11th build! Sau!
Regards, Nicolae Jascanu, Ph.D. TSD Engineering Manager
Internet Of Things Group Galati, Romania
-----Original Message----- From: Miller, Frank <Frank.Miller@windriver.com> Sent: Tuesday, May 12, 2020 19:27 To: Jascanu, Nicolae <nicolae.jascanu@intel.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] Request freeze on commits
Nicolae:
Where are the helm-charts for sanity coming from? Can you provide the link?
When one of our developers did their own flock layer build last night and installed their lab they were able to provision stx-openstack. So the helm-chart provided by a developer build is correctly picking up Sabeel's commit.
Frank
-----Original Message----- From: Jascanu, Nicolae <nicolae.jascanu@intel.com> Sent: Tuesday, May 12, 2020 12:20 PM To: Miller, Frank <Frank.Miller@windriver.com>; Saul Wold <sgw@linux.intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] Request freeze on commits
I will start the build when will become available. The layered build 20200511T230211Z failed with the same error. I will send the report in a few minutes
2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada [-] Chart deploy [kube-system-ingress] failed: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-kube-system-ingress - Tiller Message: b'Release "osh-kube-system-ingress" failed: timed out waiting for the condition' 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada Traceback (most recent call last): 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 473, in install_release 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada metadata=self.metadata) 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__ 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada return _end_unary_response_blocking(state, call, False, None) 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada raise _Rendezvous(state, None, None, deadline) 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada status = StatusCode.UNKNOWN 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada details = "release osh-kube-system-ingress failed: timed out waiting for the condition" 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada debug_error_string = "{"created":"@1589262912.237563359","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-kube-system-ingress failed: timed out waiting for the condition","grpc_status":2}" 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada >
Regards, Nicolae Jascanu, Ph.D. TSD Engineering Manager
Internet Of Things Group Galati, Romania
-----Original Message----- From: Miller, Frank <Frank.Miller@windriver.com> Sent: Tuesday, May 12, 2020 18:55 To: Saul Wold <sgw@linux.intel.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Request freeze on commits
A new build was started 30 minutes ago by Davlet to pick up a fix for the docker image build [1]. Nicolae would it be possible to schedule another sanity today when this build is available?
Once the sanity results from earlier today are available, we need to determine if the root cause is the same root cause as before [2]. If not we will need additional fixes/plan to address today's sanity error.
In parallel, Davlet is working on getting the monolithic build to generate helm charts and docker images so we have a backup plan if issues continue to be seen that can be mapped back to build layering.
Frank [1] https://review.opendev.org/#/c/727218/ [2] https://bugs.launchpad.net/starlingx/+bug/1877548
-----Original Message----- From: Saul Wold <sgw@linux.intel.com> Sent: Tuesday, May 12, 2020 11:21 AM To: Miller, Frank <Frank.Miller@windriver.com>; starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Request freeze on commits
+100!!!!
I was about to write this same email, Sanity testing is still failing from the last night's load with the Layered build (email will come after Test meeting).
So the fix that was applied yesterday for the helm chart broke the build? Or was there some other commit? I thought we had already declared a full stop last week except for required fixes to address the Sanity failures only.
Are you seeing the same failure with the helm charts as the Intel test team?
Is there a way to get the Monolithic helm charts build independent of the ISO? I think this is what Austin was getting at.
Sau!
On 5/12/20 8:06 AM, Miller, Frank wrote:
StarlingX Community:
We have struggled for the past 2+ weeks to get a sane load. At this point I have no choice but to ask all Cores to freeze commits and not allow any commits to merge unless they address a build or sanity issue.
Last night's docker image build failed due to a recent commit and the developer is providing a fix. But we also have issues prior to this in sanity due to the build layering method pulling in the wrong version of a package. If the current fixes do not result in sanity passing then I'll ask the build team to revert to the monolithic build until a better solution can be found.
**
Frank
Project Lead for Build
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss