[Starlingx-discuss] Request freeze on commits

Tue May 12 16:49:20 UTC 2020

Saul:

This isn't a sanity process issue but a build issue.  The layered builds are not re-building the openstack helm charts for each build.  The current build should have them since this build will build an ISO as well as docker images and helm charts.  But Don is looking into why the openstack helm charts aren't being generated for every build since Scott is out this week.

Frank

-----Original Message-----
From: Saul Wold <sgw at linux.intel.com> 
Sent: Tuesday, May 12, 2020 12:37 PM
To: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Request freeze on commits

On 5/12/20 9:35 AM, Jascanu, Nicolae wrote:
> The LAYERED build 20200511T230211Z  used the helm chart: 
> http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/contai
> ners/20200505T204039Z/outputs/helm-charts/helm-charts-stx-openstack-ce
> ntos-stable-versioned.tgz
> 
Please reconfirm the link!  It shows 20200505T204039Z, with is from May 5th, not the May 11th build!

Sau!

> 
> 
> Regards,
> Nicolae Jascanu, Ph.D.
> TSD Engineering Manager
> 
> 
> 
> Internet Of Things Group
> Galati, Romania
> 
> -----Original Message-----
> From: Miller, Frank <Frank.Miller at windriver.com>
> Sent: Tuesday, May 12, 2020 19:27
> To: Jascanu, Nicolae <nicolae.jascanu at intel.com>; Saul Wold 
> <sgw at linux.intel.com>; starlingx-discuss 
> <starlingx-discuss at lists.starlingx.io>
> Subject: RE: [Starlingx-discuss] Request freeze on commits
> 
> Nicolae:
> 
> Where are the helm-charts for sanity coming from? Can you provide the link?
> 
> When one of our developers did their own flock layer build last night and installed their lab they were able to provision stx-openstack.  So the helm-chart provided by a developer build is correctly picking up Sabeel's commit.
> 
> Frank
> 
> 
> -----Original Message-----
> From: Jascanu, Nicolae <nicolae.jascanu at intel.com>
> Sent: Tuesday, May 12, 2020 12:20 PM
> To: Miller, Frank <Frank.Miller at windriver.com>; Saul Wold 
> <sgw at linux.intel.com>; starlingx-discuss 
> <starlingx-discuss at lists.starlingx.io>
> Subject: RE: [Starlingx-discuss] Request freeze on commits
> 
> I will start the build when will become available.
> The layered build 20200511T230211Z failed with the same error. I will 
> send the report in a few minutes
> 
> 
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada [-] Chart deploy [kube-system-ingress] failed: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: osh-kube-system-ingress - Tiller Message: b'Release "osh-kube-system-ingress" failed: timed out waiting for the condition'
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada Traceback (most recent call last):
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 473, in install_release
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada     metadata=self.metadata)
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada     return _end_unary_response_blocking(state, call, False, None)
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada     raise _Rendezvous(state, None, None, deadline)
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada      status = StatusCode.UNKNOWN
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada      details = "release osh-kube-system-ingress failed: timed out waiting for the condition"
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada      debug_error_string = "{"created":"@1589262912.237563359","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-kube-system-ingress failed: timed out waiting for the condition","grpc_status":2}"
> 2020-05-12 05:55:12.400 36214 ERROR armada.handlers.armada >
> 
> Regards,
> Nicolae Jascanu, Ph.D.
> TSD Engineering Manager
> 
> 
> 
> Internet Of Things Group
> Galati, Romania
> 
> -----Original Message-----
> From: Miller, Frank <Frank.Miller at windriver.com>
> Sent: Tuesday, May 12, 2020 18:55
> To: Saul Wold <sgw at linux.intel.com>; starlingx-discuss 
> <starlingx-discuss at lists.starlingx.io>
> Subject: Re: [Starlingx-discuss] Request freeze on commits
> 
> A new build was started 30 minutes ago by Davlet to pick up a fix for the docker image build [1].  Nicolae would it be possible to schedule another sanity today when this build is available?
> 
> Once the sanity results from earlier today are available, we need to determine if the root cause is the same root cause as before [2].  If not we will need additional fixes/plan to address today's sanity error.
> 
> In parallel, Davlet is working on getting the monolithic build to generate helm charts and docker images so we have a backup plan if issues continue to be seen that can be mapped back to build layering.
> 
> Frank
> [1] https://review.opendev.org/#/c/727218/
> [2] https://bugs.launchpad.net/starlingx/+bug/1877548
> 
> 
> -----Original Message-----
> From: Saul Wold <sgw at linux.intel.com>
> Sent: Tuesday, May 12, 2020 11:21 AM
> To: Miller, Frank <Frank.Miller at windriver.com>; starlingx-discuss 
> <starlingx-discuss at lists.starlingx.io>
> Subject: Re: [Starlingx-discuss] Request freeze on commits
> 
> +100!!!!
> 
> I was about to write this same email, Sanity testing is still failing from the last night's load with the Layered build (email will come after Test meeting).
> 
> So the fix that was applied yesterday for the helm chart broke the build?  Or was there some other commit?  I thought we had already declared a full stop last week except for required fixes to address the Sanity failures only.
> 
> Are you seeing the same failure with the helm charts as the Intel test team?
> 
> Is there a way to get the Monolithic helm charts build independent of the ISO? I think this is what Austin was getting at.
> 
> Sau!
> 
> 
> On 5/12/20 8:06 AM, Miller, Frank wrote:
>> StarlingX Community:
>>
>> We have struggled for the past 2+ weeks to get a sane load.  At this 
>> point I have no choice but to ask all Cores to freeze commits and not 
>> allow any commits to merge unless they address a build or sanity issue.
>>
>> Last night's docker image build failed due to a recent commit and the 
>> developer is providing a fix.  But we also have issues prior to this 
>> in sanity due to the build layering method pulling in the wrong 
>> version of a package.  If the current fixes do not result in sanity 
>> passing then I'll ask the build team to revert to the monolithic 
>> build until a better solution can be found.
>>
>> **
>>
>> Frank
>>
>> Project Lead for Build
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>
> 
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> 

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss