[Starlingx-discuss] FM containerization Build issues
HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
HI Frank/Tyler/Scott
As discussed in the containerization meeting sharing the build issues details need your inputs :-
* *Issue 1*:- FM with Mario’s changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . o Please let us know where else to look for to get more details of the issue o Need your inputs to move forward * *Issue 2*:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * *Issue 3:-* Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke...
We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt)
I have attached the logs and the docker scripts . Please let us know if any further details are needed.
Thanks and Regards,
Sanjay Mukherjee
Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Smith, Tyler <tyler.smith@windriver.com> Cc: Kumar, Sharath <sharath.kumar@intel.com>; Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Smith, Tyler <tyler.smith@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image003.png@01D6A888.7BFC76D0] 1. Horizon pod got crashed. [cid:image005.png@01D6A883.D645D050] 1. Horizon pod description. [cid:image004.png@01D6A884.35A14840] 1. Pod log command gave below errors. [cid:image006.png@01D6A884.F753C4E0] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Attaching both logs. From: Kumar, Sharath Sent: Thursday, October 22, 2020 3:32 PM To: Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image001.png@01D6A888.DA4964F0] 1. Horizon pod got crashed. [cid:image002.png@01D6A888.DA4964F0] 1. Horizon pod description. [cid:image003.png@01D6A888.DA4964F0] 1. Pod log command gave below errors. [cid:image004.png@01D6A888.DA4964F0] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/ horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/ The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta... It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change. You may be able to login to a horizon container and determine what version of the code it is using. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com> Sent: Thursday, October 22, 2020 6:02 AM To: Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image003.png@01D6A888.7BFC76D0] 1. Horizon pod got crashed. [cid:image005.png@01D6A883.D645D050] 1. Horizon pod description. [cid:image004.png@01D6A884.35A14840] 1. Pod log command gave below errors. [cid:image006.png@01D6A884.F753C4E0] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there’s nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario’s changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Hi Henry, This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod. Sharing some of the screenshots with you. 1. stx-horizon.stable_docker_image file configurations. [cid:image005.png@01D6B1ED.734B5290] 1. Docker image build script. [cid:image007.png@01D6B1EF.ED1E89F0] 1. Helm chart applied successfully. [cid:image012.png@01D6B1EF.ED1E89F0] 1. Horizon pods are running successfully. [cid:image011.png@01D6B1ED.DC8FDEB0] 1. Pods are showing some warnings. [cid:image014.png@01D6B1EF.ED1E89F0] My questions: 1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application's stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings? Please let me know if you want to setup working session with me, thank you for your support and help. Regard's, Sharath From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Sent: Thursday, October 22, 2020 7:42 PM To: Kumar, Sharath <sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: FM containerization Build issues The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/ horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/ The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta... It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change. You may be able to login to a horizon container and determine what version of the code it is using. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: Thursday, October 22, 2020 6:02 AM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image001.png@01D6B1EC.FF2A5190] 1. Horizon pod got crashed. [cid:image002.png@01D6B1EC.FF2A5190] 1. Horizon pod description. [cid:image003.png@01D6B1EC.FF2A5190] 1. Pod log command gave below errors. [cid:image004.png@01D6B1EC.FF2A5190] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
I don't tend to spend much time working in containers, but my understanding for your questions * you can set overrrides for helm values https://docs.starlingx.io/operations/k8s_cluster.html * I know that the ussiri target explicitly sets Python3, which is different behaviour within Loci (the tools used to build it). I don't really know much more about the internals. As far as I can tell, you are following the instructions properly that are outlined at https://docs.starlingx.io/developer_resources/build_docker_image.html * Sometimes these docker images have a directive which is to not pull/update if a version already exists. Maybe double check if the new images you are building, are actually being pulled and used by this test procedure. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com> Sent: Tuesday, November 3, 2020 4:15 AM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues [Please note this e-mail is from an EXTERNAL e-mail address] Hi Henry, This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod. Sharing some of the screenshots with you. 1. stx-horizon.stable_docker_image file configurations. [cid:image005.png@01D6B1ED.734B5290] 1. Docker image build script. [cid:image007.png@01D6B1EF.ED1E89F0] 1. Helm chart applied successfully. [cid:image012.png@01D6B1EF.ED1E89F0] 1. Horizon pods are running successfully. [cid:image011.png@01D6B1ED.DC8FDEB0] 1. Pods are showing some warnings. [cid:image014.png@01D6B1EF.ED1E89F0] My questions: 1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application’s stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings? Please let me know if you want to setup working session with me, thank you for your support and help. Regard’s, Sharath From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> Sent: Thursday, October 22, 2020 7:42 PM To: Kumar, Sharath <sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: FM containerization Build issues The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/ horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/ The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta... It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change. You may be able to login to a horizon container and determine what version of the code it is using. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: Thursday, October 22, 2020 6:02 AM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image001.png@01D6B1EC.FF2A5190] 1. Horizon pod got crashed. [cid:image002.png@01D6B1EC.FF2A5190] 1. Horizon pod description. [cid:image003.png@01D6B1EC.FF2A5190] 1. Pod log command gave below errors. [cid:image004.png@01D6B1EC.FF2A5190] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there’s nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario’s changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
RE: question 2, From the failed ussuri build, Can you provide the stx-horizon.stable_docker_image, and the resultant ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log from the failed build Also did you run build-stx-images.sh with the the --clean option, or manually delete any pre-existing docker images, before switching your PROJECT_REF? Again the concern, as Al mentioned, is docker retaining cached 'layers' content from a previous build, especially if the build is on another branch. e.g. you might want to display the layers with 'docker history <image>' for the master vs the ussuri build, looking for shared layer uuid's. Ideally the uuid's are unique for all layers. Scott On 2020-11-03 4:15 a.m., Kumar, Sharath wrote:
**[Please note this e-mail is from an EXTERNAL e-mail address]
Hi Henry,
This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod.
Sharing some of the screenshots with you.
1. stx-horizon.stable_docker_image file configurations.
2. Docker image build script.
3. Helm chart applied successfully.
4. Horizon pods are running successfully.
5. Pods are showing some warnings.
My questions:
1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application’s stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings?
Please let me know if you want to setup working session with me, thank you for your support and help.
Regard’s, Sharath
*From:* Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> *Sent:* Thursday, October 22, 2020 7:42 PM *To:* Kumar, Sharath <sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io *Subject:* Re: FM containerization Build issues
The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/
horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/
The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit
http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta...
It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change.
You may be able to login to a horizon container and determine what version of the code it is using.
Al
------------------------------------------------------------------------
*From:*Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>> *Sent:* Thursday, October 22, 2020 6:02 AM *To:* Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Subject:* RE: FM containerization Build issues
Hi Tyler,
I had tried to reproduce the issue once again. The steps and respective errors are captured as below.
1. Applied helm chart and operation aborted during processing of horizon.
2. Horizon pod got crashed.
3. Horizon pod description.
4. Pod log command gave below errors.
Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ?
Regards, Sharath
*From:* Kumar, Sharath *Sent:* Wednesday, October 21, 2020 1:23 PM *To:* Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi Tyler,
Please find the attached log for your reference.
Regards, Sharath
*From:* Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>> *Sent:* Saturday, October 17, 2020 12:40 AM *To:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi,
For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there’s nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try.
Tyler
*From:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>> *Sent:* October 16, 2020 6:20 AM *To:* Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi Frank/Tyler,
Currently *issue 1* also blocking us, could you please help us to debug the issue.
Regards, Sharath
*From:* Scott Little <scott.little@windriver.com <mailto:scott.little@windriver.com>> *Sent:* Thursday, October 15, 2020 8:50 PM *To:* Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>> *Cc:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: FM containerization Build issues
Re Issue 2
Have you failed a launchpad?
Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung.
Include the content of your localrc and buildrc
Not knowing where you hung, it's hard to comment on which specific log would be most relevant.
Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern...
${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log
i.e. there is a separate log for each image.
You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue.
Catpuring and attaching the console/terminal session is also a good idea
Scott
On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
HI Frank/Tyler/Scott
As discussed in the containerization meeting sharing the build issues details need your inputs :-
* *Issue 1*:- FM with Mario’s changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference .
o Please let us know where else to look for to get more details of the issue o Need your inputs to move forward
* *Issue 2*:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * *Issue 3:-* Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke...
We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt)
I have attached the logs and the docker scripts . Please let us know if any further details are needed.
Thanks and Regards,
Sanjay Mukherjee
Hi Scot, As we discussed in out last meeting, please find the below links to review my code. https://review.opendev.org/#/c/761881/ https://review.opendev.org/#/c/761896/ As per your below suggestions, I will rebuild the stx-horizon image and share the logs and results with you. Regards, Sharath From: Scott Little <scott.little@windriver.com> Sent: Tuesday, November 3, 2020 9:23 PM To: Kumar, Sharath <sharath.kumar@intel.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Smith, Tyler <tyler.smith@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Penney, Don <Don.Penney@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: FM containerization Build issues RE: question 2,
From the failed ussuri build,
Can you provide the stx-horizon.stable_docker_image, and the resultant ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log from the failed build Also did you run build-stx-images.sh with the the --clean option, or manually delete any pre-existing docker images, before switching your PROJECT_REF? Again the concern, as Al mentioned, is docker retaining cached 'layers' content from a previous build, especially if the build is on another branch. e.g. you might want to display the layers with 'docker history <image>' for the master vs the ussuri build, looking for shared layer uuid's. Ideally the uuid's are unique for all layers. Scott On 2020-11-03 4:15 a.m., Kumar, Sharath wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Hi Henry, This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod. Sharing some of the screenshots with you. 1. stx-horizon.stable_docker_image file configurations. [cid:image001.png@01D6B6BA.AE44B900] 1. Docker image build script. [cid:image002.png@01D6B6BA.AE44B900] 1. Helm chart applied successfully. [cid:image003.png@01D6B6BA.AE44B900] 1. Horizon pods are running successfully. [cid:image004.png@01D6B6BA.AE44B900] 1. Pods are showing some warnings. [cid:image005.png@01D6B6BA.AE44B900] My questions: 1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application's stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings? Please let me know if you want to setup working session with me, thank you for your support and help. Regard's, Sharath From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com><mailto:Al.Bailey@windriver.com> Sent: Thursday, October 22, 2020 7:42 PM To: Kumar, Sharath <sharath.kumar@intel.com><mailto:sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com><mailto:tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com><mailto:Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com><mailto:sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com><mailto:Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com><mailto:Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com><mailto:Matt.Peters@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com><mailto:haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com><mailto:gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/ horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/ The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta... It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change. You may be able to login to a horizon container and determine what version of the code it is using. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: Thursday, October 22, 2020 6:02 AM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image006.png@01D6B6BA.AE44B900] 1. Horizon pod got crashed. [cid:image007.png@01D6B6BA.AE44B900] 1. Horizon pod description. [cid:image008.png@01D6B6BA.AE44B900] 1. Pod log command gave below errors. [cid:image009.png@01D6B6BA.AE44B900] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Hi Scott, The stx-horizon docker image has been created successfully by using stable/ussuri branch and applied to the helm chart. Pods also created and running successfully but still am seeing timeout warnings. [cid:image002.png@01D6B9B5.651E44F0] Need help: 1. I am using horizon repo with stable/ussuri branch and docker image created without any issues. But the custom FM GUI code which is part of stable/ussuri branch and it is is not reflecting in the pod. I have checkout by ssh to the running horizon pod and my code is missing there. Any suggestions to resolve this would be helpful. Code for review: https://review.opendev.org/#/c/762615/ https://review.opendev.org/#/c/761896/ Regards, Sharath From: Kumar, Sharath Sent: Monday, November 9, 2020 5:07 PM To: Scott Little <scott.little@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Smith, Tyler <tyler.smith@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Penney, Don <Don.Penney@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: FM containerization Build issues Hi Scot, As we discussed in out last meeting, please find the below links to review my code. https://review.opendev.org/#/c/761881/ https://review.opendev.org/#/c/761896/ As per your below suggestions, I will rebuild the stx-horizon image and share the logs and results with you. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Tuesday, November 3, 2020 9:23 PM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Penney, Don <Don.Penney@windriver.com<mailto:Don.Penney@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues RE: question 2,
From the failed ussuri build,
Can you provide the stx-horizon.stable_docker_image, and the resultant ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log from the failed build Also did you run build-stx-images.sh with the the --clean option, or manually delete any pre-existing docker images, before switching your PROJECT_REF? Again the concern, as Al mentioned, is docker retaining cached 'layers' content from a previous build, especially if the build is on another branch. e.g. you might want to display the layers with 'docker history <image>' for the master vs the ussuri build, looking for shared layer uuid's. Ideally the uuid's are unique for all layers. Scott On 2020-11-03 4:15 a.m., Kumar, Sharath wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Hi Henry, This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod. Sharing some of the screenshots with you. 1. stx-horizon.stable_docker_image file configurations. [cid:image012.png@01D6B900.1E49F430] 1. Docker image build script. [cid:image013.png@01D6B900.1E49F430] 1. Helm chart applied successfully. [cid:image014.png@01D6B900.1E49F430] 1. Horizon pods are running successfully. [cid:image015.png@01D6B900.1E49F430] 1. Pods are showing some warnings. [cid:image016.png@01D6B900.1E49F430] My questions: 1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application's stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings? Please let me know if you want to setup working session with me, thank you for your support and help. Regard's, Sharath From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com><mailto:Al.Bailey@windriver.com> Sent: Thursday, October 22, 2020 7:42 PM To: Kumar, Sharath <sharath.kumar@intel.com><mailto:sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com><mailto:tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com><mailto:Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com><mailto:sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com><mailto:Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com><mailto:Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com><mailto:Matt.Peters@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com><mailto:haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com><mailto:gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/ horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/ The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta... It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change. You may be able to login to a horizon container and determine what version of the code it is using. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: Thursday, October 22, 2020 6:02 AM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image017.png@01D6B900.1E49F430] 1. Horizon pod got crashed. [cid:image018.png@01D6B900.1E49F430] 1. Horizon pod description. [cid:image019.png@01D6B900.1E49F430] 1. Pod log command gave below errors. [cid:image020.png@01D6B900.1E49F430] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
Hi Scott, Adding some of the pod logs and descriptions for your reference. Earlier, I had committed on Openstack horizon project, again recommitted on starlingx project for your review. Please let me know if you need anything else. Code for review: https://review.opendev.org/#/c/762616/ https://review.opendev.org/#/c/761896/ Regards, Sharath From: Kumar, Sharath Sent: Friday, November 13, 2020 12:07 PM To: Scott Little <scott.little@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com> Subject: RE: FM containerization Build issues Hi Scott, The stx-horizon docker image has been created successfully by using stable/ussuri branch and applied to the helm chart. Pods also created and running successfully but still am seeing timeout warnings. [cid:image001.png@01D6B9D3.72644420] Need help: 1. I am using horizon repo with stable/ussuri branch and docker image created without any issues. But the custom FM GUI code which is part of stable/ussuri branch and it is is not reflecting in the pod. I have checkout by ssh to the running horizon pod and my code is missing there. Any suggestions to resolve this would be helpful. Code for review: https://review.opendev.org/#/c/762616/ https://review.opendev.org/#/c/761896/ Regards, Sharath From: Kumar, Sharath Sent: Monday, November 9, 2020 5:07 PM To: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Penney, Don <Don.Penney@windriver.com<mailto:Don.Penney@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Scot, As we discussed in out last meeting, please find the below links to review my code. https://review.opendev.org/#/c/761881/ https://review.opendev.org/#/c/761896/ As per your below suggestions, I will rebuild the stx-horizon image and share the logs and results with you. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Tuesday, November 3, 2020 9:23 PM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Penney, Don <Don.Penney@windriver.com<mailto:Don.Penney@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues RE: question 2,
From the failed ussuri build,
Can you provide the stx-horizon.stable_docker_image, and the resultant ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log from the failed build Also did you run build-stx-images.sh with the the --clean option, or manually delete any pre-existing docker images, before switching your PROJECT_REF? Again the concern, as Al mentioned, is docker retaining cached 'layers' content from a previous build, especially if the build is on another branch. e.g. you might want to display the layers with 'docker history <image>' for the master vs the ussuri build, looking for shared layer uuid's. Ideally the uuid's are unique for all layers. Scott On 2020-11-03 4:15 a.m., Kumar, Sharath wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Hi Henry, This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod. Sharing some of the screenshots with you. 1. stx-horizon.stable_docker_image file configurations. [cid:image002.png@01D6B9D3.72644420] 1. Docker image build script. [cid:image003.png@01D6B9D3.72644420] 1. Helm chart applied successfully. [cid:image004.png@01D6B9D3.72644420] 1. Horizon pods are running successfully. [cid:image005.png@01D6B9D3.72644420] 1. Pods are showing some warnings. [cid:image006.png@01D6B9D3.72644420] My questions: 1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application's stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings? Please let me know if you want to setup working session with me, thank you for your support and help. Regard's, Sharath From: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com><mailto:Al.Bailey@windriver.com> Sent: Thursday, October 22, 2020 7:42 PM To: Kumar, Sharath <sharath.kumar@intel.com><mailto:sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com><mailto:tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com><mailto:Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com><mailto:sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com><mailto:Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com><mailto:Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com><mailto:Matt.Peters@windriver.com> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com><mailto:haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com><mailto:gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/ horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/ The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta... It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change. You may be able to login to a horizon container and determine what version of the code it is using. Al ________________________________ From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: Thursday, October 22, 2020 6:02 AM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: FM containerization Build issues Hi Tyler, I had tried to reproduce the issue once again. The steps and respective errors are captured as below. 1. Applied helm chart and operation aborted during processing of horizon. [cid:image007.png@01D6B9D3.72644420] 1. Horizon pod got crashed. [cid:image008.png@01D6B9D3.72644420] 1. Horizon pod description. [cid:image009.png@01D6B9D3.72644420] 1. Pod log command gave below errors. [cid:image010.png@01D6B9D3.72644420] Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ? Regards, Sharath From: Kumar, Sharath Sent: Wednesday, October 21, 2020 1:23 PM To: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Tyler, Please find the attached log for your reference. Regards, Sharath From: Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Sent: Saturday, October 17, 2020 12:40 AM To: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi, For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there's nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try. Tyler From: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>> Sent: October 16, 2020 6:20 AM To: Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FM containerization Build issues Hi Frank/Tyler, Currently issue 1 also blocking us, could you please help us to debug the issue. Regards, Sharath From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com<mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com<mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com<mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com<mailto:tyler.smith@windriver.com>> Cc: Kumar, Sharath <sharath.kumar@intel.com<mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com<mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com<mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
I trust your are requesting help with your runtime issues on the starling-discuss list. I do build tools. I rarely do runtime and, I'm likely the least qualified to comment on the logs you provided. I was offering to help with build time issues, and was looking for your ... ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log ... I don't think you have provided that. Scott On 2020-11-13 5:31 a.m., Kumar, Sharath wrote:
**[Please note this e-mail is from an EXTERNAL e-mail address]
Hi Scott,
Adding some of the pod logs and descriptions for your reference. Earlier, I had committed on Openstack horizon project, again recommitted on starlingx project for your review. Please let me know if you need anything else.
*Code for review:*
https://review.opendev.org/#/c/762616/
https://review.opendev.org/#/c/761896/
Regards, Sharath
*From:*Kumar, Sharath *Sent:* Friday, November 13, 2020 12:07 PM *To:* Scott Little <scott.little@windriver.com> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com> *Subject:* RE: FM containerization Build issues
Hi Scott,
The stx-horizon docker image has been created successfully by using stable/ussuri branch and applied to the helm chart. Pods also created and running successfully but still am seeing timeout warnings.
*_Need help:_*
1. I am using horizon repo with stable/ussuri branch and docker image created without any issues. But the custom FM GUI code which is part of stable/ussuri branch and it is is not reflecting in the pod. I have checkout by ssh to the running horizon pod and my code is missing there. Any suggestions to resolve this would be helpful.
*Code for review:*
https://review.opendev.org/#/c/762616/
https://review.opendev.org/#/c/761896/
Regards,
Sharath
*From:*Kumar, Sharath *Sent:* Monday, November 9, 2020 5:07 PM *To:* Scott Little <scott.little@windriver.com <mailto:scott.little@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Penney, Don <Don.Penney@windriver.com <mailto:Don.Penney@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi Scot,
As we discussed in out last meeting, please find the below links to review my code.
https://review.opendev.org/#/c/761881/
https://review.opendev.org/#/c/761896/
As per your below suggestions, I will rebuild the stx-horizon image and share the logs and results with you.
Regards, Sharath
*From:*Scott Little <scott.little@windriver.com <mailto:scott.little@windriver.com>> *Sent:* Tuesday, November 3, 2020 9:23 PM *To:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Penney, Don <Don.Penney@windriver.com <mailto:Don.Penney@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: FM containerization Build issues
RE: question 2,
From the failed ussuri build,
Can you provide the stx-horizon.stable_docker_image, and the resultant ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log from the failed build
Also did you run build-stx-images.sh with the the --clean option, or manually delete any pre-existing docker images, before switching your PROJECT_REF? Again the concern, as Al mentioned, is docker retaining cached 'layers' content from a previous build, especially if the build is on another branch. e.g. you might want to display the layers with 'docker history <image>' for the master vs the ussuri build, looking for shared layer uuid's. Ideally the uuid's are unique for all layers.
Scott
On 2020-11-03 4:15 a.m., Kumar, Sharath wrote:
[Please note this e-mail is from an EXTERNAL e-mail address]
Hi Henry,
This time, I have changed the branch name from stable/ussuri to master and build the docker image and it worked on helm chart. But the custom code, we pushed on the master repo is not reflecting on the pod.
Sharing some of the screenshots with you.
1. stx-horizon.stable_docker_image file configurations.
2. Docker image build script.
3. Helm chart applied successfully.
4. Horizon pods are running successfully.
5. Pods are showing some warnings.
My questions:
1. I had ssh to the pod and the custom code is missing but docker image is referring the local repo where my custom code is existing. My question is, where should I mentioning docker image in the helm chart? In the armadas application’s stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz -> value.yaml file? 2. Why docker image build works on master branch instead of stable/ussuri ? 3. Why pods are throwing Readiness probe warnings?
Please let me know if you want to setup working session with me, thank you for your support and help.
Regard’s, Sharath
*From:* Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> <mailto:Al.Bailey@windriver.com> *Sent:* Thursday, October 22, 2020 7:42 PM *To:* Kumar, Sharath <sharath.kumar@intel.com> <mailto:sharath.kumar@intel.com>; Smith, Tyler <tyler.smith@windriver.com> <mailto:tyler.smith@windriver.com>; Little, Scott <Scott.Little@windriver.com> <mailto:Scott.Little@windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com> <mailto:sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com> <mailto:Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com> <mailto:Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com> <mailto:Matt.Peters@windriver.com> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com> <mailto:haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com> <mailto:gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: FM containerization Build issues
The list_extensions were removed from novaclient in Train. https://review.opendev.org/#/c/686516/
horizon followed suit and removed them in this review: https://review.opendev.org/#/c/690134/
The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit
http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/conta...
It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change.
You may be able to login to a horizon container and determine what version of the code it is using.
Al
------------------------------------------------------------------------
*From:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>> *Sent:* Thursday, October 22, 2020 6:02 AM *To:* Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Subject:* RE: FM containerization Build issues
Hi Tyler,
I had tried to reproduce the issue once again. The steps and respective errors are captured as below.
1. Applied helm chart and operation aborted during processing of horizon.
2. Horizon pod got crashed.
3. Horizon pod description.
4. Pod log command gave below errors.
Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ?
Regards, Sharath
*From:* Kumar, Sharath *Sent:* Wednesday, October 21, 2020 1:23 PM *To:* Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>>; Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi Tyler,
Please find the attached log for your reference.
Regards, Sharath
*From:* Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>> *Sent:* Saturday, October 17, 2020 12:40 AM *To:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>>; Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi,
For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up. If there’s nothing useful there you could try manually launching horizon from within the container and seeing what the output is. I think that would involve extending the failed container timeouts so that you have enough time to connect and try.
Tyler
*From:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>> *Sent:* October 16, 2020 6:20 AM *To:* Little, Scott <Scott.Little@windriver.com <mailto:Scott.Little@windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>> *Cc:* Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* RE: FM containerization Build issues
Hi Frank/Tyler,
Currently *issue 1* also blocking us, could you please help us to debug the issue.
Regards, Sharath
*From:* Scott Little <scott.little@windriver.com <mailto:scott.little@windriver.com>> *Sent:* Thursday, October 15, 2020 8:50 PM *To:* Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com <mailto:sanjay.k.mukherjee@intel.com>>; Miller, Frank <Frank.Miller@windriver.com <mailto:Frank.Miller@windriver.com>>; Bowness, Les <Les.Bowness@windriver.com <mailto:Les.Bowness@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com <mailto:Matt.Peters@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com <mailto:Al.Bailey@windriver.com>>; Smith, Tyler <tyler.smith@windriver.com <mailto:tyler.smith@windriver.com>> *Cc:* Kumar, Sharath <sharath.kumar@intel.com <mailto:sharath.kumar@intel.com>>; Kalvala, Haridhar <haridhar.kalvala@intel.com <mailto:haridhar.kalvala@intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com <mailto:gopalkrishna.bhat@intel.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: FM containerization Build issues
Re Issue 2
Have you failed a launchpad?
Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung.
Include the content of your localrc and buildrc
Not knowing where you hung, it's hard to comment on which specific log would be most relevant.
Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern...
${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log
i.e. there is a separate log for each image.
You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue.
Catpuring and attaching the console/terminal session is also a good idea
Scott
On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
HI Frank/Tyler/Scott
As discussed in the containerization meeting sharing the build issues details need your inputs :-
* *Issue 1*:- FM with Mario’s changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference .
o Please let us know where else to look for to get more details of the issue o Need your inputs to move forward
* *Issue 2*:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * *Issue 3:-* Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke...
We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt)
I have attached the logs and the docker scripts . Please let us know if any further details are needed.
Thanks and Regards,
Sanjay Mukherjee
Hi Scott, I have added the following bug :- https://bugs.launchpad.net/starlingx/+bug/1900126 With the requested log. Please verify and let me know any further details is required Thanks and Regards, Sanjay Mukherjee From: Scott Little <scott.little@windriver.com> Sent: Thursday, October 15, 2020 8:50 PM To: Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>; Miller, Frank <Frank.Miller@windriver.com>; Bowness, Les <Les.Bowness@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Smith, Tyler <tyler.smith@windriver.com> Cc: Kumar, Sharath <sharath.kumar@intel.com>; Kalvala, Haridhar <haridhar.kalvala@intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: FM containerization Build issues Re Issue 2 Have you failed a launchpad? Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung. Include the content of your localrc and buildrc Not knowing where you hung, it's hard to comment on which specific log would be most relevant. Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern... ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log i.e. there is a separate log for each image. You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue. Catpuring and attaching the console/terminal session is also a good idea Scott On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote: HI Frank/Tyler/Scott As discussed in the containerization meeting sharing the build issues details need your inputs :- * Issue 1:- FM with Mario's changes throws horizon not ready error on 4.0 release onwards . Attaching the Armada log for reference . * Please let us know where else to look for to get more details of the issue * Need your inputs to move forward * Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed * Issue 3:- Presence of only python 3 packages in the wheels tar. It causes issue while building in Python 2.7 environment http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docke... We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt) I have attached the logs and the docker scripts . Please let us know if any further details are needed. Thanks and Regards, Sanjay Mukherjee
participants (5)
-
Bailey, Henry Albert (Al)
-
Kumar, Sharath
-
Mukherjee, Sanjay K
-
Scott Little
-
Smith, Tyler