[Starlingx-discuss] FM containerization Build issues

Scott Little scott.little at windriver.com
Tue Nov 3 15:53:18 UTC 2020


RE: question 2,

 From the failed ussuri build,

Can you provide the stx-horizon.stable_docker_image, and the resultant 
${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log 
from the failed build

Also did you run build-stx-images.sh with the the --clean option, or 
manually delete any pre-existing docker images, before switching your 
PROJECT_REF?  Again the concern, as Al mentioned, is docker retaining 
cached 'layers' content from a previous build, especially if the build 
is on another branch.  e.g. you might want to display the layers with 
'docker history <image>' for the master vs the ussuri build, looking for 
shared layer uuid's.  Ideally the uuid's are unique for all layers.

Scott



On 2020-11-03 4:15 a.m., Kumar, Sharath wrote:
>
> **[Please note this e-mail is from an EXTERNAL e-mail address]
>
> Hi Henry,
>
> This time, I have changed the branch name from stable/ussuri to master 
> and build the docker image and it worked on helm chart. But the custom 
> code, we pushed on the master repo is not reflecting on the pod.
>
> Sharing some of the screenshots with you.
>
>  1. stx-horizon.stable_docker_image file configurations.
>
>  2. Docker image build script.
>
>  3. Helm chart applied successfully.
>
>  4. Horizon pods are running successfully.
>
>  5. Pods are showing some warnings.
>
> My questions:
>
>  1. I had ssh to the pod and the custom code is missing but docker
>     image is referring the local repo where my custom code is
>     existing. My question is, where should I mentioning docker image
>     in the helm chart? In the armadas application’s stx-openstack.yaml
>      file or in the chart of horizon-0.1.0.tgz -> value.yaml file?
>  2. Why docker image build works on master branch instead of
>     stable/ussuri ?
>  3. Why pods are throwing Readiness probe warnings?
>
> Please let me know if you want to setup working session with me, thank 
> you for your support and help.
>
> Regard’s,
> Sharath
>
> *From:* Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>
> *Sent:* Thursday, October 22, 2020 7:42 PM
> *To:* Kumar, Sharath <sharath.kumar at intel.com>; Smith, Tyler 
> <tyler.smith at windriver.com>; Little, Scott 
> <Scott.Little at windriver.com>; Mukherjee, Sanjay K 
> <sanjay.k.mukherjee at intel.com>; Miller, Frank 
> <Frank.Miller at windriver.com>; Bowness, Les 
> <Les.Bowness at windriver.com>; Peters, Matt <Matt.Peters at windriver.com>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bhat, 
> Gopalkrishna <gopalkrishna.bhat at intel.com>; 
> starlingx-discuss at lists.starlingx.io
> *Subject:* Re: FM containerization Build issues
>
> The list_extensions were removed from novaclient in Train. 
> https://review.opendev.org/#/c/686516/
>
> horizon followed suit and removed them in this review: 
> https://review.opendev.org/#/c/690134/
>
> The cengn logs for horizon indicate it pulls in novaclient 17.0.0 
> which has the novaclient commit
>
> http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/containers/logs/docker-images/
>
> It also indicates that its building using the PROJECT_REF of 
> "stable/ussuri" that includes the above horizon change.
>
> You may be able to login to a horizon container and determine what 
> version of the code it is using.
>
> Al
>
> ------------------------------------------------------------------------
>
> *From:*Kumar, Sharath <sharath.kumar at intel.com 
> <mailto:sharath.kumar at intel.com>>
> *Sent:* Thursday, October 22, 2020 6:02 AM
> *To:* Smith, Tyler <tyler.smith at windriver.com 
> <mailto:tyler.smith at windriver.com>>; Little, Scott 
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>; 
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com 
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank 
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>; 
> Bowness, Les <Les.Bowness at windriver.com 
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt 
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>; 
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com 
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com 
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna 
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io> 
> <starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Subject:* RE: FM containerization Build issues
>
> Hi Tyler,
>
> I had tried to reproduce the issue once again. The steps and 
> respective errors are captured as below.
>
>  1. Applied helm chart and operation aborted during processing of horizon.
>
>  2. Horizon pod got crashed.
>
>  3. Horizon pod description.
>
>  4. Pod log command gave below errors.
>
> Both logs are attaching for your reference, please confirm, if I open 
> new bug in launchpad and upload these logs would be fine with you ?
>
> Regards,
> Sharath
>
> *From:* Kumar, Sharath
> *Sent:* Wednesday, October 21, 2020 1:23 PM
> *To:* Smith, Tyler <tyler.smith at windriver.com 
> <mailto:tyler.smith at windriver.com>>; Little, Scott 
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>; 
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com 
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank 
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>; 
> Bowness, Les <Les.Bowness at windriver.com 
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt 
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>; 
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com 
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com 
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna 
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Tyler,
>
> Please find the attached log for your reference.
>
> Regards,
> Sharath
>
> *From:* Smith, Tyler <tyler.smith at windriver.com 
> <mailto:tyler.smith at windriver.com>>
> *Sent:* Saturday, October 17, 2020 12:40 AM
> *To:* Kumar, Sharath <sharath.kumar at intel.com 
> <mailto:sharath.kumar at intel.com>>; Little, Scott 
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>; 
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com 
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank 
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>; 
> Bowness, Les <Les.Bowness at windriver.com 
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt 
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>; 
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com 
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com 
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna 
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi,
>
> For issue 1 I would suggest looking into the container logs to see if 
> there are errors being reported when horizon tries starting up.  If 
> there’s nothing useful there you could try manually launching horizon 
> from within the container and seeing what the output is.  I think that 
> would involve extending the failed container timeouts so that you have 
> enough time to connect and try.
>
> Tyler
>
> *From:* Kumar, Sharath <sharath.kumar at intel.com 
> <mailto:sharath.kumar at intel.com>>
> *Sent:* October 16, 2020 6:20 AM
> *To:* Little, Scott <Scott.Little at windriver.com 
> <mailto:Scott.Little at windriver.com>>; Mukherjee, Sanjay K 
> <sanjay.k.mukherjee at intel.com <mailto:sanjay.k.mukherjee at intel.com>>; 
> Miller, Frank <Frank.Miller at windriver.com 
> <mailto:Frank.Miller at windriver.com>>; Bowness, Les 
> <Les.Bowness at windriver.com <mailto:Les.Bowness at windriver.com>>; 
> Peters, Matt <Matt.Peters at windriver.com 
> <mailto:Matt.Peters at windriver.com>>; Bailey, Henry Albert (Al) 
> <Al.Bailey at windriver.com <mailto:Al.Bailey at windriver.com>>; Smith, 
> Tyler <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com 
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna 
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Frank/Tyler,
>
> Currently *issue 1* also blocking us, could you please help us to 
> debug the issue.
>
> Regards,
> Sharath
>
> *From:* Scott Little <scott.little at windriver.com 
> <mailto:scott.little at windriver.com>>
> *Sent:* Thursday, October 15, 2020 8:50 PM
> *To:* Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com 
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank 
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>; 
> Bowness, Les <Les.Bowness at windriver.com 
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt 
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>; 
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com 
> <mailto:Al.Bailey at windriver.com>>; Smith, Tyler 
> <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
> *Cc:* Kumar, Sharath <sharath.kumar at intel.com 
> <mailto:sharath.kumar at intel.com>>; Kalvala, Haridhar 
> <haridhar.kalvala at intel.com <mailto:haridhar.kalvala at intel.com>>; 
> Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com 
> <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FM containerization Build issues
>
> Re Issue 2
>
> Have you failed a launchpad?
>
> Please include ALL steps required to reproduce your build issue 
> including setting environment variables, checking out source code 
> (including how to cherry pick any of your changes), setting up docker, 
> etc... and point out which command hung.
>
> Include the content of your localrc and buildrc
>
> Not knowing where you hung, it's hard to comment on which specific log 
> would be most relevant.
>
> Based on your attachment, perhaps it was build-stx-images.sh? In that 
> case it would likely be one of the logs matching the pattern...
>
> ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log
>
> i.e. there is a separate log for each image.
>
> You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed 
> up reproduction of the issue.
>
> Catpuring and attaching the console/terminal session is also a good idea
>
> Scott
>
> On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
>
>     HI Frank/Tyler/Scott
>
>       As discussed in the containerization meeting sharing the build
>     issues details  need your inputs :-
>
>       * *Issue 1*:- FM with Mario’s changes throws horizon not ready
>         error on 4.0 release onwards  . Attaching the Armada log for
>         reference .
>
>           o Please let us know where else to look for to get more
>             details of the issue
>           o Need your inputs to move forward
>
>       * *Issue 2*:- Docker build on the latest master branch is
>         hanging, let us know if this is a known issue . Let us know if
>         any specific log is needed
>       * *Issue 3:-* Presence of only python 3  packages in the wheels
>         tar. It  causes issue while building in Python 2.7 environment
>
>     http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-stable-wheels.tar
>
>     We built using python 2.7 taking the wheel packages built locally.
>     (docker_build_script_2.txt)
>
>     I have attached the logs and the docker scripts . Please let us
>     know if any further details are needed.
>
>     Thanks and Regards,
>
>     Sanjay Mukherjee
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 26054 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 286086 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image012.png
Type: image/png
Size: 167512 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image011.png
Type: image/png
Size: 13287 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image014.png
Type: image/png
Size: 175296 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 162248 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 89236 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 83218 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 126093 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0017.png>


More information about the Starlingx-discuss mailing list