[Starlingx-discuss] FM containerization Build issues

Scott Little scott.little at windriver.com
Fri Nov 13 18:58:08 UTC 2020


I trust your are requesting help with your runtime issues on the 
starling-discuss list.  I do build tools.  I rarely do runtime and, I'm 
likely the least qualified to comment on the logs you provided.

I was offering to help with build time issues, and was looking for your 
... 
${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log 
... I don't think you have provided that.

Scott


On 2020-11-13 5:31 a.m., Kumar, Sharath wrote:
>
> **[Please note this e-mail is from an EXTERNAL e-mail address]
>
> Hi Scott,
>
> Adding some of the pod logs and descriptions for your reference. 
> Earlier, I had committed on Openstack horizon project, again 
> recommitted on starlingx project for your review. Please let me know 
> if you need anything else.
>
> *Code for review:*
>
> https://review.opendev.org/#/c/762616/
>
> https://review.opendev.org/#/c/761896/
>
> Regards,
> Sharath
>
> *From:*Kumar, Sharath
> *Sent:* Friday, November 13, 2020 12:07 PM
> *To:* Scott Little <scott.little at windriver.com>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bhat, 
> Gopalkrishna <gopalkrishna.bhat at intel.com>; 
> starlingx-discuss at lists.starlingx.io; Mukherjee, Sanjay K 
> <sanjay.k.mukherjee at intel.com>
> *Subject:* RE: FM containerization Build issues
>
> Hi Scott,
>
> The stx-horizon docker image has been created successfully by using 
> stable/ussuri branch and applied to the helm chart. Pods also created 
> and running successfully but still am seeing timeout warnings.
>
> *_Need help:_*
>
>  1. I am using horizon repo with stable/ussuri branch and docker image
>     created without any issues. But the custom FM GUI code which is
>     part of stable/ussuri branch and it is  is not reflecting in the
>     pod. I have checkout by ssh to the running horizon pod and my code
>     is missing there. Any suggestions to resolve this would be helpful.
>
> *Code for review:*
>
> https://review.opendev.org/#/c/762616/
>
> https://review.opendev.org/#/c/761896/
>
> Regards,
>
> Sharath
>
> *From:*Kumar, Sharath
> *Sent:* Monday, November 9, 2020 5:07 PM
> *To:* Scott Little <scott.little at windriver.com 
> <mailto:scott.little at windriver.com>>; Bailey, Henry Albert (Al) 
> <Al.Bailey at windriver.com <mailto:Al.Bailey at windriver.com>>; Smith, 
> Tyler <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>; 
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com 
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank 
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>; 
> Bowness, Les <Les.Bowness at windriver.com 
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt 
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>; 
> Penney, Don <Don.Penney at windriver.com <mailto:Don.Penney at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com 
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna 
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Scot,
>
> As we discussed in out last meeting, please find the below links to 
> review my code.
>
> https://review.opendev.org/#/c/761881/
>
> https://review.opendev.org/#/c/761896/
>
> As per your below suggestions, I will rebuild the stx-horizon image 
> and share the logs and results with you.
>
> Regards,
> Sharath
>
> *From:*Scott Little <scott.little at windriver.com 
> <mailto:scott.little at windriver.com>>
> *Sent:* Tuesday, November 3, 2020 9:23 PM
> *To:* Kumar, Sharath <sharath.kumar at intel.com 
> <mailto:sharath.kumar at intel.com>>; Bailey, Henry Albert (Al) 
> <Al.Bailey at windriver.com <mailto:Al.Bailey at windriver.com>>; Smith, 
> Tyler <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>; 
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com 
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank 
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>; 
> Bowness, Les <Les.Bowness at windriver.com 
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt 
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>; 
> Penney, Don <Don.Penney at windriver.com <mailto:Don.Penney at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com 
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna 
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>; 
> starlingx-discuss at lists.starlingx.io 
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FM containerization Build issues
>
> RE: question 2,
>
> From the failed ussuri build,
>
> Can you provide the stx-horizon.stable_docker_image, and the resultant 
> ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log 
> from the failed build
>
> Also did you run build-stx-images.sh with the the --clean option, or 
> manually delete any pre-existing docker images, before switching your 
> PROJECT_REF?  Again the concern, as Al mentioned, is docker retaining 
> cached 'layers' content from a previous build, especially if the build 
> is on another branch.  e.g. you might want to display the layers with 
> 'docker history <image>' for the master vs the ussuri build, looking 
> for shared layer uuid's.  Ideally the uuid's are unique for all layers.
>
> Scott
>
> On 2020-11-03 4:15 a.m., Kumar, Sharath wrote:
>
>     [Please note this e-mail is from an EXTERNAL e-mail address]
>
>     Hi Henry,
>
>     This time, I have changed the branch name from stable/ussuri to
>     master and build the docker image and it worked on helm chart. But
>     the custom code, we pushed on the master repo is not reflecting on
>     the pod.
>
>     Sharing some of the screenshots with you.
>
>      1. stx-horizon.stable_docker_image file configurations.
>
>      2. Docker image build script.
>
>      3. Helm chart applied successfully.
>
>      4. Horizon pods are running successfully.
>
>      5. Pods are showing some warnings.
>
>     My questions:
>
>      1. I had ssh to the pod and the custom code is missing but docker
>         image is referring the local repo where my custom code is
>         existing. My question is, where should I mentioning docker
>         image in the helm chart? In the armadas application’s
>         stx-openstack.yaml  file or in the chart of horizon-0.1.0.tgz
>         -> value.yaml file?
>      2. Why docker image build works on master branch instead of
>         stable/ussuri ?
>      3. Why pods are throwing Readiness probe warnings?
>
>     Please let me know if you want to setup working session with me,
>     thank you for your support and help.
>
>     Regard’s,
>     Sharath
>
>     *From:* Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>
>     <mailto:Al.Bailey at windriver.com>
>     *Sent:* Thursday, October 22, 2020 7:42 PM
>     *To:* Kumar, Sharath <sharath.kumar at intel.com>
>     <mailto:sharath.kumar at intel.com>; Smith, Tyler
>     <tyler.smith at windriver.com> <mailto:tyler.smith at windriver.com>;
>     Little, Scott <Scott.Little at windriver.com>
>     <mailto:Scott.Little at windriver.com>; Mukherjee, Sanjay K
>     <sanjay.k.mukherjee at intel.com>
>     <mailto:sanjay.k.mukherjee at intel.com>; Miller, Frank
>     <Frank.Miller at windriver.com> <mailto:Frank.Miller at windriver.com>;
>     Bowness, Les <Les.Bowness at windriver.com>
>     <mailto:Les.Bowness at windriver.com>; Peters, Matt
>     <Matt.Peters at windriver.com> <mailto:Matt.Peters at windriver.com>
>     *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com>
>     <mailto:haridhar.kalvala at intel.com>; Bhat, Gopalkrishna
>     <gopalkrishna.bhat at intel.com>
>     <mailto:gopalkrishna.bhat at intel.com>;
>     starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>
>     *Subject:* Re: FM containerization Build issues
>
>     The list_extensions were removed from novaclient in Train.
>     https://review.opendev.org/#/c/686516/
>
>     horizon followed suit and removed them in this review:
>     https://review.opendev.org/#/c/690134/
>
>     The cengn logs for horizon indicate it pulls in novaclient 17.0.0
>     which has the novaclient commit
>
>     http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/containers/logs/docker-images/
>
>     It also indicates that its building using the PROJECT_REF of
>     "stable/ussuri" that includes the above horizon change.
>
>     You may be able to login to a horizon container and determine what
>     version of the code it is using.
>
>     Al
>
>     ------------------------------------------------------------------------
>
>     *From:* Kumar, Sharath <sharath.kumar at intel.com
>     <mailto:sharath.kumar at intel.com>>
>     *Sent:* Thursday, October 22, 2020 6:02 AM
>     *To:* Smith, Tyler <tyler.smith at windriver.com
>     <mailto:tyler.smith at windriver.com>>; Little, Scott
>     <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
>     Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
>     <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
>     <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
>     Bowness, Les <Les.Bowness at windriver.com
>     <mailto:Les.Bowness at windriver.com>>; Peters, Matt
>     <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
>     Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
>     <mailto:Al.Bailey at windriver.com>>
>     *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
>     <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
>     <gopalkrishna.bhat at intel.com
>     <mailto:gopalkrishna.bhat at intel.com>>;
>     starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>
>     <starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>>
>     *Subject:* RE: FM containerization Build issues
>
>     Hi Tyler,
>
>     I had tried to reproduce the issue once again. The steps and
>     respective errors are captured as below.
>
>      1. Applied helm chart and operation aborted during processing of
>         horizon.
>
>      2. Horizon pod got crashed.
>
>      3. Horizon pod description.
>
>      4. Pod log command gave below errors.
>
>     Both logs are attaching for your reference, please confirm, if I
>     open new bug in launchpad and upload these logs would be fine with
>     you ?
>
>     Regards,
>     Sharath
>
>     *From:* Kumar, Sharath
>     *Sent:* Wednesday, October 21, 2020 1:23 PM
>     *To:* Smith, Tyler <tyler.smith at windriver.com
>     <mailto:tyler.smith at windriver.com>>; Little, Scott
>     <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
>     Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
>     <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
>     <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
>     Bowness, Les <Les.Bowness at windriver.com
>     <mailto:Les.Bowness at windriver.com>>; Peters, Matt
>     <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
>     Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
>     <mailto:Al.Bailey at windriver.com>>
>     *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
>     <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
>     <gopalkrishna.bhat at intel.com
>     <mailto:gopalkrishna.bhat at intel.com>>;
>     starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>
>     *Subject:* RE: FM containerization Build issues
>
>     Hi Tyler,
>
>     Please find the attached log for your reference.
>
>     Regards,
>     Sharath
>
>     *From:* Smith, Tyler <tyler.smith at windriver.com
>     <mailto:tyler.smith at windriver.com>>
>     *Sent:* Saturday, October 17, 2020 12:40 AM
>     *To:* Kumar, Sharath <sharath.kumar at intel.com
>     <mailto:sharath.kumar at intel.com>>; Little, Scott
>     <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
>     Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
>     <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
>     <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
>     Bowness, Les <Les.Bowness at windriver.com
>     <mailto:Les.Bowness at windriver.com>>; Peters, Matt
>     <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
>     Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
>     <mailto:Al.Bailey at windriver.com>>
>     *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
>     <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
>     <gopalkrishna.bhat at intel.com
>     <mailto:gopalkrishna.bhat at intel.com>>;
>     starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>
>     *Subject:* RE: FM containerization Build issues
>
>     Hi,
>
>     For issue 1 I would suggest looking into the container logs to see
>     if there are errors being reported when horizon tries starting
>     up.  If there’s nothing useful there you could try manually
>     launching horizon from within the container and seeing what the
>     output is.  I think that would involve extending the failed
>     container timeouts so that you have enough time to connect and try.
>
>     Tyler
>
>     *From:* Kumar, Sharath <sharath.kumar at intel.com
>     <mailto:sharath.kumar at intel.com>>
>     *Sent:* October 16, 2020 6:20 AM
>     *To:* Little, Scott <Scott.Little at windriver.com
>     <mailto:Scott.Little at windriver.com>>; Mukherjee, Sanjay K
>     <sanjay.k.mukherjee at intel.com
>     <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
>     <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
>     Bowness, Les <Les.Bowness at windriver.com
>     <mailto:Les.Bowness at windriver.com>>; Peters, Matt
>     <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
>     Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
>     <mailto:Al.Bailey at windriver.com>>; Smith, Tyler
>     <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
>     *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
>     <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
>     <gopalkrishna.bhat at intel.com
>     <mailto:gopalkrishna.bhat at intel.com>>;
>     starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>
>     *Subject:* RE: FM containerization Build issues
>
>     Hi Frank/Tyler,
>
>     Currently *issue 1* also blocking us, could you please help us to
>     debug the issue.
>
>     Regards,
>     Sharath
>
>     *From:* Scott Little <scott.little at windriver.com
>     <mailto:scott.little at windriver.com>>
>     *Sent:* Thursday, October 15, 2020 8:50 PM
>     *To:* Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
>     <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
>     <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
>     Bowness, Les <Les.Bowness at windriver.com
>     <mailto:Les.Bowness at windriver.com>>; Peters, Matt
>     <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
>     Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
>     <mailto:Al.Bailey at windriver.com>>; Smith, Tyler
>     <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
>     *Cc:* Kumar, Sharath <sharath.kumar at intel.com
>     <mailto:sharath.kumar at intel.com>>; Kalvala, Haridhar
>     <haridhar.kalvala at intel.com <mailto:haridhar.kalvala at intel.com>>;
>     Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com
>     <mailto:gopalkrishna.bhat at intel.com>>;
>     starlingx-discuss at lists.starlingx.io
>     <mailto:starlingx-discuss at lists.starlingx.io>
>     *Subject:* Re: FM containerization Build issues
>
>     Re Issue 2
>
>     Have you failed a launchpad?
>
>     Please include ALL steps required to reproduce your build issue
>     including setting environment variables, checking out source code
>     (including how to cherry pick any of your changes), setting up
>     docker, etc... and point out which command hung.
>
>     Include the content of your localrc and buildrc
>
>     Not knowing where you hung, it's hard to comment on which specific
>     log would be most relevant.
>
>     Based on your attachment, perhaps it was build-stx-images.sh? In
>     that case it would likely be one of the logs matching the pattern...
>
>     ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log
>
>     i.e. there is a separate log for each image.
>
>     You can pass the '--only <LABEL>' arg to build-stx-images.sh to
>     speed up reproduction of the issue.
>
>     Catpuring and attaching the console/terminal session is also a
>     good idea
>
>     Scott
>
>     On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
>
>         HI Frank/Tyler/Scott
>
>           As discussed in the containerization meeting sharing the
>         build issues details  need your inputs :-
>
>           * *Issue 1*:- FM with Mario’s changes throws horizon not
>             ready error on 4.0 release onwards  . Attaching the Armada
>             log for reference .
>
>               o Please let us know where else to look for to get more
>                 details of the issue
>               o Need your inputs to move forward
>
>           * *Issue 2*:- Docker build on the latest master branch is
>             hanging, let us know if this is a known issue . Let us
>             know if any specific log is needed
>           * *Issue 3:-* Presence of only python 3  packages in the
>             wheels tar. It  causes issue while building in Python 2.7
>             environment
>
>         http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-stable-wheels.tar
>
>         We built using python 2.7 taking the wheel packages built
>         locally. (docker_build_script_2.txt)
>
>         I have attached the logs and the docker scripts . Please let
>         us know if any further details are needed.
>
>         Thanks and Regards,
>
>         Sanjay Mukherjee
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 381738 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 26054 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 286086 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 167512 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 13287 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 175296 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 162248 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.png
Type: image/png
Size: 89236 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image009.png
Type: image/png
Size: 83218 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.png
Type: image/png
Size: 126093 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0019.png>


More information about the Starlingx-discuss mailing list