[Starlingx-discuss] FM containerization Build issues
Scott Little
scott.little at windriver.com
Fri Nov 13 18:58:08 UTC 2020
I trust your are requesting help with your runtime issues on the
starling-discuss list. I do build tools. I rarely do runtime and, I'm
likely the least qualified to comment on the logs you provided.
I was offering to help with build time issues, and was looking for your
...
${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log
... I don't think you have provided that.
Scott
On 2020-11-13 5:31 a.m., Kumar, Sharath wrote:
>
> **[Please note this e-mail is from an EXTERNAL e-mail address]
>
> Hi Scott,
>
> Adding some of the pod logs and descriptions for your reference.
> Earlier, I had committed on Openstack horizon project, again
> recommitted on starlingx project for your review. Please let me know
> if you need anything else.
>
> *Code for review:*
>
> https://review.opendev.org/#/c/762616/
>
> https://review.opendev.org/#/c/761896/
>
> Regards,
> Sharath
>
> *From:*Kumar, Sharath
> *Sent:* Friday, November 13, 2020 12:07 PM
> *To:* Scott Little <scott.little at windriver.com>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bhat,
> Gopalkrishna <gopalkrishna.bhat at intel.com>;
> starlingx-discuss at lists.starlingx.io; Mukherjee, Sanjay K
> <sanjay.k.mukherjee at intel.com>
> *Subject:* RE: FM containerization Build issues
>
> Hi Scott,
>
> The stx-horizon docker image has been created successfully by using
> stable/ussuri branch and applied to the helm chart. Pods also created
> and running successfully but still am seeing timeout warnings.
>
> *_Need help:_*
>
> 1. I am using horizon repo with stable/ussuri branch and docker image
> created without any issues. But the custom FM GUI code which is
> part of stable/ussuri branch and it is is not reflecting in the
> pod. I have checkout by ssh to the running horizon pod and my code
> is missing there. Any suggestions to resolve this would be helpful.
>
> *Code for review:*
>
> https://review.opendev.org/#/c/762616/
>
> https://review.opendev.org/#/c/761896/
>
> Regards,
>
> Sharath
>
> *From:*Kumar, Sharath
> *Sent:* Monday, November 9, 2020 5:07 PM
> *To:* Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>; Bailey, Henry Albert (Al)
> <Al.Bailey at windriver.com <mailto:Al.Bailey at windriver.com>>; Smith,
> Tyler <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Penney, Don <Don.Penney at windriver.com <mailto:Don.Penney at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Scot,
>
> As we discussed in out last meeting, please find the below links to
> review my code.
>
> https://review.opendev.org/#/c/761881/
>
> https://review.opendev.org/#/c/761896/
>
> As per your below suggestions, I will rebuild the stx-horizon image
> and share the logs and results with you.
>
> Regards,
> Sharath
>
> *From:*Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>
> *Sent:* Tuesday, November 3, 2020 9:23 PM
> *To:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>; Bailey, Henry Albert (Al)
> <Al.Bailey at windriver.com <mailto:Al.Bailey at windriver.com>>; Smith,
> Tyler <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Penney, Don <Don.Penney at windriver.com <mailto:Don.Penney at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FM containerization Build issues
>
> RE: question 2,
>
> From the failed ussuri build,
>
> Can you provide the stx-horizon.stable_docker_image, and the resultant
> ${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log
> from the failed build
>
> Also did you run build-stx-images.sh with the the --clean option, or
> manually delete any pre-existing docker images, before switching your
> PROJECT_REF? Again the concern, as Al mentioned, is docker retaining
> cached 'layers' content from a previous build, especially if the build
> is on another branch. e.g. you might want to display the layers with
> 'docker history <image>' for the master vs the ussuri build, looking
> for shared layer uuid's. Ideally the uuid's are unique for all layers.
>
> Scott
>
> On 2020-11-03 4:15 a.m., Kumar, Sharath wrote:
>
> [Please note this e-mail is from an EXTERNAL e-mail address]
>
> Hi Henry,
>
> This time, I have changed the branch name from stable/ussuri to
> master and build the docker image and it worked on helm chart. But
> the custom code, we pushed on the master repo is not reflecting on
> the pod.
>
> Sharing some of the screenshots with you.
>
> 1. stx-horizon.stable_docker_image file configurations.
>
> 2. Docker image build script.
>
> 3. Helm chart applied successfully.
>
> 4. Horizon pods are running successfully.
>
> 5. Pods are showing some warnings.
>
> My questions:
>
> 1. I had ssh to the pod and the custom code is missing but docker
> image is referring the local repo where my custom code is
> existing. My question is, where should I mentioning docker
> image in the helm chart? In the armadas application’s
> stx-openstack.yaml file or in the chart of horizon-0.1.0.tgz
> -> value.yaml file?
> 2. Why docker image build works on master branch instead of
> stable/ussuri ?
> 3. Why pods are throwing Readiness probe warnings?
>
> Please let me know if you want to setup working session with me,
> thank you for your support and help.
>
> Regard’s,
> Sharath
>
> *From:* Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>
> <mailto:Al.Bailey at windriver.com>
> *Sent:* Thursday, October 22, 2020 7:42 PM
> *To:* Kumar, Sharath <sharath.kumar at intel.com>
> <mailto:sharath.kumar at intel.com>; Smith, Tyler
> <tyler.smith at windriver.com> <mailto:tyler.smith at windriver.com>;
> Little, Scott <Scott.Little at windriver.com>
> <mailto:Scott.Little at windriver.com>; Mukherjee, Sanjay K
> <sanjay.k.mukherjee at intel.com>
> <mailto:sanjay.k.mukherjee at intel.com>; Miller, Frank
> <Frank.Miller at windriver.com> <mailto:Frank.Miller at windriver.com>;
> Bowness, Les <Les.Bowness at windriver.com>
> <mailto:Les.Bowness at windriver.com>; Peters, Matt
> <Matt.Peters at windriver.com> <mailto:Matt.Peters at windriver.com>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com>
> <mailto:haridhar.kalvala at intel.com>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com>
> <mailto:gopalkrishna.bhat at intel.com>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FM containerization Build issues
>
> The list_extensions were removed from novaclient in Train.
> https://review.opendev.org/#/c/686516/
>
> horizon followed suit and removed them in this review:
> https://review.opendev.org/#/c/690134/
>
> The cengn logs for horizon indicate it pulls in novaclient 17.0.0
> which has the novaclient commit
>
> http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/containers/logs/docker-images/
>
> It also indicates that its building using the PROJECT_REF of
> "stable/ussuri" that includes the above horizon change.
>
> You may be able to login to a horizon container and determine what
> version of the code it is using.
>
> Al
>
> ------------------------------------------------------------------------
>
> *From:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>
> *Sent:* Thursday, October 22, 2020 6:02 AM
> *To:* Smith, Tyler <tyler.smith at windriver.com
> <mailto:tyler.smith at windriver.com>>; Little, Scott
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com
> <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Subject:* RE: FM containerization Build issues
>
> Hi Tyler,
>
> I had tried to reproduce the issue once again. The steps and
> respective errors are captured as below.
>
> 1. Applied helm chart and operation aborted during processing of
> horizon.
>
> 2. Horizon pod got crashed.
>
> 3. Horizon pod description.
>
> 4. Pod log command gave below errors.
>
> Both logs are attaching for your reference, please confirm, if I
> open new bug in launchpad and upload these logs would be fine with
> you ?
>
> Regards,
> Sharath
>
> *From:* Kumar, Sharath
> *Sent:* Wednesday, October 21, 2020 1:23 PM
> *To:* Smith, Tyler <tyler.smith at windriver.com
> <mailto:tyler.smith at windriver.com>>; Little, Scott
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com
> <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Tyler,
>
> Please find the attached log for your reference.
>
> Regards,
> Sharath
>
> *From:* Smith, Tyler <tyler.smith at windriver.com
> <mailto:tyler.smith at windriver.com>>
> *Sent:* Saturday, October 17, 2020 12:40 AM
> *To:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>; Little, Scott
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com
> <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi,
>
> For issue 1 I would suggest looking into the container logs to see
> if there are errors being reported when horizon tries starting
> up. If there’s nothing useful there you could try manually
> launching horizon from within the container and seeing what the
> output is. I think that would involve extending the failed
> container timeouts so that you have enough time to connect and try.
>
> Tyler
>
> *From:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>
> *Sent:* October 16, 2020 6:20 AM
> *To:* Little, Scott <Scott.Little at windriver.com
> <mailto:Scott.Little at windriver.com>>; Mukherjee, Sanjay K
> <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>; Smith, Tyler
> <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com
> <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Frank/Tyler,
>
> Currently *issue 1* also blocking us, could you please help us to
> debug the issue.
>
> Regards,
> Sharath
>
> *From:* Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>
> *Sent:* Thursday, October 15, 2020 8:50 PM
> *To:* Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>; Smith, Tyler
> <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
> *Cc:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>; Kalvala, Haridhar
> <haridhar.kalvala at intel.com <mailto:haridhar.kalvala at intel.com>>;
> Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com
> <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FM containerization Build issues
>
> Re Issue 2
>
> Have you failed a launchpad?
>
> Please include ALL steps required to reproduce your build issue
> including setting environment variables, checking out source code
> (including how to cherry pick any of your changes), setting up
> docker, etc... and point out which command hung.
>
> Include the content of your localrc and buildrc
>
> Not knowing where you hung, it's hard to comment on which specific
> log would be most relevant.
>
> Based on your attachment, perhaps it was build-stx-images.sh? In
> that case it would likely be one of the logs matching the pattern...
>
> ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log
>
> i.e. there is a separate log for each image.
>
> You can pass the '--only <LABEL>' arg to build-stx-images.sh to
> speed up reproduction of the issue.
>
> Catpuring and attaching the console/terminal session is also a
> good idea
>
> Scott
>
> On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
>
> HI Frank/Tyler/Scott
>
> As discussed in the containerization meeting sharing the
> build issues details need your inputs :-
>
> * *Issue 1*:- FM with Mario’s changes throws horizon not
> ready error on 4.0 release onwards . Attaching the Armada
> log for reference .
>
> o Please let us know where else to look for to get more
> details of the issue
> o Need your inputs to move forward
>
> * *Issue 2*:- Docker build on the latest master branch is
> hanging, let us know if this is a known issue . Let us
> know if any specific log is needed
> * *Issue 3:-* Presence of only python 3 packages in the
> wheels tar. It causes issue while building in Python 2.7
> environment
>
> http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-stable-wheels.tar
>
> We built using python 2.7 taking the wheel packages built
> locally. (docker_build_script_2.txt)
>
> I have attached the logs and the docker scripts . Please let
> us know if any further details are needed.
>
> Thanks and Regards,
>
> Sanjay Mukherjee
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 381738 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 26054 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 286086 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 167512 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 13287 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 175296 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 162248 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.png
Type: image/png
Size: 89236 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image009.png
Type: image/png
Size: 83218 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.png
Type: image/png
Size: 126093 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201113/db4607f0/attachment-0019.png>
More information about the Starlingx-discuss
mailing list