[Starlingx-discuss] FM containerization Build issues
Scott Little
scott.little at windriver.com
Tue Nov 3 15:53:18 UTC 2020
RE: question 2,
From the failed ussuri build,
Can you provide the stx-horizon.stable_docker_image, and the resultant
${MY_WORKSPACE}/std/build-images/docker-stx-horizon-centos-stable.log
from the failed build
Also did you run build-stx-images.sh with the the --clean option, or
manually delete any pre-existing docker images, before switching your
PROJECT_REF? Again the concern, as Al mentioned, is docker retaining
cached 'layers' content from a previous build, especially if the build
is on another branch. e.g. you might want to display the layers with
'docker history <image>' for the master vs the ussuri build, looking for
shared layer uuid's. Ideally the uuid's are unique for all layers.
Scott
On 2020-11-03 4:15 a.m., Kumar, Sharath wrote:
>
> **[Please note this e-mail is from an EXTERNAL e-mail address]
>
> Hi Henry,
>
> This time, I have changed the branch name from stable/ussuri to master
> and build the docker image and it worked on helm chart. But the custom
> code, we pushed on the master repo is not reflecting on the pod.
>
> Sharing some of the screenshots with you.
>
> 1. stx-horizon.stable_docker_image file configurations.
>
> 2. Docker image build script.
>
> 3. Helm chart applied successfully.
>
> 4. Horizon pods are running successfully.
>
> 5. Pods are showing some warnings.
>
> My questions:
>
> 1. I had ssh to the pod and the custom code is missing but docker
> image is referring the local repo where my custom code is
> existing. My question is, where should I mentioning docker image
> in the helm chart? In the armadas application’s stx-openstack.yaml
> file or in the chart of horizon-0.1.0.tgz -> value.yaml file?
> 2. Why docker image build works on master branch instead of
> stable/ussuri ?
> 3. Why pods are throwing Readiness probe warnings?
>
> Please let me know if you want to setup working session with me, thank
> you for your support and help.
>
> Regard’s,
> Sharath
>
> *From:* Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>
> *Sent:* Thursday, October 22, 2020 7:42 PM
> *To:* Kumar, Sharath <sharath.kumar at intel.com>; Smith, Tyler
> <tyler.smith at windriver.com>; Little, Scott
> <Scott.Little at windriver.com>; Mukherjee, Sanjay K
> <sanjay.k.mukherjee at intel.com>; Miller, Frank
> <Frank.Miller at windriver.com>; Bowness, Les
> <Les.Bowness at windriver.com>; Peters, Matt <Matt.Peters at windriver.com>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bhat,
> Gopalkrishna <gopalkrishna.bhat at intel.com>;
> starlingx-discuss at lists.starlingx.io
> *Subject:* Re: FM containerization Build issues
>
> The list_extensions were removed from novaclient in Train.
> https://review.opendev.org/#/c/686516/
>
> horizon followed suit and removed them in this review:
> https://review.opendev.org/#/c/690134/
>
> The cengn logs for horizon indicate it pulls in novaclient 17.0.0
> which has the novaclient commit
>
> http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/containers/logs/docker-images/
>
> It also indicates that its building using the PROJECT_REF of
> "stable/ussuri" that includes the above horizon change.
>
> You may be able to login to a horizon container and determine what
> version of the code it is using.
>
> Al
>
> ------------------------------------------------------------------------
>
> *From:*Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>
> *Sent:* Thursday, October 22, 2020 6:02 AM
> *To:* Smith, Tyler <tyler.smith at windriver.com
> <mailto:tyler.smith at windriver.com>>; Little, Scott
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Subject:* RE: FM containerization Build issues
>
> Hi Tyler,
>
> I had tried to reproduce the issue once again. The steps and
> respective errors are captured as below.
>
> 1. Applied helm chart and operation aborted during processing of horizon.
>
> 2. Horizon pod got crashed.
>
> 3. Horizon pod description.
>
> 4. Pod log command gave below errors.
>
> Both logs are attaching for your reference, please confirm, if I open
> new bug in launchpad and upload these logs would be fine with you ?
>
> Regards,
> Sharath
>
> *From:* Kumar, Sharath
> *Sent:* Wednesday, October 21, 2020 1:23 PM
> *To:* Smith, Tyler <tyler.smith at windriver.com
> <mailto:tyler.smith at windriver.com>>; Little, Scott
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Tyler,
>
> Please find the attached log for your reference.
>
> Regards,
> Sharath
>
> *From:* Smith, Tyler <tyler.smith at windriver.com
> <mailto:tyler.smith at windriver.com>>
> *Sent:* Saturday, October 17, 2020 12:40 AM
> *To:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>; Little, Scott
> <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi,
>
> For issue 1 I would suggest looking into the container logs to see if
> there are errors being reported when horizon tries starting up. If
> there’s nothing useful there you could try manually launching horizon
> from within the container and seeing what the output is. I think that
> would involve extending the failed container timeouts so that you have
> enough time to connect and try.
>
> Tyler
>
> *From:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>
> *Sent:* October 16, 2020 6:20 AM
> *To:* Little, Scott <Scott.Little at windriver.com
> <mailto:Scott.Little at windriver.com>>; Mukherjee, Sanjay K
> <sanjay.k.mukherjee at intel.com <mailto:sanjay.k.mukherjee at intel.com>>;
> Miller, Frank <Frank.Miller at windriver.com
> <mailto:Frank.Miller at windriver.com>>; Bowness, Les
> <Les.Bowness at windriver.com <mailto:Les.Bowness at windriver.com>>;
> Peters, Matt <Matt.Peters at windriver.com
> <mailto:Matt.Peters at windriver.com>>; Bailey, Henry Albert (Al)
> <Al.Bailey at windriver.com <mailto:Al.Bailey at windriver.com>>; Smith,
> Tyler <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
> *Cc:* Kalvala, Haridhar <haridhar.kalvala at intel.com
> <mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna
> <gopalkrishna.bhat at intel.com <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FM containerization Build issues
>
> Hi Frank/Tyler,
>
> Currently *issue 1* also blocking us, could you please help us to
> debug the issue.
>
> Regards,
> Sharath
>
> *From:* Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>
> *Sent:* Thursday, October 15, 2020 8:50 PM
> *To:* Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com
> <mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank
> <Frank.Miller at windriver.com <mailto:Frank.Miller at windriver.com>>;
> Bowness, Les <Les.Bowness at windriver.com
> <mailto:Les.Bowness at windriver.com>>; Peters, Matt
> <Matt.Peters at windriver.com <mailto:Matt.Peters at windriver.com>>;
> Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>; Smith, Tyler
> <tyler.smith at windriver.com <mailto:tyler.smith at windriver.com>>
> *Cc:* Kumar, Sharath <sharath.kumar at intel.com
> <mailto:sharath.kumar at intel.com>>; Kalvala, Haridhar
> <haridhar.kalvala at intel.com <mailto:haridhar.kalvala at intel.com>>;
> Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com
> <mailto:gopalkrishna.bhat at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FM containerization Build issues
>
> Re Issue 2
>
> Have you failed a launchpad?
>
> Please include ALL steps required to reproduce your build issue
> including setting environment variables, checking out source code
> (including how to cherry pick any of your changes), setting up docker,
> etc... and point out which command hung.
>
> Include the content of your localrc and buildrc
>
> Not knowing where you hung, it's hard to comment on which specific log
> would be most relevant.
>
> Based on your attachment, perhaps it was build-stx-images.sh? In that
> case it would likely be one of the logs matching the pattern...
>
> ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log
>
> i.e. there is a separate log for each image.
>
> You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed
> up reproduction of the issue.
>
> Catpuring and attaching the console/terminal session is also a good idea
>
> Scott
>
> On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:
>
> HI Frank/Tyler/Scott
>
> As discussed in the containerization meeting sharing the build
> issues details need your inputs :-
>
> * *Issue 1*:- FM with Mario’s changes throws horizon not ready
> error on 4.0 release onwards . Attaching the Armada log for
> reference .
>
> o Please let us know where else to look for to get more
> details of the issue
> o Need your inputs to move forward
>
> * *Issue 2*:- Docker build on the latest master branch is
> hanging, let us know if this is a known issue . Let us know if
> any specific log is needed
> * *Issue 3:-* Presence of only python 3 packages in the wheels
> tar. It causes issue while building in Python 2.7 environment
>
> http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-stable-wheels.tar
>
> We built using python 2.7 taking the wheel packages built locally.
> (docker_build_script_2.txt)
>
> I have attached the logs and the docker scripts . Please let us
> know if any further details are needed.
>
> Thanks and Regards,
>
> Sanjay Mukherjee
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 26054 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 286086 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image012.png
Type: image/png
Size: 167512 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image011.png
Type: image/png
Size: 13287 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image014.png
Type: image/png
Size: 175296 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 162248 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 89236 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 83218 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 126093 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201103/60439b93/attachment-0017.png>
More information about the Starlingx-discuss
mailing list