[Starlingx-discuss] FM containerization Build issues

Thu Oct 22 14:11:52 UTC 2020

The list_extensions were removed from novaclient in Train.  https://review.opendev.org/#/c/686516/
horizon followed suit and removed them in this review:  https://review.opendev.org/#/c/690134/

The cengn logs for horizon indicate it pulls in novaclient 17.0.0 which has the novaclient commit
http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/containers/logs/docker-images/

It also indicates that its building using the PROJECT_REF of "stable/ussuri" that includes the above horizon change.

You may be able to login to a horizon container and determine what version of the code it is using.

Al

________________________________
From: Kumar, Sharath <sharath.kumar at intel.com>
Sent: Thursday, October 22, 2020 6:02 AM
To: Smith, Tyler <tyler.smith at windriver.com>; Little, Scott <Scott.Little at windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Bowness, Les <Les.Bowness at windriver.com>; Peters, Matt <Matt.Peters at windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>
Cc: Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: RE: FM containerization Build issues

Hi Tyler,

I had tried to reproduce the issue once again. The steps and respective errors are captured as below.

  1.  Applied helm chart and operation aborted during processing of horizon.

[cid:image003.png at 01D6A888.7BFC76D0]

  1.  Horizon pod got crashed.

[cid:image005.png at 01D6A883.D645D050]

  1.  Horizon pod description.

[cid:image004.png at 01D6A884.35A14840]

  1.  Pod log command gave below errors.

[cid:image006.png at 01D6A884.F753C4E0]

Both logs are attaching for your reference, please confirm, if I open new bug in launchpad and upload these logs would be fine with you ?

Regards,
Sharath

From: Kumar, Sharath
Sent: Wednesday, October 21, 2020 1:23 PM
To: Smith, Tyler <tyler.smith at windriver.com>; Little, Scott <Scott.Little at windriver.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Bowness, Les <Les.Bowness at windriver.com>; Peters, Matt <Matt.Peters at windriver.com>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>
Cc: Kalvala, Haridhar <haridhar.kalvala at intel.com>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com>; starlingx-discuss at lists.starlingx.io
Subject: RE: FM containerization Build issues

Hi Tyler,

Please find the attached log for your reference.

Regards,
Sharath

From: Smith, Tyler <tyler.smith at windriver.com<mailto:tyler.smith at windriver.com>>
Sent: Saturday, October 17, 2020 12:40 AM
To: Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Little, Scott <Scott.Little at windriver.com<mailto:Scott.Little at windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank <Frank.Miller at windriver.com<mailto:Frank.Miller at windriver.com>>; Bowness, Les <Les.Bowness at windriver.com<mailto:Les.Bowness at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>
Cc: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: FM containerization Build issues

Hi,

For issue 1 I would suggest looking into the container logs to see if there are errors being reported when horizon tries starting up.  If there’s nothing useful there you could try manually launching horizon from within the container and seeing what the output is.  I think that would involve extending the failed container timeouts so that you have enough time to connect and try.

Tyler

From: Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>
Sent: October 16, 2020 6:20 AM
To: Little, Scott <Scott.Little at windriver.com<mailto:Scott.Little at windriver.com>>; Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank <Frank.Miller at windriver.com<mailto:Frank.Miller at windriver.com>>; Bowness, Les <Les.Bowness at windriver.com<mailto:Les.Bowness at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>; Smith, Tyler <tyler.smith at windriver.com<mailto:tyler.smith at windriver.com>>
Cc: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: FM containerization Build issues

Hi Frank/Tyler,

Currently issue 1 also blocking us, could you please help us to debug the issue.

Regards,
Sharath

From: Scott Little <scott.little at windriver.com<mailto:scott.little at windriver.com>>
Sent: Thursday, October 15, 2020 8:50 PM
To: Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com<mailto:sanjay.k.mukherjee at intel.com>>; Miller, Frank <Frank.Miller at windriver.com<mailto:Frank.Miller at windriver.com>>; Bowness, Les <Les.Bowness at windriver.com<mailto:Les.Bowness at windriver.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey at windriver.com<mailto:Al.Bailey at windriver.com>>; Smith, Tyler <tyler.smith at windriver.com<mailto:tyler.smith at windriver.com>>
Cc: Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>; Bhat, Gopalkrishna <gopalkrishna.bhat at intel.com<mailto:gopalkrishna.bhat at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: FM containerization Build issues

Re Issue 2

Have you failed a launchpad?

Please include ALL steps required to reproduce your build issue including setting environment variables, checking out source code (including how to cherry pick any of your changes), setting up docker, etc... and point out which command hung.

Include the content of your localrc and buildrc

Not knowing where you hung, it's hard to comment on which specific log would be most relevant.

Based on your attachment, perhaps it was build-stx-images.sh? In that case it would likely be one of the logs matching the pattern...

    ${MY_WORKSPACE}/std/build-images/docker-${LABEL}-${OS}-${BUILD_STREAM}.log

i.e. there is a separate log for each image.

You can pass the '--only <LABEL>' arg to build-stx-images.sh to speed up reproduction of the issue.

Catpuring and attaching the console/terminal session is also a good idea

Scott

On 2020-10-15 12:48 a.m., Mukherjee, Sanjay K wrote:

HI Frank/Tyler/Scott

  As discussed in the containerization meeting sharing the build issues details  need your inputs :-

  *   Issue 1:- FM with Mario’s changes throws horizon not ready error on 4.0 release onwards  . Attaching the Armada log for reference .

     *   Please let us know where else to look for to get more details of the issue
     *   Need your inputs to move forward

  *   Issue 2:- Docker build on the latest master branch is hanging, let us know if this is a known issue . Let us know if any specific log is needed
  *   Issue 3:- Presence of only python 3  packages in the wheels tar. It  causes issue while building in Python 2.7 environment

http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-stable-wheels.tar

We built using python 2.7 taking the wheel packages built locally. (docker_build_script_2.txt)

I have attached the logs and the docker scripts . Please let us know if any further details are needed.

Thanks and Regards,

Sanjay Mukherjee

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201022/2ae0f19a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 89236 bytes
Desc: image005.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201022/2ae0f19a/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 83218 bytes
Desc: image004.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201022/2ae0f19a/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 126093 bytes
Desc: image006.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201022/2ae0f19a/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 162248 bytes
Desc: image003.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20201022/2ae0f19a/attachment-0007.png>