New subject: [Starlingx-discuss] [ Test ] Sanity results - summary posted

12 Sep 2018

      Erich,

A couple of data points on our side.  For our builds, we are using stock CentOS installer for 7.4.  These are unchanged and we have not seen this issue on our side.

Have you isolated when you made the changes to the installer?  Is it the same time you started seeing this error?  Does it make sense to revert to the stock installer while we get to the bottom of this issue to continue sanity?

Regards,
Ken Y

On 2018-09-11, 8:30 PM, "Cordoba Malibran, Erich" <erich.cordoba.malibran@intel.com> wrote:

    I have narrowed down a little bit the issue, these are my findings.

      - I compared the bootimage_michel.iso with the stx-2018-09-11-
    111.iso, which is today's ISO.
      - The are not significant package differences between them, minor
    versions has changed in some packages.
      - The issue is that once the kernel has been loaded through PXE,
    anaconda is unable to find python2-qpid package. Not sure if this is
    the first package that is requested.
      - From the anaconda shell I can see the http://pxecontroller/ serving
    the files and the repodata seems to be updated there. I installed yum
    manually and I was able to install python2-qpid package, (and any other
    available).
      - The content of the pxe-network-installer package is not the same,
    WR squashfs.img file has the kernel 3.10.0-693.2.2 but ours has 3.10.0-
    862.6.3. We regenerate those files when the kernel was upgraded. (maybe
    this matches with the date where the issue started to arise..)

    So, my guess now is that the problem could be caused by the pxe-
    network-installer package. We create those files with the update-
    network-installer script. That's the same used in WR to generate these
    files?

    I'll keep diging in the generation of these files to see if there is
    any problem.

    -Erich

    On Tue, 2018-09-11 at 20:03 +0000, Lara, Cesar wrote:
    > I just wanted to make sure that we are able to reproduce the issue
    > regardless of the build environment.
    > 
    > AFAIK CentOS 7.5 upgrade is a different issue/thread, but at the end
    > we can fine tune a process in which we are in the same conditions to
    > be able to replicate errors in any given build environment. 
    > 
    > Regards 
    > Cesar Lara
    > 
    > 
    > -----Original Message-----
    > From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] 
    > Sent: Tuesday, September 11, 2018 2:02 PM
    > To: Lara, Cesar <cesar.lara@intel.com>; Alonso, Juan Carlos <juan.car
    > los.alonso@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>;
    > Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Jones, Bruce
    > E <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; 
    > starlingx-discuss@lists.starlingx.io; Thebeau, Michel <Michel.Thebeau
    > @windriver.com>; Young, Ken <Ken.Young@windriver.com>; Little, Scott
    > <Scott.Little@windriver.com>
    > Subject: RE: [ Test ] Sanity results - summary posted
    > 
    > Hi Cesar,
    > 
    > Would you elaborate on what differences you'd expect to see in an
    > empty mirror populated by the download tool, compared to an existing
    > mirror which was updated by the download tool?
    > 
    > The mirror tools were written specifically to contain multiple
    > versions of packages.  The build environment population tools take
    > the .lst file you have checked out, and places that version from the
    > mirror into your local environment guaranteeing that your build env
    > matches the .lst file.  If your mirror contains versions 1.0.0,
    > 1.2.3, and 2.0.0 of a given package, then your environment will only
    > use the version called out in the .lst file you have checked out.
    > 
    > This allows the same mirror to be used for multiple branches, for
    > multiple releases, for multiple development environments, etc, and
    > guarantees that a build can be reproduced down the road even if the
    > build inputs disappear from the Internet at some point in the
    > future.  As long as all the artifacts named in the .lst file have
    > been downloaded at some point in the past, the build environment
    > produced from a fresh mirror vs a continually updated mirror will be
    > identical.
    > 
    > The issue we are observing with the CentOS 7.5 update is that the
    > versions of packages called out in the .lst files for the centos75
    > branch of stx-tools do not match the versions of packages called out
    > in centos75 branch of the other repos, like stx-integ.  The other
    > branches are attempting to patch a (new) version of the source code,
    > but the .lst file is only bringing in (or downloading) and older
    > version of the source.
    > 
    > -Jason
    > 
    > > -----Original Message-----
    > > From: Lara, Cesar <cesar.lara@intel.com>
    > > Sent: September 11, 2018 1:44 PM
    > > To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Khalil,
    > > Ghada 
    > > <Ghada.Khalil@windriver.com>; Martinez Monroy, Elio 
    > > <elio.martinez.monroy@intel.com>; Jones, Bruce E 
    > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; 
    > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel 
    > > <Michel.Thebeau@windriver.com>
    > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
    > > posted
    > > 
    > > Hi all,
    > > 
    > > Just wanted to point out that, to be able to reproduce the issues
    > > from 
    > > Intel's ISO, somebody will need to go and create a build
    > > environment 
    > > from scratch, using external upstream repositories to generate a
    > > new 
    > > mirror, just the way we are doing it in our environment. Failing to
    > > do 
    > > this will result in a apples to oranges comparison.
    > > 
    > > Regards
    > > Cesar Lara
    > > 
    > > -----Original Message-----
    > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso@intel.com]
    > > Sent: Tuesday, September 11, 2018 10:54 AM
    > > To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez Monroy,
    > > Elio 
    > > <elio.martinez.monroy@intel.com>; Jones, Bruce E 
    > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; 
    > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel 
    > > <Michel.Thebeau@windriver.com>
    > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
    > > posted
    > > 
    > > Hi Ghada,
    > > 
    > > Yes, before shared our ISOs I followed such steps and got the same
    > > issues.
    > > 
    > > As I mention in the issue
    > > (https://bugs.launchpad.net/starlingx/+bug/1790213), I could get
    > > the 
    > > ISO from Michel and testes in our infrastructure, issues not found.
    > > I 
    > > guess there are differences between the way an ISO is built by
    > > Intel 
    > > and Wind River since Intel got issues but Wind River does not. I
    > > think we should sync about this.
    > > 
    > > Besides, our test infrastructure uses other commands to generate
    > > VMs, 
    > > add/install hosts (controllers and computes), such commands was
    > > gotten 
    > > from "tic_vb" repo, such repo was given by China team, if I am not 
    > > wrong such repo was based on Wind River documentation; our test 
    > > framework is based on "tic_vb". Anyway both ways got the same
    > > results.
    > > 
    > > Regards.
    > > Juan Carlos Alonso
    > > 
    > > -----Original Message-----
    > > From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
    > > Sent: Tuesday, September 11, 2018 8:53 AM
    > > To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Martinez 
    > > Monroy, Elio <elio.martinez.monroy@intel.com>; Jones, Bruce E 
    > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; 
    > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel 
    > > <Michel.Thebeau@windriver.com>
    > > Subject: RE: [ Test ] Sanity results - summary posted
    > > 
    > > Hello Juan,
    > > I read the notes in the Launchpad.
    > > 
    > > And, just to confirm, are you following the instructions under:
    > > https://wiki.openstack.org/wiki/StarlingX/Installation_Guide_Virtua
    > > l_E
    > > nviro
    > > nment/Controller_Storage
    > > and still reproducing the two issues below?
    > > 
    > > I was not able to definitively tell from the Launchpad notes. It's 
    > > important that both our teams are comparing apples to apples.
    > > 
    > > Thanks,
    > > Ghada
    > > 
    > > -----Original Message-----
    > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso@intel.com]
    > > Sent: Monday, September 10, 2018 1:50 PM
    > > To: Khalil, Ghada; Martinez Monroy, Elio; Jones, Bruce E;
    > > Cabrales, 
    > > Ada; starlingx-discuss@lists.starlingx.io; Thebeau, Michel
    > > Subject: RE: [ Test ] Sanity results - summary posted
    > > 
    > > Hi,
    > > 
    > > Such launchpads were updated.
    > > 
    > > A VM was set to upload the ISO tx-2018-08-28-88.iso, such ISO
    > > present 
    > > two issues. Michael will upload his ISO, then we will re test
    > > witch 
    > > each iso in our environment.
    > > 
    > > An script is being developed to get the differences between ISOs.
    > > 
    > > Regards.
    > > Juan Carlos Alonso
    > > 
    > > -----Original Message-----
    > > From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
    > > Sent: Friday, September 7, 2018 4:43 PM
    > > To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez Monroy,
    > > Elio 
    > > <elio.martinez.monroy@intel.com>; Jones, Bruce E 
    > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; 
    > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel 
    > > <Michel.Thebeau@windriver.com>
    > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
    > > posted
    > > 
    > > Hi Elio/Ada,
    > > Are there any updates from your side on these two Launchpads?
    > > https://bugs.launchpad.net/starlingx/+bug/1790213
    > > https://bugs.launchpad.net/starlingx/+bug/1790716
    > > 
    > > Michel Thebeau (Wind River) has been trying to reproduce these two 
    > > issues (following the starlingx wikis) without success for two
    > > days 
    > > now. He has been adding his findings to the bugs and communicating
    > > with your team.
    > > As a result of his investigation, he will be pushing some small
    > > fixes 
    > > to stx- tools shortly (some cleanup).
    > > 
    > > Please let us know if you need any further help from us.
    > > 
    > > Thanks,
    > > Ghada
    > > 
    > > -----Original Message-----
    > > From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
    > > Sent: Wednesday, September 05, 2018 5:39 PM
    > > To: Martinez Monroy, Elio; Jones, Bruce E; Cabrales, Ada;
    > > starlingx- 
    > > discuss@lists.starlingx.io
    > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
    > > posted
    > > 
    > > Please see inline. There are notes in all the launchpads as well.
    > > 
    > > I have tagged the last bug for the October release. The rest need
    > > more 
    > > information to determine whether they gate or not: two maybe 
    > > procedural and one requires more information.
    > > 
    > > -----Original Message-----
    > > From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com]
    > > Sent: Wednesday, September 05, 2018 2:23 PM
    > > To: Jones, Bruce E; Cabrales, Ada; 
    > > starlingx-discuss@lists.starlingx.io
    > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
    > > posted
    > > 
    > > My opinion below
    > > 
    > > -----Original Message-----
    > > From: Jones, Bruce E [mailto:bruce.e.jones@intel.com]
    > > Sent: Wednesday, September 5, 2018 12:56 PM
    > > To: Cabrales, Ada <ada.cabrales@intel.com>; starlingx- 
    > > discuss@lists.starlingx.io
    > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
    > > posted
    > > 
    > > Should the bugs coming out of this be tagged/fixed for the October
    > > release?
    > > 
    > > Related bugs:
    > > https://bugs.launchpad.net/starlingx/+bug/1790213 < --- Must, this
    > > bug 
    > > is related with 2nd controller provisioning [[GK]] Using the 
    > > instructions published on the wiki, our designer was not able to
    > > reproduce this issue.
    > > What instructions are the test team using?
    > > 
    > > https://bugs.launchpad.net/starlingx/+bug/1790514 < --- Can wait
    > > until 
    > > other issues are solved, will fail some test cases related to SSH 
    > > [[GK]] Requested more information. This cannot be debugged further
    > > without this information.
    > > 
    > > https://bugs.launchpad.net/starlingx/+bug/1790716 < --- Must, all 
    > > instances need at least one compute for multimode [[GK]] Using the 
    > > instructions published on the wiki, our designer was not able to 
    > > reproduce this issue. The test team points to internal
    > > instructions 
    > > which we have no access to. The supported install recipes need to
    > > be 
    > > on the wiki. I think it is an invalid bug
    > > 
    > > https://bugs.launchpad.net/starlingx/+bug/1790718 < --- Can wait
    > > since 
    > > is related to FM , test cases will fail [[GK]] As Brent pointed
    > > out, 
    > > this is dependent on the integration of stx-gui which is in
    > > progress.
    > > 
    > > -----Original Message-----
    > > From: Cabrales, Ada [mailto:ada.cabrales@intel.com]
    > > Sent: Wednesday, September 5, 2018 10:50 AM
    > > To: starlingx-discuss@lists.starlingx.io
    > > Subject: [Starlingx-discuss] [ Test ] Sanity results - summary
    > > posted
    > > 
    > > Hello all,
    > > 
    > >     As mentioned on the project meeting today, we're posting the 
    > > summary of the sanity test runs on the Test wiki [0]. The sanity
    > > is 
    > > triggered each time an ISO is generated (a daily task).
    > > 
    > >    Right now we are posting the results manually, and working on a
    > > way 
    > > for automating it. Logs of the run are not included, if you want
    > > to 
    > > have them, you can contact Elio to get them sent to you.
    > > 
    > > 
    > > [0]
    > > https://wiki.openstack.org/wiki/StarlingX/Test#Latest_ISO_image_San
    > > ity
    > > _S
    > > ummary
    > > 
    > > Regards
    > > Ada
    > > 
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > > _______________________________________________
    > > Starlingx-discuss mailing list
    > > Starlingx-discuss@lists.starlingx.io
    > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
    > > s
    > 
    > _______________________________________________
    > Starlingx-discuss mailing list
    > Starlingx-discuss@lists.starlingx.io
    > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

Re: [Starlingx-discuss] [ Test ] Sanity results - summary posted

Young, Ken

Young, Ken

Cordoba Malibran, Erich

Penney, Don

Cordoba Malibran, Erich

McKenna, Jason

Penney, Don

tags

participants (4)