[Starlingx-discuss] [ Test ] Sanity results - summary posted

Cordoba Malibran, Erich erich.cordoba.malibran at intel.com
Wed Sep 12 21:28:50 UTC 2018


HI,

I think I found the root cause. Running the anaconda-yum through strace I found that the webserver 
was returning a Forbidden error, then checking the lighttpd.conf differences I saw that in the WR image,
the user and group fields weren't set. In our /etc/lighttpd/lighttpd.conf we have:

## change uid to <uid> (default: don't care)
server.username            = "www"

## change uid to <uid> (default: don't care)
server.groupname           = "wrs_protected"

However on WR image, these are not set. Commenting this lines unblocks the installer. 
I'm still not sure on what changed, new package version, some puppet recipe or another thing. Even more
weird, why these files are different between ISOs.

-Erich


´╗┐On 9/12/18, 3:53 PM, "Young, Ken" <Ken.Young at windriver.com> wrote:

    Erich,
    
    One correction below: we are using a Wind River version of the installer (I am confusing this thread with another thread).  This is the installer we shipped Titanium Cloud Release 5.
    
    This begs the question: what installer were you using before the changes to the kernel?
    
    /KenY
    
    On 2018-09-12, 4:36 PM, "Young, Ken" <Ken.Young at windriver.com> wrote:
    
        Erich,
        
        A couple of data points on our side.  For our builds, we are using stock CentOS installer for 7.4.  These are unchanged and we have not seen this issue on our side.
        
        Have you isolated when you made the changes to the installer?  Is it the same time you started seeing this error?  Does it make sense to revert to the stock installer while we get to the bottom of this issue to continue sanity?
        
        Regards,
        Ken Y
        
        On 2018-09-11, 8:30 PM, "Cordoba Malibran, Erich" <erich.cordoba.malibran at intel.com> wrote:
        
            I have narrowed down a little bit the issue, these are my findings.
            
              - I compared the bootimage_michel.iso with the stx-2018-09-11-
            111.iso, which is today's ISO.
              - The are not significant package differences between them, minor
            versions has changed in some packages.
              - The issue is that once the kernel has been loaded through PXE,
            anaconda is unable to find python2-qpid package. Not sure if this is
            the first package that is requested.
              - From the anaconda shell I can see the http://pxecontroller/ serving
            the files and the repodata seems to be updated there. I installed yum
            manually and I was able to install python2-qpid package, (and any other
            available).
              - The content of the pxe-network-installer package is not the same,
            WR squashfs.img file has the kernel 3.10.0-693.2.2 but ours has 3.10.0-
            862.6.3. We regenerate those files when the kernel was upgraded. (maybe
            this matches with the date where the issue started to arise..)
            
            So, my guess now is that the problem could be caused by the pxe-
            network-installer package. We create those files with the update-
            network-installer script. That's the same used in WR to generate these
            files?
            
            I'll keep diging in the generation of these files to see if there is
            any problem.
            
            -Erich
            
            On Tue, 2018-09-11 at 20:03 +0000, Lara, Cesar wrote:
            > I just wanted to make sure that we are able to reproduce the issue
            > regardless of the build environment.
            > 
            > AFAIK CentOS 7.5 upgrade is a different issue/thread, but at the end
            > we can fine tune a process in which we are in the same conditions to
            > be able to replicate errors in any given build environment. 
            > 
            > Regards 
            > Cesar Lara
            > 
            > 
            > -----Original Message-----
            > From: McKenna, Jason [mailto:Jason.McKenna at windriver.com] 
            > Sent: Tuesday, September 11, 2018 2:02 PM
            > To: Lara, Cesar <cesar.lara at intel.com>; Alonso, Juan Carlos <juan.car
            > los.alonso at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>;
            > Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Jones, Bruce
            > E <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
            > starlingx-discuss at lists.starlingx.io; Thebeau, Michel <Michel.Thebeau
            > @windriver.com>; Young, Ken <Ken.Young at windriver.com>; Little, Scott
            > <Scott.Little at windriver.com>
            > Subject: RE: [ Test ] Sanity results - summary posted
            > 
            > Hi Cesar,
            > 
            > Would you elaborate on what differences you'd expect to see in an
            > empty mirror populated by the download tool, compared to an existing
            > mirror which was updated by the download tool?
            > 
            > The mirror tools were written specifically to contain multiple
            > versions of packages.  The build environment population tools take
            > the .lst file you have checked out, and places that version from the
            > mirror into your local environment guaranteeing that your build env
            > matches the .lst file.  If your mirror contains versions 1.0.0,
            > 1.2.3, and 2.0.0 of a given package, then your environment will only
            > use the version called out in the .lst file you have checked out.
            > 
            > This allows the same mirror to be used for multiple branches, for
            > multiple releases, for multiple development environments, etc, and
            > guarantees that a build can be reproduced down the road even if the
            > build inputs disappear from the Internet at some point in the
            > future.  As long as all the artifacts named in the .lst file have
            > been downloaded at some point in the past, the build environment
            > produced from a fresh mirror vs a continually updated mirror will be
            > identical.
            > 
            > The issue we are observing with the CentOS 7.5 update is that the
            > versions of packages called out in the .lst files for the centos75
            > branch of stx-tools do not match the versions of packages called out
            > in centos75 branch of the other repos, like stx-integ.  The other
            > branches are attempting to patch a (new) version of the source code,
            > but the .lst file is only bringing in (or downloading) and older
            > version of the source.
            > 
            > -Jason
            > 
            > > -----Original Message-----
            > > From: Lara, Cesar <cesar.lara at intel.com>
            > > Sent: September 11, 2018 1:44 PM
            > > To: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Khalil,
            > > Ghada 
            > > <Ghada.Khalil at windriver.com>; Martinez Monroy, Elio 
            > > <elio.martinez.monroy at intel.com>; Jones, Bruce E 
            > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
            > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
            > > <Michel.Thebeau at windriver.com>
            > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
            > > posted
            > > 
            > > Hi all,
            > > 
            > > Just wanted to point out that, to be able to reproduce the issues
            > > from 
            > > Intel's ISO, somebody will need to go and create a build
            > > environment 
            > > from scratch, using external upstream repositories to generate a
            > > new 
            > > mirror, just the way we are doing it in our environment. Failing to
            > > do 
            > > this will result in a apples to oranges comparison.
            > > 
            > > Regards
            > > Cesar Lara
            > > 
            > > -----Original Message-----
            > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso at intel.com]
            > > Sent: Tuesday, September 11, 2018 10:54 AM
            > > To: Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy,
            > > Elio 
            > > <elio.martinez.monroy at intel.com>; Jones, Bruce E 
            > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
            > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
            > > <Michel.Thebeau at windriver.com>
            > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
            > > posted
            > > 
            > > Hi Ghada,
            > > 
            > > Yes, before shared our ISOs I followed such steps and got the same
            > > issues.
            > > 
            > > As I mention in the issue
            > > (https://bugs.launchpad.net/starlingx/+bug/1790213), I could get
            > > the 
            > > ISO from Michel and testes in our infrastructure, issues not found.
            > > I 
            > > guess there are differences between the way an ISO is built by
            > > Intel 
            > > and Wind River since Intel got issues but Wind River does not. I
            > > think we should sync about this.
            > > 
            > > Besides, our test infrastructure uses other commands to generate
            > > VMs, 
            > > add/install hosts (controllers and computes), such commands was
            > > gotten 
            > > from "tic_vb" repo, such repo was given by China team, if I am not 
            > > wrong such repo was based on Wind River documentation; our test 
            > > framework is based on "tic_vb". Anyway both ways got the same
            > > results.
            > > 
            > > Regards.
            > > Juan Carlos Alonso
            > > 
            > > -----Original Message-----
            > > From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
            > > Sent: Tuesday, September 11, 2018 8:53 AM
            > > To: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Martinez 
            > > Monroy, Elio <elio.martinez.monroy at intel.com>; Jones, Bruce E 
            > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
            > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
            > > <Michel.Thebeau at windriver.com>
            > > Subject: RE: [ Test ] Sanity results - summary posted
            > > 
            > > Hello Juan,
            > > I read the notes in the Launchpad.
            > > 
            > > And, just to confirm, are you following the instructions under:
            > > https://wiki.openstack.org/wiki/StarlingX/Installation_Guide_Virtua
            > > l_E
            > > nviro
            > > nment/Controller_Storage
            > > and still reproducing the two issues below?
            > > 
            > > I was not able to definitively tell from the Launchpad notes. It's 
            > > important that both our teams are comparing apples to apples.
            > > 
            > > Thanks,
            > > Ghada
            > > 
            > > -----Original Message-----
            > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso at intel.com]
            > > Sent: Monday, September 10, 2018 1:50 PM
            > > To: Khalil, Ghada; Martinez Monroy, Elio; Jones, Bruce E;
            > > Cabrales, 
            > > Ada; starlingx-discuss at lists.starlingx.io; Thebeau, Michel
            > > Subject: RE: [ Test ] Sanity results - summary posted
            > > 
            > > Hi,
            > > 
            > > Such launchpads were updated.
            > > 
            > > A VM was set to upload the ISO tx-2018-08-28-88.iso, such ISO
            > > present 
            > > two issues. Michael will upload his ISO, then we will re test
            > > witch 
            > > each iso in our environment.
            > > 
            > > An script is being developed to get the differences between ISOs.
            > > 
            > > Regards.
            > > Juan Carlos Alonso
            > > 
            > > -----Original Message-----
            > > From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
            > > Sent: Friday, September 7, 2018 4:43 PM
            > > To: Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy,
            > > Elio 
            > > <elio.martinez.monroy at intel.com>; Jones, Bruce E 
            > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
            > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
            > > <Michel.Thebeau at windriver.com>
            > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
            > > posted
            > > 
            > > Hi Elio/Ada,
            > > Are there any updates from your side on these two Launchpads?
            > > https://bugs.launchpad.net/starlingx/+bug/1790213
            > > https://bugs.launchpad.net/starlingx/+bug/1790716
            > > 
            > > Michel Thebeau (Wind River) has been trying to reproduce these two 
            > > issues (following the starlingx wikis) without success for two
            > > days 
            > > now. He has been adding his findings to the bugs and communicating
            > > with your team.
            > > As a result of his investigation, he will be pushing some small
            > > fixes 
            > > to stx- tools shortly (some cleanup).
            > > 
            > > Please let us know if you need any further help from us.
            > > 
            > > Thanks,
            > > Ghada
            > > 
            > > -----Original Message-----
            > > From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
            > > Sent: Wednesday, September 05, 2018 5:39 PM
            > > To: Martinez Monroy, Elio; Jones, Bruce E; Cabrales, Ada;
            > > starlingx- 
            > > discuss at lists.starlingx.io
            > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
            > > posted
            > > 
            > > Please see inline. There are notes in all the launchpads as well.
            > > 
            > > I have tagged the last bug for the October release. The rest need
            > > more 
            > > information to determine whether they gate or not: two maybe 
            > > procedural and one requires more information.
            > > 
            > > -----Original Message-----
            > > From: Martinez Monroy, Elio [mailto:elio.martinez.monroy at intel.com]
            > > Sent: Wednesday, September 05, 2018 2:23 PM
            > > To: Jones, Bruce E; Cabrales, Ada; 
            > > starlingx-discuss at lists.starlingx.io
            > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
            > > posted
            > > 
            > > My opinion below
            > > 
            > > -----Original Message-----
            > > From: Jones, Bruce E [mailto:bruce.e.jones at intel.com]
            > > Sent: Wednesday, September 5, 2018 12:56 PM
            > > To: Cabrales, Ada <ada.cabrales at intel.com>; starlingx- 
            > > discuss at lists.starlingx.io
            > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
            > > posted
            > > 
            > > Should the bugs coming out of this be tagged/fixed for the October
            > > release?
            > > 
            > > Related bugs:
            > > https://bugs.launchpad.net/starlingx/+bug/1790213 < --- Must, this
            > > bug 
            > > is related with 2nd controller provisioning [[GK]] Using the 
            > > instructions published on the wiki, our designer was not able to
            > > reproduce this issue.
            > > What instructions are the test team using?
            > > 
            > > https://bugs.launchpad.net/starlingx/+bug/1790514 < --- Can wait
            > > until 
            > > other issues are solved, will fail some test cases related to SSH 
            > > [[GK]] Requested more information. This cannot be debugged further
            > > without this information.
            > > 
            > > https://bugs.launchpad.net/starlingx/+bug/1790716 < --- Must, all 
            > > instances need at least one compute for multimode [[GK]] Using the 
            > > instructions published on the wiki, our designer was not able to 
            > > reproduce this issue. The test team points to internal
            > > instructions 
            > > which we have no access to. The supported install recipes need to
            > > be 
            > > on the wiki. I think it is an invalid bug
            > > 
            > > https://bugs.launchpad.net/starlingx/+bug/1790718 < --- Can wait
            > > since 
            > > is related to FM , test cases will fail [[GK]] As Brent pointed
            > > out, 
            > > this is dependent on the integration of stx-gui which is in
            > > progress.
            > > 
            > > -----Original Message-----
            > > From: Cabrales, Ada [mailto:ada.cabrales at intel.com]
            > > Sent: Wednesday, September 5, 2018 10:50 AM
            > > To: starlingx-discuss at lists.starlingx.io
            > > Subject: [Starlingx-discuss] [ Test ] Sanity results - summary
            > > posted
            > > 
            > > Hello all,
            > > 
            > >     As mentioned on the project meeting today, we're posting the 
            > > summary of the sanity test runs on the Test wiki [0]. The sanity
            > > is 
            > > triggered each time an ISO is generated (a daily task).
            > > 
            > >    Right now we are posting the results manually, and working on a
            > > way 
            > > for automating it. Logs of the run are not included, if you want
            > > to 
            > > have them, you can contact Elio to get them sent to you.
            > > 
            > > 
            > > [0]
            > > https://wiki.openstack.org/wiki/StarlingX/Test#Latest_ISO_image_San
            > > ity
            > > _S
            > > ummary
            > > 
            > > Regards
            > > Ada
            > > 
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > > _______________________________________________
            > > Starlingx-discuss mailing list
            > > Starlingx-discuss at lists.starlingx.io
            > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
            > > s
            > 
            > _______________________________________________
            > Starlingx-discuss mailing list
            > Starlingx-discuss at lists.starlingx.io
            > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
        
        _______________________________________________
        Starlingx-discuss mailing list
        Starlingx-discuss at lists.starlingx.io
        http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
        
    
    



More information about the Starlingx-discuss mailing list