[Starlingx-discuss] [ Test ] Sanity results - summary posted

Penney, Don Don.Penney at windriver.com
Wed Sep 12 21:37:18 UTC 2018


The problem is one of permissions. In the /www/var/log/lighttpd-access.log file, you can see the http access request is being rejected with a 403 status code, indicating access is forbidden:

192.168.204.85 pxecontroller - [12/Sep/2018:09:30:52 +0000] "GET /feed/rel-18.08/Packages/python2-qpid-proton-0.24.0-1.el7.x86_64.rpm HTTP/1.1" 403 345 "-" "urlgrabber/3.10 yum/3.4.3"

You can see this also with a wget command:
localhost:/home/wrsroot# wget http://localhost/feed/rel-18.08/Packages/python2-qpid-proton-0.24.0-1.el7.x86_64.rpm
--2018-09-12 20:54:19--  http://localhost/feed/rel-18.08/Packages/python2-qpid-proton-0.24.0-1.el7.x86_64.rpm
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-09-12 20:54:19 ERROR 403: Forbidden.


The problem is that two of the RPMs on the ISO have 600 permissions, where all other RPMs are 644. This is carried over when the files are copied to the controller during the post-install phase:

controller-0:~$ ls -l /mnt/iso/Packages/ | grep -- ---
-rw------- 1 1001 users    258624 Aug 28 12:04 python2-qpid-proton-0.24.0-1.el7.x86_64.rpm
-rw------- 1 1001 users    190128 Aug 28 12:04 qpid-proton-c-0.24.0-1.el7.x86_64.rpm

controller-0:~$ ls -l /www/pages/feed/rel-18.08/Packages/ |grep -- ---
-rw-------. 1 root root    258624 Sep 12 20:33 python2-qpid-proton-0.24.0-1.el7.x86_64.rpm
-rw-------. 1 root root    190128 Sep 12 20:33 qpid-proton-c-0.24.0-1.el7.x86_64.rpm


The lighttpd server runs as a non-root user, and cannot access these files. Change the permissions on the files, and the installation of controller-1 should be successful.


-----Original Message-----
From: Young, Ken [mailto:Ken.Young at windriver.com] 
Sent: Wednesday, September 12, 2018 4:53 PM
To: Cordoba Malibran, Erich
Cc: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary posted

Erich,

One correction below: we are using a Wind River version of the installer (I am confusing this thread with another thread).  This is the installer we shipped Titanium Cloud Release 5.

This begs the question: what installer were you using before the changes to the kernel?

/KenY

´╗┐On 2018-09-12, 4:36 PM, "Young, Ken" <Ken.Young at windriver.com> wrote:

    Erich,
    
    A couple of data points on our side.  For our builds, we are using stock CentOS installer for 7.4.  These are unchanged and we have not seen this issue on our side.
    
    Have you isolated when you made the changes to the installer?  Is it the same time you started seeing this error?  Does it make sense to revert to the stock installer while we get to the bottom of this issue to continue sanity?
    
    Regards,
    Ken Y
    
    On 2018-09-11, 8:30 PM, "Cordoba Malibran, Erich" <erich.cordoba.malibran at intel.com> wrote:
    
        I have narrowed down a little bit the issue, these are my findings.
        
          - I compared the bootimage_michel.iso with the stx-2018-09-11-
        111.iso, which is today's ISO.
          - The are not significant package differences between them, minor
        versions has changed in some packages.
          - The issue is that once the kernel has been loaded through PXE,
        anaconda is unable to find python2-qpid package. Not sure if this is
        the first package that is requested.
          - From the anaconda shell I can see the http://pxecontroller/ serving
        the files and the repodata seems to be updated there. I installed yum
        manually and I was able to install python2-qpid package, (and any other
        available).
          - The content of the pxe-network-installer package is not the same,
        WR squashfs.img file has the kernel 3.10.0-693.2.2 but ours has 3.10.0-
        862.6.3. We regenerate those files when the kernel was upgraded. (maybe
        this matches with the date where the issue started to arise..)
        
        So, my guess now is that the problem could be caused by the pxe-
        network-installer package. We create those files with the update-
        network-installer script. That's the same used in WR to generate these
        files?
        
        I'll keep diging in the generation of these files to see if there is
        any problem.
        
        -Erich
        
        On Tue, 2018-09-11 at 20:03 +0000, Lara, Cesar wrote:
        > I just wanted to make sure that we are able to reproduce the issue
        > regardless of the build environment.
        > 
        > AFAIK CentOS 7.5 upgrade is a different issue/thread, but at the end
        > we can fine tune a process in which we are in the same conditions to
        > be able to replicate errors in any given build environment. 
        > 
        > Regards 
        > Cesar Lara
        > 
        > 
        > -----Original Message-----
        > From: McKenna, Jason [mailto:Jason.McKenna at windriver.com] 
        > Sent: Tuesday, September 11, 2018 2:02 PM
        > To: Lara, Cesar <cesar.lara at intel.com>; Alonso, Juan Carlos <juan.car
        > los.alonso at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>;
        > Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Jones, Bruce
        > E <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
        > starlingx-discuss at lists.starlingx.io; Thebeau, Michel <Michel.Thebeau
        > @windriver.com>; Young, Ken <Ken.Young at windriver.com>; Little, Scott
        > <Scott.Little at windriver.com>
        > Subject: RE: [ Test ] Sanity results - summary posted
        > 
        > Hi Cesar,
        > 
        > Would you elaborate on what differences you'd expect to see in an
        > empty mirror populated by the download tool, compared to an existing
        > mirror which was updated by the download tool?
        > 
        > The mirror tools were written specifically to contain multiple
        > versions of packages.  The build environment population tools take
        > the .lst file you have checked out, and places that version from the
        > mirror into your local environment guaranteeing that your build env
        > matches the .lst file.  If your mirror contains versions 1.0.0,
        > 1.2.3, and 2.0.0 of a given package, then your environment will only
        > use the version called out in the .lst file you have checked out.
        > 
        > This allows the same mirror to be used for multiple branches, for
        > multiple releases, for multiple development environments, etc, and
        > guarantees that a build can be reproduced down the road even if the
        > build inputs disappear from the Internet at some point in the
        > future.  As long as all the artifacts named in the .lst file have
        > been downloaded at some point in the past, the build environment
        > produced from a fresh mirror vs a continually updated mirror will be
        > identical.
        > 
        > The issue we are observing with the CentOS 7.5 update is that the
        > versions of packages called out in the .lst files for the centos75
        > branch of stx-tools do not match the versions of packages called out
        > in centos75 branch of the other repos, like stx-integ.  The other
        > branches are attempting to patch a (new) version of the source code,
        > but the .lst file is only bringing in (or downloading) and older
        > version of the source.
        > 
        > -Jason
        > 
        > > -----Original Message-----
        > > From: Lara, Cesar <cesar.lara at intel.com>
        > > Sent: September 11, 2018 1:44 PM
        > > To: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Khalil,
        > > Ghada 
        > > <Ghada.Khalil at windriver.com>; Martinez Monroy, Elio 
        > > <elio.martinez.monroy at intel.com>; Jones, Bruce E 
        > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
        > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
        > > <Michel.Thebeau at windriver.com>
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
        > > posted
        > > 
        > > Hi all,
        > > 
        > > Just wanted to point out that, to be able to reproduce the issues
        > > from 
        > > Intel's ISO, somebody will need to go and create a build
        > > environment 
        > > from scratch, using external upstream repositories to generate a
        > > new 
        > > mirror, just the way we are doing it in our environment. Failing to
        > > do 
        > > this will result in a apples to oranges comparison.
        > > 
        > > Regards
        > > Cesar Lara
        > > 
        > > -----Original Message-----
        > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso at intel.com]
        > > Sent: Tuesday, September 11, 2018 10:54 AM
        > > To: Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy,
        > > Elio 
        > > <elio.martinez.monroy at intel.com>; Jones, Bruce E 
        > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
        > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
        > > <Michel.Thebeau at windriver.com>
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
        > > posted
        > > 
        > > Hi Ghada,
        > > 
        > > Yes, before shared our ISOs I followed such steps and got the same
        > > issues.
        > > 
        > > As I mention in the issue
        > > (https://bugs.launchpad.net/starlingx/+bug/1790213), I could get
        > > the 
        > > ISO from Michel and testes in our infrastructure, issues not found.
        > > I 
        > > guess there are differences between the way an ISO is built by
        > > Intel 
        > > and Wind River since Intel got issues but Wind River does not. I
        > > think we should sync about this.
        > > 
        > > Besides, our test infrastructure uses other commands to generate
        > > VMs, 
        > > add/install hosts (controllers and computes), such commands was
        > > gotten 
        > > from "tic_vb" repo, such repo was given by China team, if I am not 
        > > wrong such repo was based on Wind River documentation; our test 
        > > framework is based on "tic_vb". Anyway both ways got the same
        > > results.
        > > 
        > > Regards.
        > > Juan Carlos Alonso
        > > 
        > > -----Original Message-----
        > > From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
        > > Sent: Tuesday, September 11, 2018 8:53 AM
        > > To: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Martinez 
        > > Monroy, Elio <elio.martinez.monroy at intel.com>; Jones, Bruce E 
        > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
        > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
        > > <Michel.Thebeau at windriver.com>
        > > Subject: RE: [ Test ] Sanity results - summary posted
        > > 
        > > Hello Juan,
        > > I read the notes in the Launchpad.
        > > 
        > > And, just to confirm, are you following the instructions under:
        > > https://wiki.openstack.org/wiki/StarlingX/Installation_Guide_Virtua
        > > l_E
        > > nviro
        > > nment/Controller_Storage
        > > and still reproducing the two issues below?
        > > 
        > > I was not able to definitively tell from the Launchpad notes. It's 
        > > important that both our teams are comparing apples to apples.
        > > 
        > > Thanks,
        > > Ghada
        > > 
        > > -----Original Message-----
        > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso at intel.com]
        > > Sent: Monday, September 10, 2018 1:50 PM
        > > To: Khalil, Ghada; Martinez Monroy, Elio; Jones, Bruce E;
        > > Cabrales, 
        > > Ada; starlingx-discuss at lists.starlingx.io; Thebeau, Michel
        > > Subject: RE: [ Test ] Sanity results - summary posted
        > > 
        > > Hi,
        > > 
        > > Such launchpads were updated.
        > > 
        > > A VM was set to upload the ISO tx-2018-08-28-88.iso, such ISO
        > > present 
        > > two issues. Michael will upload his ISO, then we will re test
        > > witch 
        > > each iso in our environment.
        > > 
        > > An script is being developed to get the differences between ISOs.
        > > 
        > > Regards.
        > > Juan Carlos Alonso
        > > 
        > > -----Original Message-----
        > > From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
        > > Sent: Friday, September 7, 2018 4:43 PM
        > > To: Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy,
        > > Elio 
        > > <elio.martinez.monroy at intel.com>; Jones, Bruce E 
        > > <bruce.e.jones at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; 
        > > starlingx-discuss at lists.starlingx.io; Thebeau, Michel 
        > > <Michel.Thebeau at windriver.com>
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
        > > posted
        > > 
        > > Hi Elio/Ada,
        > > Are there any updates from your side on these two Launchpads?
        > > https://bugs.launchpad.net/starlingx/+bug/1790213
        > > https://bugs.launchpad.net/starlingx/+bug/1790716
        > > 
        > > Michel Thebeau (Wind River) has been trying to reproduce these two 
        > > issues (following the starlingx wikis) without success for two
        > > days 
        > > now. He has been adding his findings to the bugs and communicating
        > > with your team.
        > > As a result of his investigation, he will be pushing some small
        > > fixes 
        > > to stx- tools shortly (some cleanup).
        > > 
        > > Please let us know if you need any further help from us.
        > > 
        > > Thanks,
        > > Ghada
        > > 
        > > -----Original Message-----
        > > From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
        > > Sent: Wednesday, September 05, 2018 5:39 PM
        > > To: Martinez Monroy, Elio; Jones, Bruce E; Cabrales, Ada;
        > > starlingx- 
        > > discuss at lists.starlingx.io
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
        > > posted
        > > 
        > > Please see inline. There are notes in all the launchpads as well.
        > > 
        > > I have tagged the last bug for the October release. The rest need
        > > more 
        > > information to determine whether they gate or not: two maybe 
        > > procedural and one requires more information.
        > > 
        > > -----Original Message-----
        > > From: Martinez Monroy, Elio [mailto:elio.martinez.monroy at intel.com]
        > > Sent: Wednesday, September 05, 2018 2:23 PM
        > > To: Jones, Bruce E; Cabrales, Ada; 
        > > starlingx-discuss at lists.starlingx.io
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
        > > posted
        > > 
        > > My opinion below
        > > 
        > > -----Original Message-----
        > > From: Jones, Bruce E [mailto:bruce.e.jones at intel.com]
        > > Sent: Wednesday, September 5, 2018 12:56 PM
        > > To: Cabrales, Ada <ada.cabrales at intel.com>; starlingx- 
        > > discuss at lists.starlingx.io
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary 
        > > posted
        > > 
        > > Should the bugs coming out of this be tagged/fixed for the October
        > > release?
        > > 
        > > Related bugs:
        > > https://bugs.launchpad.net/starlingx/+bug/1790213 < --- Must, this
        > > bug 
        > > is related with 2nd controller provisioning [[GK]] Using the 
        > > instructions published on the wiki, our designer was not able to
        > > reproduce this issue.
        > > What instructions are the test team using?
        > > 
        > > https://bugs.launchpad.net/starlingx/+bug/1790514 < --- Can wait
        > > until 
        > > other issues are solved, will fail some test cases related to SSH 
        > > [[GK]] Requested more information. This cannot be debugged further
        > > without this information.
        > > 
        > > https://bugs.launchpad.net/starlingx/+bug/1790716 < --- Must, all 
        > > instances need at least one compute for multimode [[GK]] Using the 
        > > instructions published on the wiki, our designer was not able to 
        > > reproduce this issue. The test team points to internal
        > > instructions 
        > > which we have no access to. The supported install recipes need to
        > > be 
        > > on the wiki. I think it is an invalid bug
        > > 
        > > https://bugs.launchpad.net/starlingx/+bug/1790718 < --- Can wait
        > > since 
        > > is related to FM , test cases will fail [[GK]] As Brent pointed
        > > out, 
        > > this is dependent on the integration of stx-gui which is in
        > > progress.
        > > 
        > > -----Original Message-----
        > > From: Cabrales, Ada [mailto:ada.cabrales at intel.com]
        > > Sent: Wednesday, September 5, 2018 10:50 AM
        > > To: starlingx-discuss at lists.starlingx.io
        > > Subject: [Starlingx-discuss] [ Test ] Sanity results - summary
        > > posted
        > > 
        > > Hello all,
        > > 
        > >     As mentioned on the project meeting today, we're posting the 
        > > summary of the sanity test runs on the Test wiki [0]. The sanity
        > > is 
        > > triggered each time an ISO is generated (a daily task).
        > > 
        > >    Right now we are posting the results manually, and working on a
        > > way 
        > > for automating it. Logs of the run are not included, if you want
        > > to 
        > > have them, you can contact Elio to get them sent to you.
        > > 
        > > 
        > > [0]
        > > https://wiki.openstack.org/wiki/StarlingX/Test#Latest_ISO_image_San
        > > ity
        > > _S
        > > ummary
        > > 
        > > Regards
        > > Ada
        > > 
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss at lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
        > > s
        > 
        > _______________________________________________
        > Starlingx-discuss mailing list
        > Starlingx-discuss at lists.starlingx.io
        > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
    
    _______________________________________________
    Starlingx-discuss mailing list
    Starlingx-discuss at lists.starlingx.io
    http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
    

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss


More information about the Starlingx-discuss mailing list