Re: [Starlingx-discuss] [ Test ] Sanity results - summary posted

13 Sep 2018

      There are a couple of places where the permissions could be checked/updated. It's never been a problem before now, but could occur again as other users come on board. The build-iso tool is responsible for creating the ISO and could ensure that all files on the ISO have the necessary permissions (and probably root/root user/group for the ISO). The controller kickstarts copy or download the files to setup the repo, and could also include a permissions check.

I think these files should continue to be root/root, but world-readable.

Cheers,
Don.

-----Original Message-----
From: McKenna, Jason 
Sent: Thursday, September 13, 2018 3:50 PM
To: Cordoba Malibran, Erich; Young, Ken; Penney, Don
Cc: starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] [ Test ] Sanity results - summary posted
...
-----Original Message-----
From: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>
Sent: September 13, 2018 2:46 PM
To: Young, Ken <Ken.Young@windriver.com>; Penney, Don
<Don.Penney@windriver.com>
Cc: starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary posted
Indeed, those two files had the 600 permissions in our mirror and the
permissions were transfered into the ISO and then to the /ww/pages/feed
folder. Changing the permissions now let the compute to be installed.
I have two questions regarding this:
- Which component is responsible to populate the /www/pages/feed folder
with the rpms. I'm wondering if some permission checking/set needs to be
done here.
+1
...
- lighttpd runs as non-root user, but the content on /www/pages/feed is
owned by root. Should this folder be owned by the same user as the one
that runs lighttpd?
The data needs to be readable by the user running lighttpd, but its okay if it's owned by someone else (including root)
...
Thanks
-Erich
On Wed, 2018-09-12 at 21:37 +0000, Penney, Don wrote:
...
The problem is one of permissions. In the /www/var/log/lighttpd-
access.log file, you can see the http access request is being rejected
with a 403 status code, indicating access is forbidden:
192.168.204.85 pxecontroller - [12/Sep/2018:09:30:52 +0000] "GET
/feed/rel-18.08/Packages/python2-qpid-proton-0.24.0-1.el7.x86_64.rpm
HTTP/1.1" 403 345 "-" "urlgrabber/3.10 yum/3.4.3"
You can see this also with a wget command:
localhost:/home/wrsroot# wget http://localhost/feed/rel-18.08/Package
s/python2-qpid-proton-0.24.0-1.el7.x86_64.rpm
--2018-09-12 20:54:19--  http://localhost/feed/rel-18.08/Packages/pyt
hon2-qpid-proton-0.24.0-1.el7.x86_64.rpm
Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to
localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2018-09-12 20:54:19 ERROR 403: Forbidden.
The problem is that two of the RPMs on the ISO have 600 permissions,
where all other RPMs are 644. This is carried over when the files are
copied to the controller during the post-install phase:
controller-0:~$ ls -l /mnt/iso/Packages/ | grep -- ---
-rw------- 1 1001 users    258624 Aug 28 12:04 python2-qpid-proton-
0.24.0-1.el7.x86_64.rpm
-rw------- 1 1001 users    190128 Aug 28 12:04 qpid-proton-c-0.24.0-
1.el7.x86_64.rpm
controller-0:~$ ls -l /www/pages/feed/rel-18.08/Packages/ |grep -- --
-
-rw-------. 1 root root    258624 Sep 12 20:33 python2-qpid-proton-
0.24.0-1.el7.x86_64.rpm
-rw-------. 1 root root    190128 Sep 12 20:33 qpid-proton-c-0.24.0-
1.el7.x86_64.rpm
The lighttpd server runs as a non-root user, and cannot access these
files. Change the permissions on the files, and the installation of
controller-1 should be successful.
-----Original Message-----
From: Young, Ken [mailto:Ken.Young@windriver.com]
Sent: Wednesday, September 12, 2018 4:53 PM
To: Cordoba Malibran, Erich
Cc: starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] [ Test ] Sanity results - summary
posted
Erich,
One correction below: we are using a Wind River version of the
installer (I am confusing this thread with another thread).  This is
the installer we shipped Titanium Cloud Release 5.
This begs the question: what installer were you using before the
changes to the kernel?
/KenY
On 2018-09-12, 4:36 PM, "Young, Ken" <Ken.Young@windriver.com>
wrote:
Erich,
A couple of data points on our side.  For our builds, we are using
stock CentOS installer for 7.4.  These are unchanged and we have not
seen this issue on our side.
Have you isolated when you made the changes to the installer?  Is
it the same time you started seeing this error?  Does it make sense to
revert to the stock installer while we get to the bottom of this issue
to continue sanity?
Regards,
    Ken Y
On 2018-09-11, 8:30 PM, "Cordoba Malibran, Erich" <erich.cordoba.
malibran@intel.com> wrote:
I have narrowed down a little bit the issue, these are my
findings.
- I compared the bootimage_michel.iso with the stx-2018-09-
11-
        111.iso, which is today's ISO.
          - The are not significant package differences between them,
minor
        versions has changed in some packages.
          - The issue is that once the kernel has been loaded through
PXE,
        anaconda is unable to find python2-qpid package. Not sure if
this is
        the first package that is requested.
          - From the anaconda shell I can see the http://pxecontrolle
r/ serving
        the files and the repodata seems to be updated there. I
installed yum
        manually and I was able to install python2-qpid package, (and
any other
        available).
          - The content of the pxe-network-installer package is not
the same,
        WR squashfs.img file has the kernel 3.10.0-693.2.2 but ours
has 3.10.0-
        862.6.3. We regenerate those files when the kernel was
upgraded. (maybe
        this matches with the date where the issue started to
arise..)
So, my guess now is that the problem could be caused by the
pxe-
        network-installer package. We create those files with the
update-
        network-installer script. That's the same used in WR to
generate these
        files?
I'll keep diging in the generation of these files to see if
there is
        any problem.
-Erich
On Tue, 2018-09-11 at 20:03 +0000, Lara, Cesar wrote:
        > I just wanted to make sure that we are able to reproduce the
issue
        > regardless of the build environment.
        >
        > AFAIK CentOS 7.5 upgrade is a different issue/thread, but at
the end
        > we can fine tune a process in which we are in the same
conditions to
        > be able to replicate errors in any given build environment.
        >
        > Regards
        > Cesar Lara
        >
        >
        > -----Original Message-----
        > From: McKenna, Jason [mailto:Jason.McKenna@windriver.com]
        > Sent: Tuesday, September 11, 2018 2:02 PM
        > To: Lara, Cesar <cesar.lara@intel.com>; Alonso, Juan Carlos
<juan.car
        > los.alonso@intel.com>; Khalil, Ghada <Ghada.Khalil@windrive
r.com>;
        > Martinez Monroy, Elio <elio.martinez.monroy@intel.com>;
Jones, Bruce
        > E <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@in
tel.com>;
        > starlingx-discuss@lists.starlingx.io; Thebeau, Michel
<Michel.Thebeau
        > @windriver.com>; Young, Ken <Ken.Young@windriver.com>;
Little, Scott
        > <Scott.Little@windriver.com>
        > Subject: RE: [ Test ] Sanity results - summary posted
        >
        > Hi Cesar,
        >
        > Would you elaborate on what differences you'd expect to see
in an
        > empty mirror populated by the download tool, compared to an
existing
        > mirror which was updated by the download tool?
        >
        > The mirror tools were written specifically to contain
multiple
        > versions of packages.  The build environment population
tools take
        > the .lst file you have checked out, and places that version
from the
        > mirror into your local environment guaranteeing that your
build env
        > matches the .lst file.  If your mirror contains versions
1.0.0,
        > 1.2.3, and 2.0.0 of a given package, then your environment
will only
        > use the version called out in the .lst file you have checked
out.
        >
        > This allows the same mirror to be used for multiple
branches, for
        > multiple releases, for multiple development environments,
etc, and
        > guarantees that a build can be reproduced down the road even
if the
        > build inputs disappear from the Internet at some point in
the
        > future.  As long as all the artifacts named in the .lst file
have
        > been downloaded at some point in the past, the build
environment
        > produced from a fresh mirror vs a continually updated mirror
will be
        > identical.
        >
        > The issue we are observing with the CentOS 7.5 update is
that the
        > versions of packages called out in the .lst files for the
centos75
        > branch of stx-tools do not match the versions of packages
called out
        > in centos75 branch of the other repos, like stx-integ.  The
other
        > branches are attempting to patch a (new) version of the
source code,
        > but the .lst file is only bringing in (or downloading) and
older
        > version of the source.
        >
        > -Jason
        >
        > > -----Original Message-----
        > > From: Lara, Cesar <cesar.lara@intel.com>
        > > Sent: September 11, 2018 1:44 PM
        > > To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>;
Khalil,
        > > Ghada
        > > <Ghada.Khalil@windriver.com>; Martinez Monroy, Elio
        > > <elio.martinez.monroy@intel.com>; Jones, Bruce E
        > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@in
tel.com>;
        > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel
        > > <Michel.Thebeau@windriver.com>
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results
- summary
        > > posted
        > >
        > > Hi all,
        > >
        > > Just wanted to point out that, to be able to reproduce the
issues
        > > from
        > > Intel's ISO, somebody will need to go and create a build
        > > environment
        > > from scratch, using external upstream repositories to
generate a
        > > new
        > > mirror, just the way we are doing it in our environment.
Failing to
        > > do
        > > this will result in a apples to oranges comparison.
        > >
        > > Regards
        > > Cesar Lara
        > >
        > > -----Original Message-----
        > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso@inte
l.com]
        > > Sent: Tuesday, September 11, 2018 10:54 AM
        > > To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez
Monroy,
        > > Elio
        > > <elio.martinez.monroy@intel.com>; Jones, Bruce E
        > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@in
tel.com>;
        > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel
        > > <Michel.Thebeau@windriver.com>
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results
- summary
        > > posted
        > >
        > > Hi Ghada,
        > >
        > > Yes, before shared our ISOs I followed such steps and got
the same
        > > issues.
        > >
        > > As I mention in the issue
        > > (https://bugs.launchpad.net/starlingx/+bug/1790213), I
could get
        > > the
        > > ISO from Michel and testes in our infrastructure, issues
not found.
        > > I
        > > guess there are differences between the way an ISO is
built by
        > > Intel
        > > and Wind River since Intel got issues but Wind River does
not. I
        > > think we should sync about this.
        > >
        > > Besides, our test infrastructure uses other commands to
generate
        > > VMs,
        > > add/install hosts (controllers and computes), such
commands was
        > > gotten
        > > from "tic_vb" repo, such repo was given by China team, if
I am not
        > > wrong such repo was based on Wind River documentation; our
test
        > > framework is based on "tic_vb". Anyway both ways got the
same
        > > results.
        > >
        > > Regards.
        > > Juan Carlos Alonso
        > >
        > > -----Original Message-----
        > > From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
        > > Sent: Tuesday, September 11, 2018 8:53 AM
        > > To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>;
Martinez
        > > Monroy, Elio <elio.martinez.monroy@intel.com>; Jones,
Bruce E
        > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@in
tel.com>;
        > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel
        > > <Michel.Thebeau@windriver.com>
        > > Subject: RE: [ Test ] Sanity results - summary posted
        > >
        > > Hello Juan,
        > > I read the notes in the Launchpad.
        > >
        > > And, just to confirm, are you following the instructions
under:
        > > https://wiki.openstack.org/wiki/StarlingX/Installation_Gu
ide_Virtua
        > > l_E
        > > nviro
        > > nment/Controller_Storage
        > > and still reproducing the two issues below?
        > >
        > > I was not able to definitively tell from the Launchpad
notes. It's
        > > important that both our teams are comparing apples to
apples.
        > >
        > > Thanks,
        > > Ghada
        > >
        > > -----Original Message-----
        > > From: Alonso, Juan Carlos [mailto:juan.carlos.alonso@inte
l.com]
        > > Sent: Monday, September 10, 2018 1:50 PM
        > > To: Khalil, Ghada; Martinez Monroy, Elio; Jones, Bruce E;
        > > Cabrales,
        > > Ada; starlingx-discuss@lists.starlingx.io; Thebeau, Michel
        > > Subject: RE: [ Test ] Sanity results - summary posted
        > >
        > > Hi,
        > >
        > > Such launchpads were updated.
        > >
        > > A VM was set to upload the ISO tx-2018-08-28-88.iso, such
ISO
        > > present
        > > two issues. Michael will upload his ISO, then we will re
test
        > > witch
        > > each iso in our environment.
        > >
        > > An script is being developed to get the differences
between ISOs.
        > >
        > > Regards.
        > > Juan Carlos Alonso
        > >
        > > -----Original Message-----
        > > From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
        > > Sent: Friday, September 7, 2018 4:43 PM
        > > To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez
Monroy,
        > > Elio
        > > <elio.martinez.monroy@intel.com>; Jones, Bruce E
        > > <bruce.e.jones@intel.com>; Cabrales, Ada <ada.cabrales@in
tel.com>;
        > > starlingx-discuss@lists.starlingx.io; Thebeau, Michel
        > > <Michel.Thebeau@windriver.com>
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results
- summary
        > > posted
        > >
        > > Hi Elio/Ada,
        > > Are there any updates from your side on these two
Launchpads?
        > > https://bugs.launchpad.net/starlingx/+bug/1790213
        > > https://bugs.launchpad.net/starlingx/+bug/1790716
        > >
        > > Michel Thebeau (Wind River) has been trying to reproduce
these two
        > > issues (following the starlingx wikis) without success for
two
        > > days
        > > now. He has been adding his findings to the bugs and
communicating
        > > with your team.
        > > As a result of his investigation, he will be pushing some
small
        > > fixes
        > > to stx- tools shortly (some cleanup).
        > >
        > > Please let us know if you need any further help from us.
        > >
        > > Thanks,
        > > Ghada
        > >
        > > -----Original Message-----
        > > From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
        > > Sent: Wednesday, September 05, 2018 5:39 PM
        > > To: Martinez Monroy, Elio; Jones, Bruce E; Cabrales, Ada;
        > > starlingx-
        > > discuss@lists.starlingx.io
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results
- summary
        > > posted
        > >
        > > Please see inline. There are notes in all the launchpads
as well.
        > >
        > > I have tagged the last bug for the October release. The
rest need
        > > more
        > > information to determine whether they gate or not: two
maybe
        > > procedural and one requires more information.
        > >
        > > -----Original Message-----
        > > From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@
intel.com]
        > > Sent: Wednesday, September 05, 2018 2:23 PM
        > > To: Jones, Bruce E; Cabrales, Ada;
        > > starlingx-discuss@lists.starlingx.io
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results
- summary
        > > posted
        > >
        > > My opinion below
        > >
        > > -----Original Message-----
        > > From: Jones, Bruce E [mailto:bruce.e.jones@intel.com]
        > > Sent: Wednesday, September 5, 2018 12:56 PM
        > > To: Cabrales, Ada <ada.cabrales@intel.com>; starlingx-
        > > discuss@lists.starlingx.io
        > > Subject: Re: [Starlingx-discuss] [ Test ] Sanity results
- summary
        > > posted
        > >
        > > Should the bugs coming out of this be tagged/fixed for the
October
        > > release?
        > >
        > > Related bugs:
        > > https://bugs.launchpad.net/starlingx/+bug/1790213 < ---
Must, this
        > > bug
        > > is related with 2nd controller provisioning [[GK]] Using
the
        > > instructions published on the wiki, our designer was not
able to
        > > reproduce this issue.
        > > What instructions are the test team using?
        > >
        > > https://bugs.launchpad.net/starlingx/+bug/1790514 < ---
Can wait
        > > until
        > > other issues are solved, will fail some test cases related
to SSH
        > > [[GK]] Requested more information. This cannot be debugged
further
        > > without this information.
        > >
        > > https://bugs.launchpad.net/starlingx/+bug/1790716 < ---
Must, all
        > > instances need at least one compute for multimode [[GK]]
Using the
        > > instructions published on the wiki, our designer was not
able to
        > > reproduce this issue. The test team points to internal
        > > instructions
        > > which we have no access to. The supported install recipes
need to
        > > be
        > > on the wiki. I think it is an invalid bug
        > >
        > > https://bugs.launchpad.net/starlingx/+bug/1790718 < ---
Can wait
        > > since
        > > is related to FM , test cases will fail [[GK]] As Brent
pointed
        > > out,
        > > this is dependent on the integration of stx-gui which is
in
        > > progress.
        > >
        > > -----Original Message-----
        > > From: Cabrales, Ada [mailto:ada.cabrales@intel.com]
        > > Sent: Wednesday, September 5, 2018 10:50 AM
        > > To: starlingx-discuss@lists.starlingx.io
        > > Subject: [Starlingx-discuss] [ Test ] Sanity results -
summary
        > > posted
        > >
        > > Hello all,
        > >
        > >     As mentioned on the project meeting today, we're
posting the
        > > summary of the sanity test runs on the Test wiki [0]. The
sanity
        > > is
        > > triggered each time an ISO is generated (a daily task).
        > >
        > >    Right now we are posting the results manually, and
working on a
        > > way
        > > for automating it. Logs of the run are not included, if
you want
        > > to
        > > have them, you can contact Elio to get them sent to you.
        > >
        > >
        > > [0]
        > > https://wiki.openstack.org/wiki/StarlingX/Test#Latest_ISO
_image_San
        > > ity
        > > _S
        > > ummary
        > >
        > > Regards
        > > Ada
        > >
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        > > _______________________________________________
        > > Starlingx-discuss mailing list
        > > Starlingx-discuss@lists.starlingx.io
        > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starli
ngx-discus
        > > s
        >
        > _______________________________________________
        > Starlingx-discuss mailing list
        > Starlingx-discuss@lists.starlingx.io
        > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starling
x-discuss
_______________________________________________
    Starlingx-discuss mailing list
    Starlingx-discuss@lists.starlingx.io
    http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-disc
uss
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss