[Starlingx-discuss] FW: [Build] Build layering reviews
Scott Little
scott.little at windriver.com
Thu Sep 26 15:38:46 UTC 2019
Returning this discussion to the mailing list.
If possible, join the Build call next week.
https://wiki.openstack.org/wiki/Starlingx/Meetings
1)
When mock installs for the first time. it uses rpms that have been
download via lst file. The mock pseudo-layer lst files capture reflect
the rpms needed by mock. The mock lst's are merged with the layers
lst's. In this way, the mock rpms don't have to be listed in each and
every layer's lst.
2)
Adding tools to generate an lst file reflacting the output of a build
layer, and adding that lst for consumprion by higher layers, is indeed a
possibility possibility I wanted to persue. Reference to that output
lst could then be added to as a new arg to the download script when
invoked by a higher layer, or a config file could specify where to find
it (cengn is the likely default supplier).
compile layer)
The compile layer is a true layer, built prior to the distro layer.
Only a few build related packages go here. Compilers, language
interpreters, rpm. I wanted those packages in place so that all distro
layer packages build with the same tools.
Scott
On 2019-09-25 10:24 p.m., Chen, Haochuan Z wrote:
>
> Still not clear about mock layer.
>
> 1, In build container, mock is already installed, why request to
> download mock required package firstly?
>
> 2, To build distro or flock layer, we could add compile layer’s tis
> rpm in distro and flock layer’s lst file. By download compile layer’s
> binary in mirror, it could also ensure user building package with same
> tool.
>
> Why introduces compile layer?
>
> Thanks!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> *From:*Scott Little <scott.little at windriver.com>
> *Sent:* Thursday, September 26, 2019 12:25 AM
> *To:* Chen, Haochuan Z <haochuan.z.chen at intel.com>
> *Subject:* Re: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> See inline ...
>
> Scott
>
> On 2019-09-23 11:00 a.m., Chen, Haochuan Z wrote:
>
> Hi Scott
>
> I studied your patch and some questions.
>
> 1, why there is mock and compile layer, what's usage or intention
> for these two layers?
>
> Build-pkgs uses a tool called mock to create a reproducible
> compilation sandbox. It is chroot like environmnet into which rpms
> are installed to construct a minimal build environment.
>
> The mock layer is not built. I created it as a placeholder for the
> lst files that downloads all the rpms that mock requires. Prior to
> building a layer, the lst files for both the layer being build, and
> the mock pseudo layer, need to be processed. Without the mock pseudo
> layer, we would have to replicate that list of mock required rpms in
> each and ever layer's lst files.
>
> The concept ran into a bit of trouble because some of the rpms needed
> by mock are themselves built by the compile or distro layers (with a
> newer revision). So the
> mock pseudo layer might actually omit those rpms from it's lst file,
> and they needed to be listed in the layer lst files instead.
>
> The compile layer is a true layer, built prior to the distro layer.
> Only a few build related packages go here. Compilers, language
> interpreters, rpm. I wanted those packages in place so that all
> distro layer packages build with the same tools.
>
>
>
> 2, for download_mirror.sh, it generate rpm list by merge file in
> config/centos/<layer>/<rpm list file name> and rpm list file in
> stx/<project name>
>
> But for generate-cgcs-centos-repo.sh, it use rpm list file in
> centos-mirror-tools folder. Rpm package should be added in two rpm
> list file.
>
> It is not good for maintenance. What about remove rpm list file
> in centos-mirror-tools folder
>
> /rpms_centos.lst/rpms_centos3rdparties.lst/rpms_3rdparties.lst/other_downloads.lst/
> tarball-dl.lst/
>
> That was considered and debated. The concern was that the many git
> repos that make up the layer would list different versions of the same
> package. Then what do we do? Throw an error? Silently use the
> newest package? So we elected to keep binary rpms needed to satisfy
> dependencies listed in a central place.
>
> At some point we may want to explore allowing lst files specify
> required package but not version (beyond a possible minimum version)
> when on the master branch, and have a way to 'lock down' the versions
> only for a release branch.
>
> 3, Question about setup yum repo for rpm tailing with “tis” such
> as xxx.*tis*.x86_64.rpm or xxx.*tis*.noarch.rpm
>
> In this patch https://review.opendev.org/#/c/681821/1
>
> /There is such line in
> config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo/
>
> /[Starlingx-cengn_distro_layer]/
>
> /name=Starlingx-cengn_distro_layer/
>
> /baseurl=http://127.0.0.1:8088/localdisk/loadbuild//*/slittle1/restructure3b/*//std/rpmbuild/RPMS//
>
> /enabled=1/
>
> /[Starlingx-cengn_distro_layer-rt]/
>
> /name=Starlingx-cengn_distro_layer/
>
> /baseurl=http://127.0.0.1:8088/localdisk/loadbuild//*/slittle1/restructure3b/*//rt/rpmbuild/RPMS//
>
> /enabled=1/
>
> also in
>
> config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> My understanding is rpm tailing with “tis” should also upload to
> cengn, rpm name listed in
> centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst
>
> So these three files should be unnecessary, with cengn update for
> layered build.
>
> config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo
>
> config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> For development testing, I had to hack together a simulated cengn by
> running lighttpd on 127.0.0.1:8088 and allowing it to serve up files
> from builds of lower layers
>
> restructure3a == layer-comiler layer build
>
> restructure3b == layer-distro layer build
>
> restructure3c == layer-flock layer build (although there shouldn't be
> a repo entry referring to that one)
>
> Finalizing the repo entries would have to wait until cengn was up an
> running.
>
> 4, In generate-cgcs-centos-repo.sh there is --layer-dir, and such
> content in your patch commit message
>
> /generate-cgcs-centos-repo.sh \/
>
> /--layer-dir=/localdisk/loadbuild/slittle1/layer-compile/std/rpmbuild/RPMS/
> \/
>
> /--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/std/rpmbuild/RPMS/
> \/
>
> /--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/rt/rpmbuild/RPMS/
> \/
>
> //import/mirrors/starlingx/
>
> For all rpm download from cengn, --layer-dir is unnecessary to
> generate link, correct?
>
> This is another way to do cross layer development testing without
> support from cengn. Again the goal is to allow an upper layer build
> to download rpms from a lower layer build on the same machine.
>
> 5, could you share
> build-tools/build-iso/centos_required_build_layer.cfg, share to me
>
> Did that not get published in my reviews? Content is very simple.
>
> compile,file:///localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc
> distro,file:///localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc
> flock,file:///localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc
>
> 6, For distro/compiler/flock, all are separate layer?
>
> Yes
>
> My current step
>
> /repo init -u https://opendev.org/starlingx/manifest.git
> <https://opendev.org/starlingx/manifest.git>/
>
> /apply this patch to .repo/manifests/
>
> /https://review.opendev.org/#/c/681914/1/
>
> /repo init -m compile.xml/
>
> /repo sync/
>
> /apply these to patch/
>
> /https://review.opendev.org/#/c/681821/1/
>
> /https://review.opendev.org/#/c/681828/1/
>
> /./download_mirror.sh -n -g -l flock -c
> config/centos/compiler/yum.conf.sample -S /
>
> /./toCopy/generate-cgts-centos-repo output/stx-r1/CentOS/pike//
>
> /build-pkgs/
>
> /build-iso/
>
> BR!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> -----Original Message-----
> From: Scott Little <scott.little at windriver.com>
> <mailto:scott.little at windriver.com>
> Sent: Thursday, September 19, 2019 9:54 PM
> To: Chen, Haochuan Z <haochuan.z.chen at intel.com>
> <mailto:haochuan.z.chen at intel.com>
> Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> Most of the required changes were already merged.
>
> I'll scan the spec when time permits to see if I missed anything.
>
> Scott
>
> On 2019-09-16 12:01 p.m., Chen, Haochuan Z wrote:
>
> > Hi Scott
>
> >
>
> > I read your patch for build layering, but could not find patch
> for build script.
>
> >
>
> > Description in spec https://review.opendev.org/#/c/672288/12/
> <https://review.opendev.org/#/c/672288/12/>
>
> > Modifications to package build scripts ... build-pkgs,
>
> > build-srpms-parallel, build-srpms-serial, build-rpms-parallel,
> build-rpms-serial (cgcs-root):
>
> >
>
> > Thanks
>
> >
>
> > Martin, Chen
>
> > SSP, Software Engineer
>
> > 021-61164330
>
> >
>
> > -----Original Message-----
>
> > From: Xie, Cindy
>
> > Sent: Friday, September 13, 2019 12:24 PM
>
> > To: Chen, Haochuan Z <haochuan.z.chen at intel.com
> <mailto:haochuan.z.chen at intel.com>>
>
> > Cc: Hu, Yong <yong.hu at intel.com <mailto:yong.hu at intel.com>>
>
> > Subject: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> >
>
> > Martin,
>
> > I am wondering if you will be interested to working on this
> task. To me, having layering build and provide efficiency to
> developers are very important for StarlingX. It also provide a
> chance that you and work together w/ WR folks being more close
> collaboration. Let me know your interested level.
>
> >
>
> > Thx. -cindy
>
> >
>
> > -----Original Message-----
>
> > From: Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>
>
> > Sent: Friday, September 13, 2019 4:55 AM
>
> > To: starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
>
> > Subject: [Starlingx-discuss] [Build] Build layering reviews
>
> >
>
> > My prototype code for the layered build has been posted to gerrit.
>
> > (listed below)
>
> >
>
> > I was briefly able to build all layers, and produce an iso with
> the same package content as the cengn build. Then I was diverted
> by non-trivial changes to the restructuring work. What I have
> posted is now several weeks out of date and will almost certainly
> will not work. The devil is in keeping the lst files in sync with
> the ever changing master branch.
>
> >
>
> > Other commitments will prevent me from resuming this work until
> November. If anyone wants to whip the lst files into shape and
> try to get the build working again, be my guest.
>
> >
>
> > Final delivery will require changes to the CENGN build scripts
> (Don or Myself), and a freeze on packaging changes will likely be
> required for aprox 3 days to finalize the lst files.
>
> >
>
> > Scott
>
> >
>
> > ---------------- <Snip> ----------------
>
> >
>
> > https://review.opendev.org/#/c/681821/1
>
> >
>
> > https://review.opendev.org/#/c/681914/1
>
> >
>
> > https://review.opendev.org/#/c/681823/1
>
> >
>
> > https://review.opendev.org/#/c/681824/1
>
> >
>
> > https://review.opendev.org/#/c/681828/1
>
> >
>
> > https://review.opendev.org/#/c/681832/2
>
> >
>
> > https://review.opendev.org/#/c/681834/1
>
> >
>
> > https://review.opendev.org/#/c/681836/1
>
> >
>
> > https://review.opendev.org/#/c/681835/1
>
> >
>
> > https://review.opendev.org/#/c/681842/1
>
> >
>
> > https://review.opendev.org/#/c/681847/1
>
> >
>
> > https://review.opendev.org/#/c/681848/1
>
> >
>
> > https://review.opendev.org/#/c/681851/1
>
> >
>
> > https://review.opendev.org/#/c/681852/1
>
> >
>
> > https://review.opendev.org/#/c/681854/1
>
> >
>
> > https://review.opendev.org/#/c/681856/1
>
> >
>
> > https://review.opendev.org/#/c/681890/1
>
> >
>
> > https://review.opendev.org/#/c/681892/1
>
> >
>
> > https://review.opendev.org/#/c/681892/1
>
> >
>
> > https://review.opendev.org/#/c/681898/1
>
> >
>
> > https://review.opendev.org/#/c/681900/1
>
> >
>
> > https://review.opendev.org/#/c/681901/1
>
> >
>
> > https://review.opendev.org/#/c/681902/1
>
> >
>
> >
>
> >
>
> >
>
> > _______________________________________________
>
> > Starlingx-discuss mailing list
>
> > Starlingx-discuss at lists.starlingx.io
> <mailto:Starlingx-discuss at lists.starlingx.io>
>
> > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190926/c0f98cfb/attachment-0001.html>
More information about the Starlingx-discuss
mailing list