Returning this discussion to the mailing list. If possible, join the Build call next week. https://wiki.openstack.org/wiki/Starlingx/Meetings 1) When mock installs for the first time. it uses rpms that have been download via lst file. The mock pseudo-layer lst files capture reflect the rpms needed by mock. The mock lst's are merged with the layers lst's. In this way, the mock rpms don't have to be listed in each and every layer's lst. 2) Adding tools to generate an lst file reflacting the output of a build layer, and adding that lst for consumprion by higher layers, is indeed a possibility possibility I wanted to persue. Reference to that output lst could then be added to as a new arg to the download script when invoked by a higher layer, or a config file could specify where to find it (cengn is the likely default supplier). compile layer) The compile layer is a true layer, built prior to the distro layer. Only a few build related packages go here. Compilers, language interpreters, rpm. I wanted those packages in place so that all distro layer packages build with the same tools. Scott On 2019-09-25 10:24 p.m., Chen, Haochuan Z wrote:
Still not clear about mock layer.
1, In build container, mock is already installed, why request to download mock required package firstly?
2, To build distro or flock layer, we could add compile layer’s tis rpm in distro and flock layer’s lst file. By download compile layer’s binary in mirror, it could also ensure user building package with same tool.
Why introduces compile layer?
Thanks!
Martin, Chen
SSP, Software Engineer
021-61164330
*From:*Scott Little <scott.little@windriver.com> *Sent:* Thursday, September 26, 2019 12:25 AM *To:* Chen, Haochuan Z <haochuan.z.chen@intel.com> *Subject:* Re: FW: [Starlingx-discuss] [Build] Build layering reviews
See inline ...
Scott
On 2019-09-23 11:00 a.m., Chen, Haochuan Z wrote:
Hi Scott
I studied your patch and some questions.
1, why there is mock and compile layer, what's usage or intention for these two layers?
Build-pkgs uses a tool called mock to create a reproducible compilation sandbox. It is chroot like environmnet into which rpms are installed to construct a minimal build environment.
The mock layer is not built. I created it as a placeholder for the lst files that downloads all the rpms that mock requires. Prior to building a layer, the lst files for both the layer being build, and the mock pseudo layer, need to be processed. Without the mock pseudo layer, we would have to replicate that list of mock required rpms in each and ever layer's lst files.
The concept ran into a bit of trouble because some of the rpms needed by mock are themselves built by the compile or distro layers (with a newer revision). So the mock pseudo layer might actually omit those rpms from it's lst file, and they needed to be listed in the layer lst files instead.
The compile layer is a true layer, built prior to the distro layer. Only a few build related packages go here. Compilers, language interpreters, rpm. I wanted those packages in place so that all distro layer packages build with the same tools.
2, for download_mirror.sh, it generate rpm list by merge file in config/centos/<layer>/<rpm list file name> and rpm list file in stx/<project name>
But for generate-cgcs-centos-repo.sh, it use rpm list file in centos-mirror-tools folder. Rpm package should be added in two rpm list file.
It is not good for maintenance. What about remove rpm list file in centos-mirror-tools folder
/rpms_centos.lst/rpms_centos3rdparties.lst/rpms_3rdparties.lst/other_downloads.lst/ tarball-dl.lst/
That was considered and debated. The concern was that the many git repos that make up the layer would list different versions of the same package. Then what do we do? Throw an error? Silently use the newest package? So we elected to keep binary rpms needed to satisfy dependencies listed in a central place.
At some point we may want to explore allowing lst files specify required package but not version (beyond a possible minimum version) when on the master branch, and have a way to 'lock down' the versions only for a release branch.
3, Question about setup yum repo for rpm tailing with “tis” such as xxx.*tis*.x86_64.rpm or xxx.*tis*.noarch.rpm
In this patch https://review.opendev.org/#/c/681821/1
/There is such line in config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo/
/[Starlingx-cengn_distro_layer]/
/name=Starlingx-cengn_distro_layer/
/baseurl=http://127.0.0.1:8088/localdisk/loadbuild//*/slittle1/restructure3b/*//std/r...
/enabled=1/
/[Starlingx-cengn_distro_layer-rt]/
/name=Starlingx-cengn_distro_layer/
/baseurl=http://127.0.0.1:8088/localdisk/loadbuild//*/slittle1/restructure3b/*//rt/rp...
/enabled=1/
also in
config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
My understanding is rpm tailing with “tis” should also upload to cengn, rpm name listed in centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst
So these three files should be unnecessary, with cengn update for layered build.
config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo
config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
For development testing, I had to hack together a simulated cengn by running lighttpd on 127.0.0.1:8088 and allowing it to serve up files from builds of lower layers
restructure3a == layer-comiler layer build
restructure3b == layer-distro layer build
restructure3c == layer-flock layer build (although there shouldn't be a repo entry referring to that one)
Finalizing the repo entries would have to wait until cengn was up an running.
4, In generate-cgcs-centos-repo.sh there is --layer-dir, and such content in your patch commit message
/generate-cgcs-centos-repo.sh \/
/--layer-dir=/localdisk/loadbuild/slittle1/layer-compile/std/rpmbuild/RPMS/ \/
/--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/std/rpmbuild/RPMS/ \/
/--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/rt/rpmbuild/RPMS/ \/
//import/mirrors/starlingx/
For all rpm download from cengn, --layer-dir is unnecessary to generate link, correct?
This is another way to do cross layer development testing without support from cengn. Again the goal is to allow an upper layer build to download rpms from a lower layer build on the same machine.
5, could you share build-tools/build-iso/centos_required_build_layer.cfg, share to me
Did that not get published in my reviews? Content is very simple.
compile,file:///localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc distro,file:///localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc flock,file:///localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc
6, For distro/compiler/flock, all are separate layer?
Yes
My current step
/repo init -u https://opendev.org/starlingx/manifest.git <https://opendev.org/starlingx/manifest.git>/
/apply this patch to .repo/manifests/
/https://review.opendev.org/#/c/681914/1/
/repo init -m compile.xml/
/repo sync/
/apply these to patch/
/https://review.opendev.org/#/c/681821/1/
/https://review.opendev.org/#/c/681828/1/
/./download_mirror.sh -n -g -l flock -c config/centos/compiler/yum.conf.sample -S /
/./toCopy/generate-cgts-centos-repo output/stx-r1/CentOS/pike//
/build-pkgs/
/build-iso/
BR!
Martin, Chen
SSP, Software Engineer
021-61164330
-----Original Message----- From: Scott Little <scott.little@windriver.com> <mailto:scott.little@windriver.com> Sent: Thursday, September 19, 2019 9:54 PM To: Chen, Haochuan Z <haochuan.z.chen@intel.com> <mailto:haochuan.z.chen@intel.com> Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews
Most of the required changes were already merged.
I'll scan the spec when time permits to see if I missed anything.
Scott
On 2019-09-16 12:01 p.m., Chen, Haochuan Z wrote:
> Hi Scott
>
> I read your patch for build layering, but could not find patch for build script.
>
> Description in spec https://review.opendev.org/#/c/672288/12/ <https://review.opendev.org/#/c/672288/12/>
> Modifications to package build scripts ... build-pkgs,
> build-srpms-parallel, build-srpms-serial, build-rpms-parallel, build-rpms-serial (cgcs-root):
>
> Thanks
>
> Martin, Chen
> SSP, Software Engineer
> 021-61164330
>
> -----Original Message-----
> From: Xie, Cindy
> Sent: Friday, September 13, 2019 12:24 PM
> To: Chen, Haochuan Z <haochuan.z.chen@intel.com <mailto:haochuan.z.chen@intel.com>>
> Cc: Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>
> Subject: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> Martin,
> I am wondering if you will be interested to working on this task. To me, having layering build and provide efficiency to developers are very important for StarlingX. It also provide a chance that you and work together w/ WR folks being more close collaboration. Let me know your interested level.
>
> Thx. -cindy
>
> -----Original Message-----
> From: Scott Little <scott.little@windriver.com <mailto:scott.little@windriver.com>>
> Sent: Friday, September 13, 2019 4:55 AM
> To: starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>
> Subject: [Starlingx-discuss] [Build] Build layering reviews
>
> My prototype code for the layered build has been posted to gerrit.
> (listed below)
>
> I was briefly able to build all layers, and produce an iso with the same package content as the cengn build. Then I was diverted by non-trivial changes to the restructuring work. What I have posted is now several weeks out of date and will almost certainly will not work. The devil is in keeping the lst files in sync with the ever changing master branch.
>
> Other commitments will prevent me from resuming this work until November. If anyone wants to whip the lst files into shape and try to get the build working again, be my guest.
>
> Final delivery will require changes to the CENGN build scripts (Don or Myself), and a freeze on packaging changes will likely be required for aprox 3 days to finalize the lst files.
>
> Scott
>
> ---------------- <Snip> ----------------
>
> https://review.opendev.org/#/c/681821/1
>
> https://review.opendev.org/#/c/681914/1
>
> https://review.opendev.org/#/c/681823/1
>
> https://review.opendev.org/#/c/681824/1
>
> https://review.opendev.org/#/c/681828/1
>
> https://review.opendev.org/#/c/681832/2
>
> https://review.opendev.org/#/c/681834/1
>
> https://review.opendev.org/#/c/681836/1
>
> https://review.opendev.org/#/c/681835/1
>
> https://review.opendev.org/#/c/681842/1
>
> https://review.opendev.org/#/c/681847/1
>
> https://review.opendev.org/#/c/681848/1
>
> https://review.opendev.org/#/c/681851/1
>
> https://review.opendev.org/#/c/681852/1
>
> https://review.opendev.org/#/c/681854/1
>
> https://review.opendev.org/#/c/681856/1
>
> https://review.opendev.org/#/c/681890/1
>
> https://review.opendev.org/#/c/681892/1
>
> https://review.opendev.org/#/c/681892/1
>
> https://review.opendev.org/#/c/681898/1
>
> https://review.opendev.org/#/c/681900/1
>
> https://review.opendev.org/#/c/681901/1
>
> https://review.opendev.org/#/c/681902/1
>
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io>
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss