[Starlingx-discuss] FW: [Build] Build layering reviews
Bailey, Henry Albert (Al)
Al.Bailey at windriver.com
Mon Sep 30 14:55:44 UTC 2019
I think Cgts-client (renamed to system-client) should be in its own repo.
Al
From: Chen, Haochuan Z [mailto:haochuan.z.chen at intel.com]
Sent: Monday, September 30, 2019 2:06 AM
To: Little, Scott; 'starlingx-discuss at lists.starlingx.io'
Subject: Re: [Starlingx-discuss] FW: [Build] Build layering reviews
Hi Scott
I think have some question or concern.
1, I find distro layer has dependency to cgts-client, as there is a addin for Horizon, which enables look up flock service info in Horizon.
So there is task is must for layer building, which is out of my expectation. I begin to tackle these task.
• Relocate from git 'config', subdir 'sysinv/cgts-client', to git 'utilities', subdir 'system-client', with full git history.
• Rename rpm package from 'cgts-client' to 'system-client'
• Fix rpm packages that list cgts-client as a require or build require
• Rename python module from 'cgts-client' to 'system-client'
• Scan function names and/or variables for any that contain 'cgts' ... replace with 'system' ... can we do this in a backward compatible way?
• Fix users of the client to import using the new name, and reference any renamed functions/data.
• Fix ISO and/or docker images that list this package for inclusion.
2, “all” layer maybe doesn’t work, warning!
For current patch for generate-cgts-centos-repo.sh, all layer’s rpms_lst_file is generated by merge “compiler/distro/flock” three layers, config/centos/all doesn’t be referenced, which is none sense. Which means package such bash-4.2.46-31.el7.tis.4.x86_64.rpm, named with “tis” is also in mirror. This maybe interference source package building. I think request some fix in generate-cgts-centos-repo.sh to omit package with “tis” in name.
3, Layered build introduces rpms_cemtos.lst dependency on TIS_PATCH_VER in build_srpm.data, which make developer easily disrupt layered building mechanism.
For distro and flock layer, it has dependency to starlingX customized package, for example, distro layer require bash.tis.<tis patch version>.<version>.rpm, which will be add in config/centos/distro/rpms_centos3rdparty.lst
If new developer forget to update and build all package, it maybe disrupt layered building.
BR!
Martin, Chen
SSP, Software Engineer
021-61164330
From: Chen, Haochuan Z
Sent: Friday, September 27, 2019 4:40 PM
To: Scott Little <scott.little at windriver.com>; starlingx-discuss at lists.starlingx.io
Subject: RE: FW: [Starlingx-discuss] [Build] Build layering reviews
Thanks, Scott. I have studied so much.
BR!
Martin, Chen
SSP, Software Engineer
021-61164330
From: Scott Little <scott.little at windriver.com<mailto:scott.little at windriver.com>>
Sent: Thursday, September 26, 2019 11:39 PM
To: Chen, Haochuan Z <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews
Returning this discussion to the mailing list.
If possible, join the Build call next week. https://wiki.openstack.org/wiki/Starlingx/Meetings
1)
When mock installs for the first time. it uses rpms that have been download via lst file. The mock pseudo-layer lst files capture reflect the rpms needed by mock. The mock lst's are merged with the layers lst's. In this way, the mock rpms don't have to be listed in each and every layer's lst.
2)
Adding tools to generate an lst file reflacting the output of a build layer, and adding that lst for consumprion by higher layers, is indeed a possibility possibility I wanted to persue. Reference to that output lst could then be added to as a new arg to the download script when invoked by a higher layer, or a config file could specify where to find it (cengn is the likely default supplier).
compile layer)
The compile layer is a true layer, built prior to the distro layer. Only a few build related packages go here. Compilers, language interpreters, rpm. I wanted those packages in place so that all distro layer packages build with the same tools.
Scott
On 2019-09-25 10:24 p.m., Chen, Haochuan Z wrote:
Still not clear about mock layer.
1, In build container, mock is already installed, why request to download mock required package firstly?
2, To build distro or flock layer, we could add compile layer’s tis rpm in distro and flock layer’s lst file. By download compile layer’s binary in mirror, it could also ensure user building package with same tool.
Why introduces compile layer?
Thanks!
Martin, Chen
SSP, Software Engineer
021-61164330
From: Scott Little <scott.little at windriver.com><mailto:scott.little at windriver.com>
Sent: Thursday, September 26, 2019 12:25 AM
To: Chen, Haochuan Z <haochuan.z.chen at intel.com><mailto:haochuan.z.chen at intel.com>
Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews
See inline ...
Scott
On 2019-09-23 11:00 a.m., Chen, Haochuan Z wrote:
Hi Scott
I studied your patch and some questions.
1, why there is mock and compile layer, what's usage or intention for these two layers?
Build-pkgs uses a tool called mock to create a reproducible compilation sandbox. It is chroot like environmnet into which rpms are installed to construct a minimal build environment.
The mock layer is not built. I created it as a placeholder for the lst files that downloads all the rpms that mock requires. Prior to building a layer, the lst files for both the layer being build, and the mock pseudo layer, need to be processed. Without the mock pseudo layer, we would have to replicate that list of mock required rpms in each and ever layer's lst files.
The concept ran into a bit of trouble because some of the rpms needed by mock are themselves built by the compile or distro layers (with a newer revision). So the
mock pseudo layer might actually omit those rpms from it's lst file, and they needed to be listed in the layer lst files instead.
The compile layer is a true layer, built prior to the distro layer. Only a few build related packages go here. Compilers, language interpreters, rpm. I wanted those packages in place so that all distro layer packages build with the same tools.
2, for download_mirror.sh, it generate rpm list by merge file in config/centos/<layer>/<rpm list file name> and rpm list file in stx/<project name>
But for generate-cgcs-centos-repo.sh, it use rpm list file in centos-mirror-tools folder. Rpm package should be added in two rpm list file.
It is not good for maintenance. What about remove rpm list file in centos-mirror-tools folder
rpms_centos.lst/rpms_centos3rdparties.lst/rpms_3rdparties.lst/other_downloads.lst/ tarball-dl.lst
That was considered and debated. The concern was that the many git repos that make up the layer would list different versions of the same package. Then what do we do? Throw an error? Silently use the newest package? So we elected to keep binary rpms needed to satisfy dependencies listed in a central place.
At some point we may want to explore allowing lst files specify required package but not version (beyond a possible minimum version) when on the master branch, and have a way to 'lock down' the versions only for a release branch.
3, Question about setup yum repo for rpm tailing with “tis” such as xxx.tis.x86_64.rpm or xxx.tis.noarch.rpm
In this patch https://review.opendev.org/#/c/681821/1
There is such line in config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo
[Starlingx-cengn_distro_layer]
name=Starlingx-cengn_distro_layer
baseurl=http://127.0.0.1:8088/localdisk/loadbuild/slittle1/restructure3b/std/rpmbuild/RPMS/
enabled=1
[Starlingx-cengn_distro_layer-rt]
name=Starlingx-cengn_distro_layer
baseurl=http://127.0.0.1:8088/localdisk/loadbuild/slittle1/restructure3b/rt/rpmbuild/RPMS/
enabled=1
also in
config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
My understanding is rpm tailing with “tis” should also upload to cengn, rpm name listed in centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst
So these three files should be unnecessary, with cengn update for layered build.
config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo
config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
For development testing, I had to hack together a simulated cengn by running lighttpd on 127.0.0.1:8088 and allowing it to serve up files from builds of lower layers
restructure3a == layer-comiler layer build
restructure3b == layer-distro layer build
restructure3c == layer-flock layer build (although there shouldn't be a repo entry referring to that one)
Finalizing the repo entries would have to wait until cengn was up an running.
4, In generate-cgcs-centos-repo.sh there is --layer-dir, and such content in your patch commit message
generate-cgcs-centos-repo.sh \
--layer-dir=/localdisk/loadbuild/slittle1/layer-compile/std/rpmbuild/RPMS/ \
--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/std/rpmbuild/RPMS/ \
--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/rt/rpmbuild/RPMS/ \
/import/mirrors/starlingx
For all rpm download from cengn, --layer-dir is unnecessary to generate link, correct?
This is another way to do cross layer development testing without support from cengn. Again the goal is to allow an upper layer build to download rpms from a lower layer build on the same machine.
5, could you share build-tools/build-iso/centos_required_build_layer.cfg, share to me
Did that not get published in my reviews? Content is very simple.
compile,file:///localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc<file:///\\localdisk\loadbuild\slittle1\restructure3a\centos_compile_iso_image.inc>
distro,file:///localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc<file:///\\localdisk\loadbuild\slittle1\restructure3b\centos_distro_iso_image.inc>
flock,file:///localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc<file:///\\localdisk\loadbuild\slittle1\restructure3c\centos_flock_iso_image.inc>
6, For distro/compiler/flock, all are separate layer?
Yes
My current step
repo init -u https://opendev.org/starlingx/manifest.git
apply this patch to .repo/manifests
https://review.opendev.org/#/c/681914/1
repo init -m compile.xml
repo sync
apply these to patch
https://review.opendev.org/#/c/681821/1
https://review.opendev.org/#/c/681828/1
./download_mirror.sh -n -g -l flock -c config/centos/compiler/yum.conf.sample -S
./toCopy/generate-cgts-centos-repo output/stx-r1/CentOS/pike/
build-pkgs
build-iso
BR!
Martin, Chen
SSP, Software Engineer
021-61164330
-----Original Message-----
From: Scott Little <scott.little at windriver.com><mailto:scott.little at windriver.com>
Sent: Thursday, September 19, 2019 9:54 PM
To: Chen, Haochuan Z <haochuan.z.chen at intel.com><mailto:haochuan.z.chen at intel.com>
Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews
Most of the required changes were already merged.
I'll scan the spec when time permits to see if I missed anything.
Scott
On 2019-09-16 12:01 p.m., Chen, Haochuan Z wrote:
> Hi Scott
>
> I read your patch for build layering, but could not find patch for build script.
>
> Description in spec https://review.opendev.org/#/c/672288/12/
> Modifications to package build scripts ... build-pkgs,
> build-srpms-parallel, build-srpms-serial, build-rpms-parallel, build-rpms-serial (cgcs-root):
>
> Thanks
>
> Martin, Chen
> SSP, Software Engineer
> 021-61164330
>
> -----Original Message-----
> From: Xie, Cindy
> Sent: Friday, September 13, 2019 12:24 PM
> To: Chen, Haochuan Z <haochuan.z.chen at intel.com<mailto:haochuan.z.chen at intel.com>>
> Cc: Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>
> Subject: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> Martin,
> I am wondering if you will be interested to working on this task. To me, having layering build and provide efficiency to developers are very important for StarlingX. It also provide a chance that you and work together w/ WR folks being more close collaboration. Let me know your interested level.
>
> Thx. -cindy
>
> -----Original Message-----
> From: Scott Little <scott.little at windriver.com<mailto:scott.little at windriver.com>>
> Sent: Friday, September 13, 2019 4:55 AM
> To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
> Subject: [Starlingx-discuss] [Build] Build layering reviews
>
> My prototype code for the layered build has been posted to gerrit.
> (listed below)
>
> I was briefly able to build all layers, and produce an iso with the same package content as the cengn build. Then I was diverted by non-trivial changes to the restructuring work. What I have posted is now several weeks out of date and will almost certainly will not work. The devil is in keeping the lst files in sync with the ever changing master branch.
>
> Other commitments will prevent me from resuming this work until November. If anyone wants to whip the lst files into shape and try to get the build working again, be my guest.
>
> Final delivery will require changes to the CENGN build scripts (Don or Myself), and a freeze on packaging changes will likely be required for aprox 3 days to finalize the lst files.
>
> Scott
>
> ---------------- <Snip> ----------------
>
> https://review.opendev.org/#/c/681821/1
>
> https://review.opendev.org/#/c/681914/1
>
> https://review.opendev.org/#/c/681823/1
>
> https://review.opendev.org/#/c/681824/1
>
> https://review.opendev.org/#/c/681828/1
>
> https://review.opendev.org/#/c/681832/2
>
> https://review.opendev.org/#/c/681834/1
>
> https://review.opendev.org/#/c/681836/1
>
> https://review.opendev.org/#/c/681835/1
>
> https://review.opendev.org/#/c/681842/1
>
> https://review.opendev.org/#/c/681847/1
>
> https://review.opendev.org/#/c/681848/1
>
> https://review.opendev.org/#/c/681851/1
>
> https://review.opendev.org/#/c/681852/1
>
> https://review.opendev.org/#/c/681854/1
>
> https://review.opendev.org/#/c/681856/1
>
> https://review.opendev.org/#/c/681890/1
>
> https://review.opendev.org/#/c/681892/1
>
> https://review.opendev.org/#/c/681892/1
>
> https://review.opendev.org/#/c/681898/1
>
> https://review.opendev.org/#/c/681900/1
>
> https://review.opendev.org/#/c/681901/1
>
> https://review.opendev.org/#/c/681902/1
>
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190930/b3ae4663/attachment-0001.html>
More information about the Starlingx-discuss
mailing list