Hi Penney and Scott 1, As I explained yesterday, you should update these file depending on your cenga setup tools/centos-mirror-tools/config/centos/flock/required_build_tis_pkg.cfg tools/centos-mirror-tools/config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo tools/centos-mirror-tools/config/centos/flock/yum.repos.d/ StarlingX_cengn_distro_layer.repo tools/centos-mirror-tools/config/centos/distro/required_build_tis_pkg.cfg tools/centos-mirror-tools/config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo root/build-tools/build_iso/centos_required_build_layer.cfg 2, I add a parameter for tools/toCOPY/generate-cgcs-centos-repo.sh, “—layer=<layer>”. Take attention. Thanks! Martin, Chen SSP, Software Engineer 021-61164330 From: Chen, Haochuan Z Sent: Monday, October 21, 2019 10:20 PM To: 'Penney, Don' <Don.Penney@windriver.com>; 'Bailey, Henry Albert (Al)' <Al.Bailey@windriver.com>; 'Little, Scott' <Scott.Little@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] FW: [Build] Build layering reviews Hi I have update patch again. Last week, I raise this concern 3, update script download_mirror.sh and generate-cgts-centos-repo Currently all rpm name in lst is listed with fixed rpm version, which makes layer build easily to be corrupted. For example, any time TIS_PATCH_VER in build_srpm.data update, rpm name in lst should be updated. Prefer rpm named with “tis” should be query candidate on cenga, which request download_mirror.sh and generate-cgts-centos-repo update. Prefer we could finish such update in this story, as I think this is a critical issue. So I update patch in stx-tools again to fix this issue. You can check download_mirror.sh I add a step to generate rpm_tis.lst for layer build case. In such case, it will download tis_pkg.lst from cenga. What listed in tis_pkg.lst is distro or compile layer’s built rpm with named xxx.tis.xxx.rpm. So no matter src rpm or tarball upgrade, rpm_centos3party.lst is unnecessary to update. And I update build-iso, image.inc should concatenate with distro and compile layer’s image.in. As image.inc not only for install package, but also request for generate comps.xml. Please check my update and review my patch Thanks! Martin, Chen SSP, Software Engineer 021-61164330 From: Chen, Haochuan Z Sent: Tuesday, October 15, 2019 10:27 PM To: Penney, Don <Don.Penney@windriver.com<mailto:Don.Penney@windriver.com>>; Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] FW: [Build] Build layering reviews Hi I have updated patch for layer build https://review.opendev.org/#/c/681821/ https://review.opendev.org/#/c/681823/ https://review.opendev.org/#/c/681824/ https://review.opendev.org/#/c/681828/ https://review.opendev.org/#/c/681832/ https://review.opendev.org/#/c/681834/ https://review.opendev.org/#/c/681835/ https://review.opendev.org/#/c/681836/ https://review.opendev.org/#/c/681842/ https://review.opendev.org/#/c/681847/ https://review.opendev.org/#/c/681848/ https://review.opendev.org/#/c/681851/ https://review.opendev.org/#/c/681852/ https://review.opendev.org/#/c/681854/ https://review.opendev.org/#/c/681856/ https://review.opendev.org/#/c/681890/ https://review.opendev.org/#/c/681892/ https://review.opendev.org/#/c/681894/ https://review.opendev.org/#/c/681898/ https://review.opendev.org/#/c/681900/ https://review.opendev.org/#/c/681901/ https://review.opendev.org/#/c/681902/ https://review.opendev.org/#/c/681914/ https://review.opendev.org/#/c/688598/ And I think we still left three issue. 1, request build-tools/build_iso/centos_required_build_layer.cfg Current for self validation, we could add a file with such content. But we should create a file depending on cenga update compile,file:///localdisk/loadbuild/martin/starlingx/centos_compile_iso_image.inc distro,file:///localdisk/loadbuild/martin/distro/centos_distro_iso_image.inc flock,file:///localdisk/loadbuild/martin/flock/centos_flock_iso_image.inc 2, propose use one copy of yum.conf.sample and yum.repo.d/* Currently there is 4 copies of yum.conf.sample and yum.repo.d/*. 3 copies in stx-tools/centos-mirror-tools/config/centos/<layer>/ and 1 copy in stx-tools/centos-mirror-tools/ Propose to remove 3 copies in stx-tools/centos-mirror-tools/config/centos/<layer>/. To avoid user confusing 3, update script download_mirror.sh and generate-cgts-centos-repo Currently all rpm name in lst is listed with fixed rpm version, which makes layer build easily to be corrupted. For example, any time TIS_PATCH_VER in build_srpm.data update, rpm name in lst should be updated. Prefer rpm named with “tis” should be query candidate on cenga, which request download_mirror.sh and generate-cgts-centos-repo update. Prefer we could finish such update in this story, as I think this is a critical issue. Thanks! Martin, Chen SSP, Software Engineer 021-61164330 From: Penney, Don <Don.Penney@windriver.com<mailto:Don.Penney@windriver.com>> Sent: Monday, September 30, 2019 11:33 PM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com<mailto:Al.Bailey@windriver.com>>; Chen, Haochuan Z <haochuan.z.chen@intel.com<mailto:haochuan.z.chen@intel.com>>; Little, Scott <Scott.Little@windriver.com<mailto:Scott.Little@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] FW: [Build] Build layering reviews Is this the issue? openstack/python-horizon/centos/python-django-horizon.spec:BuildRequires: cgts-client I don’t see anything in this package that should need this dependency, and it should be safe to remove it. From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Monday, September 30, 2019 10:56 AM To: Chen, Haochuan Z; Little, Scott; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] FW: [Build] Build layering reviews I think Cgts-client (renamed to system-client) should be in its own repo. Al From: Chen, Haochuan Z [mailto:haochuan.z.chen@intel.com] Sent: Monday, September 30, 2019 2:06 AM To: Little, Scott; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] FW: [Build] Build layering reviews Hi Scott I think have some question or concern. 1, I find distro layer has dependency to cgts-client, as there is a addin for Horizon, which enables look up flock service info in Horizon. So there is task is must for layer building, which is out of my expectation. I begin to tackle these task. • Relocate from git 'config', subdir 'sysinv/cgts-client', to git 'utilities', subdir 'system-client', with full git history. • Rename rpm package from 'cgts-client' to 'system-client' • Fix rpm packages that list cgts-client as a require or build require • Rename python module from 'cgts-client' to 'system-client' • Scan function names and/or variables for any that contain 'cgts' ... replace with 'system' ... can we do this in a backward compatible way? • Fix users of the client to import using the new name, and reference any renamed functions/data. • Fix ISO and/or docker images that list this package for inclusion. 2, “all” layer maybe doesn’t work, warning! For current patch for generate-cgts-centos-repo.sh, all layer’s rpms_lst_file is generated by merge “compiler/distro/flock” three layers, config/centos/all doesn’t be referenced, which is none sense. Which means package such bash-4.2.46-31.el7.tis.4.x86_64.rpm, named with “tis” is also in mirror. This maybe interference source package building. I think request some fix in generate-cgts-centos-repo.sh to omit package with “tis” in name. 3, Layered build introduces rpms_cemtos.lst dependency on TIS_PATCH_VER in build_srpm.data, which make developer easily disrupt layered building mechanism. For distro and flock layer, it has dependency to starlingX customized package, for example, distro layer require bash.tis.<tis patch version>.<version>.rpm, which will be add in config/centos/distro/rpms_centos3rdparty.lst If new developer forget to update and build all package, it maybe disrupt layered building. BR! Martin, Chen SSP, Software Engineer 021-61164330 From: Chen, Haochuan Z Sent: Friday, September 27, 2019 4:40 PM To: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: FW: [Starlingx-discuss] [Build] Build layering reviews Thanks, Scott. I have studied so much. BR! Martin, Chen SSP, Software Engineer 021-61164330 From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Thursday, September 26, 2019 11:39 PM To: Chen, Haochuan Z <haochuan.z.chen@intel.com<mailto:haochuan.z.chen@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews Returning this discussion to the mailing list. If possible, join the Build call next week. https://wiki.openstack.org/wiki/Starlingx/Meetings 1) When mock installs for the first time. it uses rpms that have been download via lst file. The mock pseudo-layer lst files capture reflect the rpms needed by mock. The mock lst's are merged with the layers lst's. In this way, the mock rpms don't have to be listed in each and every layer's lst. 2) Adding tools to generate an lst file reflacting the output of a build layer, and adding that lst for consumprion by higher layers, is indeed a possibility possibility I wanted to persue. Reference to that output lst could then be added to as a new arg to the download script when invoked by a higher layer, or a config file could specify where to find it (cengn is the likely default supplier). compile layer) The compile layer is a true layer, built prior to the distro layer. Only a few build related packages go here. Compilers, language interpreters, rpm. I wanted those packages in place so that all distro layer packages build with the same tools. Scott On 2019-09-25 10:24 p.m., Chen, Haochuan Z wrote: Still not clear about mock layer. 1, In build container, mock is already installed, why request to download mock required package firstly? 2, To build distro or flock layer, we could add compile layer’s tis rpm in distro and flock layer’s lst file. By download compile layer’s binary in mirror, it could also ensure user building package with same tool. Why introduces compile layer? Thanks! Martin, Chen SSP, Software Engineer 021-61164330 From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: Thursday, September 26, 2019 12:25 AM To: Chen, Haochuan Z <haochuan.z.chen@intel.com><mailto:haochuan.z.chen@intel.com> Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews See inline ... Scott On 2019-09-23 11:00 a.m., Chen, Haochuan Z wrote: Hi Scott I studied your patch and some questions. 1, why there is mock and compile layer, what's usage or intention for these two layers? Build-pkgs uses a tool called mock to create a reproducible compilation sandbox. It is chroot like environmnet into which rpms are installed to construct a minimal build environment. The mock layer is not built. I created it as a placeholder for the lst files that downloads all the rpms that mock requires. Prior to building a layer, the lst files for both the layer being build, and the mock pseudo layer, need to be processed. Without the mock pseudo layer, we would have to replicate that list of mock required rpms in each and ever layer's lst files. The concept ran into a bit of trouble because some of the rpms needed by mock are themselves built by the compile or distro layers (with a newer revision). So the mock pseudo layer might actually omit those rpms from it's lst file, and they needed to be listed in the layer lst files instead. The compile layer is a true layer, built prior to the distro layer. Only a few build related packages go here. Compilers, language interpreters, rpm. I wanted those packages in place so that all distro layer packages build with the same tools. 2, for download_mirror.sh, it generate rpm list by merge file in config/centos/<layer>/<rpm list file name> and rpm list file in stx/<project name> But for generate-cgcs-centos-repo.sh, it use rpm list file in centos-mirror-tools folder. Rpm package should be added in two rpm list file. It is not good for maintenance. What about remove rpm list file in centos-mirror-tools folder rpms_centos.lst/rpms_centos3rdparties.lst/rpms_3rdparties.lst/other_downloads.lst/ tarball-dl.lst That was considered and debated. The concern was that the many git repos that make up the layer would list different versions of the same package. Then what do we do? Throw an error? Silently use the newest package? So we elected to keep binary rpms needed to satisfy dependencies listed in a central place. At some point we may want to explore allowing lst files specify required package but not version (beyond a possible minimum version) when on the master branch, and have a way to 'lock down' the versions only for a release branch. 3, Question about setup yum repo for rpm tailing with “tis” such as xxx.tis.x86_64.rpm or xxx.tis.noarch.rpm In this patch https://review.opendev.org/#/c/681821/1 There is such line in config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo [Starlingx-cengn_distro_layer] name=Starlingx-cengn_distro_layer baseurl=http://127.0.0.1:8088/localdisk/loadbuild/slittle1/restructure3b/std/rpmbuil... enabled=1 [Starlingx-cengn_distro_layer-rt] name=Starlingx-cengn_distro_layer baseurl=http://127.0.0.1:8088/localdisk/loadbuild/slittle1/restructure3b/rt/rpmbuild... enabled=1 also in config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo My understanding is rpm tailing with “tis” should also upload to cengn, rpm name listed in centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst So these three files should be unnecessary, with cengn update for layered build. config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo For development testing, I had to hack together a simulated cengn by running lighttpd on 127.0.0.1:8088 and allowing it to serve up files from builds of lower layers restructure3a == layer-comiler layer build restructure3b == layer-distro layer build restructure3c == layer-flock layer build (although there shouldn't be a repo entry referring to that one) Finalizing the repo entries would have to wait until cengn was up an running. 4, In generate-cgcs-centos-repo.sh there is --layer-dir, and such content in your patch commit message generate-cgcs-centos-repo.sh \ --layer-dir=/localdisk/loadbuild/slittle1/layer-compile/std/rpmbuild/RPMS/ \ --layer-dir=/localdisk/loadbuild/slittle1/layer-distro/std/rpmbuild/RPMS/ \ --layer-dir=/localdisk/loadbuild/slittle1/layer-distro/rt/rpmbuild/RPMS/ \ /import/mirrors/starlingx For all rpm download from cengn, --layer-dir is unnecessary to generate link, correct? This is another way to do cross layer development testing without support from cengn. Again the goal is to allow an upper layer build to download rpms from a lower layer build on the same machine. 5, could you share build-tools/build-iso/centos_required_build_layer.cfg, share to me Did that not get published in my reviews? Content is very simple. compile,file:///localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc<file://localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc> distro,file:///localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc<file://localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc> flock,file:///localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc<file://localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc> 6, For distro/compiler/flock, all are separate layer? Yes My current step repo init -u https://opendev.org/starlingx/manifest.git apply this patch to .repo/manifests https://review.opendev.org/#/c/681914/1 repo init -m compile.xml repo sync apply these to patch https://review.opendev.org/#/c/681821/1 https://review.opendev.org/#/c/681828/1 ./download_mirror.sh -n -g -l flock -c config/centos/compiler/yum.conf.sample -S ./toCopy/generate-cgts-centos-repo output/stx-r1/CentOS/pike/ build-pkgs build-iso BR! Martin, Chen SSP, Software Engineer 021-61164330 -----Original Message----- From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: Thursday, September 19, 2019 9:54 PM To: Chen, Haochuan Z <haochuan.z.chen@intel.com><mailto:haochuan.z.chen@intel.com> Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews Most of the required changes were already merged. I'll scan the spec when time permits to see if I missed anything. Scott On 2019-09-16 12:01 p.m., Chen, Haochuan Z wrote:
Hi Scott
I read your patch for build layering, but could not find patch for build script.
Description in spec https://review.opendev.org/#/c/672288/12/ Modifications to package build scripts ... build-pkgs, build-srpms-parallel, build-srpms-serial, build-rpms-parallel, build-rpms-serial (cgcs-root):
Thanks
Martin, Chen SSP, Software Engineer 021-61164330
-----Original Message----- From: Xie, Cindy Sent: Friday, September 13, 2019 12:24 PM To: Chen, Haochuan Z <haochuan.z.chen@intel.com<mailto:haochuan.z.chen@intel.com>> Cc: Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> Subject: FW: [Starlingx-discuss] [Build] Build layering reviews
Martin, I am wondering if you will be interested to working on this task. To me, having layering build and provide efficiency to developers are very important for StarlingX. It also provide a chance that you and work together w/ WR folks being more close collaboration. Let me know your interested level.
Thx. -cindy
-----Original Message----- From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Friday, September 13, 2019 4:55 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [Build] Build layering reviews
My prototype code for the layered build has been posted to gerrit. (listed below)
I was briefly able to build all layers, and produce an iso with the same package content as the cengn build. Then I was diverted by non-trivial changes to the restructuring work. What I have posted is now several weeks out of date and will almost certainly will not work. The devil is in keeping the lst files in sync with the ever changing master branch.
Other commitments will prevent me from resuming this work until November. If anyone wants to whip the lst files into shape and try to get the build working again, be my guest.
Final delivery will require changes to the CENGN build scripts (Don or Myself), and a freeze on packaging changes will likely be required for aprox 3 days to finalize the lst files.
Scott
---------------- <Snip> ----------------
https://review.opendev.org/#/c/681821/1
https://review.opendev.org/#/c/681914/1
https://review.opendev.org/#/c/681823/1
https://review.opendev.org/#/c/681824/1
https://review.opendev.org/#/c/681828/1
https://review.opendev.org/#/c/681832/2
https://review.opendev.org/#/c/681834/1
https://review.opendev.org/#/c/681836/1
https://review.opendev.org/#/c/681835/1
https://review.opendev.org/#/c/681842/1
https://review.opendev.org/#/c/681847/1
https://review.opendev.org/#/c/681848/1
https://review.opendev.org/#/c/681851/1
https://review.opendev.org/#/c/681852/1
https://review.opendev.org/#/c/681854/1
https://review.opendev.org/#/c/681856/1
https://review.opendev.org/#/c/681890/1
https://review.opendev.org/#/c/681892/1
https://review.opendev.org/#/c/681892/1
https://review.opendev.org/#/c/681898/1
https://review.opendev.org/#/c/681900/1
https://review.opendev.org/#/c/681901/1
https://review.opendev.org/#/c/681902/1
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss