[Starlingx-discuss] FW: [Build] Build layering reviews
Saul Wold
sgw at linux.intel.com
Mon Oct 21 23:32:24 UTC 2019
Martin,
I am still wondering how your testing this, if you have a completely
updated manifest that points to all the updated gerrit reviews?
As I mentioned in some reviews earlier I was having issues with building
due to missing files.
On 10/21/19 7:20 AM, Chen, Haochuan Z wrote:
> Hi
>
> I have update patch again. Last week, I raise this concern//
>
> 3, update script download_mirror.sh and generate-cgts-centos-repo
>
> Currently all rpm name in lst is listed with fixed rpm version, which
> makes layer build easily to be corrupted. For example, any time
> TIS_PATCH_VER in build_srpm.data update, rpm name in lst should be updated.
>
> Prefer rpm named with “tis” should be query candidate on cenga, which
> request download_mirror.sh and generate-cgts-centos-repo update. Prefer
> we could finish such update in this story, as I think this is a critical
> issue.
>
> So I update patch in stx-tools again to fix this issue. You can check
> download_mirror.sh
>
> I add a step to generate rpm_tis.lst for layer build case. In such case,
> it will download tis_pkg.lst from cenga. What listed in tis_pkg.lst is
> distro or compile layer’s built rpm with named xxx.tis.xxx.rpm.
>
> So no matter src rpm or tarball upgrade, rpm_centos3party.lst is
> unnecessary to update.
>
> And I update build-iso, image.inc should concatenate with distro and
> compile layer’s image.in. As image.inc not only for install package, but
> also request for generate comps.xml.
>
> Please check my update and review my patch
>
I am looking at this patch and doing some testing.
Sau!
> Thanks!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> *From:*Chen, Haochuan Z
> *Sent:* Tuesday, October 15, 2019 10:27 PM
> *To:* Penney, Don <Don.Penney at windriver.com>; Bailey, Henry Albert (Al)
> <Al.Bailey at windriver.com>; Little, Scott <Scott.Little at windriver.com>;
> 'starlingx-discuss at lists.starlingx.io'
> <starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: [Starlingx-discuss] FW: [Build] Build layering reviews
>
> Hi
>
> I have updated patch for layer build
>
> https://review.opendev.org/#/c/681821/
>
> https://review.opendev.org/#/c/681823/
>
> https://review.opendev.org/#/c/681824/
>
> https://review.opendev.org/#/c/681828/
>
> https://review.opendev.org/#/c/681832/
>
> https://review.opendev.org/#/c/681834/
>
> https://review.opendev.org/#/c/681835/
>
> https://review.opendev.org/#/c/681836/
>
> https://review.opendev.org/#/c/681842/
>
> https://review.opendev.org/#/c/681847/
>
> https://review.opendev.org/#/c/681848/
>
> https://review.opendev.org/#/c/681851/
>
> https://review.opendev.org/#/c/681852/
>
> https://review.opendev.org/#/c/681854/
>
> https://review.opendev.org/#/c/681856/
>
> https://review.opendev.org/#/c/681890/
>
> https://review.opendev.org/#/c/681892/
>
> https://review.opendev.org/#/c/681894/
>
> https://review.opendev.org/#/c/681898/
>
> https://review.opendev.org/#/c/681900/
>
> https://review.opendev.org/#/c/681901/
>
> https://review.opendev.org/#/c/681902/
>
> https://review.opendev.org/#/c/681914/
>
> https://review.opendev.org/#/c/688598/
>
> And I think we still left three issue.
>
> 1, request build-tools/build_iso/centos_required_build_layer.cfg
>
> Current for self validation, we could add a file with such content. But
> we should create a file depending on cenga update
>
> compile,file:///localdisk/loadbuild/martin/starlingx/centos_compile_iso_image.inc
>
> distro,file:///localdisk/loadbuild/martin/distro/centos_distro_iso_image.inc
>
> flock,file:///localdisk/loadbuild/martin/flock/centos_flock_iso_image.inc
>
> 2, propose use one copy of yum.conf.sample and yum.repo.d/*
>
> Currently there is 4 copies of yum.conf.sample and yum.repo.d/*. 3
> copies in stx-tools/centos-mirror-tools/config/centos/<layer>/ and 1
> copy in stx-tools/centos-mirror-tools/
>
> Propose to remove 3 copies in
> stx-tools/centos-mirror-tools/config/centos/<layer>/. To avoid user
> confusing
>
> 3, update script download_mirror.sh and generate-cgts-centos-repo
>
> Currently all rpm name in lst is listed with fixed rpm version, which
> makes layer build easily to be corrupted. For example, any time
> TIS_PATCH_VER in build_srpm.data update, rpm name in lst should be updated.
>
> Prefer rpm named with “tis” should be query candidate on cenga, which
> request download_mirror.sh and generate-cgts-centos-repo update. Prefer
> we could finish such update in this story, as I think this is a critical
> issue.
>
> Thanks!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> *From:*Penney, Don <Don.Penney at windriver.com
> <mailto:Don.Penney at windriver.com>>
> *Sent:* Monday, September 30, 2019 11:33 PM
> *To:* Bailey, Henry Albert (Al) <Al.Bailey at windriver.com
> <mailto:Al.Bailey at windriver.com>>; Chen, Haochuan Z
> <haochuan.z.chen at intel.com <mailto:haochuan.z.chen at intel.com>>; Little,
> Scott <Scott.Little at windriver.com <mailto:Scott.Little at windriver.com>>;
> 'starlingx-discuss at lists.starlingx.io'
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Subject:* RE: [Starlingx-discuss] FW: [Build] Build layering reviews
>
> Is this the issue?
>
> openstack/python-horizon/centos/python-django-horizon.spec:BuildRequires: cgts-client
>
> I don’t see anything in this package that should need this dependency,
> and it should be safe to remove it.
>
> *From:*Bailey, Henry Albert (Al) [mailto:Al.Bailey at windriver.com]
> *Sent:* Monday, September 30, 2019 10:56 AM
> *To:* Chen, Haochuan Z; Little, Scott;
> 'starlingx-discuss at lists.starlingx.io'
> *Subject:* Re: [Starlingx-discuss] FW: [Build] Build layering reviews
>
> I think Cgts-client (renamed to system-client) should be in its own repo.
>
> Al
>
> *From:*Chen, Haochuan Z [mailto:haochuan.z.chen at intel.com]
> *Sent:* Monday, September 30, 2019 2:06 AM
> *To:* Little, Scott; 'starlingx-discuss at lists.starlingx.io'
> *Subject:* Re: [Starlingx-discuss] FW: [Build] Build layering reviews
>
> Hi Scott
>
> I think have some question or concern.
>
> 1, I find distro layer has dependency to cgts-client, as there is a
> addin for Horizon, which enables look up flock service info in Horizon.
>
> So there is task is must for layer building, which is out of my
> expectation. I begin to tackle these task.
>
> ·Relocate from git 'config', subdir 'sysinv/cgts-client', to git
> 'utilities', subdir 'system-client', *with full git history.*
>
> ·Rename rpm package from 'cgts-client' to 'system-client'
>
> ·Fix rpm packages that list cgts-client as a require or build require
>
> ·Rename python module from 'cgts-client' to 'system-client'
>
> ·Scan function names and/or variables for any that contain 'cgts' ...
> replace with 'system' ... can we do this in a backward compatible way?
>
> ·Fix users of the client to import using the new name, and reference any
> renamed functions/data.
>
> ·Fix ISO and/or docker images that list this package for inclusion.
>
> 2, “all” layer maybe doesn’t work, warning!
>
> For current patch for generate-cgts-centos-repo.sh, all layer’s
> rpms_lst_file is generated by merge “compiler/distro/flock” three
> layers, config/centos/all doesn’t be referenced, which is none sense.
> Which means package such bash-4.2.46-31.el7.tis.4.x86_64.rpm, named with
> “tis” is also in mirror. This maybe interference source package
> building. I think request some fix in generate-cgts-centos-repo.sh to
> omit package with “tis” in name.
>
> 3, Layered build introduces rpms_cemtos.lst dependency on TIS_PATCH_VER
> in build_srpm.data, which make developer easily disrupt layered building
> mechanism.
>
> For distro and flock layer, it has dependency to starlingX customized
> package, for example, distro layer require bash.tis.<tis patch
> version>.<version>.rpm, which will be add in
> config/centos/distro/rpms_centos3rdparty.lst
>
> If new developer forget to update and build all package, it maybe
> disrupt layered building.
>
> BR!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> *From:*Chen, Haochuan Z
> *Sent:* Friday, September 27, 2019 4:40 PM
> *To:* Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* RE: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> Thanks, Scott. I have studied so much.
>
> BR!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> *From:*Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>
> *Sent:* Thursday, September 26, 2019 11:39 PM
> *To:* Chen, Haochuan Z <haochuan.z.chen at intel.com
> <mailto:haochuan.z.chen at intel.com>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Subject:* Re: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> Returning this discussion to the mailing list.
>
> If possible, join the Build call next week.
> https://wiki.openstack.org/wiki/Starlingx/Meetings
>
> 1)
>
> When mock installs for the first time. it uses rpms that have been
> download via lst file. The mock pseudo-layer lst files capture reflect
> the rpms needed by mock. The mock lst's are merged with the layers
> lst's. In this way, the mock rpms don't have to be listed in each and
> every layer's lst.
>
> 2)
>
> Adding tools to generate an lst file reflacting the output of a build
> layer, and adding that lst for consumprion by higher layers, is indeed a
> possibility possibility I wanted to persue. Reference to that output
> lst could then be added to as a new arg to the download script when
> invoked by a higher layer, or a config file could specify where to find
> it (cengn is the likely default supplier).
>
> compile layer)
>
> The compile layer is a true layer, built prior to the distro layer.
> Only a few build related packages go here. Compilers, language
> interpreters, rpm. I wanted those packages in place so that all distro
> layer packages build with the same tools.
>
> Scott
>
> On 2019-09-25 10:24 p.m., Chen, Haochuan Z wrote:
>
> Still not clear about mock layer.
>
> 1, In build container, mock is already installed, why request to
> download mock required package firstly?
>
> 2, To build distro or flock layer, we could add compile layer’s tis
> rpm in distro and flock layer’s lst file. By download compile
> layer’s binary in mirror, it could also ensure user building package
> with same tool.
>
> Why introduces compile layer?
>
> Thanks!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> *From:*Scott Little <scott.little at windriver.com>
> <mailto:scott.little at windriver.com>
> *Sent:* Thursday, September 26, 2019 12:25 AM
> *To:* Chen, Haochuan Z <haochuan.z.chen at intel.com>
> <mailto:haochuan.z.chen at intel.com>
> *Subject:* Re: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> See inline ...
>
> Scott
>
> On 2019-09-23 11:00 a.m., Chen, Haochuan Z wrote:
>
> Hi Scott
>
> I studied your patch and some questions.
>
> 1, why there is mock and compile layer, what's usage or
> intention for these two layers?
>
> Build-pkgs uses a tool called mock to create a reproducible
> compilation sandbox. It is chroot like environmnet into which rpms
> are installed to construct a minimal build environment.
>
> The mock layer is not built. I created it as a placeholder for the
> lst files that downloads all the rpms that mock requires. Prior to
> building a layer, the lst files for both the layer being build, and
> the mock pseudo layer, need to be processed. Without the mock pseudo
> layer, we would have to replicate that list of mock required rpms in
> each and ever layer's lst files.
>
> The concept ran into a bit of trouble because some of the rpms
> needed by mock are themselves built by the compile or distro layers
> (with a newer revision). So the
> mock pseudo layer might actually omit those rpms from it's lst file,
> and they needed to be listed in the layer lst files instead.
>
> The compile layer is a true layer, built prior to the distro layer.
> Only a few build related packages go here. Compilers, language
> interpreters, rpm. I wanted those packages in place so that all
> distro layer packages build with the same tools.
>
> 2, for download_mirror.sh, it generate rpm list by merge file in
> config/centos/<layer>/<rpm list file name> and rpm list file in
> stx/<project name>
>
> But for generate-cgcs-centos-repo.sh, it use rpm list file
> in centos-mirror-tools folder. Rpm package should be added in
> two rpm list file.
>
> It is not good for maintenance. What about remove rpm list
> file in centos-mirror-tools folder
>
> /rpms_centos.lst/rpms_centos3rdparties.lst/rpms_3rdparties.lst/other_downloads.lst/
> tarball-dl.lst/
>
> That was considered and debated. The concern was that the many git
> repos that make up the layer would list different versions of the
> same package. Then what do we do? Throw an error? Silently use
> the newest package? So we elected to keep binary rpms needed to
> satisfy dependencies listed in a central place.
>
> At some point we may want to explore allowing lst files specify
> required package but not version (beyond a possible minimum version)
> when on the master branch, and have a way to 'lock down' the
> versions only for a release branch.
>
> 3, Question about setup yum repo for rpm tailing with “tis” such
> as xxx.*tis*.x86_64.rpm or xxx.*tis*.noarch.rpm
>
> In this patch https://review.opendev.org/#/c/681821/1
>
> /There is such line in
> config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo/
>
> /[Starlingx-cengn_distro_layer]/
>
> /name=Starlingx-cengn_distro_layer/
>
> /baseurl=http://127.0.0.1:8088/localdisk/loadbuild//*/slittle1/restructure3b/*//std/rpmbuild/RPMS//
>
> /enabled=1/
>
> /[Starlingx-cengn_distro_layer-rt]/
>
> /name=Starlingx-cengn_distro_layer/
>
> /baseurl=http://127.0.0.1:8088/localdisk/loadbuild//*/slittle1/restructure3b/*//rt/rpmbuild/RPMS//
>
> /enabled=1/
>
> also in
>
> config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> My understanding is rpm tailing with “tis” should also upload to
> cengn, rpm name listed in
> centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst
>
> So these three files should be unnecessary, with cengn update
> for layered build.
>
> config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo
>
> config/centos/flock/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> config/centos/distro/yum.repos.d/StarlingX_cengn_compile_layer.repo
>
> For development testing, I had to hack together a simulated cengn by
> running lighttpd on 127.0.0.1:8088 and allowing it to serve up files
> from builds of lower layers
>
> restructure3a == layer-comiler layer build
>
> restructure3b == layer-distro layer build
>
> restructure3c == layer-flock layer build (although there shouldn't
> be a repo entry referring to that one)
>
> Finalizing the repo entries would have to wait until cengn was up an
> running.
>
> 4, In generate-cgcs-centos-repo.sh there is --layer-dir, and
> such content in your patch commit message
>
> /generate-cgcs-centos-repo.sh \/
>
> /--layer-dir=/localdisk/loadbuild/slittle1/layer-compile/std/rpmbuild/RPMS/
> \/
>
> /--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/std/rpmbuild/RPMS/
> \/
>
> /--layer-dir=/localdisk/loadbuild/slittle1/layer-distro/rt/rpmbuild/RPMS/
> \/
>
> //import/mirrors/starlingx/
>
> For all rpm download from cengn, --layer-dir is unnecessary
> to generate link, correct?
>
> This is another way to do cross layer development testing without
> support from cengn. Again the goal is to allow an upper layer build
> to download rpms from a lower layer build on the same machine.
>
> 5, could you share
> build-tools/build-iso/centos_required_build_layer.cfg, share to me
>
> Did that not get published in my reviews? Content is very simple.
>
> compile,file:///localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc
> <file://localdisk/loadbuild/slittle1/restructure3a/centos_compile_iso_image.inc>
> distro,file:///localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc
> <file://localdisk/loadbuild/slittle1/restructure3b/centos_distro_iso_image.inc>
> flock,file:///localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc
> <file://localdisk/loadbuild/slittle1/restructure3c/centos_flock_iso_image.inc>
>
> 6, For distro/compiler/flock, all are separate layer?
>
> Yes
>
> My current step
>
> /repo init -u https://opendev.org/starlingx/manifest.git/
>
> /apply this patch to .repo/manifests/
>
> /https://review.opendev.org/#/c/681914/1/
>
> /repo init -m compile.xml/
>
> /repo sync/
>
> /apply these to patch/
>
> /https://review.opendev.org/#/c/681821/1/
>
> /https://review.opendev.org/#/c/681828/1/
>
> /./download_mirror.sh -n -g -l flock -c
> config/centos/compiler/yum.conf.sample -S /
>
> /./toCopy/generate-cgts-centos-repo output/stx-r1/CentOS/pike//
>
> /build-pkgs/
>
> /build-iso/
>
> BR!
>
> Martin, Chen
>
> SSP, Software Engineer
>
> 021-61164330
>
> -----Original Message-----
> From: Scott Little <scott.little at windriver.com>
> <mailto:scott.little at windriver.com>
> Sent: Thursday, September 19, 2019 9:54 PM
> To: Chen, Haochuan Z <haochuan.z.chen at intel.com>
> <mailto:haochuan.z.chen at intel.com>
> Subject: Re: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> Most of the required changes were already merged.
>
> I'll scan the spec when time permits to see if I missed anything.
>
> Scott
>
> On 2019-09-16 12:01 p.m., Chen, Haochuan Z wrote:
>
> > Hi Scott
>
> >
>
> > I read your patch for build layering, but could not find
> patch for build script.
>
> >
>
> > Description in spec https://review.opendev.org/#/c/672288/12/
>
> > Modifications to package build scripts ... build-pkgs,
>
> > build-srpms-parallel, build-srpms-serial,
> build-rpms-parallel, build-rpms-serial (cgcs-root):
>
> >
>
> > Thanks
>
> >
>
> > Martin, Chen
>
> > SSP, Software Engineer
>
> > 021-61164330
>
> >
>
> > -----Original Message-----
>
> > From: Xie, Cindy
>
> > Sent: Friday, September 13, 2019 12:24 PM
>
> > To: Chen, Haochuan Z <haochuan.z.chen at intel.com
> <mailto:haochuan.z.chen at intel.com>>
>
> > Cc: Hu, Yong <yong.hu at intel.com <mailto:yong.hu at intel.com>>
>
> > Subject: FW: [Starlingx-discuss] [Build] Build layering reviews
>
> >
>
> > Martin,
>
> > I am wondering if you will be interested to working on this
> task. To me, having layering build and provide efficiency to
> developers are very important for StarlingX. It also provide a
> chance that you and work together w/ WR folks being more close
> collaboration. Let me know your interested level.
>
> >
>
> > Thx. -cindy
>
> >
>
> > -----Original Message-----
>
> > From: Scott Little <scott.little at windriver.com
> <mailto:scott.little at windriver.com>>
>
> > Sent: Friday, September 13, 2019 4:55 AM
>
> > To: starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
>
> > Subject: [Starlingx-discuss] [Build] Build layering reviews
>
> >
>
> > My prototype code for the layered build has been posted to
> gerrit.
>
> > (listed below)
>
> >
>
> > I was briefly able to build all layers, and produce an iso
> with the same package content as the cengn build. Then I was
> diverted by non-trivial changes to the restructuring work. What
> I have posted is now several weeks out of date and will almost
> certainly will not work. The devil is in keeping the lst files
> in sync with the ever changing master branch.
>
> >
>
> > Other commitments will prevent me from resuming this work
> until November. If anyone wants to whip the lst files into
> shape and try to get the build working again, be my guest.
>
> >
>
> > Final delivery will require changes to the CENGN build
> scripts (Don or Myself), and a freeze on packaging changes will
> likely be required for aprox 3 days to finalize the lst files.
>
> >
>
> > Scott
>
> >
>
> > ---------------- <Snip> ----------------
>
> >
>
> > https://review.opendev.org/#/c/681821/1
>
> >
>
> > https://review.opendev.org/#/c/681914/1
>
> >
>
> > https://review.opendev.org/#/c/681823/1
>
> >
>
> > https://review.opendev.org/#/c/681824/1
>
> >
>
> > https://review.opendev.org/#/c/681828/1
>
> >
>
> > https://review.opendev.org/#/c/681832/2
>
> >
>
> > https://review.opendev.org/#/c/681834/1
>
> >
>
> > https://review.opendev.org/#/c/681836/1
>
> >
>
> > https://review.opendev.org/#/c/681835/1
>
> >
>
> > https://review.opendev.org/#/c/681842/1
>
> >
>
> > https://review.opendev.org/#/c/681847/1
>
> >
>
> > https://review.opendev.org/#/c/681848/1
>
> >
>
> > https://review.opendev.org/#/c/681851/1
>
> >
>
> > https://review.opendev.org/#/c/681852/1
>
> >
>
> > https://review.opendev.org/#/c/681854/1
>
> >
>
> > https://review.opendev.org/#/c/681856/1
>
> >
>
> > https://review.opendev.org/#/c/681890/1
>
> >
>
> > https://review.opendev.org/#/c/681892/1
>
> >
>
> > https://review.opendev.org/#/c/681892/1
>
> >
>
> > https://review.opendev.org/#/c/681898/1
>
> >
>
> > https://review.opendev.org/#/c/681900/1
>
> >
>
> > https://review.opendev.org/#/c/681901/1
>
> >
>
> > https://review.opendev.org/#/c/681902/1
>
> >
>
> >
>
> >
>
> >
>
> > _______________________________________________
>
> > Starlingx-discuss mailing list
>
> > Starlingx-discuss at lists.starlingx.io
> <mailto:Starlingx-discuss at lists.starlingx.io>
>
> >
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
More information about the Starlingx-discuss
mailing list