[Starlingx-discuss] [Build] Build Avoidance

[Scott Little]

I've posted the Build Avoidance wiki

https://wiki.openstack.org/wiki/StarlingX/Developer_Guide#Build_Avoidance

Reviews welcome.

We can relocate it if there is a better home folks can suggest.

Scott



On 18-08-27 04:17 PM, Scott Little wrote:
>
> Build Avoidance, a build tool improvement.*
> *
>
> *Purpose:*
>
>    Greatly reduce build times after a repo sync for designers working 
> within a regional office.  For a new workspace, build-pkgs typically 
> requires 3+ hours, build avoidance typically reduces this step to ~20min.
>
> *Limitations:*
>
>    Little or no benefit for designers who refresh a pre-existing 
> workspace at least daily. (download_mirror.sh, repo sync, 
> generate-cgcs-centos-repo.sh, build-pkgs, build-iso).
>
>    Not likely to be useful to solo designers, or teleworkers that wish 
> to compile on there home computers.  WAN speeds are generally to slow.
>
> *Method (in brief):*
>
> 1) Reference builds
>
>    - A server performs a regular (daily?), automated builds using 
> existing methods.  Call these the reference builds.
>    - The builds are timestamped, and preserved for some time. (weeks?)
>    - A build CONTEXT is captured, consisting of the SHA of each and 
> every git that contributed to the build.
>    - For each package built, a file shall capture he md5sums of all 
> the source code inputs to the build of that package.
>    - All these build products are accessible locally (e.g. a regional 
> office) via rsync (other protocols can be added later)
>
> 2) Designers
>    - build-pkgs --build-avoidance   ... will request a build avoidance 
> build.
>    - Additional arguments, and/or environment variables, and/or a 
> config file unique to the regional office, are used to specify a URL 
> to the reference builds.
>    - build-pkgs will:
>      = From newest to oldest, scan the CONTEXTs of the various 
> reference builds.  Select the first (most recent) context which 
> satisfies:  For every git, the SHA specified in the CONTEXT is present.
>      = The selected context might be slightly out of date, but not by 
> more than a day (assuming daily reference builds).
>      = If the context has not been previously downloaded, then 
> download it now.  Meaning download select portions of the reference 
> build workspace into the designer's workspace.  This includes all the 
> SRPMS, RPMS, MD5SUMS, and misc supporting files. (~10 min over office LAN)
>      = The designer may have additional commits not present in the 
> reference build, or uncommitted changes.   Affected packages will 
> identified by the differing md5sum's, and the package is re-built. (5+ 
> min, depending on what packages have changed)
>
>
> *Requirement:*
>
>    - The regional office implements an automated build that pulls the 
> latest StarlingX software and builds it on a regular basis. e.g. a 
> daily.  Perhaps implemented by Jenkins, cron, or similar tools.
>    - Each build is saved to a unique directory, and preserved for a 
> time that is reflective of how long a designer might be expected to 
> work on a private branch without syncronizing with the master 
> branch.   e.g. 2 weeks.
>    - The MY_WORKSPACE directory for the build shall have a common root 
> directory, and a leaf directory that is a time stamp of format 
> YYYY-MM-DD_hh-mm-ss.  e.g. 
> MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/2018-07-19_11-30-21
>    - Designers can access all build products over the internal network 
> of the regional office.  The current prototype employs rsync. Other 
> protocols that can efficiently share/copy/transfer large directories 
> of content can be added as needed.
>
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181012/6554d54e/attachment-0001.html>