[Starlingx-discuss] [Build] Build Avoidance

Scott Little scott.little at windriver.com
Mon Aug 27 20:17:55 UTC 2018


Build Avoidance, a build tool improvement.*
*

*Purpose:*

    Greatly reduce build times after a repo sync for designers working 
within a regional office.  For a new workspace, build-pkgs typically 
requires 3+ hours, build avoidance typically reduces this step to ~20min.

*Limitations:*

    Little or no benefit for designers who refresh a pre-existing 
workspace at least daily. (download_mirror.sh, repo sync, 
generate-cgcs-centos-repo.sh, build-pkgs, build-iso).

    Not likely to be useful to solo designers, or teleworkers that wish 
to compile on there home computers.  WAN speeds are generally to slow.

*Method (in brief):*

1) Reference builds

    - A server performs a regular (daily?), automated builds using 
existing methods.  Call these the reference builds.
    - The builds are timestamped, and preserved for some time. (weeks?)
    - A build CONTEXT is captured, consisting of the SHA of each and 
every git that contributed to the build.
    - For each package built, a file shall capture he md5sums of all the 
source code inputs to the build of that package.
    - All these build products are accessible locally (e.g. a regional 
office) via rsync (other protocols can be added later)

2) Designers
    - build-pkgs --build-avoidance   ... will request a build avoidance 
build.
    - Additional arguments, and/or environment variables, and/or a 
config file unique to the regional office, are used to specify a URL to 
the reference builds.
    - build-pkgs will:
      = From newest to oldest, scan the CONTEXTs of the various 
reference builds.  Select the first (most recent) context which 
satisfies:  For every git, the SHA specified in the CONTEXT is present.
      = The selected context might be slightly out of date, but not by 
more than a day (assuming daily reference builds).
      = If the context has not been previously downloaded, then download 
it now.  Meaning download select portions of the reference build 
workspace into the designer's workspace.  This includes all the SRPMS, 
RPMS, MD5SUMS, and misc supporting files.  (~10 min over office LAN)
      = The designer may have additional commits not present in the 
reference build, or uncommitted changes.   Affected packages will 
identified by the differing md5sum's, and the package is re-built. (5+ 
min, depending on what packages have changed)


*Requirement:*

    - The regional office implements an automated build that pulls the 
latest StarlingX software and builds it on a regular basis. e.g. a 
daily.  Perhaps implemented by Jenkins, cron, or similar tools.
    - Each build is saved to a unique directory, and preserved for a 
time that is reflective of how long a designer might be expected to work 
on a private branch without syncronizing with the master branch.   e.g. 
2 weeks.
    - The MY_WORKSPACE directory for the build shall have a common root 
directory, and a leaf directory that is a time stamp of format 
YYYY-MM-DD_hh-mm-ss.  e.g. 
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/2018-07-19_11-30-21
    - Designers can access all build products over the internal network 
of the regional office.  The current prototype employs rsync. Other 
protocols that can efficiently share/copy/transfer large directories of 
content can be added as needed.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20180827/6ba893e1/attachment.html>


More information about the Starlingx-discuss mailing list