[Starlingx-discuss] [Build] Build Avoidance algorithm

Scott Little scott.little at windriver.com
Thu Apr 16 13:44:50 UTC 2020


For those that are curious about the current Build Avoidance algorithm...

Note: The foundational principle of build avoidance is that it is faster 
to download the rpms than it is to build them.  This is likely only true 
the host for reference builds and the consumer are close to each other 
and share a high speed link.  It is not practical for 
mirror.starlingx.cengn.ca to serve as provider of reference builds for 
the world.  The real goal is for a corporate office to have a provider 
of reference builds to the designers within their corporate network.

Which reference build server to use?

- A data file, not supplied by the STX gits, is required to describe your reference build server. It's location should be ...
   $MY_REPO/local-build-data/build_avoidance_source
- The required fields, and hypothetical values are ...

    BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
    BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
    BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
    BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
    BUILD_AVOIDANCE_DATE_UTC=0

    BUILD_AVOIDANCE_FILE_TRANSFER="rsync"

    BUILD_AVOIDANCE_USR="jenkins"
    BUILD_AVOIDANCE_HOST="my-builder.my-company.com"
    BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/master"

Reference build server

- The reference build server should build regularly, e.g. daily
- The MY_WORKSPACE variable set prior to a reference build should be ...
   TIMESTAMP=$(date +${BUILD_AVOIDANCE_DATE_FORMAT}${BUILD_AVOIDANCE_DATE_TIME_DELIM}${BUILD_AVOIDANCE_TIME_FORMAT}${BUILD_AVOIDANCE_DATE_TIME_POSTFIX})
   export MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${TIMESTAMP}
- Builds should be preserved for a useful period of time.  e.g. at least two weeks

Which reference build to use?

- The reference builds side
   = Build contexts of all builds are collected into a common directory.
   = Context files are prefixed by the build time stamp allowing chronological traversal of the files.
- On the consumer side
   = The set of available reference build context are downloaded
   = Traverse the set of available build contexts from newest to oldest.
     + If all sha of all gits in a candidate reference build are also present in our local git context we stop traversal and will use this reference build.
   = If selected reference build is newer than the last (if any) reference build that we downloaded, then download the selected build context, else do nothing.

Downloading a selected reference build

- The list of artifacts to download is captured in datafile $MY_REPO/build-data/build_avoidance_source.
   = The following paths are relative to $MY_WORKSPACE/$BUILD_TYPE

   BUILD_AVOIDANCE_SRPM_DIRECTORIES="inputs srpm_assemble rpmbuild/SRPMS rpmbuild/SOURCES"
   BUILD_AVOIDANCE_SRPM_FILES=""
   BUILD_AVOIDANCE_RPM_DIRECTORIES="results rpmbuild/RPMS rpmbuild/SPECS repo/cgcs-tis-repo/dependancy-cache"
   BUILD_AVOIDANCE_RPM_FILES=".platform_release"

   = details of the files and directories downloaded

   inputs = Working directory used to assemble srpms from git or tarball
   srpm_assemble = working directory used to assemble srpms from upstream srpms
   rpmbuild/SRPMS = Assembled stx src.rpms that we will build
   rpmbuild/SOURCES = Additional per package metadata data collected to support build avoidance
      rpmbuild/SOURCES/<package-name>/srpm_reference.md5 = md5sums of all files that go into building the STX src.rpm
   results = per package build logs and artifacts generated by mockchain
   rpmbuild/RPMS = Build rpms
   rpmbuild/SPECS = spec files of build rpms
   repo/cgcs-tis-repo/dependancy-cache = build-pkgs data summarazing
       = The 'Requires' of rpms
       = the 'BuildRequires' of src.rpms
       = What rpms are derived from what src.rpms
   .platform_release = the platform release value

- The reference builds side
   = The only extra step to support build avoidance is to generate rpmbuild/SOURCES/<package-name>/srpm_reference.md5 files

- On the consumer side
   = For each build type we want...
     + For each file or subdirectory listed in $MY_REPO/build-data/build_avoidance_source
       * rsync the file or directory with options to preserve the file time stamp

Build src.rpms
- for each build type
   = For each package
     + Generate a list of input files for the current package
     + Generate a srpm_reference.md5 file for the current inputs.
     + compare srpm_reference.md5 files for current and reference builds.
       * If different (list of files, or md5sum of those files), we need to rebuild this src.rpm

Build RPMS
- for each build type
   = For each package
     + or list of rpms build by src.rpm
       * if rpm is missing, must rebuild package
       * if rpm is wrong version, must rebuild package
       * if rpm older than src.rpm, then must rebuild package
         NOTE: Assumes reference build and consumer are on NTP time, and any drift is well below the download time for the reference build

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200416/dff4ff06/attachment-0001.html>


More information about the Starlingx-discuss mailing list