[Starlingx-discuss] [build] go packages, version 2

[McKenna, Jason]

Hi build team,

At the previous build meeting I had identified an issue with the way some go based packages were being built be built (they required internet access), and promised I'd update the mailing list on a potential way forward that we were prototyping.

Some preliminary points:

-          go usually attempts to resolve dependencies at build time, by going out to the internet and fetching stuff (like dependency source code) using the "go get" command

-          Sometimes the stuff fetched by "go get" isn't appropriate (i.e. "go get" fetches the latest version, but deprecated APIs may have been removed, etc)

-          Different versions of go packages may require different versions of dependencies

-          We want builds to be reproducible without unexpected code changes (i.e. we want to know what we're compiling in)

-          Some people build in environments where they don't have Internet access

The initial solution (which didn't take into account the Internet access problem) was to use "dep".  "dep" is an external tool which was an "official experiment" of the go project.  Rather than fetch the latest dependencies from the internet (like "go get"), it allowed specific revisions of dependencies to be captured.  "dep" fetched those versions from the Internet.  This solved the deprecated API issue, the reproducible build issue, and the issue of not using rpms for the dependencies.  However, if someone was attempting to build in an internet-less context, the system would fail.

Enter this second revision.

The dependency packages are now downloaded at download-mirrors.sh time as tarballs.  The tarballs are produced as of a specific commit for each dependency.  This allows us to hit all our bullet points - code is snapshotted, reproducible, we don't end up having to create a bunch of new rpms with dependency source code and potential version conflicts, and it requires no internet access (other than at download-mirrors.sh time)

Jerry has posted a preview code review showing his work using this mechanism.  I've marked the reviews as workflow -1 to give the build team a chance to see the mechanism.
https://review.openstack.org/#/c/631001/
https://review.openstack.org/#/c/631002/

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190115/c05a1da0/attachment.html>