[Starlingx-discuss] [build] go packages, version 2

[McKenna, Jason]

Hi Victor, great feedback.  Inline.

> -----Original Message-----
> From: Victor Rodriguez <vm.rod25 at gmail.com>
> Sent: January 16, 2019 10:49 AM
> To: McKenna, Jason <Jason.McKenna at windriver.com>
> Cc: starlingx-discuss at lists.starlingx.io
> Subject: Re: [Starlingx-discuss] [build] go packages, version 2
> 
> Hi Jason
> 
> 
> On Tue, Jan 15, 2019 at 11:57 AM McKenna, Jason
> <Jason.McKenna at windriver.com> wrote:
> >
> > Hi build team,
> >
> >
> >
> > At the previous build meeting I had identified an issue with the way some
> go based packages were being built be built (they required internet access),
> and promised I’d update the mailing list on a potential way forward that we
> were prototyping.
> >
> >
> >
> > Some preliminary points:
> >
> > -          go usually attempts to resolve dependencies at build time, by going
> out to the internet and fetching stuff (like dependency source code) using
> the “go get” command
> >
> > -          Sometimes the stuff fetched by “go get” isn’t appropriate (i.e. “go
> get” fetches the latest version, but deprecated APIs may have been
> removed, etc)
> >
> > -          Different versions of go packages may require different versions of
> dependencies
> >
> > -          We want builds to be reproducible without unexpected code changes
> (i.e. we want to know what we’re compiling in)
> >
> > -          Some people build in environments where they don’t have Internet
> access
> >
> >
> >
> > The initial solution (which didn’t take into account the Internet access
> problem) was to use “dep”.  “dep” is an external tool which was an “official
> experiment” of the go project.  Rather than fetch the latest dependencies
> from the internet (like “go get”), it allowed specific revisions of
> dependencies to be captured.  “dep” fetched those versions from the
> Internet.  This solved the deprecated API issue, the reproducible build issue,
> and the issue of not using rpms for the dependencies.  However, if someone
> was attempting to build in an internet-less context, the system would fail.
> >
> >
> >
> > Enter this second revision.
> >
> >
> >
> > The dependency packages are now downloaded at download-mirrors.sh
> time as tarballs.  The tarballs are produced as of a specific commit for each
> dependency.
> 
> I agree taht this is a good solution , in terms of make it work , my concern is
> that we are mantaining a lot of tar balls , and that scares me a bit

Agreed.  This actually is an opportunity to leverage the work that Marcela is doing where she is refactoring the download tarballs from a single .lst file into a more manageable form.  If we can separate the tarballs/rpms/srpms/etc from a single file to a per-repo file (or a per-package file for build-time artifacts...) then maintaining the tarball downloads becomes a lot cleaner.

> 
>  This allows us to hit all our bullet points – code is snapshotted, reproducible,
> we don’t end up having to create a bunch of new rpms with dependency
> source code and potential version conflicts, and it requires no internet access
> (other than at download-mirrors.sh time)
> 
> If it works for you is fine , my concern will be to mantain changes of broken
> links in the future ( that sure go dep will fix )
> 
> +1 from my part as long as it does not geneerate a lot of mantainance
> does not create a problem for ourselvs
> 
> How hard to aloud internet during build ?

Allowing Internet during build is easy if you have Internet access (edit the .cfg for your mock environment), but obviously troublesome if you're behind a firewall which blocks access.  While this doesn't affect me personally, I am under the impression that you folks have several sites which are configured this way (I could be wrong, but I seem to recall early in the project a few packages breaking because reworked packages assumed Internet).  Even if I'm mistaken about that point, I do know of several organizations which require clean-room builds for security and reproducibility reasons.  Better to not limit potential adopters if we can help it :)
 
> 
> Regards
> 
> >
> >
> >
> > Jerry has posted a preview code review showing his work using this
> mechanism.  I’ve marked the reviews as workflow -1 to give the build team a
> chance to see the mechanism.
> >
> > https://review.openstack.org/#/c/631001/
> >
> > https://review.openstack.org/#/c/631002/
> >
> >
> >
> > -Jason
> >
> > _______________________________________________
> > Starlingx-discuss mailing list
> > Starlingx-discuss at lists.starlingx.io
> > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss