[Starlingx-discuss] [build] go packages, version 2

[Victor Rodriguez]

On Thu, Jan 17, 2019 at 9:14 AM McKenna, Jason
<Jason.McKenna at windriver.com> wrote:
>
> Hi Victor, great feedback.  Inline.
>
> > -----Original Message-----
> > From: Victor Rodriguez <vm.rod25 at gmail.com>
> > Sent: January 16, 2019 10:49 AM
> > To: McKenna, Jason <Jason.McKenna at windriver.com>
> > Cc: starlingx-discuss at lists.starlingx.io
> > Subject: Re: [Starlingx-discuss] [build] go packages, version 2
> >
> > Hi Jason
> >
> >
> > On Tue, Jan 15, 2019 at 11:57 AM McKenna, Jason
> > <Jason.McKenna at windriver.com> wrote:
> > >
> > > Hi build team,
> > >
> > >
> > >
> > > At the previous build meeting I had identified an issue with the way some
> > go based packages were being built be built (they required internet access),
> > and promised I’d update the mailing list on a potential way forward that we
> > were prototyping.
> > >
> > >
> > >
> > > Some preliminary points:
> > >
> > > -          go usually attempts to resolve dependencies at build time, by going
> > out to the internet and fetching stuff (like dependency source code) using
> > the “go get” command
> > >
> > > -          Sometimes the stuff fetched by “go get” isn’t appropriate (i.e. “go
> > get” fetches the latest version, but deprecated APIs may have been
> > removed, etc)
> > >
> > > -          Different versions of go packages may require different versions of
> > dependencies
> > >
> > > -          We want builds to be reproducible without unexpected code changes
> > (i.e. we want to know what we’re compiling in)
> > >
> > > -          Some people build in environments where they don’t have Internet
> > access
> > >
> > >
> > >
> > > The initial solution (which didn’t take into account the Internet access
> > problem) was to use “dep”.  “dep” is an external tool which was an “official
> > experiment” of the go project.  Rather than fetch the latest dependencies
> > from the internet (like “go get”), it allowed specific revisions of
> > dependencies to be captured.  “dep” fetched those versions from the
> > Internet.  This solved the deprecated API issue, the reproducible build issue,
> > and the issue of not using rpms for the dependencies.  However, if someone
> > was attempting to build in an internet-less context, the system would fail.
> > >
> > >
> > >
> > > Enter this second revision.
> > >
> > >
> > >
> > > The dependency packages are now downloaded at download-mirrors.sh
> > time as tarballs.  The tarballs are produced as of a specific commit for each
> > dependency.
> >
> > I agree taht this is a good solution , in terms of make it work , my concern is
> > that we are mantaining a lot of tar balls , and that scares me a bit
>
> Agreed.  This actually is an opportunity to leverage the work that Marcela is doing where she is refactoring the download tarballs from a single .lst file into a more manageable form.  If we can separate the tarballs/rpms/srpms/etc from a single file to a per-repo file (or a per-package file for build-time artifacts...) then maintaining the tarball downloads becomes a lot cleaner.

Ok , i think is a good oportunity for the work that Marcela is doing

>
> >
> >  This allows us to hit all our bullet points – code is snapshotted, reproducible,
> > we don’t end up having to create a bunch of new rpms with dependency
> > source code and potential version conflicts, and it requires no internet access
> > (other than at download-mirrors.sh time)
> >
> > If it works for you is fine , my concern will be to mantain changes of broken
> > links in the future ( that sure go dep will fix )
> >
> > +1 from my part as long as it does not geneerate a lot of mantainance
> > does not create a problem for ourselvs
> >
> > How hard to aloud internet during build ?
>
> Allowing Internet during build is easy if you have Internet access (edit the .cfg for your mock environment), but obviously troublesome if you're behind a firewall which blocks access.  While this doesn't affect me personally, I am under the impression that you folks have several sites which are configured this way (I could be wrong, but I seem to recall early in the project a few packages breaking because reworked packages assumed Internet).  Even if I'm mistaken about that point, I do know of several organizations which require clean-room builds for security and reproducibility reasons.  Better to not limit potential adopters if we can help it :)

Ok, I see your point, well even if is behind a firewall it should have
some proxy solution but yes, let's assume that we have a user that
does not have internet.  My only concern will be that I am not a fan
of having many tarballs/patches or change that we have to maintain
/host. If we are going to host them, Do we make a copy of them in the
official CENGN repo? and point our scripts over there to be 100% sure
that the download will work all the time?

regards


>
> >
> > Regards
> >
> > >
> > >
> > >
> > > Jerry has posted a preview code review showing his work using this
> > mechanism.  I’ve marked the reviews as workflow -1 to give the build team a
> > chance to see the mechanism.
> > >
> > > https://review.openstack.org/#/c/631001/
> > >
> > > https://review.openstack.org/#/c/631002/
> > >
> > >
> > >
> > > -Jason
> > >
> > > _______________________________________________
> > > Starlingx-discuss mailing list
> > > Starlingx-discuss at lists.starlingx.io
> > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss