[Starlingx-discuss] [build] go packages, version 2
Hi build team, At the previous build meeting I had identified an issue with the way some go based packages were being built be built (they required internet access), and promised I'd update the mailing list on a potential way forward that we were prototyping. Some preliminary points: - go usually attempts to resolve dependencies at build time, by going out to the internet and fetching stuff (like dependency source code) using the "go get" command - Sometimes the stuff fetched by "go get" isn't appropriate (i.e. "go get" fetches the latest version, but deprecated APIs may have been removed, etc) - Different versions of go packages may require different versions of dependencies - We want builds to be reproducible without unexpected code changes (i.e. we want to know what we're compiling in) - Some people build in environments where they don't have Internet access The initial solution (which didn't take into account the Internet access problem) was to use "dep". "dep" is an external tool which was an "official experiment" of the go project. Rather than fetch the latest dependencies from the internet (like "go get"), it allowed specific revisions of dependencies to be captured. "dep" fetched those versions from the Internet. This solved the deprecated API issue, the reproducible build issue, and the issue of not using rpms for the dependencies. However, if someone was attempting to build in an internet-less context, the system would fail. Enter this second revision. The dependency packages are now downloaded at download-mirrors.sh time as tarballs. The tarballs are produced as of a specific commit for each dependency. This allows us to hit all our bullet points - code is snapshotted, reproducible, we don't end up having to create a bunch of new rpms with dependency source code and potential version conflicts, and it requires no internet access (other than at download-mirrors.sh time) Jerry has posted a preview code review showing his work using this mechanism. I've marked the reviews as workflow -1 to give the build team a chance to see the mechanism. https://review.openstack.org/#/c/631001/ https://review.openstack.org/#/c/631002/ -Jason
We will need a wiki update on the procedure to use. i.e. how do I develop a list of tarballs I need to add to the .lst? On 2019-01-15 1:56 p.m., McKenna, Jason wrote:
Hi build team,
At the previous build meeting I had identified an issue with the way some go based packages were being built be built (they required internet access), and promised I’d update the mailing list on a potential way forward that we were prototyping.
Some preliminary points:
-go usually attempts to resolve dependencies at build time, by going out to the internet and fetching stuff (like dependency source code) using the “go get” command
-Sometimes the stuff fetched by “go get” isn’t appropriate (i.e. “go get” fetches the latest version, but deprecated APIs may have been removed, etc)
-Different versions of go packages may require different versions of dependencies
-We want builds to be reproducible without unexpected code changes (i.e. we want to know what we’re compiling in)
-Some people build in environments where they don’t have Internet access
The initial solution (which didn’t take into account the Internet access problem) was to use “dep”. “dep” is an external tool which was an “official experiment” of the go project. Rather than fetch the latest dependencies from the internet (like “go get”), it allowed specific revisions of dependencies to be captured. “dep” fetched those versions from the Internet. This solved the deprecated API issue, the reproducible build issue, and the issue of not using rpms for the dependencies. However, if someone was attempting to build in an internet-less context, the system would fail.
Enter this second revision.
The dependency packages are now downloaded at download-mirrors.sh time as tarballs. The tarballs are produced as of a specific commit for each dependency. This allows us to hit all our bullet points – code is snapshotted, reproducible, we don’t end up having to create a bunch of new rpms with dependency source code and potential version conflicts, and it requires no internet access (other than at download-mirrors.sh time)
Jerry has posted a preview code review showing his work using this mechanism. I’ve marked the reviews as workflow -1 to give the build team a chance to see the mechanism.
https://review.openstack.org/#/c/631001/
https://review.openstack.org/#/c/631002/
-Jason
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Jason On Tue, Jan 15, 2019 at 11:57 AM McKenna, Jason <Jason.McKenna@windriver.com> wrote:
Hi build team,
At the previous build meeting I had identified an issue with the way some go based packages were being built be built (they required internet access), and promised I’d update the mailing list on a potential way forward that we were prototyping.
Some preliminary points:
- go usually attempts to resolve dependencies at build time, by going out to the internet and fetching stuff (like dependency source code) using the “go get” command
- Sometimes the stuff fetched by “go get” isn’t appropriate (i.e. “go get” fetches the latest version, but deprecated APIs may have been removed, etc)
- Different versions of go packages may require different versions of dependencies
- We want builds to be reproducible without unexpected code changes (i.e. we want to know what we’re compiling in)
- Some people build in environments where they don’t have Internet access
The initial solution (which didn’t take into account the Internet access problem) was to use “dep”. “dep” is an external tool which was an “official experiment” of the go project. Rather than fetch the latest dependencies from the internet (like “go get”), it allowed specific revisions of dependencies to be captured. “dep” fetched those versions from the Internet. This solved the deprecated API issue, the reproducible build issue, and the issue of not using rpms for the dependencies. However, if someone was attempting to build in an internet-less context, the system would fail.
Enter this second revision.
The dependency packages are now downloaded at download-mirrors.sh time as tarballs. The tarballs are produced as of a specific commit for each dependency.
I agree taht this is a good solution , in terms of make it work , my concern is that we are mantaining a lot of tar balls , and that scares me a bit This allows us to hit all our bullet points – code is snapshotted, reproducible, we don’t end up having to create a bunch of new rpms with dependency source code and potential version conflicts, and it requires no internet access (other than at download-mirrors.sh time) If it works for you is fine , my concern will be to mantain changes of broken links in the future ( that sure go dep will fix ) +1 from my part as long as it does not geneerate a lot of mantainance does not create a problem for ourselvs How hard to aloud internet during build ? Regards
Jerry has posted a preview code review showing his work using this mechanism. I’ve marked the reviews as workflow -1 to give the build team a chance to see the mechanism.
https://review.openstack.org/#/c/631001/
https://review.openstack.org/#/c/631002/
-Jason
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Victor, great feedback. Inline.
-----Original Message----- From: Victor Rodriguez <vm.rod25@gmail.com> Sent: January 16, 2019 10:49 AM To: McKenna, Jason <Jason.McKenna@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [build] go packages, version 2
Hi Jason
On Tue, Jan 15, 2019 at 11:57 AM McKenna, Jason <Jason.McKenna@windriver.com> wrote:
Hi build team,
At the previous build meeting I had identified an issue with the way some
go based packages were being built be built (they required internet access), and promised I’d update the mailing list on a potential way forward that we were prototyping.
Some preliminary points:
- go usually attempts to resolve dependencies at build time, by going
out to the internet and fetching stuff (like dependency source code) using the “go get” command
- Sometimes the stuff fetched by “go get” isn’t appropriate (i.e. “go
get” fetches the latest version, but deprecated APIs may have been removed, etc)
- Different versions of go packages may require different versions of
dependencies
- We want builds to be reproducible without unexpected code changes
(i.e. we want to know what we’re compiling in)
- Some people build in environments where they don’t have Internet
access
The initial solution (which didn’t take into account the Internet access
problem) was to use “dep”. “dep” is an external tool which was an “official experiment” of the go project. Rather than fetch the latest dependencies from the internet (like “go get”), it allowed specific revisions of dependencies to be captured. “dep” fetched those versions from the Internet. This solved the deprecated API issue, the reproducible build issue, and the issue of not using rpms for the dependencies. However, if someone was attempting to build in an internet-less context, the system would fail.
Enter this second revision.
The dependency packages are now downloaded at download-mirrors.sh
time as tarballs. The tarballs are produced as of a specific commit for each dependency.
I agree taht this is a good solution , in terms of make it work , my concern is that we are mantaining a lot of tar balls , and that scares me a bit
Agreed. This actually is an opportunity to leverage the work that Marcela is doing where she is refactoring the download tarballs from a single .lst file into a more manageable form. If we can separate the tarballs/rpms/srpms/etc from a single file to a per-repo file (or a per-package file for build-time artifacts...) then maintaining the tarball downloads becomes a lot cleaner.
This allows us to hit all our bullet points – code is snapshotted, reproducible, we don’t end up having to create a bunch of new rpms with dependency source code and potential version conflicts, and it requires no internet access (other than at download-mirrors.sh time)
If it works for you is fine , my concern will be to mantain changes of broken links in the future ( that sure go dep will fix )
+1 from my part as long as it does not geneerate a lot of mantainance does not create a problem for ourselvs
How hard to aloud internet during build ?
Allowing Internet during build is easy if you have Internet access (edit the .cfg for your mock environment), but obviously troublesome if you're behind a firewall which blocks access. While this doesn't affect me personally, I am under the impression that you folks have several sites which are configured this way (I could be wrong, but I seem to recall early in the project a few packages breaking because reworked packages assumed Internet). Even if I'm mistaken about that point, I do know of several organizations which require clean-room builds for security and reproducibility reasons. Better to not limit potential adopters if we can help it :)
Regards
Jerry has posted a preview code review showing his work using this
mechanism. I’ve marked the reviews as workflow -1 to give the build team a chance to see the mechanism.
https://review.openstack.org/#/c/631001/
https://review.openstack.org/#/c/631002/
-Jason
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Thu, Jan 17, 2019 at 9:14 AM McKenna, Jason <Jason.McKenna@windriver.com> wrote:
Hi Victor, great feedback. Inline.
-----Original Message----- From: Victor Rodriguez <vm.rod25@gmail.com> Sent: January 16, 2019 10:49 AM To: McKenna, Jason <Jason.McKenna@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [build] go packages, version 2
Hi Jason
On Tue, Jan 15, 2019 at 11:57 AM McKenna, Jason <Jason.McKenna@windriver.com> wrote:
Hi build team,
At the previous build meeting I had identified an issue with the way some
go based packages were being built be built (they required internet access), and promised I’d update the mailing list on a potential way forward that we were prototyping.
Some preliminary points:
- go usually attempts to resolve dependencies at build time, by going
out to the internet and fetching stuff (like dependency source code) using the “go get” command
- Sometimes the stuff fetched by “go get” isn’t appropriate (i.e. “go
get” fetches the latest version, but deprecated APIs may have been removed, etc)
- Different versions of go packages may require different versions of
dependencies
- We want builds to be reproducible without unexpected code changes
(i.e. we want to know what we’re compiling in)
- Some people build in environments where they don’t have Internet
access
The initial solution (which didn’t take into account the Internet access
problem) was to use “dep”. “dep” is an external tool which was an “official experiment” of the go project. Rather than fetch the latest dependencies from the internet (like “go get”), it allowed specific revisions of dependencies to be captured. “dep” fetched those versions from the Internet. This solved the deprecated API issue, the reproducible build issue, and the issue of not using rpms for the dependencies. However, if someone was attempting to build in an internet-less context, the system would fail.
Enter this second revision.
The dependency packages are now downloaded at download-mirrors.sh
time as tarballs. The tarballs are produced as of a specific commit for each dependency.
I agree taht this is a good solution , in terms of make it work , my concern is that we are mantaining a lot of tar balls , and that scares me a bit
Agreed. This actually is an opportunity to leverage the work that Marcela is doing where she is refactoring the download tarballs from a single .lst file into a more manageable form. If we can separate the tarballs/rpms/srpms/etc from a single file to a per-repo file (or a per-package file for build-time artifacts...) then maintaining the tarball downloads becomes a lot cleaner.
Ok , i think is a good oportunity for the work that Marcela is doing
This allows us to hit all our bullet points – code is snapshotted, reproducible, we don’t end up having to create a bunch of new rpms with dependency source code and potential version conflicts, and it requires no internet access (other than at download-mirrors.sh time)
If it works for you is fine , my concern will be to mantain changes of broken links in the future ( that sure go dep will fix )
+1 from my part as long as it does not geneerate a lot of mantainance does not create a problem for ourselvs
How hard to aloud internet during build ?
Allowing Internet during build is easy if you have Internet access (edit the .cfg for your mock environment), but obviously troublesome if you're behind a firewall which blocks access. While this doesn't affect me personally, I am under the impression that you folks have several sites which are configured this way (I could be wrong, but I seem to recall early in the project a few packages breaking because reworked packages assumed Internet). Even if I'm mistaken about that point, I do know of several organizations which require clean-room builds for security and reproducibility reasons. Better to not limit potential adopters if we can help it :)
Ok, I see your point, well even if is behind a firewall it should have some proxy solution but yes, let's assume that we have a user that does not have internet. My only concern will be that I am not a fan of having many tarballs/patches or change that we have to maintain /host. If we are going to host them, Do we make a copy of them in the official CENGN repo? and point our scripts over there to be 100% sure that the download will work all the time? regards
Regards
Jerry has posted a preview code review showing his work using this
mechanism. I’ve marked the reviews as workflow -1 to give the build team a chance to see the mechanism.
https://review.openstack.org/#/c/631001/
https://review.openstack.org/#/c/631002/
-Jason
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
McKenna, Jason
-
Scott Little
-
Victor Rodriguez