[Starlingx-discuss] [Build] go loanguage
I have a designer that want to introduce a GO language package into our build. This raises the question of how to handle GO's 'imports'. First a disclaimer. I'm not a 'GO' exports. The following discussion points are based on my brief research. Go imports are roughly akin to a 'C' language include, with one or few major differences. 1) It points to a directory rather than a file, which could be local or relative to your GOPATH 2) The convention is to use a path that is a crude approximation of the url from which it can be obtained. e.g. import ( "github.com/docker/distribution/registry/api/errcode" ) 3) If the directory is NOT already present, it can probably be downloaded automatically with tools like 'go get' or 'dep' (aka golang-dep). These tools attempt transform the path into a URI, trying several vcs download protocols (git, mercurial, subversion...). Downloads are stored under a local 'vendor' sub-directory. There are a few wrinkles, like the ability of a sire to respond with a re-direct to another site. There are centos rpms for a handful of core libraries, delivering code to /usr/share/gocode (part of your GOPATH). However most go code has never been published as an rpm, and much of the remainder seems to be a one off rpm, with no ongoing maintenance. 'go get' just seems to grab the latest code. Reproducibility is a big concern. 'dep init' solves the reproducibility concern. It grabs the latest code, but also generates a 'lock' file that capture a CVS commit identifier (e.g. a git SHA). Deliver the lock file with your code and 'dep ensure' will use the lock file to download the same code every time. Dep does not currently ship as an RPM. I have a working spec file for dep 0.4.1. Dep 0.5.0 needs more work. Licenses of public go libraries seem to be permissive, at least for a statically linked binary, which is what we would want to ship. Not so sure about the inclusion of library source as an embedded vendor sub-directory in our own src.rpm packages. That would be one for the lawyers. So our options seem to be. 1) Create rpms of all the go libraries we need, populating /usr/share/gocode. There might be some tooling available to help with this. Pro: Don't require network access to build. Pro: Reproducible builds Pro: License of each go rpm is hopefully clear. Con: We'll be adding a lot of go libraries to our manifest. Assuming all are available via git. Con: Maintenance headache. Create spec files and compile rpms for the transitive set of imports. When do we upversion the rpms? Con: More packages we need to build 2) Allow our new go packages to include a pre-populated vendor sub-directory. Pro: Don't require network access to build Pro: Reproducible builds Con: We store snapshots of 3rd party code in the same src.rpm as our own code. Con: More complicated licensing statements. Con: Maintenance headache. When do we upversion the snapshots? Who want's to code inspect the mess? Con: If we have many go packages, there may multiple copies of the same library spread through our code base. Con: Is StarlingX, and OpenStack, ok with all the go code snapshots being added to our gits? 3) Use 'dep init' to create a lock file. Deliver only the lock file alongside our go code. Use 'dep ensure' during the build to download the required libraries. Pro: Lowest maintenance on our part Pro: Don't need to store a snapshot of any go libraries, nor add it too our manifest. Con: Need network access to build, not just in the docker container, but all the way down to the mock instance. Con: Relies on the upstream server continuing to host the package. Con: Relies on stability of upstream server and it's network connectivity. Vast majority of packages seem to come from golang.org, github and go.googlesource.com, so not likely a concern. I'm inclined to option 3. Opinions? Scott
On Fri, Nov 23, 2018 at 1:48 PM Scott Little <scott.little@windriver.com> wrote:
I have a designer that want to introduce a GO language package into our build. This raises the question of how to handle GO's 'imports'.
First a disclaimer. I'm not a 'GO' exports. The following discussion points are based on my brief research.
Go imports are roughly akin to a 'C' language include, with one or few major differences. 1) It points to a directory rather than a file, which could be local or relative to your GOPATH 2) The convention is to use a path that is a crude approximation of the url from which it can be obtained. e.g. import ( "github.com/docker/distribution/registry/api/errcode" ) 3) If the directory is NOT already present, it can probably be downloaded automatically with tools like 'go get' or 'dep' (aka golang-dep). These tools attempt transform the path into a URI, trying several vcs download protocols (git, mercurial, subversion...). Downloads are stored under a local 'vendor' sub-directory. There are a few wrinkles, like the ability of a sire to respond with a re-direct to another site.
There are centos rpms for a handful of core libraries, delivering code to /usr/share/gocode (part of your GOPATH). However most go code has never been published as an rpm, and much of the remainder seems to be a one off rpm, with no ongoing maintenance.
'go get' just seems to grab the latest code. Reproducibility is a big concern.
'dep init' solves the reproducibility concern. It grabs the latest code, but also generates a 'lock' file that capture a CVS commit
identifier (e.g. a git SHA). Deliver the lock file with your code and 'dep ensure' will use the lock file to download the same code every time. Dep does not currently ship as an RPM. I have a working spec file for dep 0.4.1. Dep 0.5.0 needs more work.
For me it works : curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
Licenses of public go libraries seem to be permissive, at least for a statically linked binary, which is what we would want to ship. Not so sure about the inclusion of library source as an embedded vendor sub-directory in our own src.rpm packages. That would be one for the lawyers.
So our options seem to be. 1) Create rpms of all the go libraries we need, populating /usr/share/gocode. There might be some tooling available to help with this. Pro: Don't require network access to build. Pro: Reproducible builds Pro: License of each go rpm is hopefully clear. Con: We'll be adding a lot of go libraries to our manifest. Assuming all are available via git. Con: Maintenance headache. Create spec files and compile rpms for the transitive set of imports. When do we upversion the rpms? Con: More packages we need to build
2) Allow our new go packages to include a pre-populated vendor sub-directory. Pro: Don't require network access to build Pro: Reproducible builds Con: We store snapshots of 3rd party code in the same src.rpm as our own code. Con: More complicated licensing statements. Con: Maintenance headache. When do we upversion the snapshots? Who want's to code inspect the mess? Con: If we have many go packages, there may multiple copies of the same library spread through our code base. Con: Is StarlingX, and OpenStack, ok with all the go code snapshots being added to our gits?
3) Use 'dep init' to create a lock file. Deliver only the lock file alongside our go code. Use 'dep ensure' during the build to download the required libraries. Pro: Lowest maintenance on our part Pro: Don't need to store a snapshot of any go libraries, nor add it too our manifest. Con: Need network access to build, not just in the docker container, but all the way down to the mock instance. Con: Relies on the upstream server continuing to host the package. Con: Relies on stability of upstream server and it's network connectivity. Vast majority of packages seem to come from golang.org, github and go.googlesource.com, so not likely a concern.
I'm inclined to option 3.
I am in favor of providing both go get and dep since they are for different reasons https://github.com/golang/dep/issues/376
Opinions?
Scott
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Fri, Nov 23, 2018 at 4:30 PM Victor Rodriguez <vm.rod25@gmail.com> wrote:
On Fri, Nov 23, 2018 at 1:48 PM Scott Little <scott.little@windriver.com> wrote:
I have a designer that want to introduce a GO language package into our build. This raises the question of how to handle GO's 'imports'.
First a disclaimer. I'm not a 'GO' exports. The following discussion points are based on my brief research.
Go imports are roughly akin to a 'C' language include, with one or few major differences. 1) It points to a directory rather than a file, which could be local or relative to your GOPATH 2) The convention is to use a path that is a crude approximation of the url from which it can be obtained. e.g. import ( "github.com/docker/distribution/registry/api/errcode" ) 3) If the directory is NOT already present, it can probably be downloaded automatically with tools like 'go get' or 'dep' (aka golang-dep). These tools attempt transform the path into a URI, trying several vcs download protocols (git, mercurial, subversion...). Downloads are stored under a local 'vendor' sub-directory. There are a few wrinkles, like the ability of a sire to respond with a re-direct to another site.
There are centos rpms for a handful of core libraries, delivering code to /usr/share/gocode (part of your GOPATH). However most go code has never been published as an rpm, and much of the remainder seems to be a one off rpm, with no ongoing maintenance.
'go get' just seems to grab the latest code. Reproducibility is a big concern.
'dep init' solves the reproducibility concern. It grabs the latest code, but also generates a 'lock' file that capture a CVS commit
identifier (e.g. a git SHA). Deliver the lock file with your code and 'dep ensure' will use the lock file to download the same code every time. Dep does not currently ship as an RPM. I have a working spec file for dep 0.4.1. Dep 0.5.0 needs more work.
For me it works : curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
Licenses of public go libraries seem to be permissive, at least for a statically linked binary, which is what we would want to ship. Not so sure about the inclusion of library source as an embedded vendor sub-directory in our own src.rpm packages. That would be one for the lawyers.
So our options seem to be. 1) Create rpms of all the go libraries we need, populating /usr/share/gocode. There might be some tooling available to help with this. Pro: Don't require network access to build. Pro: Reproducible builds Pro: License of each go rpm is hopefully clear. Con: We'll be adding a lot of go libraries to our manifest. Assuming all are available via git. Con: Maintenance headache. Create spec files and compile rpms for the transitive set of imports. When do we upversion the rpms? Con: More packages we need to build
2) Allow our new go packages to include a pre-populated vendor sub-directory. Pro: Don't require network access to build Pro: Reproducible builds Con: We store snapshots of 3rd party code in the same src.rpm as our own code. Con: More complicated licensing statements. Con: Maintenance headache. When do we upversion the snapshots? Who want's to code inspect the mess? Con: If we have many go packages, there may multiple copies of the same library spread through our code base. Con: Is StarlingX, and OpenStack, ok with all the go code snapshots being added to our gits?
3) Use 'dep init' to create a lock file. Deliver only the lock file alongside our go code. Use 'dep ensure' during the build to download the required libraries. Pro: Lowest maintenance on our part Pro: Don't need to store a snapshot of any go libraries, nor add it too our manifest. Con: Need network access to build, not just in the docker container, but all the way down to the mock instance. Con: Relies on the upstream server continuing to host the package. Con: Relies on stability of upstream server and it's network connectivity. Vast majority of packages seem to come from golang.org, github and go.googlesource.com, so not likely a concern.
I'm inclined to option 3.
I am in favor of providing both go get and dep since they are for different reasons
diference between go get and dep ensure : go get: I want to download the source code for a go project so that I can work on it myself, or to install a tool. This clones the repo under GOPATH for all to use. dep ensure: I have imported a new dependency in my code and want to download the dependency so I can start using it. My workflow is "add the import to the code, and then run dep ensure so that the manifest/lock/vendor are updated". This clones the repo under my project's vendor directory, and remembers the revision used so that everyone who works on my project is guaranteed to be using the same version of dependencies.
Opinions?
Scott
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (2)
-
Scott Little
-
Victor Rodriguez