[Starlingx-discuss] [Build] Feedback on stxb demo
Hi build team, Many thanks for the demo today, I think we're on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer: - I really like how the creation of the docker image is streamlined. - The system as currently demo'd as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx-tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/. There should only one repo downloaded. o As a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image's filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/.../ then I would expect the build command to detect and use my changes if I do a build. - I'm a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the -n and -p options. Would you provide examples of what commands I'd execute for use cases 1 and 2 below? I'm pretty sure these usages are supported, but am just seeking clarification. - Very happy do see it down to 1 docker image, rather than 2 :) - Native build - Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container - Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn't be able to log to /var/log. - Question - what privileges are required to create the docker image? Does the script assume the user has sudo privileges? - I'm unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet? Thanks again for the demo, -Jason Use Case 1: I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don't have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx Use Case 2: I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet. Use Case 3A: After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed? Use Case 3B: After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)? Use Case 4: I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated? Use Case 5: I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don't have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
Initial documentation on Regional Office > Regional Mirror found at https://wiki.openstack.org/wiki/StarlingX/Developer_Guide/Regional_Mirror
Sorry I missed this today, I am in Denver. Can you point me at the repo you are working in please? On 09/10/2018 09:02 AM, McKenna, Jason wrote:
Hi build team,
Many thanks for the demo today, I think we’re on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer:
-I really like how the creation of the docker image is streamlined.
-The system as currently demo’d as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx-tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/. There should only one repo downloaded.
oAs a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image’s filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/…/ then I would expect the build command to detect and use my changes if I do a build.
-I’m a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the –n and –p options. Would you provide examples of what commands I’d execute for use cases 1 and 2 below? I’m pretty sure these usages are supported, but am just seeking clarification.
-Very happy do see it down to 1 docker image, rather than 2 J
-Native build – Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container
-Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn’t be able to log to /var/log.
-Question – what privileges are required to create the docker image? Does the script assume the user has sudo privileges?
-I’m unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet?
Jason, Thanks for this feedback and the use case suggestions below, see additional comments.
Thanks again for the demo,
-Jason
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don’t have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Based on your usage of /export and /import below, /export is on a server machine and /import is on the local developer build machine, just for clarification. Also, this job would run native on the host Linux OS not containerized?
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
To further clarify this one, the remote mirror could contain all binary rpms along with their cooresponding src rpm, such that if no changes all the building of an ISO would be done from the mirror rpms, no actual local rpm build required unless it changes (use case 4/5 below).
Use Case 3A:
After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed?
Use Case 3B:
After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)?
Use Case 4:
I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated?
I would say not just puppet manifest, but any change to files in the repos.
Use Case 5:
I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
I look forward to seeing the demo also. Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi folks, I have been seen excellent ideas that we can to add to the tool, however I think we are walking a little out of the first scope, what do you think if we include this as first version a we can post our ideas in the storyboard?: https://storyboard.openstack.org/#!/story/2003712 Then we can track all ideas suggested :) Best regards. Mario. ________________________________________ From: Saul Wold [sgw@linux.intel.com] Sent: Monday, September 10, 2018 10:33 AM To: McKenna, Jason; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo Sorry I missed this today, I am in Denver. Can you point me at the repo you are working in please? On 09/10/2018 09:02 AM, McKenna, Jason wrote:
Hi build team,
Many thanks for the demo today, I think we’re on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer:
-I really like how the creation of the docker image is streamlined.
-The system as currently demo’d as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx-tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/. There should only one repo downloaded.
oAs a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image’s filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/…/ then I would expect the build command to detect and use my changes if I do a build.
-I’m a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the –n and –p options. Would you provide examples of what commands I’d execute for use cases 1 and 2 below? I’m pretty sure these usages are supported, but am just seeking clarification.
-Very happy do see it down to 1 docker image, rather than 2 J
-Native build – Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container
-Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn’t be able to log to /var/log.
-Question – what privileges are required to create the docker image? Does the script assume the user has sudo privileges?
-I’m unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet?
Jason, Thanks for this feedback and the use case suggestions below, see additional comments.
Thanks again for the demo,
-Jason
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don’t have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Based on your usage of /export and /import below, /export is on a server machine and /import is on the local developer build machine, just for clarification. Also, this job would run native on the host Linux OS not containerized?
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
To further clarify this one, the remote mirror could contain all binary rpms along with their cooresponding src rpm, such that if no changes all the building of an ISO would be done from the mirror rpms, no actual local rpm build required unless it changes (use case 4/5 below).
Use Case 3A:
After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed?
Use Case 3B:
After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)?
Use Case 4:
I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated?
I would say not just puppet manifest, but any change to files in the repos.
Use Case 5:
I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
I look forward to seeing the demo also. Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Mario,
-----Original Message----- From: Arevalo, Mario Alfredo C <mario.alfredo.c.arevalo@intel.com> Sent: September 10, 2018 4:57 PM To: Saul Wold <sgw@linux.intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Build] Feedback on stxb demo
Hi folks,
I have been seen excellent ideas that we can to add to the tool, however I think we are walking a little out of the first scope, what do you think if we include this as first version a we can post our ideas in the storyboard?:
I agree that we can put it out as a first preview version, however I would have strong reservations about changing the wiki documentation to suggest that this is "the official way to build" at this time.
+1 to the consolidation of docker images, and it sounds like that part's done. Good job!
Then we can track all ideas suggested :)
Yeah, I think we should have ideally had a discussion about use cases and requirements before the tool was designed. In particular, I do not think the tool (as demoed) is usable by developers. I understand that your requirements are to come up with an end-to-end tool that would produce a build with as few commands as possible, but the current architecture of having a second repo inside the docker container is a deal-breaker for any developer trying to work on the code. I'm all for designing and using better tools, but we shouldn't make this the official way of doing things if it places a new non-trivial burden on the devs. There might be some wiggle room on my suggestion to support a native (rather than containerized) build. As a tangential note, are you able to confirm that the use of a local mirror of packages rather than a download into the container from the Internet is working? We don't want to introduce a change that would unnecessarily add hours to a build.
Best regards. Mario. ________________________________________ From: Saul Wold [sgw@linux.intel.com] Sent: Monday, September 10, 2018 10:33 AM To: McKenna, Jason; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo
Sorry I missed this today, I am in Denver.
Can you point me at the repo you are working in please?
On 09/10/2018 09:02 AM, McKenna, Jason wrote:
Hi build team,
Many thanks for the demo today, I think we're on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer:
-I really like how the creation of the docker image is streamlined.
-The system as currently demo'd as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx- tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/. There should only one repo downloaded.
oAs a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image's filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/.../ then I would expect the build command to detect and use my changes if I do a build.
-I'm a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the -n and -p options. Would you provide examples of what commands I'd execute for use cases 1 and 2 below? I'm pretty sure these usages are supported, but am just seeking clarification.
-Very happy do see it down to 1 docker image, rather than 2 J
-Native build - Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container
-Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn't be able to log to /var/log.
-Question - what privileges are required to create the docker image? Does the script assume the user has sudo privileges?
-I'm unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet?
Jason,
Thanks for this feedback and the use case suggestions below, see additional comments.
Thanks again for the demo,
-Jason
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don't have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Based on your usage of /export and /import below, /export is on a server machine and /import is on the local developer build machine, just for clarification.
Also, this job would run native on the host Linux OS not containerized?
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
To further clarify this one, the remote mirror could contain all binary rpms along with their cooresponding src rpm, such that if no changes all the building of an ISO would be done from the mirror rpms, no actual local rpm build required unless it changes (use case 4/5 below).
Use Case 3A:
After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed?
Use Case 3B:
After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)?
Use Case 4:
I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated?
I would say not just puppet manifest, but any change to files in the repos.
Use Case 5:
I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
I look forward to seeing the demo also.
Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hello all, +1 for Jason's suggestion ...we cannot have the community transition to this tool as the official way of building at this time. We need to all agree as a community when to trial new tools to get feedback and when to make them official. We are driving hard to achieve code freeze for the October Release. We need to ensure everyone is building the same official way to avoid inconsistent results. As a side note, based on the review with the Build Project Lead (Cesar), there were no build improvements planned for the October release except for https://storyboard.openstack.org/#!/story/2002835 As the StarlingX release prime, I have tagged the new story for the March release (stx.2019.03). Please plan accordingly. Code submissions can still go in master, but cannot alter the documented build process and tools currently used. This story should not be considered complete until there is agreement to move the community to it and update the official documentation at some point after the October release. Regards, Ghada -----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:32 PM To: Arevalo, Mario Alfredo C; Saul Wold; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo Hi Mario,
-----Original Message----- From: Arevalo, Mario Alfredo C <mario.alfredo.c.arevalo@intel.com> Sent: September 10, 2018 4:57 PM To: Saul Wold <sgw@linux.intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Build] Feedback on stxb demo
Hi folks,
I have been seen excellent ideas that we can to add to the tool, however I think we are walking a little out of the first scope, what do you think if we include this as first version a we can post our ideas in the storyboard?:
I agree that we can put it out as a first preview version, however I would have strong reservations about changing the wiki documentation to suggest that this is "the official way to build" at this time.
+1 to the consolidation of docker images, and it sounds like that part's done. Good job!
Then we can track all ideas suggested :)
Yeah, I think we should have ideally had a discussion about use cases and requirements before the tool was designed. In particular, I do not think the tool (as demoed) is usable by developers. I understand that your requirements are to come up with an end-to-end tool that would produce a build with as few commands as possible, but the current architecture of having a second repo inside the docker container is a deal-breaker for any developer trying to work on the code. I'm all for designing and using better tools, but we shouldn't make this the official way of doing things if it places a new non-trivial burden on the devs. There might be some wiggle room on my suggestion to support a native (rather than containerized) build. As a tangential note, are you able to confirm that the use of a local mirror of packages rather than a download into the container from the Internet is working? We don't want to introduce a change that would unnecessarily add hours to a build.
Best regards. Mario. ________________________________________ From: Saul Wold [sgw@linux.intel.com] Sent: Monday, September 10, 2018 10:33 AM To: McKenna, Jason; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo
Sorry I missed this today, I am in Denver.
Can you point me at the repo you are working in please?
On 09/10/2018 09:02 AM, McKenna, Jason wrote:
Hi build team,
Many thanks for the demo today, I think we're on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer:
-I really like how the creation of the docker image is streamlined.
-The system as currently demo'd as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx- tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/. There should only one repo downloaded.
oAs a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image's filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/.../ then I would expect the build command to detect and use my changes if I do a build.
-I'm a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the -n and -p options. Would you provide examples of what commands I'd execute for use cases 1 and 2 below? I'm pretty sure these usages are supported, but am just seeking clarification.
-Very happy do see it down to 1 docker image, rather than 2 J
-Native build - Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container
-Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn't be able to log to /var/log.
-Question - what privileges are required to create the docker image? Does the script assume the user has sudo privileges?
-I'm unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet?
Jason,
Thanks for this feedback and the use case suggestions below, see additional comments.
Thanks again for the demo,
-Jason
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don't have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Based on your usage of /export and /import below, /export is on a server machine and /import is on the local developer build machine, just for clarification.
Also, this job would run native on the host Linux OS not containerized?
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
To further clarify this one, the remote mirror could contain all binary rpms along with their cooresponding src rpm, such that if no changes all the building of an ISO would be done from the mirror rpms, no actual local rpm build required unless it changes (use case 4/5 below).
Use Case 3A:
After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed?
Use Case 3B:
After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)?
Use Case 4:
I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated?
I would say not just puppet manifest, but any change to files in the repos.
Use Case 5:
I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
I look forward to seeing the demo also.
Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Wed, 2018-09-12 at 13:08 +0000, Khalil, Ghada wrote: Hi Folks,
Hello all, +1 for Jason's suggestion ...we cannot have the community transition to this tool as the official way of building at this time. We need to all agree as a community when to trial new tools to get feedback and when to make them official.
If you take a look to the tool's PRs, the first objective is not to do a transition for the moment, basically the ISO process involves a series of steps that might be painful for new users, and we have noticed that in the mailing list. For us, StarlingX team, those steps are easy because we deal with them every day, but we need to think of those who are not interested in become a developer for the moment and just want to make an ISO image, that is the first use case for the stxb tool. The current building process will continue working in the same way, stxb tool just automates those steps. The biggest change in the stxb tool development and the building tools core is the unification of dockerfiles, however it does not involve huge changes in documentation and it is open to feedback.
We are driving hard to achieve code freeze for the October Release. We need to ensure everyone is building the same official way to avoid inconsistent results.
Completely agree :)
As a side note, based on the review with the Build Project Lead (Cesar), there were no build improvements planned for the October release except for https://storyboard.openstack.org/#!/story/2002835
As the StarlingX release prime, I have tagged the new story for the March release (stx.2019.03). Please plan accordingly. Code submissions can still go in master, but cannot alter the documented build process and tools currently used. This story should not be considered complete until there is agreement to move the community to it and update the official documentation at some point after the October release.
Best Regards. Mario.
Regards, Ghada
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:32 PM To: Arevalo, Mario Alfredo C; Saul Wold; 'starlingx-discuss@lists.sta rlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo
Hi Mario,
-----Original Message----- From: Arevalo, Mario Alfredo C <mario.alfredo.c.arevalo@intel.com> Sent: September 10, 2018 4:57 PM To: Saul Wold <sgw@linux.intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; 'starlingx-discuss@lists.starlingx.i o' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Build] Feedback on stxb demo
Hi folks,
I have been seen excellent ideas that we can to add to the tool, however I think we are walking a little out of the first scope, what do you think if we include this as first version a we can post our ideas in the storyboard?:
I agree that we can put it out as a first preview version, however I would have strong reservations about changing the wiki documentation to suggest that this is "the official way to build" at this time.
+1 to the consolidation of docker images, and it sounds like that part's done. Good job!
Then we can track all ideas suggested :)
Yeah, I think we should have ideally had a discussion about use cases and requirements before the tool was designed. In particular, I do not think the tool (as demoed) is usable by developers. I understand that your requirements are to come up with an end-to-end tool that would produce a build with as few commands as possible, but the current architecture of having a second repo inside the docker container is a deal-breaker for any developer trying to work on the code. I'm all for designing and using better tools, but we shouldn't make this the official way of doing things if it places a new non- trivial burden on the devs. There might be some wiggle room on my suggestion to support a native (rather than containerized) build.
As a tangential note, are you able to confirm that the use of a local mirror of packages rather than a download into the container from the Internet is working? We don't want to introduce a change that would unnecessarily add hours to a build.
Best regards. Mario. ________________________________________ From: Saul Wold [sgw@linux.intel.com] Sent: Monday, September 10, 2018 10:33 AM To: McKenna, Jason; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo
Sorry I missed this today, I am in Denver.
Can you point me at the repo you are working in please?
On 09/10/2018 09:02 AM, McKenna, Jason wrote:
Hi build team,
Many thanks for the demo today, I think we're on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer:
-I really like how the creation of the docker image is streamlined.
-The system as currently demo'd as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx-
tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/ .
There should only one repo downloaded.
oAs a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image's filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/.../ then I would expect the build command to detect and use my changes if I do a build.
-I'm a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the -n and -p options. Would you provide examples of what commands I'd execute for use cases 1 and 2 below? I'm pretty sure these usages are supported, but am just seeking clarification.
-Very happy do see it down to 1 docker image, rather than 2 J
-Native build - Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container
-Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn't be able to log to /var/log.
-Question - what privileges are required to create the docker image? Does the script assume the user has sudo privileges?
-I'm unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet?
Jason,
Thanks for this feedback and the use case suggestions below, see additional comments.
Thanks again for the demo,
-Jason
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don't have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Based on your usage of /export and /import below, /export is on a server machine and /import is on the local developer build machine, just for clarification.
Also, this job would run native on the host Linux OS not containerized?
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
To further clarify this one, the remote mirror could contain all binary rpms along with their cooresponding src rpm, such that if no changes all the building of an ISO would be done from the mirror rpms, no actual local rpm build required unless it changes (use case 4/5 below).
Use Case 3A:
After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed?
Use Case 3B:
After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)?
Use Case 4:
I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated?
I would say not just puppet manifest, but any change to files in the repos.
Use Case 5:
I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
I look forward to seeing the demo also.
Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-disc uss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Kudo's on the container unification. I'de like to see the underlying steps captured on the wiki. I'll second all of Jason's concerns. There are a lot of assumptions built into the current draft of the tool that are a deal killer for a working designer, doubly so for designer working on a shared machine. Might be acceptable for a newcomer who wants to build his first load to try it out ... if he has a private box ... and sudo powers ... and doesn't really plan to change anything. As for everyone building the same way... correct goal. But this tool is unusable in our shared build environments, so goal not yet achieved. Scott On 18-09-12 09:57 AM, Arevalo, Mario Alfredo C wrote:
On Wed, 2018-09-12 at 13:08 +0000, Khalil, Ghada wrote:
Hi Folks,
Hello all, +1 for Jason's suggestion ...we cannot have the community transition to this tool as the official way of building at this time. We need to all agree as a community when to trial new tools to get feedback and when to make them official. If you take a look to the tool's PRs, the first objective is not to do a transition for the moment, basically the ISO process involves a series of steps that might be painful for new users, and we have noticed that in the mailing list.
For us, StarlingX team, those steps are easy because we deal with them every day, but we need to think of those who are not interested in become a developer for the moment and just want to make an ISO image, that is the first use case for the stxb tool.
The current building process will continue working in the same way, stxb tool just automates those steps. The biggest change in the stxb tool development and the building tools core is the unification of dockerfiles, however it does not involve huge changes in documentation and it is open to feedback.
We are driving hard to achieve code freeze for the October Release. We need to ensure everyone is building the same official way to avoid inconsistent results. Completely agree :)
As a side note, based on the review with the Build Project Lead (Cesar), there were no build improvements planned for the October release except for https://storyboard.openstack.org/#!/story/2002835
As the StarlingX release prime, I have tagged the new story for the March release (stx.2019.03). Please plan accordingly. Code submissions can still go in master, but cannot alter the documented build process and tools currently used. This story should not be considered complete until there is agreement to move the community to it and update the official documentation at some point after the October release. Best Regards. Mario.
Regards, Ghada
-----Original Message----- From: McKenna, Jason [mailto:Jason.McKenna@windriver.com] Sent: Tuesday, September 11, 2018 12:32 PM To: Arevalo, Mario Alfredo C; Saul Wold; 'starlingx-discuss@lists.sta rlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo
Hi Mario,
-----Original Message----- From: Arevalo, Mario Alfredo C <mario.alfredo.c.arevalo@intel.com> Sent: September 10, 2018 4:57 PM To: Saul Wold <sgw@linux.intel.com>; McKenna, Jason <Jason.McKenna@windriver.com>; 'starlingx-discuss@lists.starlingx.i o' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [Build] Feedback on stxb demo
Hi folks,
I have been seen excellent ideas that we can to add to the tool, however I think we are walking a little out of the first scope, what do you think if we include this as first version a we can post our ideas in the storyboard?: I agree that we can put it out as a first preview version, however I would have strong reservations about changing the wiki documentation to suggest that this is "the official way to build" at this time.
+1 to the consolidation of docker images, and it sounds like that part's done. Good job!
Then we can track all ideas suggested :)
Yeah, I think we should have ideally had a discussion about use cases and requirements before the tool was designed. In particular, I do not think the tool (as demoed) is usable by developers. I understand that your requirements are to come up with an end-to-end tool that would produce a build with as few commands as possible, but the current architecture of having a second repo inside the docker container is a deal-breaker for any developer trying to work on the code. I'm all for designing and using better tools, but we shouldn't make this the official way of doing things if it places a new non- trivial burden on the devs. There might be some wiggle room on my suggestion to support a native (rather than containerized) build.
As a tangential note, are you able to confirm that the use of a local mirror of packages rather than a download into the container from the Internet is working? We don't want to introduce a change that would unnecessarily add hours to a build.
Best regards. Mario. ________________________________________ From: Saul Wold [sgw@linux.intel.com] Sent: Monday, September 10, 2018 10:33 AM To: McKenna, Jason; 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo
Sorry I missed this today, I am in Denver.
Can you point me at the repo you are working in please?
On 09/10/2018 09:02 AM, McKenna, Jason wrote:
Hi build team,
Many thanks for the demo today, I think we're on the right track. As promised, here are a few feedback points on the demo, from the perspective of a developer:
-I really like how the creation of the docker image is streamlined.
-The system as currently demo'd as two copies of the repos. One in /(developers_path)/, and a second in /(developers_path)/stx- tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/ . There should only one repo downloaded.
oAs a related point, a developer working on changes to a file would expect to make their changes to the repo in /(developers_path)/, rather than to the version in the docker image's filesystem. If I edited a puppet manifest (for example) /(developers_path)/cgcs-root/stx/stx-config/.../ then I would expect the build command to detect and use my changes if I do a build.
-I'm a bit confused on the syntax regarding specifying and using localized mirrors, specifically with the -n and -p options. Would you provide examples of what commands I'd execute for use cases 1 and 2 below? I'm pretty sure these usages are supported, but am just seeking clarification.
-Very happy do see it down to 1 docker image, rather than 2 J
-Native build - Being able to build in a docker image is great, but the would ideally be an option to perform a native build, rather than have to use a container
-Minor nit - logs should be in a user directory (like $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged user wouldn't be able to log to /var/log.
-Question - what privileges are required to create the docker image? Does the script assume the user has sudo privileges?
-I'm unclear on what would happen with use cases 3-5 below. Have these types of uses been considered yet?
Jason,
Thanks for this feedback and the use case suggestions below, see additional comments.
Thanks again for the demo,
-Jason
Use Case 1:
I am an organization with a site who is working with StarlingX. I want to provide a local mirror for my employees to use so they don't have to download all artifacts from the external internet every time. I want to create an automated job which I will run daily to download all artifacts from the Internet (if they do not already exist) and place them in directory /export/mirrors/starlingx. Assume the user running the automated job has permissions to write to /export/mirrors/starlingx
Based on your usage of /export and /import below, /export is on a server machine and /import is on the local developer build machine, just for clarification.
Also, this job would run native on the host Linux OS not containerized?
Use Case 2:
I am a developer within an organization which has a local mirror of artifacts available in /import/mirrors/starlingx. I want to build a StarlingX ISO without downloading rpms or src.rpms from the external internet.
To further clarify this one, the remote mirror could contain all binary rpms along with their cooresponding src rpm, such that if no changes all the building of an ISO would be done from the mirror rpms, no actual local rpm build required unless it changes (use case 4/5 below).
Use Case 3A:
After doing a build, I just performed a repo sync, and the .lst files were not updated. If I perform another build of the ISO, will the system attempt to redownload external artifacts, even though nothing has changed?
Use Case 3B:
After doing a build, I just performed a repo sync, and noticed that the .lst files have been updated. If I perform another build of the ISO, will the stale mirror content be detected and the new additions downloaded (while not re-downloading anything which has not changed)?
Use Case 4:
I have manually made a change to a puppet manifest in sysinv (stx-config git) but want to test before I commit anything. How can I build a new ISO with my changes incorporated?
I would say not just puppet manifest, but any change to files in the repos.
Use Case 5:
I want to test StarlingX with my own additional program (foobar.x86_64.rpm) on the ISO. How would I perform a build with this file added?
I look forward to seeing the demo also.
Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-disc uss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus s
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
From: Saul Wold <sgw@linux.intel.com> Date: Monday, September 10, 2018 at 12:33 PM To: "McKenna, Jason" <Jason.McKenna@windriver.com>, "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [Build] Feedback on stxb demo Sorry I missed this today, I am in Denver. Can you point me at the repo you are working in please? https://review.openstack.org/#/c/601366/ https://review.openstack.org/#/c/601306/ https://review.openstack.org/#/c/601308/ https://review.openstack.org/#/c/601309/ On 09/10/2018 09:02 AM, McKenna, Jason wrote: > Hi build team, > > Many thanks for the demo today, I think we’re on the right track. As > promised, here are a few feedback points on the demo, from the > perspective of a developer: > > -I really like how the creation of the docker image is streamlined. > > -The system as currently demo’d as two copies of the repos. One in > /(developers_path)/, and a second in > /(developers_path)/stx-tools/(docker_image_root_dir)/localdisk/designer/builder/starlingx/. > There should only one repo downloaded. > oAs a related point, a developer working on changes to a file would > expect to make their changes to the repo in /(developers_path)/, rather > than to the version in the docker image’s filesystem. If I edited a > puppet manifest (for example) Hi, I think there are a lot of user scenarios that we need to consider, and we Should be adding functionality step by step, so far the commits mentioned simplify The building process to new users and can be used to add more functionality Four our next steps, I think we should create an storyboard to extend the Functionality by adding the scenario that need to be covered, starting with the ones Jason mentioned in this email: What about using two flags set by opts , one to use a local repo path and another one And another one to specify if the user wants to use a repo from a different Manifest and url. > /(developers_path)/cgcs-root/stx/stx-config/…/ then I would expect the > build command to detect and use my changes if I do a build. It could be an update flag, it the value is false, then it could skip the Updates/downloads and use the local content only > -I’m a bit confused on the syntax regarding specifying and using > localized mirrors, specifically with the –n and –p options. Would you > provide examples of what commands I’d execute for use cases 1 and 2 > below? I’m pretty sure these usages are supported, but am just seeking > clarification. `stxb build mirror`creates a defauilt mirror in the local workspace/mirror if no opts are given, In the current path ($PWD/starlingx/mirror), If the `stx build mirror -n my-staging-mirror` is given the path would be $PWD/starlingx/my-staging-mirror If the `stx build mirror -p /mnt/nfs` is given the path would be /mnt/nfs/mirror If the `stx build mirror -p /mnt/nfs/ww11/ -n monday` is given the path would be /mnt/nfs/mirror/www11/Monday > -Very happy do see it down to 1 docker image, rather than 2 J > > -Native build – Being able to build in a docker image is great, but the > would ideally be an option to perform a native build, rather than have > to use a container > -Minor nit - logs should be in a user directory (like > $MY_REPO_ROOT_DIR/logs/) rather than in /var/log, as a non-privileged > user wouldn’t be able to log to /var/log. The PR changed the path to use the ${WORKSPACE}, and specify another Another path if needed with the opt `-p` > -Question – what privileges are required to create the docker image? > Does the script assume the user has sudo privileges? So far I've running it without using sudo, using a user with docker group Permissions > -I’m unclear on what would happen with use cases 3-5 below. Have these > types of uses been considered yet? >
participants (7)
-
Arce Moreno, Abraham
-
Arevalo, Mario Alfredo C
-
Khalil, Ghada
-
McKenna, Jason
-
Ornelas Aguayo, Jesus
-
Saul Wold
-
Scott Little