[Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
Victor Rodriguez
vm.rod25 at gmail.com
Thu Jul 25 15:19:01 UTC 2019
Hi Shuquan
Months ago we had the open to have the capability in starling x to
take a personal/upstream change to nova and create a new image w/o the
need to rebuild all the images/project.
Don point about this new tool:
https://wiki.openstack.org/wiki/StarlingX/Containers/BuildingImages#Incremental_Image_Updates
Which allowed making this functionality.
Please let us know if this fulfills your needs and if not, what else
can be done on the build side
Thanks
Victor R
On Tue, Jul 23, 2019 at 1:16 AM Liu, ZhipengS <zhipengs.liu at intel.com> wrote:
>
> Hi Don,
> Thanks for your help!
> Now it works as below! Not sure why it did not work yesterday.
>
> ~/starlingx/cgcs-root/build-tools/build-docker-images$ bash update-stx-image.sh --from starlingx/stx-nova:master-centos-stable-20190715T233000Z.0 --module-src ~/starlingx/workspace/localdisk/loadbuild/zhipengl/nova-update/stx-nova --user zhipengl
> .........
>
> Successfully built nova
> Installing collected packages: nova
> Found existing installation: nova 19.0.1.dev116
> Uninstalling nova-19.0.1.dev116:
> Created temporary directory: /tmp/pip-uninstall-KrVTRI
> Removing file or directory /var/lib/openstack/bin/nova-api
> Removing file or directory /var/lib/openstack/bin/nova-api-metadata
> Removing file or directory /var/lib/openstack/bin/nova-api-os-compute
> Removing file or directory /var/lib/openstack/bin/nova-api-wsgi
> Removing file or directory /var/lib/openstack/bin/nova-cells
> Removing file or directory /var/lib/openstack/bin/nova-compute
> Removing file or directory /var/lib/openstack/bin/nova-conductor
> Removing file or directory /var/lib/openstack/bin/nova-console
> Removing file or directory /var/lib/openstack/bin/nova-consoleauth
> Removing file or directory /var/lib/openstack/bin/nova-dhcpbridge
> Removing file or directory /var/lib/openstack/bin/nova-manage
> Removing file or directory /var/lib/openstack/bin/nova-metadata-wsgi
> Removing file or directory /var/lib/openstack/bin/nova-network
> Removing file or directory /var/lib/openstack/bin/nova-novncproxy
> Removing file or directory /var/lib/openstack/bin/nova-placement-api
> Removing file or directory /var/lib/openstack/bin/nova-policy
> Removing file or directory /var/lib/openstack/bin/nova-rootwrap
> Removing file or directory /var/lib/openstack/bin/nova-rootwrap-daemon
> Removing file or directory /var/lib/openstack/bin/nova-scheduler
> Removing file or directory /var/lib/openstack/bin/nova-serialproxy
> Removing file or directory /var/lib/openstack/bin/nova-spicehtml5proxy
> Removing file or directory /var/lib/openstack/bin/nova-status
> Removing file or directory /var/lib/openstack/bin/nova-xvpvncproxy
> Created temporary directory: /var/lib/openstack/etc/~ova
> Removing file or directory /var/lib/openstack/etc/nova/
> Created temporary directory: /var/lib/openstack/lib/python2.7/site-packages/~ova-19.0.1.dev116.dist-info
> Removing file or directory /var/lib/openstack/lib/python2.7/site-packages/nova-19.0.1.dev116.dist-info/
> Created temporary directory: /var/lib/openstack/lib/python2.7/site-packages/~ova
> Removing file or directory /var/lib/openstack/lib/python2.7/site-packages/nova/
> Successfully uninstalled nova-19.0.1.dev116
> Successfully installed nova-19.0.1.dev117
> Cleaning up...
> Removed build tracker '/tmp/pip-req-tracker-OryF4V'
> + '[' 0 -ne 0 ']'
> + run_customization_script
> + '[' -x /image-update/customize.sh ']'
> + exit 0
> sha256:89c9ae9e438412f08f5ae0b85e562e657cf78542e763c595c52083bc574f9b31
> Updated image: zhipengl/stx-nova:master-centos-stable-20190715T233000Z.1
>
> Zhipeng
>
> -----Original Message-----
> From: Penney, Don [mailto:Don.Penney at windriver.com]
> Sent: 2019年7月23日 12:06
> To: Liu, ZhipengS <zhipengs.liu at intel.com>; Dean Troyer <dtroyer at gmail.com>; starlingx-discuss at lists.starlingx.io
> Cc: zhu.boxiang at 99cloud.net
> Subject: RE: [Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
>
> The extra text in the output of your --version vs mine is concerning, and I'm thinking that's getting picked up by the script when the hardcode_python_module_version function runs and causing some corruption. It appears to be INFO logs, which don't get printed to stdout by default on our systems. The function is expecting the only thing on stdout when running the --version command is the version itself.
>
> Is there some python config file on your host that's setting the default logging level to logging.INFO or logging.DEBUG maybe, instead of logging.WARN?
>
> -----Original Message-----
> From: Liu, ZhipengS [mailto:zhipengs.liu at intel.com]
> Sent: Monday, July 22, 2019 11:22 PM
> To: Penney, Don; Dean Troyer; starlingx-discuss at lists.starlingx.io
> Cc: zhu.boxiang at 99cloud.net
> Subject: RE: [Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
>
> Hi Don,
>
> ~/starlingx/workspace/localdisk/loadbuild/zhipengl/nova-update/stx-nova$ python ./setup.py --version
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
> warnings.warn(msg)
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
> warnings.warn(msg)
>
> Installed /home/wrsroot/starlingx/workspace/localdisk/loadbuild/zhipengl/nova-update/stx-nova/.eggs/pbr-5.4.1-py2.7.egg
> [pbr] Generating ChangeLog
> 19.0.1.dev117
>
> Zhipeng
> -----Original Message-----
> From: Penney, Don [mailto:Don.Penney at windriver.com]
> Sent: 2019年7月22日 21:59
> To: Liu, ZhipengS <zhipengs.liu at intel.com>; Dean Troyer <dtroyer at gmail.com>; starlingx-discuss at lists.starlingx.io
> Cc: zhu.boxiang at 99cloud.net
> Subject: RE: [Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
>
> Hi Zhipeng,
>
> Can you verify that you can run the following in your cloned repo?
>
> [/localdisk/loadbuild/dpenney/nova-update/stx-nova]$ python ./setup.py --version
> 19.0.1.dev116
>
>
>
> -----Original Message-----
> From: Liu, ZhipengS [mailto:zhipengs.liu at intel.com]
> Sent: Monday, July 22, 2019 3:15 AM
> To: Penney, Don; Dean Troyer; starlingx-discuss at lists.starlingx.io
> Cc: zhu.boxiang at 99cloud.net
> Subject: RE: [Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
>
> Hi Don,
>
> I failed to update image. Could you give me some help, thanks!
>
> ~/starlingx/cgcs-root/build-tools/build-docker-images$ bash update-stx-image.sh \
> > --from starlingx/stx-nova:master-centos-stable-20190715T233000Z.0 \
> > --module-src
> > ~/starlingx/workspace/localdisk/loadbuild/zhipengl/nova-update/stx-nov
> > a --user zhipengl
> ~/starlingx/cgcs-root/build-tools/build-docker-images ~/starlingx/cgcs-root/build-tools/build-docker-images
> ~/starlingx/workspace/localdisk/loadbuild/zhipengl/starlingx/std/update-images/unnamed-update/stx-nova_master-centos-stable-20190715T233000Z.1/pip-packages/wheels ~/starlingx/cgcs-root/build-tools/build-docker-images ~/starlingx/cgcs-root/build-tools/build-docker-images
> Running: wget http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels//stx-centos-stable-wheels.tar
> --2019-07-22 02:57:20-- http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels//stx-centos-stable-wheels.tar
> Resolving mirror.starlingx.cengn.ca (mirror.starlingx.cengn.ca)... 135.84.104.40 Connecting to mirror.starlingx.cengn.ca (mirror.starlingx.cengn.ca)|135.84.104.40|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 63354880 (60M) [application/octet-stream] Saving to: 'stx-centos-stable-wheels.tar'
>
> stx-centos-stable-wheels.tar 100%[===========================================================================>] 60.42M 7.41MB/s in 14s
>
> 2019-07-22 02:57:34 (4.42 MB/s) - 'stx-centos-stable-wheels.tar' saved [63354880/63354880]
>
> ~/starlingx/workspace/localdisk/loadbuild/zhipengl/starlingx/std/update-images/unnamed-update/stx-nova_master-centos-stable-20190715T233000Z.1/pip-packages/modules/stx-nova ~/starlingx/workspace/localdisk/loadbuild/zhipengl/starlingx/std/update-images/unnamed-update/stx-nova_master-centos-stable-20190715T233000Z.1/pip-packages/wheels ~/starlingx/cgcs-root/build-tools/build-docker-images ~/starlingx/cgcs-root/build-tools/build-docker-images
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
> warnings.warn(msg)
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
> warnings.warn(msg)
> sed: -e expression #1, char 20: unterminated `s' command ~/starlingx/workspace/localdisk/loadbuild/zhipengl/starlingx/std/update-images/unnamed-update/stx-nova_master-centos-stable-20190715T233000Z.1/pip-packages/wheels ~/starlingx/cgcs-root/build-tools/build-docker-images ~/starlingx/cgcs-root/build-tools/build-docker-images
> ~/starlingx/cgcs-root/build-tools/build-docker-images ~/starlingx/cgcs-root/build-tools/build-docker-images
> Running: docker image pull starlingx/stx-nova:master-centos-stable-20190715T233000Z.0
> master-centos-stable-20190715T233000Z.0: Pulling from starlingx/stx-nova
> 5ad559c5ae16: Already exists
> 3d7ee0b84e39: Pull complete
> e1047bfd73cf: Pull complete
> 16f151ad544f: Pull complete
> 844126c15d7e: Pull complete
> 869460797821: Pull complete
> 0c319da8f09d: Pull complete
> 2e884702ad07: Pull complete
> Digest: sha256:4032358f3ab208e76c2737854fdcf4f7bd582ef8f91e150193ded8feb662fdbe
> Status: Downloaded newer image for starlingx/stx-nova:master-centos-stable-20190715T233000Z.0
> + bash -x /image-update/internal-update-stx-image.sh
> + UPDATES_DIR=/image-update
> + PIP_PACKAGES_DIR=/image-update/pip-packages
> + DIST_PACKAGES_DIR=/image-update/dist-packages
> + CUSTOMIZATION_SCRIPT=/image-update/customize.sh
> ++ source /etc/os-release
> +++ NAME='CentOS Linux'
> +++ VERSION='7 (Core)'
> +++ ID=centos
> +++ ID_LIKE='rhel fedora'
> +++ VERSION_ID=7
> +++ PRETTY_NAME='CentOS Linux 7 (Core)'
> +++ ANSI_COLOR='0;31'
> +++ CPE_NAME=cpe:/o:centos:centos:7
> +++ HOME_URL=https://www.centos.org/
> +++ BUG_REPORT_URL=https://bugs.centos.org/
> +++ CENTOS_MANTISBT_PROJECT=CentOS-7
> +++ CENTOS_MANTISBT_PROJECT_VERSION=7
> +++ REDHAT_SUPPORT_PRODUCT=centos
> +++ REDHAT_SUPPORT_PRODUCT_VERSION=7
> ++ echo CentOS Linux
> + OS_NAME='CentOS Linux'
> ++ getopt -o h -l help: --
> + OPTS=' --'
> + '[' 0 -ne 0 ']'
> + eval set -- ' --'
> ++ set -- --
> + true
> + case $1 in
> + shift
> + break
> + install_dist_packages
> + local -i file_count=0
> ++ find /image-update/dist-packages -type f wc -l
> + file_count=0
> + '[' 0 -eq 0 ']'
> + return 0
> + install_pip_packages
> + local modules
> + local wheels
> ++ find /image-update/pip-packages/modules/stx-nova -maxdepth 0 -type d
> + modules=/image-update/pip-packages/modules/stx-nova
> ++ find /image-update/pip-packages/wheels/ -type f -name '*.whl'
> + wheels=
> + '[' -z /image-update/pip-packages/modules/stx-nova -a -z '' ']'
> + pip install -vvv --no-deps --no-index --pre --no-cache-dir
> + --only-binary :all: --no-compile --force-reinstall
> + /image-update/pip-packages/modules/stx-nova
> DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
> Ignoring indexes: https://pypi.org/simple Created temporary directory: /tmp/pip-ephem-wheel-cache-Pld6sg Created temporary directory: /tmp/pip-req-tracker-5OQGeX Created requirements tracker '/tmp/pip-req-tracker-5OQGeX'
> Created temporary directory: /tmp/pip-install-8tWna2 Processing /image-update/pip-packages/modules/stx-nova
> Created temporary directory: /tmp/pip-req-build-EngJ2Q
> Added file:///image-update/pip-packages/modules/stx-nova to build tracker '/tmp/pip-req-tracker-5OQGeX'
> Running setup.py (path:/tmp/pip-req-build-EngJ2Q/setup.py) egg_info for package from file:///image-update/pip-packages/modules/stx-nova
> Running command python setup.py egg_info
> ERROR:root:Error parsing
> Traceback (most recent call last):
> File "/var/lib/openstack/lib/python2.7/site-packages/pbr/core.py", line 96, in pbr
> attrs = util.cfg_to_args(path, dist.script_args)
> File "/var/lib/openstack/lib/python2.7/site-packages/pbr/util.py", line 256, in cfg_to_args
> pbr.hooks.setup_hook(config)
> File "/var/lib/openstack/lib/python2.7/site-packages/pbr/hooks/__init__.py", line 25, in setup_hook
> metadata_config.run()
> File "/var/lib/openstack/lib/python2.7/site-packages/pbr/hooks/base.py", line 27, in run
> self.hook()
> File "/var/lib/openstack/lib/python2.7/site-packages/pbr/hooks/metadata.py", line 26, in hook
> self.config['name'], self.config.get('version', None))
> File "/var/lib/openstack/lib/python2.7/site-packages/pbr/packaging.py", line 849, in get_version
> name=package_name))
> Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name nova was given, but was not able to be found.
> error in setup command: Error parsing /tmp/pip-req-build-EngJ2Q/setup.cfg: Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name nova was given, but was not able to be found.
> Cleaning up...
> Removing source in /tmp/pip-req-build-EngJ2Q Removed file:///image-update/pip-packages/modules/stx-nova from build tracker '/tmp/pip-req-tracker-5OQGeX'
> Removed build tracker '/tmp/pip-req-tracker-5OQGeX'
> ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-EngJ2Q/ Exception information:
> Traceback (most recent call last):
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main
> status = self.run(options, args)
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 352, in run
> resolver.resolve(requirement_set)
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
> self._resolve_one(requirement_set, req)
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
> abstract_dist = self._get_abstract_dist_for(req_to_install)
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
> self.require_hashes
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 362, in prepare_linked_requirement
> abstract_dist.prep_for_dist(finder, self.build_isolation)
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 171, in prep_for_dist
> self.req.prepare_metadata()
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 537, in prepare_metadata
> self.run_egg_info()
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 615, in run_egg_info
> command_desc='python setup.py egg_info')
> File "/var/lib/openstack/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 776, in call_subprocess
> % (command_desc, proc.returncode, cwd))
> InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-EngJ2Q/
> + '[' 1 -ne 0 ']'
> + echo 'Failed pip install'
> Failed pip install
> + exit 1
> Failed to update image: starlingx/stx-nova:master-centos-stable-20190715T233000Z.0
>
>
> Zhipeng
> -----Original Message-----
> From: Penney, Don [mailto:Don.Penney at windriver.com]
> Sent: 2019年7月20日 2:04
> To: Dean Troyer <dtroyer at gmail.com>; starlingx-discuss at lists.starlingx.io
> Cc: zhu.boxiang at 99cloud.net
> Subject: Re: [Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
>
> For updating an image for testing, take a look at:
> https://wiki.openstack.org/wiki/StarlingX/Containers/BuildingImages#Incremental_Image_Updates
>
> As Dean notes, clone the repo and cherry-pick the commit, and then do something like:
>
> time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
> --from starlingx/stx-nova:master-centos-stable-20190715T233000Z.0 \
> --module-src ${path_to_cloned_repo}/stx-nova
>
>
> ie:
>
> # clone stx-nova, get stx/stein.2
> cd /localdisk/loadbuild/dpenney/
> mkdir nova-update
> cd nova-update/
> git clone https://github.com/starlingx-staging/stx-nova.git
> cd stx-nova/
> git fetch https://github.com/starlingx-staging/stx-nova.git stx/stein.2 git checkout FETCH_HEAD
>
> # cherry-pick update
> git fetch https://review.opendev.org/openstack/nova refs/changes/69/651969/13 && git cherry-pick FETCH_HEAD
>
> # Fix up conflicts, etc
>
> # Build updated image, from 20190715T233000Z build as base, # specifying cloned/modified repo time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
> --from starlingx/stx-nova:master-centos-stable-20190715T233000Z.0 \
> --module-src /localdisk/loadbuild/dpenney/nova-update/stx-nova \
> --user dpenney
>
>
> This produces an updated image in the local registry:
> Updated image: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
>
> If you also set a registry with --registry and use the --push option, the command will push the updated image to that registry.
>
> This also produces an image record file:
> $ cat ${MY_WORKSPACE}/std/update-images/unnamed-update/image-updates.lst
> dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
>
>
> Which you can pass as an argument to build-helm-charts.sh when building your application tarball for testing:
>
> build-helm-charts.sh \
> --image-record http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/docker-images/images-centos-stable-versioned.lst \
> --image-record ${MY_WORKSPACE}/std/update-images/unnamed-update/image-updates.lst \
> --label centos-stable-versioned
>
>
>
> If you look at the yaml file in the tarball, you can see that it now references the updated image:
>
> $ tar xzf ${MY_WORKSPACE}/std/build-helm/stx/stx-openstack-1.0-17-centos-stable-versioned.tgz -O ./stx-openstack.yaml | grep stx-nova:
> nova_api: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_cell_setup: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_compute: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_compute_ironic: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_compute_ssh: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_conductor: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_consoleauth: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_db_sync: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_novncproxy: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_placement: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_scheduler: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_spiceproxy: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
> nova_spiceproxy_assets: dpenney/stx-nova:master-centos-stable-20190715T233000Z.1
>
>
>
>
>
> -----Original Message-----
> From: Dean Troyer [mailto:dtroyer at gmail.com]
> Sent: Friday, July 19, 2019 1:08 PM
> To: starlingx-discuss at lists.starlingx.io
> Cc: zhu.boxiang at 99cloud.net
> Subject: Re: [Starlingx-discuss] about cherry-pick a patch from OpenStack upstream for testing issue in StarlingX
>
> On Fri, Jul 19, 2019 at 10:57 AM Yong Hu <yong.hu at intel.com> wrote:
> > For LP[0], there is a patch (https://review.opendev.org/#/c/651969/)
> > in Nova upstream, what's the method/process for StarlingX to
> > cherry-pick it for testing a LP reported in StarlingX?
> >
> > [0]:https://bugs.launchpad.net/starlingx/+bug/1820882
>
> At a high level you would clone the stx-nova repo stx/stein.2 branch [0] and cherry-pick/backport the commit you want to test, and rebuild the Nova docker image and test that.
>
> I do have some Zuul jobs in starlingx/tis-repo that will pull that Nova branch and run the unit, functional and pep8 tox jobs to test in OpenStack CI.
>
> Given that the stx/stein branches also carry the NUMA live migration patches you would probably want to do whatever other live migration testing would be done upstream to validate this in the StarlingX context.
>
> dt
>
> [0] https://github.com/starlingx-staging/stx-nova/tree/stx/stein.2
>
> --
> Dean Troyer
> dtroyer at gmail.com
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
More information about the Starlingx-discuss
mailing list