[Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing!
Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false
Centos 8.3 was released yesterday. One of the changes is to repo names, and that broke at least one container build. e.g. stx-fm-trap-subagent's Dockerfile was referencing 'PowerTools' which is now 'powertools' I've posted a fix for this one: https://review.opendev.org/c/starlingx/snmp-armada-app/+/766035 To all container owners: Please verify your containers still build! Scott On 2020-12-08 1:08 a.m., build.starlingx@gmail.com wrote:
[Please note this e-mail is from an EXTERNAL e-mail address]
Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master
Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters
BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
There is second container build failure in the logs. The stx-horizon docker image can't find a version of Django that it can use. https://bugs.launchpad.net/starlingx/+bug/1907290 Can someone volunteer to investigate this one? Scott On 2020-12-08 1:08 a.m., build.starlingx@gmail.com wrote:
[Please note this e-mail is from an EXTERNAL e-mail address]
Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master
Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters
BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
That seems odd. It certainly looks like pypi.org has a wheel that should satisfy the requirement Django<3.0,>=2.2: https://pypi.org/project/Django/2.2.17/#files Django-2.2.17-py3-none-any.whl <https://files.pythonhosted.org/packages/82/2b/75f2909ba02a3b0e343e560863101aa3d43f58357e7c053596aa29d1cce7/Django-2.2.17-py3-none-any.whl> This is a py3 wheel, and stx-horizon has been updated to python3: https://opendev.org/starlingx/upstream/src/branch/master/openstack/python-ho... From: Scott Little <scott.little@windriver.com> Sent: Tuesday, December 8, 2020 12:46 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! There is second container build failure in the logs. The stx-horizon docker image can't find a version of Django that it can use. https://bugs.launchpad.net/starlingx/+bug/1907290 Can someone volunteer to investigate this one? Scott On 2020-12-08 1:08 a.m., build.starlingx@gmail.com<mailto:build.starlingx@gmail.com> wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
This is because of the new pip https://discuss.python.org/t/announcement-pip-20-3-release/5948 In python3 it uses a diff resolver. Which seems to fail for any component ending in a number (Django2, iso8601, etc..) virtualenv on Dec7 was also updated to use that version of pip by default, which is what is affecting some tox jobs. there is supposed to be a way to add --use-deprecated=legacy-resolver to the pip command, which is what I am trying to do to get sysinv zuul to pass, I havent had much luck yet. Al ________________________________ From: Penney, Don <Don.Penney@windriver.com> Sent: Tuesday, December 8, 2020 12:54 PM To: Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! That seems odd. It certainly looks like pypi.org has a wheel that should satisfy the requirement Django<3.0,>=2.2: https://pypi.org/project/Django/2.2.17/#files Django-2.2.17-py3-none-any.whl <https://files.pythonhosted.org/packages/82/2b/75f2909ba02a3b0e343e560863101aa3d43f58357e7c053596aa29d1cce7/Django-2.2.17-py3-none-any.whl> This is a py3 wheel, and stx-horizon has been updated to python3: https://opendev.org/starlingx/upstream/src/branch/master/openstack/python-ho... From: Scott Little <scott.little@windriver.com> Sent: Tuesday, December 8, 2020 12:46 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! There is second container build failure in the logs. The stx-horizon docker image can't find a version of Django that it can use. https://bugs.launchpad.net/starlingx/+bug/1907290 Can someone volunteer to investigate this one? Scott On 2020-12-08 1:08 a.m., build.starlingx@gmail.com<mailto:build.starlingx@gmail.com> wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 2020-12-08 20:32:23 +0000 (+0000), Bailey, Henry Albert (Al) wrote: [...]
In python3 it uses a diff resolver. Which seems to fail for any component ending in a number (Django2, iso8601, etc..) [...]
Interesting. I've helped a lot of folks fix latent dependency version conflicts exposed by the new solver, but this is the first I've heard that it cares whether package names end in digits. Can you point to where you saw that asserted? -- Jeremy Stanley
I think the trailing number was just a wild goose chase. The resolver will complain if a min version and a max version don't line up. I was able to successfully use some of the entities in a minimal setup (boto3, iso8601) that had been failing when hitting conflicts in the larger env. Al ________________________________ From: Jeremy Stanley <fungi@yuggoth.org> Sent: Tuesday, December 8, 2020 4:12 PM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! [Please note this e-mail is from an EXTERNAL e-mail address] On 2020-12-08 20:32:23 +0000 (+0000), Bailey, Henry Albert (Al) wrote: [...]
In python3 it uses a diff resolver. Which seems to fail for any component ending in a number (Django2, iso8601, etc..) [...]
Interesting. I've helped a lot of folks fix latent dependency version conflicts exposed by the new solver, but this is the first I've heard that it cares whether package names end in digits. Can you point to where you saw that asserted? -- Jeremy Stanley
I've posted a fix for the stx-horizon build issue: https://review.opendev.org/c/starlingx/root/+/766357 Inclusion of a specific version of Django in our wheels tarball for the loci build resulted in a conflict between the upper-constraints.txt in the tarball and the requirements.txt from openstack/horizon. The new pip resolver exposed this conflict and causes the stx-horizon image build to fail. Removing the entry from our stable-wheels.cfg avoids the conflict, and I'm able to successfully build the image. Once the review is approved and merged, we can kick a new image build from cengn. Cheers, Don. From: Penney, Don <Don.Penney@windriver.com> Sent: Tuesday, December 8, 2020 12:55 PM To: Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! That seems odd. It certainly looks like pypi.org has a wheel that should satisfy the requirement Django<3.0,>=2.2: https://pypi.org/project/Django/2.2.17/#files Django-2.2.17-py3-none-any.whl <https://files.pythonhosted.org/packages/82/2b/75f2909ba02a3b0e343e560863101aa3d43f58357e7c053596aa29d1cce7/Django-2.2.17-py3-none-any.whl> This is a py3 wheel, and stx-horizon has been updated to python3: https://opendev.org/starlingx/upstream/src/branch/master/openstack/python-ho... From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Tuesday, December 8, 2020 12:46 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! There is second container build failure in the logs. The stx-horizon docker image can't find a version of Django that it can use. https://bugs.launchpad.net/starlingx/+bug/1907290 Can someone volunteer to investigate this one? Scott On 2020-12-08 1:08 a.m., build.starlingx@gmail.com<mailto:build.starlingx@gmail.com> wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Scott, Even I am seeing this error since last two days. It was building fine(same code/repo) prior to 7th Dec. Anything changed in upstream open stack reference ? Thank you, Haridhar Kalvala From: Scott Little <scott.little@windriver.com> Sent: Tuesday, December 8, 2020 11:16 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [build-report] master STX_build_layer_containers_master_master - Build # 90 - Still Failing! There is second container build failure in the logs. The stx-horizon docker image can't find a version of Django that it can use. https://bugs.launchpad.net/starlingx/+bug/1907290 Can someone volunteer to investigate this one? Scott On 2020-12-08 1:08 a.m., build.starlingx@gmail.com<mailto:build.starlingx@gmail.com> wrote: [Please note this e-mail is from an EXTERNAL e-mail address] Project: STX_build_layer_containers_master_master Build #: 90 Status: Still Failing Timestamp: 20201208T014249Z Branch: master Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/2... -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: false _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 2020-12-09 04:07:21 +0000 (+0000), Kalvala, Haridhar wrote:
Even I am seeing this error since last two days. It was building fine(same code/repo) prior to 7th Dec. Anything changed in upstream open stack reference ? [...]
Not in upstream OpenStack specifically, but across the Python packaging ecosystem as a whole. Pip 20.3.0, released on November 30, turned on a new and much more thorough dependency resolver. Earlier versions of pip did not try particularly hard to make sure the dependencies claimed by packages were all satisfied. Virtualenv 20.2.2, released December 7, increased the version of pip it's vendoring to a version which uses the new solver as well. These changes mean that latent version conflicts are now being correctly identified as bugs, and so your builds will do a far better job of confirming the declared versions of dependencies are actually used if possible (and generate an error if not). The error in the bug you linked looks like it's probably a version conflict between what's in the constraints file you're applying and what horizon says it wants to satisfy its django requirement. Earlier releases of pip included in earlier releases of virtualenv may have not been capable of noticing this latent conflict, so if loci relies on virtualenv (I don't know much about it to be honest), then starting to see this bug exposed coincident with the release of virtualenv 20.2.2 would make sense. Projects in OpenStack are currently working to solve these sorts of emergent problems in their own builds as well, though for the most part keeping the global upper constraints list in sync with the corresponding project branches has shielded them from major issues and this is mainly cropping up in places where projects are supplying their own bespoke constraints lists which aren't built from a guaranteed consistent set. I don't know what the state of the source code and constraints lists being used in your builds is, so it's hard for me to speculate. If it's just asking loci to generate images from upstream stable/ussuri branches in combination with the upstream stable/ussuri upper-constraints.txt file, then perhaps this is a stable branch bug which needs to get solved in loci or even deeper in the stack. -- Jeremy Stanley
participants (6)
-
Bailey, Henry Albert (Al)
-
build.starlingx@gmail.com
-
Jeremy Stanley
-
Kalvala, Haridhar
-
Penney, Don
-
Scott Little