[Starlingx-discuss] [Zuul] failure on review in stx-config project
Hi, I have a review https://review.openstack.org/#/c/642488/2 with small changes to sysinv. sysinv-tox-py27 and sysinv-tox-py35 failed for this review. However the failure doesn't seem to be related to my changes. The test output has traceback like: 2019-03-14 15:56:10.816504 | ubuntu-bionic | tests.db.sqlalchemy.test_migrations.TestMigrations.test_postgresql_connect_fail 2019-03-14 15:56:10.816685 | ubuntu-bionic | ------------------------------------------------------------------------------- 2019-03-14 15:56:10.816716 | ubuntu-bionic | 2019-03-14 15:56:10.816781 | ubuntu-bionic | Captured traceback: 2019-03-14 15:56:10.816845 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~ 2019-03-14 15:56:10.816945 | ubuntu-bionic | Traceback (most recent call last): 2019-03-14 15:56:10.817260 | ubuntu-bionic | File "/home/zuul/src/git.openstack.org/openstack/stx-config/sysinv/sysinv/sysinv/sysinv/tests/db/sqlalchemy/test_migrations.py", line 523, in setUp 2019-03-14 15:56:10.817370 | ubuntu-bionic | super(TestMigrations, self).setUp() 2019-03-14 15:56:10.817679 | ubuntu-bionic | File "/home/zuul/src/git.openstack.org/openstack/stx-config/sysinv/sysinv/sysinv/sysinv/tests/db/sqlalchemy/test_migrations.py", line 188, in setUp 2019-03-14 15:56:10.817817 | ubuntu-bionic | self.engines[key] = sqlalchemy.create_engine(value) 2019-03-14 15:56:10.818086 | ubuntu-bionic | File "/tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine 2019-03-14 15:56:10.818203 | ubuntu-bionic | return strategy.create(*args, **kwargs) 2019-03-14 15:56:10.818457 | ubuntu-bionic | File "/tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in create 2019-03-14 15:56:10.818574 | ubuntu-bionic | dbapi = dialect_cls.dbapi(**dbapi_args) 2019-03-14 15:56:10.818848 | ubuntu-bionic | File "/tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 554, in dbapi 2019-03-14 15:56:10.818920 | ubuntu-bionic | import psycopg2 2019-03-14 15:56:10.819158 | ubuntu-bionic | File "/tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 50, in <module> 2019-03-14 15:56:10.819308 | ubuntu-bionic | from psycopg2._psycopg import ( # noqa 2019-03-14 15:56:10.819740 | ubuntu-bionic | ImportError: /tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/psycopg2/.libs/libresolv-2-c4c53def.5.so: symbol __res_maybe_init version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference 2019-03-14 15:56:10.819785 | ubuntu-bionic | Also, local run of tox py27 and py35 both passed. Anybody knows what's going wrong? Any fix is in progress? Thanks, Andy -- Andy Ning Cube: 3071 Tel: 613-9631408 (int: 4408) Skype: andy.ning.wr
On 2019-03-14 15:39:23 -0400 (-0400), Andy Ning wrote:
I have a review https://review.openstack.org/#/c/642488/2 with small changes to sysinv. sysinv-tox-py27 and sysinv-tox-py35 failed for this review. However the failure doesn't seem to be related to my changes. The test output has traceback like: [...] 2019-03-14 15:56:10.819308 | ubuntu-bionic | from psycopg2._psycopg import ( # noqa 2019-03-14 15:56:10.819740 | ubuntu-bionic | ImportError: /tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/psycopg2/.libs/libresolv-2-c4c53def.5.so: symbol __res_maybe_init version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference 2019-03-14 15:56:10.819785 | ubuntu-bionic |
Also, local run of tox py27 and py35 both passed.
Anybody knows what's going wrong? Any fix is in progress?
As of roughly 22:00 UTC (on Wednesday, March 13), the default nodeset in the opendev/base-jobs repository was updated from ubuntu-xenial (16.04 LTS) to ubuntu-bionic (18.04 LTS). In the case of the sysinv-tox-py35 job you almost certainly need to set an explicit "nodeset: ubuntu-xenial" so that it runs on a platform with actually provides a Python 3.5 interpreter (ubuntu-bionic only provides 3.6 and 3.7). For the sysinv-tox-py27 psycopg2 error above, it looks like it's attempting to use a libresolv build which references symbols from a different libc. Are you maybe installing bundled libraries previously built on an ubuntu-xenial node? -- Jeremy Stanley
Thanks Jeremy, see my comments inline ... Andy On 2019-03-14 04:02 PM, Jeremy Stanley wrote:
On 2019-03-14 15:39:23 -0400 (-0400), Andy Ning wrote:
I have a review https://review.openstack.org/#/c/642488/2 with small changes to sysinv. sysinv-tox-py27 and sysinv-tox-py35 failed for this review. However the failure doesn't seem to be related to my changes. The test output has traceback like: [...] 2019-03-14 15:56:10.819308 | ubuntu-bionic | from psycopg2._psycopg import ( # noqa 2019-03-14 15:56:10.819740 | ubuntu-bionic | ImportError: /tmp/zuul_sysinvtox/py27/local/lib/python2.7/site-packages/psycopg2/.libs/libresolv-2-c4c53def.5.so: symbol __res_maybe_init version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference 2019-03-14 15:56:10.819785 | ubuntu-bionic |
Also, local run of tox py27 and py35 both passed.
Anybody knows what's going wrong? Any fix is in progress? As of roughly 22:00 UTC (on Wednesday, March 13), the default nodeset in the opendev/base-jobs repository was updated from ubuntu-xenial (16.04 LTS) to ubuntu-bionic (18.04 LTS).
In the case of the sysinv-tox-py35 job you almost certainly need to set an explicit "nodeset: ubuntu-xenial" so that it runs on a platform with actually provides a Python 3.5 interpreter (ubuntu-bionic only provides 3.6 and 3.7).
For the sysinv-tox-py27 psycopg2 error above, it looks like it's attempting to use a libresolv build which references symbols from a different libc. Are you maybe installing bundled libraries previously built on an ubuntu-xenial node?
[AN]: As developer I don't install anything, I submit my source code changes to gerrit as review, and Zuul is triggered to run to verify if my change break tox. Where tox runs the tests and what its environment like is transparent to (as least for me until now :)). So I'm not sure how I could do what you suggest.
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Thu, Mar 14, 2019 at 3:58 PM Andy Ning <andy.ning@windriver.com> wrote:
[AN]: As developer I don't install anything, I submit my source code changes to gerrit as review, and Zuul is triggered to run to verify if my change break tox. Where tox runs the tests and what its environment like is transparent to (as least for me until now :)). So I'm not sure how I could do what you suggest.
Andy, I am working through the STX repos to clean up some of these py35 things following the bionic update upstream. As Jeremy points out, there are two issues at play here, running on xenial will work around both, and I will start pushing that out. dt -- Dean Troyer dtroyer@gmail.com
On 2019-03-14 16:57:09 -0400 (-0400), Andy Ning wrote: [...]
[AN]: As developer I don't install anything, I submit my source code changes to gerrit as review, and Zuul is triggered to run to verify if my change break tox. Where tox runs the tests and what its environment like is transparent to (as least for me until now :)). So I'm not sure how I could do what you suggest.
By "you" in this case I meant developers on StarlingX. Your CI jobs should absolutely not be a black box to you, especially since the job in question is defined in your repository: http://git.starlingx.io/cgit/stx-config/tree/.zuul.yaml#n66 Just a matter of adding "nodeset: ubuntu-xenial" to that job definition. The change which adds it should be self-testing so you'll be able to see in the CI results whether it worked or not. -- Jeremy Stanley
On 2019-03-14 05:23 PM, Jeremy Stanley wrote:
On 2019-03-14 16:57:09 -0400 (-0400), Andy Ning wrote: [...]
[AN]: As developer I don't install anything, I submit my source code changes to gerrit as review, and Zuul is triggered to run to verify if my change break tox. Where tox runs the tests and what its environment like is transparent to (as least for me until now :)). So I'm not sure how I could do what you suggest. By "you" in this case I meant developers on StarlingX. Your CI jobs should absolutely not be a black box to you, especially since the job in question is defined in your repository:
http://git.starlingx.io/cgit/stx-config/tree/.zuul.yaml#n66
Just a matter of adding "nodeset: ubuntu-xenial" to that job definition. The change which adds it should be self-testing so you'll be able to see in the CI results whether it worked or not.
Good to know that we can somehow control the job by the .zuul.yaml file. I would think adding a nodeset to the job should be a temporary workaround. Overall I'm not sure we want to specify where a particular job is running (will that be a load balancing issue for Zuul for example?). Plus most of us want to focus on the production code, so hiding Zuul job details may not be a bad idea for developers (maybe that's the reason why .zuul.yaml is hidden file?) In terms of self-testing, I usually run tox locally on our build machine and that works fairly well. Is there a way we can trigger Zuul jobs on our change before we sumit the review? The idea is developers run tox in the same environment as Zuul runs it. Thanks, Andy
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
-- Andy Ning Cube: 3071 Tel: 613-9631408 (int: 4408) Skype: andy.ning.wr
On 2019-03-15 09:26:55 -0400 (-0400), Andy Ning wrote: [...]
Good to know that we can somehow control the job by the .zuul.yaml file. I would think adding a nodeset to the job should be a temporary workaround.
Yes, I think this is what Dean was going to propose for both the jobs you noted.
Overall I'm not sure we want to specify where a particular job is running (will that be a load balancing issue for Zuul for example?).
The nodeset doesn't specify a location, just what sort of environment should be booted for the system in which the job will be run. Aside from some highly-specialized nodesets we have which are provider-specific or specific to non x86-64 processors, our generic $distro-$release nodesets can be booted in any of our Nodepool providers.
Plus most of us want to focus on the production code, so hiding Zuul job details may not be a bad idea for developers (maybe that's the reason why .zuul.yaml is hidden file?)
Zuul will load[0] zuul.yaml or zuul.d/*.yaml with or without a leading '.' so it's not an architectural choice to make those files hidden, and they can be renamed to remove the leading '.' in the file or directory name with no change in behavior. The goal with Zuul is that your job definitions are part of your repository so they can be available to anyone to inspect and alter (and with a few security-related exceptions, will even run speculatively on proposed alterations to those configuration files so they can be proven to work before they're merged).
In terms of self-testing, I usually run tox locally on our build machine and that works fairly well. Is there a way we can trigger Zuul jobs on our change before we sumit the review? The idea is developers run tox in the same environment as Zuul runs it.
As the errors you raised demonstrate, the details/dependencies of some tests rely strongly on the characteristics of the system on which they're being run. You can of course download[1] the images we build for our test systems and boot one in a virtual machine context or run a script[2] we provide to build one yourself with or without modifications. However, it's not just the images themselves which can affect job characteristics but also the underlying machine, so we provide a breakdown[3] of the most relevant known (and unknown) properties for the providers/flavors we use. At present, exactly replicating every detail of a Zuul job without running Zuul itself is nontrivial, since job definitions are often distributed and components inherited from multiple git repositories. There is some work underway to provide tooling to make this task much easier, but most times it's sufficient for tox-based jobs to just emulate them in an appropriate system (with images described above) by checking out the repository in question, installing any system packages bindep says are missing for its "test" profile, running any additional tools/test-setup.sh script that project provides, and then invoking tox with the desired parameters. That said, if you're looking to have the Zuul service we're operating test your changes before you push them to Gerrit for review, I don't see the point. We standardize on making a "work in progress" option available to all change owners (currently implemented as a -1 vote for the Workflow label) so they can communicate to reviewers that a change is not yet ready to be reviewed. Zuul will still run all configured jobs on such changes and report results back in a review comment just like for any other proposed change. [0] https://zuul-ci.org/docs/zuul/user/config.html#configuration-loading [1] https://nb01.openstack.org/images/ [2] https://opendev.org/openstack-infra/project-config/src/branch/master/tools/b... [3] https://docs.openstack.org/infra/manual/testing.html -- Jeremy Stanley
On 2019-03-14 20:02:36 +0000 (+0000), Jeremy Stanley wrote: [...]
For the sysinv-tox-py27 psycopg2 error above, it looks like it's attempting to use a libresolv build which references symbols from a different libc. Are you maybe installing bundled libraries previously built on an ubuntu-xenial node?
After a bit more digging this seems to be https://github.com/psycopg/psycopg2-wheels/issues/2 which can be remedied by upgrading the version of psycopg2 you're using or by setting that job to also use an ubuntu-xenial nodeset for now. -- Jeremy Stanley
participants (3)
-
Andy Ning
-
Dean Troyer
-
Jeremy Stanley