I raised the bug for this https://bugs.launchpad.net/starlingx/+bug/1840292 My attempted change (dropping nova/neutron) did not work, so I have removed devstack as a zuul job from config. https://review.opendev.org/#/c/676660/ If approved, this will almost definitely need to be cherry picked to R2.0, since I suspect any attempted cherry-picks for already committed bugs back to the R2 config branch will also fail zuul. Al From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Thursday, August 15, 2019 11:39 AM To: Rowsell, Brent; starlingx-discuss Subject: Re: [Starlingx-discuss] StarlingX devstack jobs randomly failing So I think that because the devstack job for config is requiring nova and neutron, and nova pulls in glance, that is why. I think those jobs were started before containerized openstack support, so I don't think we need nova and neutron anymore. I'm trying this review https://review.opendev.org/#/c/676719/ If that does not work, we likely just need to turn off devstack job for config. Our env (sysinv) uses postgres, and perhaps postgres support has been dropped in train. Al From: Rowsell, Brent Sent: Thursday, August 15, 2019 11:31 AM To: Bailey, Henry Albert (Al); starlingx-discuss Subject: RE: StarlingX devstack jobs randomly failing Why would config be setting up a glance db ? Brent From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: Thursday, August 15, 2019 11:12 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] StarlingX devstack jobs randomly failing The devstack zuul jobs for config are failing. The error looks to be a database migration failure when initializing the glance DB These are the DB components being installed psycopg2 2.8.3 SQLAlchemy 1.3.6 Sqlalchemy_migrate 0.12.0 I think this may be due to this upstream commit which merged this morning https://review.opendev.org/#/c/665606/ I don't have any idea on how to fix this. We can disable the jobs, or remove glance from our devstack jobs, but we are blocked until this is resolved. It may affect other flock repos as well, if they also have devstack jobs that are setting up glance. Example stacktrace: File "/opt/stack/glance/glance/db/sqlalchemy/alembic_migrations/data_migrations/train_migrate01_backend_to_store.py", line 28, in has_migrations metadata_backend = con.execute(sql_query) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 982, in execute return self._execute_text(object_, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _execute_text parameters, File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context e, statement, parameters, cursor, context File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception util.raise_from_cause(newraise, exc_info) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context cursor, statement, parameters, context File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 552, in do_execute cursor.execute(statement, parameters) DBError: (psycopg2.errors.UndefinedFunction) function instr(text, unknown) does not exist LINE 1: select meta_data from image_locations where INSTR(meta_data,... ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. [SQL: select meta_data from image_locations where INSTR(meta_data, '"backend":') > 0] (Background on this error at: http://sqlalche.me/e/f405) Upgraded database to: train_expand01, current revision(s): train_expand01