<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">I raised the bug for this<o:p></o:p></span></p>
<p class="MsoNormal"><a href="https://bugs.launchpad.net/starlingx/+bug/1840292">https://bugs.launchpad.net/starlingx/+bug/1840292</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My attempted change (dropping nova/neutron) did not work, so I have removed devstack as a zuul job from config.
<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><a href="https://review.opendev.org/#/c/676660/">https://review.opendev.org/#/c/676660/</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If approved, this will almost definitely need to be cherry picked to R2.0, since I suspect any attempted cherry-picks for already committed bugs back to the R2 config branch will also fail zuul.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Al<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com]
<br>
<b>Sent:</b> Thursday, August 15, 2019 11:39 AM<br>
<b>To:</b> Rowsell, Brent; starlingx-discuss<br>
<b>Subject:</b> Re: [Starlingx-discuss] StarlingX devstack jobs randomly failing<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">So I think that because the devstack job for config is requiring nova and neutron, and nova pulls in glance, that is why.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I think those jobs were started before containerized openstack support, so I don’t think we need nova and neutron anymore.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I’m trying this review<o:p></o:p></span></p>
<p class="MsoNormal"><a href="https://review.opendev.org/#/c/676719/">https://review.opendev.org/#/c/676719/</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If that does not work, we likely just need to turn off devstack job for config.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our env (sysinv) uses postgres, and perhaps postgres support has been dropped in train.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Al<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Rowsell, Brent
<br>
<b>Sent:</b> Thursday, August 15, 2019 11:31 AM<br>
<b>To:</b> Bailey, Henry Albert (Al); starlingx-discuss<br>
<b>Subject:</b> RE: StarlingX devstack jobs randomly failing<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Why would config be setting up a glance db ?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brent <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com]
<br>
<b>Sent:</b> Thursday, August 15, 2019 11:12 AM<br>
<b>To:</b> starlingx-discuss <starlingx-discuss@lists.starlingx.io><br>
<b>Subject:</b> [Starlingx-discuss] StarlingX devstack jobs randomly failing<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The devstack zuul jobs for config are failing.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The error looks to be a database migration failure when initializing the glance DB<o:p></o:p></p>
<p class="MsoNormal">These are the DB components being installed<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">psycopg2 2.8.3<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">SQLAlchemy 1.3.6<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Sqlalchemy_migrate 0.12.0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think this may be due to this upstream commit which merged this morning<o:p></o:p></p>
<p class="MsoNormal"><a href="https://review.opendev.org/#/c/665606/">https://review.opendev.org/#/c/665606/</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don’t have any idea on how to fix this. We can disable the jobs, or remove glance from our devstack jobs, but we are blocked until this is resolved.<o:p></o:p></p>
<p class="MsoNormal">It may affect other flock repos as well, if they also have devstack jobs that are setting up glance.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Example stacktrace:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#363636;background:white">File "/opt/stack/glance/glance/db/sqlalchemy/alembic_migrations/data_migrations/train_migrate01_backend_to_store.py", line 28, in has_migrations<br>
metadata_backend = con.execute(sql_query)<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 982, in execute<br>
return self._execute_text(object_, multiparams, params)<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _execute_text<br>
parameters,<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context<br>
e, statement, parameters, cursor, context<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception<br>
util.raise_from_cause(newraise, exc_info)<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause<br>
reraise(type(exception), exception, tb=exc_tb, cause=cause)<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context<br>
cursor, statement, parameters, context<br>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 552, in do_execute<br>
cursor.execute(statement, parameters)<br>
DBError: (psycopg2.errors.UndefinedFunction) function instr(text, unknown) does not exist<br>
LINE 1: select meta_data from image_locations where INSTR(meta_data,...<br>
^<br>
HINT: No function matches the given name and argument types. You might need to add explicit type casts.<br>
<br>
[SQL: select meta_data from image_locations where INSTR(meta_data, '"backend":') > 0]<br>
(Background on this error at: <a href="http://sqlalche.me/e/f405">http://sqlalche.me/e/f405</a>)<br>
Upgraded database to: train_expand01, current revision(s): train_expand01</span><o:p></o:p></p>
</div>
</body>
</html>