<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";
mso-fareast-language:EN-GB;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I’ve have starling-x 8 release (Debian-based) installation with duplex controllers + 2 workers and I’ve ran into issue with following configuration:<br>
All nodes are on baremetal with 2x10G interfaces. <o:p></o:p></span></p>
<p class="MsoNormal">Interfaces are bonded with LACP, and VLAN-interfaces are created for different networks<span lang="EN-US">:<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |<o:p></o:p></p>
<p class="MsoNormal">+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">| 1c259ab0-c08f-4860-976c-6cc00e4cfaed | mgmt0 | platform | vlan | 201 | [] | ['ae0'] | [] | MTU=1500 |<o:p></o:p></p>
<p class="MsoNormal">| 4b86faef-509f-4efb-a53a-3b2d1d9b98e8 | oam0 | platform | vlan | 100 | [] | ['ae0'] | [] | MTU=1500 |<o:p></o:p></p>
<p class="MsoNormal">| 5edb4655-a048-4839-b01d-f2ec64e5ad01 | enp2s0f1 | None | ethernet | None | ['enp2s0f1'] | [] | ['ae0'] | MTU=1500 |<o:p></o:p></p>
<p class="MsoNormal">| 792d08ec-63cc-4ffa-aba2-8978e411433e | ae0 | None | ae | None | [] | ['enp2s0f0', 'enp2s0f1'] | ['clusterhost0', 'mgmt0', 'oam0'] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2+3 |<o:p></o:p></p>
<p class="MsoNormal">| 8d7fe091-4c63-4ef7-8983-cd2007470443 | enp2s0f0 | None | ethernet | None | ['enp2s0f0'] | [] | ['ae0'] | MTU=1500 |<o:p></o:p></p>
<p class="MsoNormal">| e6ff860f-a6ef-49df-bde7-437665625970 | clusterhost0 | platform | vlan | 101 | [] | ['ae0'] | [] | MTU=1500 |<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Worker nodes after reboots start correctly, but after 24 hours mgmt IP is removed from interface and they become unavailable.<o:p></o:p></p>
<p class="MsoNormal">It appeared that networking.service is in a failed state:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">sysadmin@worker-0:~$ systemctl status networking.service<o:p></o:p></p>
<p class="MsoNormal">system/networking.service; enabled; vendor preset: enabled)<o:p></o:p></p>
<p class="MsoNormal"> Active: failed (Result: exit-code) since Tue 2023-02-28 13:26:44 UTC; 46s ago<o:p></o:p></p>
<p class="MsoNormal"> Docs: man:interfaces(5)<o:p></o:p></p>
<p class="MsoNormal"> Process: 2068 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)<o:p></o:p></p>
<p class="MsoNormal"> Main PID: 2068 (code=exited, status=1/FAILURE)<o:p></o:p></p>
<p class="MsoNormal"> CPU: 561ms<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Because of that, dhclient process is not running, ‘valid_lft’ timeout is never refreshed and after lease time expiry IP address is removed from interface.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">Runing ‘ifup -a’ manually gave me an error:<o:p></o:p></span></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in;box-sizing: inherit;--saf-0:rgba(var(--sk_foreground_low,29,28,29),0.13);overflow-wrap: break-word;font-variant-ligatures: none;tab-size: 4;white-space:pre-wrap;background:rgba(var(--sk_foreground_min,29,28,29),0.04);border:var(--saf-0);border-radius: 4px;overflow-y: hidden;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;font-variant-caps: normal;orphans: 2;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"><o:p> </o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">$ sudo /sbin/ifup -a --read-environment<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">No iface stanza found for master ae0<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">ifup: failed to bring up enp2s0f0<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">No iface stanza found for master ae0<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">ifup: failed to bring up enp2s0f1<o:p></o:p></span></pre>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">Googling this error led me to a number of messages about ethernet bonding somewhat broken in debian bullseye at some point: <a href="https://blog.rtsp.us/debian-11-bullseye-bonding-problem-9d8d8866117e" target="_blank">https://blog.rtsp.us/debian-11-bullseye-bonding-problem-9d8d8866117e</a><br style="box-sizing: inherit">
<br>
<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">Hacking /etc/network/if-pre-up.d/ifenslave as suggested in the article solved the problem<o:p></o:p></span></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in;box-sizing: inherit;--saf-0:rgba(var(--sk_foreground_low,29,28,29),0.13);overflow-wrap: break-word;font-variant-ligatures: none;tab-size: 4;white-space:pre-wrap;background:rgba(var(--sk_foreground_min,29,28,29),0.04);border:var(--saf-0);border-radius: 4px;overflow-y: hidden;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;font-variant-caps: normal;orphans: 2;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"><o:p> </o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">sed -i 's/ifstate -l/ifquery -l/g' /etc/network/if-pre-up.d/ifenslave<o:p></o:p></span></pre>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">After reboot networking service is in active state and is managing dhclient processes:<o:p></o:p></span></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in;box-sizing: inherit;--saf-0:rgba(var(--sk_foreground_low,29,28,29),0.13);overflow-wrap: break-word;font-variant-ligatures: none;tab-size: 4;white-space:pre-wrap;background:rgba(var(--sk_foreground_min,29,28,29),0.04);border:var(--saf-0);border-radius: 4px;overflow-y: hidden;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;font-variant-caps: normal;orphans: 2;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">sysadmin@worker-0:~$ systemctl status networking.service<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">● networking.service - Raise network interfaces<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Active: active (exited) since Tue 2023-02-28 14:28:43 UTC; 52s ago<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Docs: man:interfaces(5)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Process: 2064 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Main PID: 2064 (code=exited, status=0/SUCCESS)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Tasks: 8 (limit: 306419)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Memory: 10.0M<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> CPU: 678ms<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> CGroup: /system.slice/networking.service<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> ├─2436 /sbin/dhclient -4 -v -i -pf /run/dhclient.vlan101.pid -lf /var/lib/dhcp/dhclient.vlan101.leases -I -df /var/lib/dhcp/dhclient6.vlan102.leases vlan101<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> └瘢雹─2569 /sbin/dhclient -4 -v -i -pf /run/dhclient.vlan201.pid -lf /var/lib/dhcp/dhclient.vlan201.leases -I -df /var/lib/dhcp/dhclient6.vlan202.leases vlan201<o:p></o:p></span></pre>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Could someone please validate the issue?<o:p></o:p></p>
<p class="MsoNormal">What is preferred way for a fix in production environment until it could be fixed upstream?<o:p></o:p></p>
</div>
</body>
</html>