<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"MS PGothic";
panose-1:2 11 6 0 7 2 5 8 2 4;}
@font-face
{font-family:"\@MS PGothic";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
mso-fareast-language:JA;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";
mso-fareast-language:EN-GB;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello, this does look like a valid issue. The following STX bug has been raised:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://bugs.launchpad.net/starlingx/+bug/2010119">https://bugs.launchpad.net/starlingx/+bug/2010119</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The issue appears to have been resolved in ifenslave v2.13 (STX 8 is on v2.12)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The workaround noted in your mail could be applied to a system and wouldn’t be overwritten until the ifenslave package were updated via system upgrade or some other patch that touches those files. The noted workaround seems to produce
essentially the same result that went into fixing this issue officially in v2.13.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The proper way to patch this in a production STX 8 environment would be to create and install a patch after pulling in ifenslave v2.13 to be explicitly built (this is likely exactly what the bug fix will be if you want to have a go at doing
it in the master branch). The STX documentation [1] for patching seems to need some updating for Debian (references to RPMs), but the sw-patch commands are still valid and the references [2] and [3] should help if you intend to go this route.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Steve<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://docs.starlingx.io/developer_resources/starlingx_patching.html">
https://docs.starlingx.io/developer_resources/starlingx_patching.html</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://review.opendev.org/c/starlingx/update/+/845169">
https://review.opendev.org/c/starlingx/update/+/845169</a><o:p></o:p></p>
<p class="MsoNormal">[3] <a href="https://opendev.org/starlingx/update/src/branch/master/sw-patch/cgcs-patch/cgcs_make_patch">
https://opendev.org/starlingx/update/src/branch/master/sw-patch/cgcs-patch/cgcs_make_patch</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="mso-fareast-language:JA">From:</span></b><span style="mso-fareast-language:JA"> Sergei Akhmatov <sergei.akhmatov@xunison.com>
<br>
<b>Sent:</b> Wednesday, March 1, 2023 8:18 AM<br>
<b>To:</b> starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> [Starlingx-discuss] Stx8 interface bonding issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="background:#FCFC03"><b><span lang="EN-GB" style="font-family:"MS PGothic",sans-serif;color:black;mso-fareast-language:JA">CAUTION: This email comes from a non Wind River email account!</span></b><span lang="EN-GB" style="font-size:12.0pt;font-family:"MS PGothic",sans-serif;color:black;mso-fareast-language:JA"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"MS PGothic",sans-serif;color:black;mso-fareast-language:JA">Do not click links or open attachments unless you recognize the sender and know the content is safe.</span><span lang="EN-GB" style="font-size:12.0pt;font-family:"MS PGothic",sans-serif;mso-fareast-language:JA"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hello.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve have starling-x 8 release (Debian-based) installation with duplex controllers + 2 workers and I’ve ran into issue with following configuration:<br>
All nodes are on baremetal with 2x10G interfaces. <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Interfaces are bonded with LACP, and VLAN-interfaces are created for different networks</span>:<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| 1c259ab0-c08f-4860-976c-6cc00e4cfaed | mgmt0 | platform | vlan | 201 | [] | ['ae0'] | [] | MTU=1500 |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| 4b86faef-509f-4efb-a53a-3b2d1d9b98e8 | oam0 | platform | vlan | 100 | [] | ['ae0'] | [] | MTU=1500 |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| 5edb4655-a048-4839-b01d-f2ec64e5ad01 | enp2s0f1 | None | ethernet | None | ['enp2s0f1'] | [] | ['ae0'] | MTU=1500 |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| 792d08ec-63cc-4ffa-aba2-8978e411433e | ae0 | None | ae | None | [] | ['enp2s0f0', 'enp2s0f1'] | ['clusterhost0', 'mgmt0', 'oam0'] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2+3 |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| 8d7fe091-4c63-4ef7-8983-cd2007470443 | enp2s0f0 | None | ethernet | None | ['enp2s0f0'] | [] | ['ae0'] | MTU=1500 |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">| e6ff860f-a6ef-49df-bde7-437665625970 | clusterhost0 | platform | vlan | 101 | [] | ['ae0'] | [] | MTU=1500 |<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Worker nodes after reboots start correctly, but after 24 hours mgmt IP is removed from interface and they become unavailable.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">It appeared that networking.service is in a failed state:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">sysadmin@worker-0:~$ systemctl status networking.service<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">system/networking.service; enabled; vendor preset: enabled)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> Active: failed (Result: exit-code) since Tue 2023-02-28 13:26:44 UTC; 46s ago<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> Docs: man:interfaces(5)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> Process: 2068 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> Main PID: 2068 (code=exited, status=1/FAILURE)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> CPU: 561ms<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Because of that, dhclient process is not running, ‘valid_lft’ timeout is never refreshed and after lease time expiry IP address is removed from interface.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">Runing ‘ifup -a’ manually gave me an error:<o:p></o:p></span></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in;box-sizing: inherit;--saf-0:rgba(var(--sk_foreground_low,29,28,29),0.13);overflow-wrap: break-word;font-variant-ligatures: none;tab-size: 4;white-space:pre-wrap;background:rgba(var(--sk_foreground_min,29,28,29),0.04);border:var(--saf-0);border-radius: 4px;overflow-y: hidden;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;font-variant-caps: normal;orphans: 2;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"><o:p> </o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">$ sudo /sbin/ifup -a --read-environment<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">No iface stanza found for master ae0<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">ifup: failed to bring up enp2s0f0<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">No iface stanza found for master ae0<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">ifup: failed to bring up enp2s0f1<o:p></o:p></span></pre>
<p class="MsoNormal" style="background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt;background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">Googling this error led me to a number of messages about ethernet bonding somewhat broken in debian bullseye
at some point: <a href="https://urldefense.com/v3/__https:/blog.rtsp.us/debian-11-bullseye-bonding-problem-9d8d8866117e__;!!AjveYdw8EvQ!fIokdGuoletNb8bl_iIAVJcTwh9mC2aHa4ylbvRzZoA5kHV0mVOJ7O0G0laEEa60wamwqzQTGzGhMRdzWPTuVZLWONLktkn3KD88uQ$" target="_blank">https://blog.rtsp.us/debian-11-bullseye-bonding-problem-9d8d8866117e</a><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">Hacking /etc/network/if-pre-up.d/ifenslave as suggested in the article solved the problem<o:p></o:p></span></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in;box-sizing: inherit;--saf-0:rgba(var(--sk_foreground_low,29,28,29),0.13);overflow-wrap: break-word;font-variant-ligatures: none;tab-size: 4;white-space:pre-wrap;background:rgba(var(--sk_foreground_min,29,28,29),0.04);border:var(--saf-0);border-radius: 4px;overflow-y: hidden;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;font-variant-caps: normal;orphans: 2;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"><o:p> </o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">sed -i 's/ifstate -l/ifquery -l/g' /etc/network/if-pre-up.d/ifenslave<o:p></o:p></span></pre>
<p class="MsoNormal" style="background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="EN-GB" style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D">After reboot networking service is in active state and is managing dhclient processes:<o:p></o:p></span></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in;box-sizing: inherit;--saf-0:rgba(var(--sk_foreground_low,29,28,29),0.13);overflow-wrap: break-word;font-variant-ligatures: none;tab-size: 4;white-space:pre-wrap;background:rgba(var(--sk_foreground_min,29,28,29),0.04);border:var(--saf-0);border-radius: 4px;overflow-y: hidden;counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0;font-variant-caps: normal;orphans: 2;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-thickness: initial;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">sysadmin@worker-0:~$ systemctl status networking.service<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D">● networking.service - Raise network interfaces<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Active: active (exited) since Tue 2023-02-28 14:28:43 UTC; 52s ago<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Docs: man:interfaces(5)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Process: 2064 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Main PID: 2064 (code=exited, status=0/SUCCESS)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Tasks: 8 (limit: 306419)<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> Memory: 10.0M<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> CPU: 678ms<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> CGroup: /system.slice/networking.service<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> ├─2436 /sbin/dhclient -4 -v -i -pf /run/<a href="https://urldefense.com/v3/__http:/dhclient.vlan101.pid__;!!AjveYdw8EvQ!fIokdGuoletNb8bl_iIAVJcTwh9mC2aHa4ylbvRzZoA5kHV0mVOJ7O0G0laEEa60wamwqzQTGzGhMRdzWPTuVZLWONLktkkGiiNJjg$">dhclient.vlan101.pid</a> -lf /var/lib/dhcp/dhclient.vlan101.leases -I -df /var/lib/dhcp/dhclient6.vlan102.leases vlan101<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1D1C1D"> └瘢雹─2569 /sbin/dhclient -4 -v -i -pf /run/<a href="https://urldefense.com/v3/__http:/dhclient.vlan201.pid__;!!AjveYdw8EvQ!fIokdGuoletNb8bl_iIAVJcTwh9mC2aHa4ylbvRzZoA5kHV0mVOJ7O0G0laEEa60wamwqzQTGzGhMRdzWPTuVZLWONLktknsiJzrwA$">dhclient.vlan201.pid</a> -lf /var/lib/dhcp/dhclient.vlan201.leases -I -df /var/lib/dhcp/dhclient6.vlan202.leases vlan201<o:p></o:p></span></pre>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Could someone please validate the issue?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">What is preferred way for a fix in production environment until it could be fixed upstream?<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>