Hello.
I’ve have starling-x 8 release (Debian-based) installation with duplex controllers + 2 workers and I’ve ran into issue with following configuration:
All nodes are on baremetal with 2x10G interfaces.
Interfaces are bonded with LACP, and VLAN-interfaces are created for different networks:
+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+
| 1c259ab0-c08f-4860-976c-6cc00e4cfaed | mgmt0 | platform | vlan | 201 | [] | ['ae0'] | [] | MTU=1500 |
| 4b86faef-509f-4efb-a53a-3b2d1d9b98e8 | oam0 | platform | vlan | 100 | [] | ['ae0'] | [] | MTU=1500 |
| 5edb4655-a048-4839-b01d-f2ec64e5ad01 | enp2s0f1 | None | ethernet | None | ['enp2s0f1'] | [] | ['ae0'] | MTU=1500 |
| 792d08ec-63cc-4ffa-aba2-8978e411433e | ae0 | None | ae | None | [] | ['enp2s0f0', 'enp2s0f1'] | ['clusterhost0', 'mgmt0', 'oam0'] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2+3 |
| 8d7fe091-4c63-4ef7-8983-cd2007470443 | enp2s0f0 | None | ethernet | None | ['enp2s0f0'] | [] | ['ae0'] | MTU=1500 |
| e6ff860f-a6ef-49df-bde7-437665625970 | clusterhost0 | platform | vlan | 101 | [] | ['ae0'] | [] | MTU=1500 |
Worker nodes after reboots start correctly, but after 24 hours mgmt IP is removed from interface and they become unavailable.
It appeared that networking.service is in a failed state:
sysadmin@worker-0:~$ systemctl status networking.service
system/networking.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-02-28 13:26:44 UTC; 46s ago
Docs: man:interfaces(5)
Process: 2068 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
Main PID: 2068 (code=exited, status=1/FAILURE)
CPU: 561ms
Because of that, dhclient process is not running, ‘valid_lft’ timeout is never refreshed and after lease time expiry IP address is removed from interface.
Runing ‘ifup -a’ manually gave me an error:
$ sudo /sbin/ifup -a --read-environment
No iface stanza found for master ae0
run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
ifup: failed to bring up enp2s0f0
No iface stanza found for master ae0
run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
ifup: failed to bring up enp2s0f1
Googling this error led me to a number of messages about ethernet bonding somewhat broken in debian bullseye at some point: https://blog.rtsp.us/debian-11-bullseye-bonding-problem-9d8d8866117e
Hacking /etc/network/if-pre-up.d/ifenslave as suggested in the article solved the problem
sed -i 's/ifstate -l/ifquery -l/g' /etc/network/if-pre-up.d/ifenslave
After reboot networking service is in active state and is managing dhclient processes:
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Active: active (exited) since Tue 2023-02-28 14:28:43 UTC; 52s ago
Docs: man:interfaces(5)
Process: 2064 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
Main PID: 2064 (code=exited, status=0/SUCCESS)
Tasks: 8 (limit: 306419)
Memory: 10.0M
CPU: 678ms
CGroup: /system.slice/networking.service
├─2436 /sbin/dhclient -4 -v -i -pf /run/dhclient.vlan101.pid -lf /var/lib/dhcp/dhclient.vlan101.leases -I -df /var/lib/dhcp/dhclient6.vlan102.leases vlan101
└─2569 /sbin/dhclient -4 -v -i -pf /run/dhclient.vlan201.pid -lf /var/lib/dhcp/dhclient.vlan201.leases -I -df /var/lib/dhcp/dhclient6.vlan202.leases vlan201
Could someone please validate the issue?
What is preferred way for a fix in production environment until it could be fixed upstream?