Hello.

 

I’ve have starling-x 8 release (Debian-based) installation with duplex controllers + 2 workers and I’ve ran into issue with following configuration:
All nodes are on baremetal with 2x10G interfaces.

Interfaces are bonded with LACP, and VLAN-interfaces are created for different networks:

 

+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+

| uuid                                 | name         | class    | type     | vlan id | ports        | uses i/f                 | used by i/f                       | attributes                                       |

+--------------------------------------+--------------+----------+----------+---------+--------------+--------------------------+-----------------------------------+--------------------------------------------------+

| 1c259ab0-c08f-4860-976c-6cc00e4cfaed | mgmt0        | platform | vlan     | 201     | []           | ['ae0']                  | []                                | MTU=1500                                         |

| 4b86faef-509f-4efb-a53a-3b2d1d9b98e8 | oam0         | platform | vlan     | 100     | []           | ['ae0']                  | []                                | MTU=1500                                         |

| 5edb4655-a048-4839-b01d-f2ec64e5ad01 | enp2s0f1     | None     | ethernet | None    | ['enp2s0f1'] | []                       | ['ae0']                           | MTU=1500                                         |

| 792d08ec-63cc-4ffa-aba2-8978e411433e | ae0          | None     | ae       | None    | []           | ['enp2s0f0', 'enp2s0f1'] | ['clusterhost0', 'mgmt0', 'oam0'] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2+3 |

| 8d7fe091-4c63-4ef7-8983-cd2007470443 | enp2s0f0     | None     | ethernet | None    | ['enp2s0f0'] | []                       | ['ae0']                           | MTU=1500                                         |

| e6ff860f-a6ef-49df-bde7-437665625970 | clusterhost0 | platform | vlan     | 101     | []           | ['ae0']                  | []                                | MTU=1500                                         |

 

Worker nodes after reboots start correctly, but after 24 hours mgmt IP is removed from interface and they become unavailable.

It appeared that networking.service is in a failed state:

 

sysadmin@worker-0:~$ systemctl status networking.service

system/networking.service; enabled; vendor preset: enabled)

     Active: failed (Result: exit-code) since Tue 2023-02-28 13:26:44 UTC; 46s ago

       Docs: man:interfaces(5)

    Process: 2068 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)

   Main PID: 2068 (code=exited, status=1/FAILURE)

        CPU: 561ms

 

Because of that, dhclient process is not running, ‘valid_lft’ timeout is never refreshed and after lease time expiry IP address is removed from interface.

 

Runing ‘ifup -a’ manually gave me an error:

 
$ sudo /sbin/ifup -a --read-environment
No iface stanza found for master ae0
run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
ifup: failed to bring up enp2s0f0
No iface stanza found for master ae0
run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
ifup: failed to bring up enp2s0f1

 

Googling this error led me to a number of messages about ethernet bonding somewhat broken in debian bullseye at some point: https://blog.rtsp.us/debian-11-bullseye-bonding-problem-9d8d8866117e

Hacking /etc/network/if-pre-up.d/ifenslave as suggested in the article solved the problem

 
sed -i 's/ifstate -l/ifquery -l/g' /etc/network/if-pre-up.d/ifenslave

 

 

After reboot networking service is in active state and is managing dhclient processes:

sysadmin@worker-0:~$ systemctl status networking.service
● networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: active (exited) since Tue 2023-02-28 14:28:43 UTC; 52s ago
       Docs: man:interfaces(5)
    Process: 2064 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
   Main PID: 2064 (code=exited, status=0/SUCCESS)
      Tasks: 8 (limit: 306419)
     Memory: 10.0M
        CPU: 678ms
     CGroup: /system.slice/networking.service
             ├─2436 /sbin/dhclient -4 -v -i -pf /run/dhclient.vlan101.pid -lf /var/lib/dhcp/dhclient.vlan101.leases -I -df /var/lib/dhcp/dhclient6.vlan102.leases vlan101
             └─2569 /sbin/dhclient -4 -v -i -pf /run/dhclient.vlan201.pid -lf /var/lib/dhcp/dhclient.vlan201.leases -I -df /var/lib/dhcp/dhclient6.vlan202.leases vlan201

 

Could someone please validate the issue?

What is preferred way for a fix in production environment until it could be fixed upstream?