https://bugs.launchpad.net/starlingx/+bug/1913582 i did also find this which should let me manage where pxe booting occurs, as in our network config you can only pxe from the 10.16.48 network, so some config changes are needed which ill be also trying today On Thu, May 12, 2022 at 2:11 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

Hey Scott,

Wrt what we were talking about in the community meeting:

- ROOK Deployment for say an AIO-DX + Workers system > ROOK deployment on Controllers * see https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/aio_du... - notice that there are multiple steps to setup rook in this guide > adding of high-level ceph-rook backend for system, while configuring controller-0 > adding of ceph-mon-placement and ceph-mgr-placement labels on each controller (controller-0 and controller-1) > adding of OSDs to each controller (controller-0 and controller-1) > ROOK OSD deployment on Workers * DOH ... we are missing that ( I'll get the starlingx DOC TEAM to look into this ) * in https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/aio_du... - we forgot to add ... at the very end ... AFTER the unlocking of the worker nodes, the section to optionally add OSDs to the worker nodes - you can just use the section at the bottom of https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/aio_du... (i.e. If using Rook container-based Ceph, finish configuring the ceph-rook Persistent Storage Backend) but substitute in the worker hostnames.

- PXE BOOT details

( NOTE ... I can't remember how we decided you needed a pxeboot network. What was the original problem that you were trying to solve ? e.g. if it was just to change the IP Address Subnet for mgmt., you can do that simply with a bootstrap override, see below )

> As discussed, only the first time the very first controller is installed (controller-0) can an external PXEBOOT server be used. * The docs on how to do this are here: - https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/config... - https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/access...

> Also as discussed, * For all subsequent pxeboot installs of starlingx hosts, it is done by the active controller (controller-0 or controller-1) * By default, StarlingX uses the MGMT network for PXEBOOTING * But for scenarios where the MGMT network can not be used ( i.e. MGMT network needs to be vlan-tagged for some reason or MGMT network needs to be IPv6 ) then one can configure a PXEBOOT network who's sole purpose is for pxebooting. * This is discussed in many of the 'Network Planning' PLANNING sections of StarlingX DOCS e.g. https://docs.starlingx.io/planning/kubernetes/network-requirements.html ... and every page in-between ... https://docs.starlingx.io/planning/kubernetes/network-planning-the-pxe-boot-... * DOH ... we don't have this well documented wrt how to configure this - again, I'll get the DOC team to look at this - for now here's the info on how to use it.

Using 'pxeboot' network --------------------------------- - the 'pxeboot' network (and all networks I think) are always present in the system e.g. you can see them with system network-list system addrpool-list - the easiest way to configure the pxeboot IP Address Subnet is at bootstrap time in the localhost.yml (bootstrap override file) e.g. system_mode: duplex

dns_servers: - 8.8.8.8 - 8.8.4.4

pxeboot_subnet: 169.254.202.0/24 # pxeboot_start_address: # pxeboot_end_address:

management_subnet: 192.168.204.0/24 # management_start_address: # management_end_address:

external_oam_subnet: <OAM-IP-SUBNET>/<OAM-IP-SUBNET-LENGTH> external_oam_gateway_address: <OAM-GATEWAY-IP-ADDRESS> external_oam_floating_address: <OAM-FLOATING-IP-ADDRESS> external_oam_node_0_address: <OAM-CONTROLLER-0-IP-ADDRESS> external_oam_node_1_address: <OAM-CONTROLLER-1-IP-ADDRESS>

admin_username: admin admin_password: <admin-password> ansible_become_pass: <sysadmin-password>

# OPTIONALLY provide a ROOT CA certificate and key for k8s root ca, # if not specified, one will be auto-generated, # see 'Kubernetes Root CA Certificate' in Security Guide for details. k8s_root_ca_cert: < your_root_ca_cert.pem > k8s_root_ca_key: < your_root_ca_key.pem > apiserver_cert_sans: - < your_hostname_for_oam_floating.your_domain >

( see all parameters and their defaults in /usr/share/ansible/stx-ansible/playbooks/host_vars/bootstrap/default.yml on target )

- you then need to configure a host interface on controller-0 and attach it to the 'pxeboot' network IMPORTANT: a host's pxeboot interface MUST be a port-based/untagged vlan AND MUST be on the same physical interface as the mgmt interface (which would be vlan-tagged). e.g. // the config of the mgmt. interface and pxeboot interface for controller-0 MGMT_PORT=<MGMT-PORT> system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_PORT -c platform system interface-network-assign controller-0 $MGMT_PORT pxeboot

system host-if-add controller-0 -V 22 -c platform mgmt0 vlan $MGMT_PORT system interface-network-assign controller-0 mgmt0 mgmt system interface-network-assign controller-0 mgmt0 cluster-host

- dnsmasq (boot server on controllers) is configured to answer dhcp on the pxeboot and mgmt interface IPs

- when adding new hosts (e.g. second controller or workers), you need to ensure in BIOS that these nodes are trying to network boot off of the interface physically attached to the port-based/vlan-untagged pxeboot network. > and then after new host boots, you need to configure its host interfaces correctly ... e.g. with pxeboot interface on port-based/vlan-untagged physical interface, and the mgmt. interface as a vlan interface on that same physical interface.

Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 12:00 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

scenario...

i have a host say controller-0

prior to any ansible run

i need to create a bond, and bridges and vlans

sure.... Add a bond device as root:

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up

Add the bridge device and attach VLAN to it: ip link add br0 type bridge ip link set bond0.1648 master br0 ip link set bond0.1664 master br0 ip link set bond0.1680 master br0 ip link set br0 up

so i see where in starlingx

system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0

does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge.

where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664

OAM_IF=bond0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0 MGMT_IF=bond0.1664 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

the reason for this being our switches are

# MGMT interface vlan1648 address 10.16.48.2/24 address-virtual 44:38:39:FF:00:02 10.16.48.1 vlan-id 1648 vlan-raw-device bridge

interface vlan1672 address 10.16.72.2/24 address-virtual 44:38:39:FF:00:03 10.16.72.1 vlan-id 1672 vlan-raw-device bridge

interface vlan1680 address 10.16.80.2/24 address-virtual 44:38:39:FF:00:03 10.16.80.1 vlan-id 1680 vlan-raw-device bridge

interface vlan1696 address 10.16.96.2/24 address-virtual 44:38:39:FF:00:03 10.16.96.1 vlan-id 1696 vlan-raw-device bridge

interface vlan1664 address 10.16.64.2/24 address-virtual 44:38:39:FF:00:07 10.16.64.1 vlan-id 1664 vlan-raw-device bridge

and further down DATAIF_0=bond0.1680

the reason being we are trying to have starlingx conform to our networks topology I also noted: in https://docs.starlingx.io/deploy_install_guides/r6_release/ansible_bootstrap... ...

Network Properties I listed at the bottom

can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.

Network Properties pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_addres cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 10.16.48.112/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.114/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 8: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1664:12 valid_lft forever preferred_lft forever inet 169.254.202.1/24 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 9: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 10: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 11: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever

_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

I think even allowing for special conditions of bond0 to allow OAM_IF, MGMT_IF, CLUSTER_IF, PXE_IF to all be set on the same bond0, and even dropping the vlans for corner cases then i would only need to set the Install-time-only parameters: Network Properties pxeboot_subnet: 10.16.48.1 pxeboot_start_address 10.16.48.100 pxeboot_end_address 10.16.48.125 management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up Set VLAN on the bond device: ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up and modify the host details as per below: OAM_IF=bond0.1648 MGMT_IF=bond0.1664 CLUSTER_IF=bond0.1680 PXE_IF=bond0 <- this puts pxe on bond0 On Thu, May 12, 2022 at 2:25 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

replying to answer your questions from email below, see in-lined below, Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 12:00 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

scenario...

i have a host say controller-0

prior to any ansible run [Greg] I assume you mean the bootstrap ansible playbook

i need to create a bond, and bridges and vlans [Greg] You do need an interface to the outside world … e.g. in order to download container images from docker hub. [Greg] Why can you not simply create a single interface (one link of bond) with a vlan ?

sure.... Add a bond device as root:

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up

Add the bridge device and attach VLAN to it: ip link add br0 type bridge ip link set bond0.1648 master br0 ip link set bond0.1664 master br0 ip link set bond0.1680 master br0 ip link set br0 up

so i see where in starlingx [Greg] the following commands are only possible AFTER bootstrap

system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0

does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge. [Greg] No … there is no requirement for a bridge with StarlingX.

where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664 [Greg] Yes … i.e. OAM on port-based/untagged-vlan of bond and MGMT on vlan-tag=1664 on bond ( BUT here is where you need the pxeboot network because your MGMT network is vlan-tagged … and you can't pxe boot over that )

OAM_IF=bond0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0 MGMT_IF=bond0.1664 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform [Greg] don't think you actually need this command system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

the reason for this being our switches are

# MGMT interface vlan1648 address 10.16.48.2/24 address-virtual 44:38:39:FF:00:02 10.16.48.1 vlan-id 1648 vlan-raw-device bridge

interface vlan1672 address 10.16.72.2/24 address-virtual 44:38:39:FF:00:03 10.16.72.1 vlan-id 1672 vlan-raw-device bridge

interface vlan1680 address 10.16.80.2/24 address-virtual 44:38:39:FF:00:03 10.16.80.1 vlan-id 1680 vlan-raw-device bridge

interface vlan1696 address 10.16.96.2/24 address-virtual 44:38:39:FF:00:03 10.16.96.1 vlan-id 1696 vlan-raw-device bridge

interface vlan1664 address 10.16.64.2/24 address-virtual 44:38:39:FF:00:07 10.16.64.1 vlan-id 1664 vlan-raw-device bridge

and further down DATAIF_0=bond0.1680

the reason being we are trying to have starlingx conform to our networks topology I also noted: in https://docs.starlingx.io/deploy_install_guides/r6_release/ansible_bootstrap... ...

Network Properties I listed at the bottom

can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.

Network Properties pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_addres cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 10.16.48.112/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.114/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 8: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1664:12 valid_lft forever preferred_lft forever inet 169.254.202.1/24 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 9: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 10: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 11: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever

_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

working through the configuration now based on findings, and yes i only have to do the ip link commands once prior to bootstrap i did get bond0 and vlans on a previous try to be configured after system host-unlock controller-0 they were just in the wrong order, so rebuilding the primary node, if it works and put the interfaces and networks on proper interfaces and i can bootstrap controller-1 and get past unlocking that also... i think it will be a win! On Thu, May 12, 2022 at 7:32 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

Were you successful ?

( One question ... you are only having to do the 'ip link ...' commands BEFORE bootstrap in order to have IP Connectivity to the outside world for bootstrapping .. correct ? )

Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 8:21 PM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

I think even allowing for special conditions of bond0 to allow OAM_IF, MGMT_IF, CLUSTER_IF, PXE_IF to all be set on the same bond0, and even dropping the vlans for corner cases then i would only need to set the Install-time-only parameters: Network Properties

pxeboot_subnet: 10.16.48.1 pxeboot_start_address 10.16.48.100 pxeboot_end_address 10.16.48.125 management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up

and modify the host details as per below:

OAM_IF=bond0.1648 MGMT_IF=bond0.1664 CLUSTER_IF=bond0.1680 PXE_IF=bond0 <- this puts pxe on bond0

On Thu, May 12, 2022 at 2:25 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

...

replying to answer your questions from email below, see in-lined below, Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 12:00 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

scenario...

i have a host say controller-0

prior to any ansible run [Greg] I assume you mean the bootstrap ansible playbook

i need to create a bond, and bridges and vlans [Greg] You do need an interface to the outside world … e.g. in order to download container images from docker hub. [Greg] Why can you not simply create a single interface (one link of bond) with a vlan ?

sure.... Add a bond device as root:

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up

Add the bridge device and attach VLAN to it: ip link add br0 type bridge ip link set bond0.1648 master br0 ip link set bond0.1664 master br0 ip link set bond0.1680 master br0 ip link set br0 up

so i see where in starlingx [Greg] the following commands are only possible AFTER bootstrap

system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0

does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge. [Greg] No … there is no requirement for a bridge with StarlingX.

where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664 [Greg] Yes … i.e. OAM on port-based/untagged-vlan of bond and MGMT on vlan-tag=1664 on bond ( BUT here is where you need the pxeboot network because your MGMT network is vlan-tagged … and you can't pxe boot over that )

OAM_IF=bond0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0 MGMT_IF=bond0.1664 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform [Greg] don't think you actually need this command system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

the reason for this being our switches are

# MGMT interface vlan1648 address 10.16.48.2/24 address-virtual 44:38:39:FF:00:02 10.16.48.1 vlan-id 1648 vlan-raw-device bridge

interface vlan1672 address 10.16.72.2/24 address-virtual 44:38:39:FF:00:03 10.16.72.1 vlan-id 1672 vlan-raw-device bridge

interface vlan1680 address 10.16.80.2/24 address-virtual 44:38:39:FF:00:03 10.16.80.1 vlan-id 1680 vlan-raw-device bridge

interface vlan1696 address 10.16.96.2/24 address-virtual 44:38:39:FF:00:03 10.16.96.1 vlan-id 1696 vlan-raw-device bridge

interface vlan1664 address 10.16.64.2/24 address-virtual 44:38:39:FF:00:07 10.16.64.1 vlan-id 1664 vlan-raw-device bridge

and further down DATAIF_0=bond0.1680

the reason being we are trying to have starlingx conform to our networks topology I also noted: in https://docs.starlingx.io/deploy_install_guides/r6_release/ansible_boo tstrap_configs.html#install-time-only-params-r6 ...

Network Properties I listed at the bottom

can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.

Network Properties pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_addres cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 10.16.48.112/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.114/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 8: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1664:12 valid_lft forever preferred_lft forever inet 169.254.202.1/24 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 9: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 10: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 11: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever

_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

I doubt if this is causing issues but shouldn't this: pxeboot_subnet: 10.16.48.1/24 be pxeboot_subnet: 10.16.48.0/24 ? ( and similar issue for oam_subnet ) ACTUALLY ... you have the pxeboot_subnet and the oam_subnet being the same ? external_oam_subnet: 10.16.48.1/24 pxeboot_subnet: 10.16.48.1/24 That is wrong ... they have to be separate IP subnets. You should also remove: external_oam_node_2_address: 10.16.48.116 external_oam_node_3_address: 10.16.48.117 external_oam_node_4_address: 10.16.48.118 Other comment on your config commands ... system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host // you should remove this, as you assign 'cluster-host' network below to a separate interface ... system host-if-modify controller-0 $CLUSTER_IF -c platform system interface-network-assign controller-0 $CLUSTER_IF cluster-host ... I would redo install using a unique IP Subnet for pxeboot and oam networks, Greg. -----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Thursday, May 12, 2022 4:30 AM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking [Please note: This e-mail is from an EXTERNAL e-mail address] ok, so getting closer its about the ip space and preset variables now and ... after installing software on controller-0 last run i used note i set pxeboot_ vars to put the right network and interfaces on bond0 [sysadmin@controller-0 ~(keystone_admin)]$ cat localhost.yml system_mode: duplex dns_servers: - 8.8.8.8 - 8.8.4.4 external_oam_subnet: 10.16.48.1/24 external_oam_gateway_address: 10.16.48.1 external_oam_floating_address: 10.16.48.110 external_oam_node_0_address: 10.16.48.114 external_oam_node_1_address: 10.16.48.115 external_oam_node_2_address: 10.16.48.116 external_oam_node_3_address: 10.16.48.117 external_oam_node_4_address: 10.16.48.118 admin_username: admin admin_password: somepass ansible_become_pass: somepass # Add these lines to configure Docker to use a proxy server # # docker_http_proxy: http://my.proxy.com:1080 # # docker_https_proxy: https://my.proxy.com:1443 # # docker_no_proxy: # # - 1.2.3.4 # kubernetes_version: 1.21.3 pxeboot_subnet: 10.16.48.1/24 pxeboot_start_address: 10.16.48.100 pxeboot_end_address: 10.16.48.151 then ran ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml and ... it was successful... onto configuring source /etc/platform/openrc system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 OAM_IF=bond0.1648 PXE_IF=bond0 MGMT_IF=bond0.1680 CLUSTER_IF=bond0.1664 ping 8.8.8.8 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID}; done system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host system host-if-modify controller-0 $PXE_IF -c platform system interface-network-assign controller-0 $PXE_IF pxeboot system host-if-modify controller-0 $CLUSTER_IF -c platform system interface-network-assign controller-0 $CLUSTER_IF cluster-host system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org system host-label-assign controller-0 openstack-control-plane=enabled system host-label-assign controller-0 ceph-mon-placement=enabled system host-label-assign controller-0 ceph-mgr-placement=enabled system storage-backend-add ceph-rook --confirmed system host-unlock controller-0 where controller-0 does reboot and do its boot sequence, then comes up on the correct OAM_IF IP, and DOES have also the correct floating address assigned to bond0.1648 i can actually login, i them waited some minutes and source /etc/platform/openrc [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ functionally working... with the following network config [sysadmin@controller-0 ~(keystone_admin)]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:60:97:52 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:60:97:53 brd ff:ff:ff:ff:ff:ff 4: enp33s0: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff 5: enp49s0: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff permaddr b8:59:9f:12:2c:fc 6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:d5:7a:2c:4c brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 10.16.48.101/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.100/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 8: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 10.16.48.114/24 brd 10.16.48.255 scope global vlan1648 valid_lft forever preferred_lft forever inet 10.16.48.110/24 scope global secondary vlan1648 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 9: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 10: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1680 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1680:12 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1680 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1680 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1680 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever ------------------snip---------------------- i pxe booted 2 more nodes, they did pxe fine from controller bond0 with 10.16.48.x as specified in localhost.yml they did show in system host-list... where i set their personalities. [sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | | 3 | worker-0 | worker | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ they both then proceeded to boot.... but now appear hung...... 1+ hours and i think the problem just might be... 192.168.204 and 192.168.206 addresses on bond0.1680 so.... they need to be 10.16.64.x which is what all our pods talk across for bond0.1664 or 10.16.80.x for bond0.1680 so now they question is which is what in the variables:, i believe i have the proper pxeboot_ and oam with 10.16.48.x on bond0 and bond0.1648 though you show oam and managment_ as different but i dont think i completely grasp whats mgmt_, cluster_host, cluster_pod, cluster_service and management multicast, so whats what ? in your ip space compare to what mine should be. pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address On Thu, May 12, 2022 at 7:44 AM Outback Dingo <outbackdingo@gmail.com> wrote:

working through the configuration now based on findings, and yes i only have to do the ip link commands once prior to bootstrap

i did get bond0 and vlans on a previous try to be configured after system host-unlock controller-0 they were just in the wrong order, so rebuilding the primary node, if it works and put the interfaces and networks on proper interfaces and i can bootstrap controller-1 and get past unlocking that also... i think it will be a win!

On Thu, May 12, 2022 at 7:32 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

...

Were you successful ?

( One question ... you are only having to do the 'ip link ...' commands BEFORE bootstrap in order to have IP Connectivity to the outside world for bootstrapping .. correct ? )

Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 8:21 PM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

I think even allowing for special conditions of bond0 to allow OAM_IF, MGMT_IF, CLUSTER_IF, PXE_IF to all be set on the same bond0, and even dropping the vlans for corner cases then i would only need to set the Install-time-only parameters: Network Properties

pxeboot_subnet: 10.16.48.1 pxeboot_start_address 10.16.48.100 pxeboot_end_address 10.16.48.125 management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up

and modify the host details as per below:

OAM_IF=bond0.1648 MGMT_IF=bond0.1664 CLUSTER_IF=bond0.1680 PXE_IF=bond0 <- this puts pxe on bond0

On Thu, May 12, 2022 at 2:25 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

...

replying to answer your questions from email below, see in-lined below, Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 12:00 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

scenario...

i have a host say controller-0

prior to any ansible run [Greg] I assume you mean the bootstrap ansible playbook

i need to create a bond, and bridges and vlans [Greg] You do need an interface to the outside world … e.g. in order to download container images from docker hub. [Greg] Why can you not simply create a single interface (one link of bond) with a vlan ?

sure.... Add a bond device as root:

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up

Add the bridge device and attach VLAN to it: ip link add br0 type bridge ip link set bond0.1648 master br0 ip link set bond0.1664 master br0 ip link set bond0.1680 master br0 ip link set br0 up

so i see where in starlingx [Greg] the following commands are only possible AFTER bootstrap

system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0

does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge. [Greg] No … there is no requirement for a bridge with StarlingX.

where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664 [Greg] Yes … i.e. OAM on port-based/untagged-vlan of bond and MGMT on vlan-tag=1664 on bond ( BUT here is where you need the pxeboot network because your MGMT network is vlan-tagged … and you can't pxe boot over that )

OAM_IF=bond0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0 MGMT_IF=bond0.1664 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform [Greg] don't think you actually need this command system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

the reason for this being our switches are

# MGMT interface vlan1648 address 10.16.48.2/24 address-virtual 44:38:39:FF:00:02 10.16.48.1 vlan-id 1648 vlan-raw-device bridge

interface vlan1672 address 10.16.72.2/24 address-virtual 44:38:39:FF:00:03 10.16.72.1 vlan-id 1672 vlan-raw-device bridge

interface vlan1680 address 10.16.80.2/24 address-virtual 44:38:39:FF:00:03 10.16.80.1 vlan-id 1680 vlan-raw-device bridge

interface vlan1696 address 10.16.96.2/24 address-virtual 44:38:39:FF:00:03 10.16.96.1 vlan-id 1696 vlan-raw-device bridge

interface vlan1664 address 10.16.64.2/24 address-virtual 44:38:39:FF:00:07 10.16.64.1 vlan-id 1664 vlan-raw-device bridge

and further down DATAIF_0=bond0.1680

the reason being we are trying to have starlingx conform to our networks topology I also noted: in https://docs.starlingx.io/deploy_install_guides/r6_release/ansible _boo tstrap_configs.html#install-time-only-params-r6 ...

Network Properties I listed at the bottom

can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.

Network Properties pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_addres cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 10.16.48.112/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.114/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 8: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1664:12 valid_lft forever preferred_lft forever inet 169.254.202.1/24 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 9: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 10: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 11: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever

_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discu ss

okay heres the curiosity, why do they require to be separate subnets, if their on the seperate interfaces bond0 / bond0.1648 even though its the same really as everything goes over the bond at some point. we really use 10.16.48.x for our management backplane, and it is the only network we allow pxe on. as shown in previous netplans

pxeboot and mgmt are on two separate layer 2 networks pxeboot on the untagged vlan of bond0, and mgmt on vlan=1648 of bond0 If those two networks are connected to the same routing instance (which they are in the case of StarlingX Platform Networking), then they must have unique IP Subnets, based on basic rules for IP Routing. So you say "10.16.48.x" is the only network that you support pxebooting on. Questions: - Does your management network for your StarlingX hosts need to be vlan tagged ? ( if yes, not a problem, just curious why ? ) - If YES * use pxeboot (bond0) = 10.16.48.0/24 mgmt. (bond0.1648) = 10.16.49.0/24 ( I just picked 49 as an arbitrary other network ... - if NO * DON"T use pxeboot * use mgmt. (bond0) = 10.16.48.0/24 Greg. -----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Thursday, May 12, 2022 10:03 PM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking [Please note: This e-mail is from an EXTERNAL e-mail address] On Thu, May 12, 2022 at 6:49 PM Waines, Greg <Greg.Waines@windriver.com> wrote:

I doubt if this is causing issues but shouldn't this: pxeboot_subnet: 10.16.48.1/24 be pxeboot_subnet: 10.16.48.0/24 ?

ok changed to > pxeboot_subnet: 10.16.48.0/24

( and similar issue for oam_subnet )

ACTUALLY ... you have the pxeboot_subnet and the oam_subnet being the same ? external_oam_subnet: 10.16.48.1/24 pxeboot_subnet: 10.16.48.1/24 That is wrong ... they have to be separate IP subnets.

okay heres the curiosity, why do they require to be separate subnets, if their on the seperate interfaces bond0 / bond0.1648 even though its the same really as everything goes over the bond at some point. we really use 10.16.48.x for our management backplane, and it is the only network we allow pxe on. as shown in previous netplans # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: bonds: bond0: interfaces: - enp33s0 - enp101s0 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 parameters: down-delay: 0 lacp-rate: fast mii-monitor-interval: 100 mode: 802.3ad transmit-hash-policy: layer3+4 up-delay: 0 bridges: br-mgmt: addresses: - 10.16.48.10/24 gateway4: 10.16.48.1 interfaces: - bond0 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 nameservers: addresses: - 10.16.48.10 - 10.16.48.11 - 1.1.1.1 search: - maas parameters: forward-delay: 15 stp: false br-storage: addresses: - 10.16.72.21/24 interfaces: - bond0.1672 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 nameservers: addresses: - 10.16.48.10 - 1.1.1.1 search: - maas parameters: forward-delay: 15 stp: false br-vxlan: addresses: - 10.16.80.21/24 interfaces: - bond0.1680 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 nameservers: addresses: - 10.16.48.10 - 1.1.1.1 search: - maas parameters: forward-delay: 15 stp: false ethernets: eno1np0: dhcp4: true match: macaddress: 00:25:90:b9:71:8c mtu: 9000 set-name: enp17s0f0 eno2np1: dhcp4: true match: macaddress: 00:25:90:b9:71:8d mtu: 9000 set-name: enp17s0f1 enp33s0: match: macaddress: 98:03:9b:54:9c:f4 mtu: 9000 set-name: enp33s0 enp101s0: match: macaddress: 98:03:9b:54:9c:e4 mtu: 9000 set-name: enp101s0 vlans: bond0.1672: id: 1672 link: bond0 mtu: 9000 bond0.1680: id: 1680 link: bond0 mtu: 9000 version: 2

You should also remove: external_oam_node_2_address: 10.16.48.116 external_oam_node_3_address: 10.16.48.117 external_oam_node_4_address: 10.16.48.118

done....

Other comment on your config commands ... system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host // you should remove this, as you assign 'cluster-host' network below to a separate interface ... system host-if-modify controller-0 $CLUSTER_IF -c platform system interface-network-assign controller-0 $CLUSTER_IF cluster-host ...

also fixed ...

I would redo install using a unique IP Subnet for pxeboot and oam networks, Greg.

redeploying now, can i ask, whats the logical difference between cluster_host_subnet: 10.16.96.0/24 and cluster_pod_subnet: 10.16.64.0/16 all our pods should exist / be accessible from 10.16.64, do they really require a completely separated interface then hosts?? feels kinda vxlan type magic running new install again...

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Thursday, May 12, 2022 4:30 AM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

ok, so getting closer its about the ip space and preset variables now

and ... after installing software on controller-0 last run i used note i set pxeboot_ vars to put the right network and interfaces on bond0 [sysadmin@controller-0 ~(keystone_admin)]$ cat localhost.yml system_mode: duplex

dns_servers: - 8.8.8.8 - 8.8.4.4

external_oam_subnet: 10.16.48.1/24 external_oam_gateway_address: 10.16.48.1 external_oam_floating_address: 10.16.48.110 external_oam_node_0_address: 10.16.48.114 external_oam_node_1_address: 10.16.48.115 external_oam_node_2_address: 10.16.48.116 external_oam_node_3_address: 10.16.48.117 external_oam_node_4_address: 10.16.48.118

admin_username: admin admin_password: somepass ansible_become_pass: somepass

# Add these lines to configure Docker to use a proxy server # # docker_http_proxy: http://my.proxy.com:1080 # # docker_https_proxy: https://my.proxy.com:1443 # # docker_no_proxy: # # - 1.2.3.4 # kubernetes_version: 1.21.3 pxeboot_subnet: 10.16.48.1/24 pxeboot_start_address: 10.16.48.100 pxeboot_end_address: 10.16.48.151

then ran ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml and ... it was successful... onto configuring source /etc/platform/openrc system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 OAM_IF=bond0.1648 PXE_IF=bond0 MGMT_IF=bond0.1680 CLUSTER_IF=bond0.1664 ping 8.8.8.8

system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID}; done

system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam

system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

system host-if-modify controller-0 $PXE_IF -c platform system interface-network-assign controller-0 $PXE_IF pxeboot

system host-if-modify controller-0 $CLUSTER_IF -c platform system interface-network-assign controller-0 $CLUSTER_IF cluster-host

system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org system host-label-assign controller-0 openstack-control-plane=enabled system host-label-assign controller-0 ceph-mon-placement=enabled system host-label-assign controller-0 ceph-mgr-placement=enabled system storage-backend-add ceph-rook --confirmed system host-unlock controller-0

where controller-0 does reboot and do its boot sequence, then comes up on the correct OAM_IF IP, and DOES have also the correct floating address assigned to bond0.1648

i can actually login, i them waited some minutes and

source /etc/platform/openrc

[sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+

functionally working... with the following network config

[sysadmin@controller-0 ~(keystone_admin)]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:60:97:52 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:60:97:53 brd ff:ff:ff:ff:ff:ff 4: enp33s0: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff 5: enp49s0: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff permaddr b8:59:9f:12:2c:fc 6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:d5:7a:2c:4c brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 10.16.48.101/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.100/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 8: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 10.16.48.114/24 brd 10.16.48.255 scope global vlan1648 valid_lft forever preferred_lft forever inet 10.16.48.110/24 scope global secondary vlan1648 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 9: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 10: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1680 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1680:12 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1680 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1680 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1680 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever ------------------snip----------------------

i pxe booted 2 more nodes, they did pxe fine from controller bond0 with 10.16.48.x as specified in localhost.yml

they did show in system host-list... where i set their personalities.

[sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | | 3 | worker-0 | worker | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+

they both then proceeded to boot.... but now appear hung...... 1+ hours

and i think the problem just might be... 192.168.204 and 192.168.206 addresses on bond0.1680 so.... they need to be 10.16.64.x which is what all our pods talk across for bond0.1664 or 10.16.80.x for bond0.1680

so now they question is which is what in the variables:, i believe i have the proper pxeboot_ and oam with 10.16.48.x on bond0 and bond0.1648 though you show oam and managment_ as different but i dont think i completely grasp whats mgmt_, cluster_host, cluster_pod, cluster_service and management multicast, so whats what ? in your ip space compare to what mine should be.

pxeboot_subnet pxeboot_start_address pxeboot_end_address

management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

On Thu, May 12, 2022 at 7:44 AM Outback Dingo <outbackdingo@gmail.com> wrote:

...

working through the configuration now based on findings, and yes i only have to do the ip link commands once prior to bootstrap

i did get bond0 and vlans on a previous try to be configured after system host-unlock controller-0 they were just in the wrong order, so rebuilding the primary node, if it works and put the interfaces and networks on proper interfaces and i can bootstrap controller-1 and get past unlocking that also... i think it will be a win!

On Thu, May 12, 2022 at 7:32 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

...

Were you successful ?

( One question ... you are only having to do the 'ip link ...' commands BEFORE bootstrap in order to have IP Connectivity to the outside world for bootstrapping .. correct ? )

Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 8:21 PM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

I think even allowing for special conditions of bond0 to allow OAM_IF, MGMT_IF, CLUSTER_IF, PXE_IF to all be set on the same bond0, and even dropping the vlans for corner cases then i would only need to set the Install-time-only parameters: Network Properties

pxeboot_subnet: 10.16.48.1 pxeboot_start_address 10.16.48.100 pxeboot_end_address 10.16.48.125 management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up

and modify the host details as per below:

OAM_IF=bond0.1648 MGMT_IF=bond0.1664 CLUSTER_IF=bond0.1680 PXE_IF=bond0 <- this puts pxe on bond0

On Thu, May 12, 2022 at 2:25 AM Waines, Greg <Greg.Waines@windriver.com> wrote:

...

replying to answer your questions from email below, see in-lined below, Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com> Sent: Wednesday, May 11, 2022 12:00 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

scenario...

i have a host say controller-0

prior to any ansible run [Greg] I assume you mean the bootstrap ansible playbook

i need to create a bond, and bridges and vlans [Greg] You do need an interface to the outside world … e.g. in order to download container images from docker hub. [Greg] Why can you not simply create a single interface (one link of bond) with a vlan ?

sure.... Add a bond device as root:

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up

Add the bridge device and attach VLAN to it: ip link add br0 type bridge ip link set bond0.1648 master br0 ip link set bond0.1664 master br0 ip link set bond0.1680 master br0 ip link set br0 up

so i see where in starlingx [Greg] the following commands are only possible AFTER bootstrap

system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0

does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge. [Greg] No … there is no requirement for a bridge with StarlingX.

where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664 [Greg] Yes … i.e. OAM on port-based/untagged-vlan of bond and MGMT on vlan-tag=1664 on bond ( BUT here is where you need the pxeboot network because your MGMT network is vlan-tagged … and you can't pxe boot over that )

OAM_IF=bond0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0 MGMT_IF=bond0.1664 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform [Greg] don't think you actually need this command system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

the reason for this being our switches are

# MGMT interface vlan1648 address 10.16.48.2/24 address-virtual 44:38:39:FF:00:02 10.16.48.1 vlan-id 1648 vlan-raw-device bridge

interface vlan1672 address 10.16.72.2/24 address-virtual 44:38:39:FF:00:03 10.16.72.1 vlan-id 1672 vlan-raw-device bridge

interface vlan1680 address 10.16.80.2/24 address-virtual 44:38:39:FF:00:03 10.16.80.1 vlan-id 1680 vlan-raw-device bridge

interface vlan1696 address 10.16.96.2/24 address-virtual 44:38:39:FF:00:03 10.16.96.1 vlan-id 1696 vlan-raw-device bridge

interface vlan1664 address 10.16.64.2/24 address-virtual 44:38:39:FF:00:07 10.16.64.1 vlan-id 1664 vlan-raw-device bridge

and further down DATAIF_0=bond0.1680

the reason being we are trying to have starlingx conform to our networks topology I also noted: in https://docs.starlingx.io/deploy_install_guides/r6_release/ansib le _boo tstrap_configs.html#install-time-only-params-r6 ...

Network Properties I listed at the bottom

can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.

Network Properties pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_addres cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 10.16.48.112/24 brd 10.16.48.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.114/24 scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 8: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1664:12 valid_lft forever preferred_lft forever inet 169.254.202.1/24 scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.1/24 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1664 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 9: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 10: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 11: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever

_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis cu ss

Greg So wait.... Are you saying that I can put both pxeboot and mgmt on untagged 10.16.48.x no need to separate them?

Well the options are: * Option 1: use only mgmt network * mgmt. untagged, with address 10.16.48.0/24 * where mgmt. would be used for * pxebooting between StarlingX Controller(s) and other StarlingX hosts, and * internal StarlingX infrastructure mgmt traffic * Option 2: use both pxeboot and mgmt. network * pxeboot untagged, with address 10.16.48.0/24 * to be used for pxebooting between StarlingX Controller(s) and other StarlingX hosts * mgmt. tagged, with address <something different> * to be used for internal StarlingX infrastructure mgmt. traffic. Greg. From: Outback Dingo <outbackdingo@gmail.com> Sent: Friday, May 13, 2022 7:46 AM To: Waines, Greg <Greg.Waines@windriver.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] STX networking [Please note: This e-mail is from an EXTERNAL e-mail address] On Fri, May 13, 2022, 6:22 PM Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> wrote:

okay heres the curiosity, why do they require to be separate subnets, if their on the seperate interfaces bond0 / bond0.1648 even though its the same really as everything goes over the bond at some point. we really use 10.16.48.x for our management backplane, and it is the only network we allow pxe on. as shown in previous netplans

pxeboot and mgmt are on two separate layer 2 networks pxeboot on the untagged vlan of bond0, and mgmt on vlan=1648 of bond0 If those two networks are connected to the same routing instance (which they are in the case of StarlingX Platform Networking), then they must have unique IP Subnets, based on basic rules for IP Routing. So you say "10.16.48.x" is the only network that you support pxebooting on. Questions: - Does your management network for your StarlingX hosts need to be vlan tagged ? ( if yes, not a problem, just curious why ? ) - If YES * use pxeboot (bond0) = 10.16.48.0/24<https://urldefense.com/v3/__http:/10.16.48.0/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirOYHFtBw$> mgmt. (bond0.1648) = 10.16.49.0/24<https://urldefense.com/v3/__http:/10.16.49.0/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OircUfI_cw$> ( I just picked 49 as an arbitrary other network ... - if NO * DON"T use pxeboot * use mgmt. (bond0) = 10.16.48.0/24<https://urldefense.com/v3/__http:/10.16.48.0/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirOYHFtBw$> Greg So wait.... Are you saying that I can put both pxeboot and mgmt on untagged 10.16.48.x no need to separate them? -----Original Message----- From: Outback Dingo <outbackdingo@gmail.com<mailto:outbackdingo@gmail.com>> Sent: Thursday, May 12, 2022 10:03 PM To: Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] STX networking [Please note: This e-mail is from an EXTERNAL e-mail address] On Thu, May 12, 2022 at 6:49 PM Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> wrote:

I doubt if this is causing issues but shouldn't this: pxeboot_subnet: 10.16.48.1/24<https://urldefense.com/v3/__http:/10.16.48.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipZWWT9KA$> be pxeboot_subnet: 10.16.48.0/24<https://urldefense.com/v3/__http:/10.16.48.0/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirOYHFtBw$> ?

ok changed to > pxeboot_subnet: 10.16.48.0/24<https://urldefense.com/v3/__http:/10.16.48.0/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirOYHFtBw$>

( and similar issue for oam_subnet )

ACTUALLY ... you have the pxeboot_subnet and the oam_subnet being the same ? external_oam_subnet: 10.16.48.1/24<https://urldefense.com/v3/__http:/10.16.48.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipZWWT9KA$> pxeboot_subnet: 10.16.48.1/24<https://urldefense.com/v3/__http:/10.16.48.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipZWWT9KA$> That is wrong ... they have to be separate IP subnets.

okay heres the curiosity, why do they require to be separate subnets, if their on the seperate interfaces bond0 / bond0.1648 even though its the same really as everything goes over the bond at some point. we really use 10.16.48.x for our management backplane, and it is the only network we allow pxe on. as shown in previous netplans # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: bonds: bond0: interfaces: - enp33s0 - enp101s0 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 parameters: down-delay: 0 lacp-rate: fast mii-monitor-interval: 100 mode: 802.3ad transmit-hash-policy: layer3+4 up-delay: 0 bridges: br-mgmt: addresses: - 10.16.48.10/24<https://urldefense.com/v3/__http:/10.16.48.10/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirSW8Nvjw$> gateway4: 10.16.48.1<https://urldefense.com/v3/__http:/10.16.48.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipkWgJpsA$> interfaces: - bond0 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 nameservers: addresses: - 10.16.48.10<https://urldefense.com/v3/__http:/10.16.48.10__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OioXn5DFPQ$> - 10.16.48.11<https://urldefense.com/v3/__http:/10.16.48.11__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipVvf_XWQ$> - 1.1.1.1<https://urldefense.com/v3/__http:/1.1.1.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiryJe30tg$> search: - maas parameters: forward-delay: 15 stp: false br-storage: addresses: - 10.16.72.21/24<https://urldefense.com/v3/__http:/10.16.72.21/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirvgpgXCQ$> interfaces: - bond0.1672 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 nameservers: addresses: - 10.16.48.10<https://urldefense.com/v3/__http:/10.16.48.10__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OioXn5DFPQ$> - 1.1.1.1<https://urldefense.com/v3/__http:/1.1.1.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiryJe30tg$> search: - maas parameters: forward-delay: 15 stp: false br-vxlan: addresses: - 10.16.80.21/24<https://urldefense.com/v3/__http:/10.16.80.21/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqUiTYCgQ$> interfaces: - bond0.1680 macaddress: 98:03:9b:54:9c:f4 mtu: 9000 nameservers: addresses: - 10.16.48.10<https://urldefense.com/v3/__http:/10.16.48.10__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OioXn5DFPQ$> - 1.1.1.1<https://urldefense.com/v3/__http:/1.1.1.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiryJe30tg$> search: - maas parameters: forward-delay: 15 stp: false ethernets: eno1np0: dhcp4: true match: macaddress: 00:25:90:b9:71:8c mtu: 9000 set-name: enp17s0f0 eno2np1: dhcp4: true match: macaddress: 00:25:90:b9:71:8d mtu: 9000 set-name: enp17s0f1 enp33s0: match: macaddress: 98:03:9b:54:9c:f4 mtu: 9000 set-name: enp33s0 enp101s0: match: macaddress: 98:03:9b:54:9c:e4 mtu: 9000 set-name: enp101s0 vlans: bond0.1672: id: 1672 link: bond0 mtu: 9000 bond0.1680: id: 1680 link: bond0 mtu: 9000 version: 2

You should also remove: external_oam_node_2_address: 10.16.48.116<https://urldefense.com/v3/__http:/10.16.48.116__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipCxW-vUQ$> external_oam_node_3_address: 10.16.48.117<https://urldefense.com/v3/__http:/10.16.48.117__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipbrBm2nw$> external_oam_node_4_address: 10.16.48.118<https://urldefense.com/v3/__http:/10.16.48.118__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirsA72lvg$>

done....

Other comment on your config commands ... system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host // you should remove this, as you assign 'cluster-host' network below to a separate interface ... system host-if-modify controller-0 $CLUSTER_IF -c platform system interface-network-assign controller-0 $CLUSTER_IF cluster-host ...

also fixed ...

I would redo install using a unique IP Subnet for pxeboot and oam networks, Greg.

redeploying now, can i ask, whats the logical difference between cluster_host_subnet: 10.16.96.0/24<https://urldefense.com/v3/__http:/10.16.96.0/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipE-ZlXxQ$> and cluster_pod_subnet: 10.16.64.0/16<https://urldefense.com/v3/__http:/10.16.64.0/16__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiovior4xg$> all our pods should exist / be accessible from 10.16.64, do they really require a completely separated interface then hosts?? feels kinda vxlan type magic running new install again...

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com<mailto:outbackdingo@gmail.com>> Sent: Thursday, May 12, 2022 4:30 AM To: Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

ok, so getting closer its about the ip space and preset variables now

and ... after installing software on controller-0 last run i used note i set pxeboot_ vars to put the right network and interfaces on bond0 [sysadmin@controller-0 ~(keystone_admin)]$ cat localhost.yml system_mode: duplex

dns_servers: - 8.8.8.8<https://urldefense.com/v3/__http:/8.8.8.8__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipT1FtENA$> - 8.8.4.4<https://urldefense.com/v3/__http:/8.8.4.4__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirhnjwX3Q$>

external_oam_subnet: 10.16.48.1/24<https://urldefense.com/v3/__http:/10.16.48.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipZWWT9KA$> external_oam_gateway_address: 10.16.48.1<https://urldefense.com/v3/__http:/10.16.48.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipkWgJpsA$> external_oam_floating_address: 10.16.48.110<https://urldefense.com/v3/__http:/10.16.48.110__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqFEzom4Q$> external_oam_node_0_address: 10.16.48.114<https://urldefense.com/v3/__http:/10.16.48.114__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqWIvQdOQ$> external_oam_node_1_address: 10.16.48.115<https://urldefense.com/v3/__http:/10.16.48.115__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OioCvKvKDw$> external_oam_node_2_address: 10.16.48.116<https://urldefense.com/v3/__http:/10.16.48.116__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipCxW-vUQ$> external_oam_node_3_address: 10.16.48.117<https://urldefense.com/v3/__http:/10.16.48.117__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipbrBm2nw$> external_oam_node_4_address: 10.16.48.118<https://urldefense.com/v3/__http:/10.16.48.118__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirsA72lvg$>

admin_username: admin admin_password: somepass ansible_become_pass: somepass

# Add these lines to configure Docker to use a proxy server # # docker_http_proxy: http://my.proxy.com:1080<https://urldefense.com/v3/__http:/my.proxy.com:1080__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiq_WMWq_g$> # # docker_https_proxy: https://my.proxy.com:1443<https://urldefense.com/v3/__https:/my.proxy.com:1443__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiryFDWfSQ$> # # docker_no_proxy: # # - 1.2.3.4<https://urldefense.com/v3/__http:/1.2.3.4__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipN2KNc2Q$> # kubernetes_version: 1.21.3 pxeboot_subnet: 10.16.48.1/24<https://urldefense.com/v3/__http:/10.16.48.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipZWWT9KA$> pxeboot_start_address: 10.16.48.100<https://urldefense.com/v3/__http:/10.16.48.100__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipnVymXSg$> pxeboot_end_address: 10.16.48.151<https://urldefense.com/v3/__http:/10.16.48.151__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oir2ih0rbA$>

then ran ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml and ... it was successful... onto configuring source /etc/platform/openrc system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 OAM_IF=bond0.1648 PXE_IF=bond0 MGMT_IF=bond0.1680 CLUSTER_IF=bond0.1664 ping 8.8.8.8<https://urldefense.com/v3/__http:/8.8.8.8__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipT1FtENA$>

system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID}; done

system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam

system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

system host-if-modify controller-0 $PXE_IF -c platform system interface-network-assign controller-0 $PXE_IF pxeboot

system host-if-modify controller-0 $CLUSTER_IF -c platform system interface-network-assign controller-0 $CLUSTER_IF cluster-host

system ntp-modify ntpservers=0.pool.ntp.org<https://urldefense.com/v3/__http:/0.pool.ntp.org__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirhFcTwMA$>,1.pool.ntp.org<https://urldefense.com/v3/__http:/1.pool.ntp.org__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqrPag3zw$> system host-label-assign controller-0 openstack-control-plane=enabled system host-label-assign controller-0 ceph-mon-placement=enabled system host-label-assign controller-0 ceph-mgr-placement=enabled system storage-backend-add ceph-rook --confirmed system host-unlock controller-0

where controller-0 does reboot and do its boot sequence, then comes up on the correct OAM_IF IP, and DOES have also the correct floating address assigned to bond0.1648

i can actually login, i them waited some minutes and

source /etc/platform/openrc

[sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+

functionally working... with the following network config

[sysadmin@controller-0 ~(keystone_admin)]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8<https://urldefense.com/v3/__http:/127.0.0.1/8__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqmwIApcA$> scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:60:97:52 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:60:97:53 brd ff:ff:ff:ff:ff:ff 4: enp33s0: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff 5: enp49s0: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff permaddr b8:59:9f:12:2c:fc 6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:d5:7a:2c:4c brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16<https://urldefense.com/v3/__http:/172.17.0.1/16__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipJoxDVjQ$> brd 172.17.255.255<https://urldefense.com/v3/__http:/172.17.255.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OioUVldRZw$> scope global docker0 valid_lft forever preferred_lft forever 7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 10.16.48.101/24<https://urldefense.com/v3/__http:/10.16.48.101/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oipgb3yjPw$> brd 10.16.48.255<https://urldefense.com/v3/__http:/10.16.48.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqUcSWXuQ$> scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.100/24<https://urldefense.com/v3/__http:/10.16.48.100/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oio8yY9TIg$> scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 8: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 10.16.48.114/24<https://urldefense.com/v3/__http:/10.16.48.114/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqOIx4VBA$> brd 10.16.48.255<https://urldefense.com/v3/__http:/10.16.48.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqUcSWXuQ$> scope global vlan1648 valid_lft forever preferred_lft forever inet 10.16.48.110/24<https://urldefense.com/v3/__http:/10.16.48.110/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqqLHb4SQ$> scope global secondary vlan1648 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 9: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever 10: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:32:78 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24<https://urldefense.com/v3/__http:/192.168.204.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiryME1-Fw$> brd 192.168.204.255<https://urldefense.com/v3/__http:/192.168.204.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiqkd5p_gw$> scope global vlan1680 valid_lft forever preferred_lft forever inet 192.168.206.2/24<https://urldefense.com/v3/__http:/192.168.206.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiol2DzQ6w$> brd 192.168.206.255<https://urldefense.com/v3/__http:/192.168.206.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oip5-vJN2A$> scope global vlan1680:12 valid_lft forever preferred_lft forever inet 192.168.206.1/24<https://urldefense.com/v3/__http:/192.168.206.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqrgX7SJQ$> scope global secondary vlan1680 valid_lft forever preferred_lft forever inet 192.168.204.1/24<https://urldefense.com/v3/__http:/192.168.204.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipGA54MSA$> scope global secondary vlan1680 valid_lft forever preferred_lft forever inet 192.168.204.4/24<https://urldefense.com/v3/__http:/192.168.204.4/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oir1x4jH-A$> brd 192.168.204.255<https://urldefense.com/v3/__http:/192.168.204.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiqkd5p_gw$> scope global secondary vlan1680 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:3278/64 scope link valid_lft forever preferred_lft forever ------------------snip----------------------

i pxe booted 2 more nodes, they did pxe fine from controller bond0 with 10.16.48.x as specified in localhost.yml

they did show in system host-list... where i set their personalities.

[sysadmin@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | | 3 | worker-0 | worker | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+

they both then proceeded to boot.... but now appear hung...... 1+ hours

and i think the problem just might be... 192.168.204 and 192.168.206 addresses on bond0.1680 so.... they need to be 10.16.64.x which is what all our pods talk across for bond0.1664 or 10.16.80.x for bond0.1680

so now they question is which is what in the variables:, i believe i have the proper pxeboot_ and oam with 10.16.48.x on bond0 and bond0.1648 though you show oam and managment_ as different but i dont think i completely grasp whats mgmt_, cluster_host, cluster_pod, cluster_service and management multicast, so whats what ? in your ip space compare to what mine should be.

pxeboot_subnet pxeboot_start_address pxeboot_end_address

management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

On Thu, May 12, 2022 at 7:44 AM Outback Dingo <outbackdingo@gmail.com<mailto:outbackdingo@gmail.com>> wrote:

...

working through the configuration now based on findings, and yes i only have to do the ip link commands once prior to bootstrap

i did get bond0 and vlans on a previous try to be configured after system host-unlock controller-0 they were just in the wrong order, so rebuilding the primary node, if it works and put the interfaces and networks on proper interfaces and i can bootstrap controller-1 and get past unlocking that also... i think it will be a win!

On Thu, May 12, 2022 at 7:32 AM Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> wrote:

...

Were you successful ?

( One question ... you are only having to do the 'ip link ...' commands BEFORE bootstrap in order to have IP Connectivity to the outside world for bootstrapping .. correct ? )

Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com<mailto:outbackdingo@gmail.com>> Sent: Wednesday, May 11, 2022 8:21 PM To: Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

I think even allowing for special conditions of bond0 to allow OAM_IF, MGMT_IF, CLUSTER_IF, PXE_IF to all be set on the same bond0, and even dropping the vlans for corner cases then i would only need to set the Install-time-only parameters: Network Properties

pxeboot_subnet: 10.16.48.1<https://urldefense.com/v3/__http:/10.16.48.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipkWgJpsA$> pxeboot_start_address 10.16.48.100<https://urldefense.com/v3/__http:/10.16.48.100__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipnVymXSg$> pxeboot_end_address 10.16.48.125<https://urldefense.com/v3/__http:/10.16.48.125__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipLwudf0Q$> management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_address cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad<https://urldefense.com/v3/__http:/80211.ad__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiouX7DOwA$> ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up

and modify the host details as per below:

OAM_IF=bond0.1648 MGMT_IF=bond0.1664 CLUSTER_IF=bond0.1680 PXE_IF=bond0 <- this puts pxe on bond0

On Thu, May 12, 2022 at 2:25 AM Waines, Greg <Greg.Waines@windriver.com<mailto:Greg.Waines@windriver.com>> wrote:

...

replying to answer your questions from email below, see in-lined below, Greg.

-----Original Message----- From: Outback Dingo <outbackdingo@gmail.com<mailto:outbackdingo@gmail.com>> Sent: Wednesday, May 11, 2022 12:00 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] STX networking

[Please note: This e-mail is from an EXTERNAL e-mail address]

scenario...

i have a host say controller-0

prior to any ansible run [Greg] I assume you mean the bootstrap ansible playbook

i need to create a bond, and bridges and vlans [Greg] You do need an interface to the outside world … e.g. in order to download container images from docker hub. [Greg] Why can you not simply create a single interface (one link of bond) with a vlan ?

sure.... Add a bond device as root:

ip link add bond0 type bond ip link set bond0 type bond miimon 100 mode 80211.ad<https://urldefense.com/v3/__http:/80211.ad__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiouX7DOwA$> ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up

Set VLAN on the bond device:

ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up

Add the bridge device and attach VLAN to it: ip link add br0 type bridge ip link set bond0.1648 master br0 ip link set bond0.1664 master br0 ip link set bond0.1680 master br0 ip link set br0 up

so i see where in starlingx [Greg] the following commands are only possible AFTER bootstrap

system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0

does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge. [Greg] No … there is no requirement for a bridge with StarlingX.

where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664 [Greg] Yes … i.e. OAM on port-based/untagged-vlan of bond and MGMT on vlan-tag=1664 on bond ( BUT here is where you need the pxeboot network because your MGMT network is vlan-tagged … and you can't pxe boot over that )

OAM_IF=bond0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0 MGMT_IF=bond0.1664 system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') for UUID in $IFNET_UUIDS; do system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform [Greg] don't think you actually need this command system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host

the reason for this being our switches are

# MGMT interface vlan1648 address 10.16.48.2/24<https://urldefense.com/v3/__http:/10.16.48.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqEpeb3Ww$> address-virtual 44:38:39:FF:00:02 10.16.48.1<https://urldefense.com/v3/__http:/10.16.48.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipkWgJpsA$> vlan-id 1648 vlan-raw-device bridge

interface vlan1672 address 10.16.72.2/24<https://urldefense.com/v3/__http:/10.16.72.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oir9m7LeeA$> address-virtual 44:38:39:FF:00:03 10.16.72.1<https://urldefense.com/v3/__http:/10.16.72.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oios8svY-w$> vlan-id 1672 vlan-raw-device bridge

interface vlan1680 address 10.16.80.2/24<https://urldefense.com/v3/__http:/10.16.80.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipTXjlJfA$> address-virtual 44:38:39:FF:00:03 10.16.80.1<https://urldefense.com/v3/__http:/10.16.80.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiot2oAdHQ$> vlan-id 1680 vlan-raw-device bridge

interface vlan1696 address 10.16.96.2/24<https://urldefense.com/v3/__http:/10.16.96.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipkpapXRw$> address-virtual 44:38:39:FF:00:03 10.16.96.1<https://urldefense.com/v3/__http:/10.16.96.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oir1yDGeZw$> vlan-id 1696 vlan-raw-device bridge

interface vlan1664 address 10.16.64.2/24<https://urldefense.com/v3/__http:/10.16.64.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqMrfHEaQ$> address-virtual 44:38:39:FF:00:07 10.16.64.1<https://urldefense.com/v3/__http:/10.16.64.1__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OioS6n4GFQ$> vlan-id 1664 vlan-raw-device bridge

and further down DATAIF_0=bond0.1680

the reason being we are trying to have starlingx conform to our networks topology I also noted: in https://docs.starlingx.io/deploy_install_guides/r6_release/ansib<https://urldefense.com/v3/__https:/docs.starlingx.io/deploy_install_guides/r6_release/ansib__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqOEfNT8A$> le _boo tstrap_configs.html#install-time-only-params-r6 ...

Network Properties I listed at the bottom

can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.

Network Properties pxeboot_subnet pxeboot_start_address pxeboot_end_address management_subnet management_start_address management_end_address cluster_host_subnet cluster_host_start_address cluster_host_end_address cluster_pod_subnet cluster_pod_start_addres cluster_pod_end_address cluster_service_subnet cluster_service_start_address cluster_service_end_address management_multicast_subnet management_multicast_start_address management_multicast_end_address

7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 10.16.48.112/24<https://urldefense.com/v3/__http:/10.16.48.112/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oioo1TARzw$> brd 10.16.48.255<https://urldefense.com/v3/__http:/10.16.48.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqUcSWXuQ$> scope global bond0 valid_lft forever preferred_lft forever inet 10.16.48.114/24<https://urldefense.com/v3/__http:/10.16.48.114/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqOIx4VBA$> scope global secondary bond0 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 8: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.204.2/24<https://urldefense.com/v3/__http:/192.168.204.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiryME1-Fw$> brd 192.168.204.255<https://urldefense.com/v3/__http:/192.168.204.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiqkd5p_gw$> scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.2/24<https://urldefense.com/v3/__http:/192.168.206.2/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiol2DzQ6w$> brd 192.168.206.255<https://urldefense.com/v3/__http:/192.168.206.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oip5-vJN2A$> scope global vlan1664:12 valid_lft forever preferred_lft forever inet 169.254.202.1/24<https://urldefense.com/v3/__http:/169.254.202.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqJeefziA$> scope global vlan1664 valid_lft forever preferred_lft forever inet 192.168.206.1/24<https://urldefense.com/v3/__http:/192.168.206.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OiqrgX7SJQ$> scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.1/24<https://urldefense.com/v3/__http:/192.168.204.1/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OipGA54MSA$> scope global secondary vlan1664 valid_lft forever preferred_lft forever inet 192.168.204.4/24<https://urldefense.com/v3/__http:/192.168.204.4/24__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oir1x4jH-A$> brd 192.168.204.255<https://urldefense.com/v3/__http:/192.168.204.255__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__Oiqkd5p_gw$> scope global secondary vlan1664 valid_lft forever preferred_lft forever inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 9: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 10: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever 11: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ba59:9fff:fe12:34f0/64 scope link valid_lft forever preferred_lft forever

_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis<https://urldefense.com/v3/__http:/lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-dis__;!!AjveYdw8EvQ!bwB4zytXZUx1agXWwc5M7BC35UM0tDi8IFJ2IiW5JX4P7FenF-8KO9IIjGvpkt6ymw5hfPna5GQOfO__OirQ3XdXHw$> cu ss