[Starlingx-discuss] STX networking

Outback Dingo outbackdingo at gmail.com
Wed May 11 23:05:20 UTC 2022


https://bugs.launchpad.net/starlingx/+bug/1913582

i did also find this which should let me manage where pxe booting
occurs, as in our
network config you can only pxe from the 10.16.48 network, so some
config changes are needed
which ill be also trying today

On Thu, May 12, 2022 at 2:11 AM Waines, Greg <Greg.Waines at windriver.com> wrote:
>
> Hey Scott,
>
> Wrt what we were talking about in the community meeting:
>
> - ROOK Deployment for say an AIO-DX + Workers system
>        > ROOK deployment on Controllers
>               * see https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/aio_duplex_install_kubernetes.html
>                        - notice that there are multiple steps to setup rook in this guide
>                               > adding of high-level ceph-rook backend for system, while configuring controller-0
>                               > adding of ceph-mon-placement and ceph-mgr-placement labels on each controller  (controller-0 and controller-1)
>                               > adding of OSDs to each controller (controller-0 and controller-1)
>         > ROOK OSD deployment on Workers
>               * DOH ... we are missing that      ( I'll get the starlingx DOC TEAM to look into this )
>               * in https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/aio_duplex_extend.html
>                        - we forgot to add ... at the very end ... AFTER the unlocking of the worker nodes,
>                           the section to optionally add OSDs to the worker nodes
>                        - you can just use the section at the bottom of
>                           https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/aio_duplex_install_kubernetes.html
>                           (i.e. If using Rook container-based Ceph, finish configuring the ceph-rook Persistent Storage Backend)
>                           but substitute in the worker hostnames.
>
>
> - PXE BOOT details
>
>       ( NOTE ... I can't remember how we decided you needed a pxeboot network.  What was the original problem that you were trying to solve ?
>                         e.g. if it was just to change the IP Address Subnet for mgmt., you can do that simply with a bootstrap override, see below )
>
>        > As discussed, only the first time the very first controller is installed (controller-0) can an external PXEBOOT server be used.
>               * The docs on how to do this are here:
>                        - https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/configuring-a-pxe-boot-server.html
>                        - https://docs.starlingx.io/deploy_install_guides/r6_release/bare_metal/accessing-pxe-boot-server-files-for-a-custom-configuration.html
>
>       > Also as discussed,
>                * For all subsequent pxeboot installs of starlingx hosts, it is done by the active controller (controller-0 or controller-1)
>                * By default, StarlingX uses the MGMT network for PXEBOOTING
>                * But for scenarios where the MGMT network can not be used
>                   ( i.e. MGMT network needs to be vlan-tagged for some reason or MGMT network needs to be IPv6 )
>                   then one can configure a PXEBOOT network who's sole purpose is for pxebooting.
>                * This is discussed in many of the 'Network Planning' PLANNING sections of StarlingX DOCS
>                   e.g.
>                                https://docs.starlingx.io/planning/kubernetes/network-requirements.html
>                                ... and every page in-between ...
>                                https://docs.starlingx.io/planning/kubernetes/network-planning-the-pxe-boot-network.html
>                * DOH ... we don't have this well documented wrt how to configure this
>                          - again, I'll get the DOC team to look at this
>                          - for now here's the info on how to use it.
>
> Using 'pxeboot' network
> ---------------------------------
> - the 'pxeboot' network (and all networks I think) are always present in the system
>   e.g. you can see them with
>                system network-list
>                system addrpool-list
> - the easiest way to configure the pxeboot IP Address Subnet is at bootstrap time in the localhost.yml (bootstrap override file)
>   e.g.
>           system_mode: duplex
>
>           dns_servers:
>             - 8.8.8.8
>             - 8.8.4.4
>
>           pxeboot_subnet: 169.254.202.0/24
>           # pxeboot_start_address:
>           # pxeboot_end_address:
>
>           management_subnet: 192.168.204.0/24
>           # management_start_address:
>           # management_end_address:
>
>           external_oam_subnet: <OAM-IP-SUBNET>/<OAM-IP-SUBNET-LENGTH>
>           external_oam_gateway_address: <OAM-GATEWAY-IP-ADDRESS>
>           external_oam_floating_address: <OAM-FLOATING-IP-ADDRESS>
>           external_oam_node_0_address: <OAM-CONTROLLER-0-IP-ADDRESS>
>           external_oam_node_1_address: <OAM-CONTROLLER-1-IP-ADDRESS>
>
>           admin_username: admin
>           admin_password: <admin-password>
>           ansible_become_pass: <sysadmin-password>
>
>           # OPTIONALLY provide a ROOT CA certificate and key for k8s root ca,
>           # if not specified, one will be auto-generated,
>           # see 'Kubernetes Root CA Certificate' in Security Guide for details.
>           k8s_root_ca_cert: < your_root_ca_cert.pem >
>           k8s_root_ca_key: < your_root_ca_key.pem >
>           apiserver_cert_sans:
>             - < your_hostname_for_oam_floating.your_domain >
>
>       ( see all parameters and their defaults in /usr/share/ansible/stx-ansible/playbooks/host_vars/bootstrap/default.yml on target )
>
>
> - you then need to configure a host interface on controller-0 and attach it to the 'pxeboot' network
>   IMPORTANT:  a host's pxeboot interface MUST be a port-based/untagged vlan AND MUST be on the same physical interface as the mgmt interface (which would be vlan-tagged).
>   e.g.  // the config of the mgmt. interface and pxeboot interface for controller-0
>                MGMT_PORT=<MGMT-PORT>
>                system host-if-modify controller-0 lo -c none
>                IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
>                for UUID in $IFNET_UUIDS; do
>                    system interface-network-remove ${UUID}
>                done
>                system host-if-modify controller-0 $MGMT_PORT -c platform
>                system interface-network-assign controller-0 $MGMT_PORT pxeboot
>
>                system host-if-add controller-0 -V 22 -c platform mgmt0 vlan $MGMT_PORT
>                system interface-network-assign controller-0 mgmt0 mgmt
>                system interface-network-assign controller-0 mgmt0 cluster-host
>
> - dnsmasq (boot server on controllers) is configured to answer dhcp on the pxeboot and mgmt interface IPs
>
> - when adding new hosts (e.g. second controller or workers),
>   you need to ensure in BIOS that these nodes are trying to network boot off of the interface physically attached to the port-based/vlan-untagged pxeboot network.
>         > and then after new host boots,
>            you need to configure its host interfaces correctly ... e.g. with pxeboot interface on port-based/vlan-untagged physical interface, and the mgmt. interface as a vlan interface on that same physical interface.
>
>
>
> Greg.
>
>
> -----Original Message-----
> From: Outback Dingo <outbackdingo at gmail.com>
> Sent: Wednesday, May 11, 2022 12:00 AM
> To: starlingx-discuss at lists.starlingx.io
> Subject: [Starlingx-discuss] STX networking
>
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> scenario...
>
> i have a host say controller-0
>
> prior to any ansible run
>
> i need to create a bond, and bridges and vlans
>
> sure....
> Add a bond device as root:
>
> ip link add bond0 type bond
> ip link set bond0 type bond miimon 100 mode 80211.ad ip link set enp33s0 down ip link set snp33s0 master bond0 ip link set enp44s0 down ip link set enp44s0 master bond0 ip link set bond0 up
>
> Set VLAN on the bond device:
>
> ip link add link bond0 name bond0.1648 type vlan id 1648 ip link set bond0.1648 up ip link add link bond0 name bond0.1664 type vlan id 1664 ip link set bond0.1664 up ip link add link bond0 name bond0.1680 type vlan id 1680 ip link set bond0.1680 up
>
> Add the bridge device and attach VLAN to it:
> ip link add br0 type bridge
> ip link set bond0.1648 master br0
> ip link set bond0.1664 master br0
> ip link set bond0.1680 master br0
> ip link set br0 up
>
> so i see where in starlingx
>
> system host-if-add -c platform -a 802.3ad -x layer2 controller-0 bond0 ae enp33s0 enp49s0 system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0
>
> does allow for me to create the bond0 and the vlans? but I dont see any documentation for bridges anywhere ? Do I even need the bridge.
>
> where i want to set example, since each needs its own interface can i set OAM_IF=bond0, and say MGMT_IF=bond0.1664
>
> OAM_IF=bond0
> system host-if-modify controller-0 $OAM_IF -c platform system interface-network-assign controller-0 $OAM_IF oam system host-if-add controller-0 -V 1648 -c platform bond0.1648 vlan bond0 system host-if-add controller-0 -V 1664 -c platform bond0.1664 vlan bond0 system host-if-add controller-0 -V 1680 -c platform bond0.1680 vlan bond0 system host-if-add controller-0 -V 1672 -c platform bond0.1672 vlan bond0
> MGMT_IF=bond0.1664
> system host-if-modify controller-0 lo -c none IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if
> ($6=="lo") print $4;}')
> for UUID in $IFNET_UUIDS; do
>     system interface-network-remove ${UUID} done system host-if-modify controller-0 $MGMT_IF -c platform system interface-network-assign controller-0 $MGMT_IF mgmt system interface-network-assign controller-0 $MGMT_IF cluster-host
>
> the reason for this being our switches are
>
> # MGMT
> interface vlan1648
>   address 10.16.48.2/24
>   address-virtual 44:38:39:FF:00:02 10.16.48.1
>   vlan-id 1648
>   vlan-raw-device bridge
>
> interface vlan1672
>   address 10.16.72.2/24
>   address-virtual 44:38:39:FF:00:03 10.16.72.1
>   vlan-id 1672
>   vlan-raw-device bridge
>
> interface vlan1680
>   address 10.16.80.2/24
>   address-virtual 44:38:39:FF:00:03 10.16.80.1
>   vlan-id 1680
>   vlan-raw-device bridge
>
> interface vlan1696
>   address 10.16.96.2/24
>   address-virtual 44:38:39:FF:00:03 10.16.96.1
>   vlan-id 1696
>   vlan-raw-device bridge
>
> interface vlan1664
>   address 10.16.64.2/24
>   address-virtual 44:38:39:FF:00:07 10.16.64.1
>   vlan-id 1664
>   vlan-raw-device bridge
>
> and further down DATAIF_0=bond0.1680
>
> the reason being we are trying to have starlingx conform to our networks topology I also noted: in
> https://docs.starlingx.io/deploy_install_guides/r6_release/ansible_bootstrap_configs.html#install-time-only-params-r6
> ...
>
> Network Properties I listed at the bottom
>
> can i modify these addresses to conform to our networks, as our switches wount pass the traffic you set as defaults, as seen in my first attempt at the bottom. Though i dont believe still dhcp/pxe will work on a vlan interface.
>
> Network Properties
> pxeboot_subnet
> pxeboot_start_address
> pxeboot_end_address
> management_subnet
> management_start_address
> management_end_address
> cluster_host_subnet
> cluster_host_start_address
> cluster_host_end_address
> cluster_pod_subnet
> cluster_pod_start_addres
> cluster_pod_end_address
> cluster_service_subnet
> cluster_service_start_address
> cluster_service_end_address
> management_multicast_subnet
> management_multicast_start_address
> management_multicast_end_address
>
> 7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
>     link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff
>     inet 10.16.48.112/24 brd 10.16.48.255 scope global bond0
>        valid_lft forever preferred_lft forever
>     inet 10.16.48.114/24 scope global secondary bond0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::ba59:9fff:fe12:34f0/64 scope link
>        valid_lft forever preferred_lft forever
> 8: vlan1664 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
>     link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.204.2/24 brd 192.168.204.255 scope global vlan1664
>        valid_lft forever preferred_lft forever
>     inet 192.168.206.2/24 brd 192.168.206.255 scope global vlan1664:12
>        valid_lft forever preferred_lft forever
>     inet 169.254.202.1/24 scope global vlan1664
>        valid_lft forever preferred_lft forever
>     inet 192.168.206.1/24 scope global secondary vlan1664
>        valid_lft forever preferred_lft forever
>     inet 192.168.204.1/24 scope global secondary vlan1664
>        valid_lft forever preferred_lft forever
>     inet 192.168.204.4/24 brd 192.168.204.255 scope global secondary vlan1664
>        valid_lft forever preferred_lft forever
>     inet6 fe80::ba59:9fff:fe12:34f0/64 scope link
>        valid_lft forever preferred_lft forever
> 9: vlan1672 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>     link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::ba59:9fff:fe12:34f0/64 scope link
>        valid_lft forever preferred_lft forever
> 10: vlan1648 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>     link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::ba59:9fff:fe12:34f0/64 scope link
>        valid_lft forever preferred_lft forever
> 11: vlan1680 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>     link/ether b8:59:9f:12:34:f0 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::ba59:9fff:fe12:34f0/64 scope link
>        valid_lft forever preferred_lft forever
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss



More information about the Starlingx-discuss mailing list