Thanks Pratik. There is no openstack-neutron component. We have only one k8s cluster in the edge cloud in one subnet. Some of the docs that I read suggest assigning mac addresses to vf's using "ip link" command. I'm not sure if that's the only way forward. Regards, Sriram On Thu, Oct 1, 2020 at 7:22 AM Pratik M. <pvmpublic@gmail.com> wrote:
Hi, a.: I believe you should be able to lock, apply these steps, and unlock.
b.: I don't know if StarlingX has an option to specify per node pools. Will defer to experts. But if the nodes are in one L2, they would typically be in one cluster and thus the VF MAC assignment would be arbitrated by cluster wide neutron/CNI, right? If each node is one cluster, maybe the user would need to annotate the pods with static MACs.
BR
On Tue, 29 Sep, 2020, 23:23 Sriram, <sriram.ec@gmail.com> wrote:
Hi,
How do we ensure the uniqueness of VF mac addresses across all the nodes in the k8s cluster formed on edge nodes. Please let me know if this problem is addressed by starlingX or if it is taken care of by some other means.
Regards, Sriram
On Fri, Sep 25, 2020 at 1:20 PM Sriram <sriram.ec@gmail.com> wrote:
Hi Pratik,
Thanks for your reply.
a. Can these steps be done after the installation is complete, now that I have already installed. b. How do we ensure the uniqueness of VF mac addresses across all the nodes in the k8s cluster formed on edge nodes.
Regards, Sriram
On Fri, Sep 25, 2020 at 1:02 PM Pratik M. <pvmpublic@gmail.com> wrote:
Hi, You would need to do: # system host-label-assign controller-0 sriovdp=enabled # system host-if-modify controller-0 <interface> -c pci-sriov -n sriov0 -N <num vfs> # system interface-datanetwork-assign controller-0 <interface> <datanetwork> # system host-unlock
And that should populate the /etc/pcidp/config.json
Ref: https://wiki.openstack.org/wiki/StarlingX/Networking Steven Webster's helpful comments in https://bugs.launchpad.net/starlingx/+bug/1891889
Thanks
On Thu, Sep 24, 2020 at 3:04 PM Sriram <sriram.ec@gmail.com> wrote:
Hi,
I have installed distributed starlingx 4.0 in "All in one Duplex" mode. There are two nodes in the central cloud and two in the edge cloud.
I have enabled SRIOV in bios settings of edge cloud nodes and set total VFs as 16.
After that, while installing starlingX I followed the steps to enable SRIOV.
system host-label-assign controller-0 sriovdp=enabled
system host-memory-modify controller-0 0 -1G 100 system host-memory-modify controller-0 1 -1G 100
and ran these steps for controller-1 as well.
As I understand the first step would label the node "controller-0 and controller-1" as "sriovdp=enabled" and set the number of 1G huge pages to 200. Once the installation was complete, I saw that k8s sriov-device plugin was not coming up. It complained that the resource list was empty.
I had to set "/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/sriov_numvfs" to 8 (I needed 8 virtual interfaces) and update the resource list in /etc/pcidp/config.json
{
"resourceList": [ { "resourceName": "bcm_sriov_netdevice", "selectors": { "vendors": ["14e4"], "devices": ["16dc"], "drivers": ["bnxt_en"], "pfNames": ["enp59s0f0#0-7"] } } ] }
to see that sriov-dp comes up properly.
Is there any way to pass on the number of VF's( sriov_numvfs )required per node and resourcelist during the time of installation when we label the nodes as sriovdp=enabled
Regards, Sriram
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss