Hi,

a.: I believe you should be able to lock, apply these steps, and unlock.

b.: I don't know if StarlingX has an option to specify per node pools. Will defer to experts. But if the nodes are in one L2, they would typically be in one cluster and thus the VF MAC assignment would be arbitrated by cluster wide neutron/CNI, right? If each node is one cluster, maybe the user would need to annotate the pods with static MACs.

BR

On Tue, 29 Sep, 2020, 23:23 Sriram, <sriram.ec@gmail.com> wrote:

Hi,

How do we ensure the uniqueness of VF mac addresses across all the nodes in the k8s cluster formed on edge nodes. 

Please let me know if this problem is addressed by starlingX or if it is taken care of by some other means.

Regards,

Sriram

On Fri, Sep 25, 2020 at 1:20 PM Sriram <sriram.ec@gmail.com> wrote:

Hi Pratik,

Thanks for your reply. 

a. Can these steps be done after the installation is complete, now that I have already installed.

b. How do we ensure the uniqueness of VF mac addresses across all the nodes in the k8s cluster formed on edge nodes.

Regards,
Sriram

On Fri, Sep 25, 2020 at 1:02 PM Pratik M. <pvmpublic@gmail.com> wrote:

Hi,
You would need to do:
# system host-label-assign controller-0 sriovdp=enabled  
# system host-if-modify controller-0 <interface> -c pci-sriov -n sriov0 -N <num vfs>  
# system interface-datanetwork-assign controller-0 <interface> <datanetwork>  
# system host-unlock

And that should populate the /etc/pcidp/config.json

Ref:
https://wiki.openstack.org/wiki/StarlingX/Networking
Steven Webster's helpful comments in https://bugs.launchpad.net/starlingx/+bug/1891889

Thanks

On Thu, Sep 24, 2020 at 3:04 PM Sriram <sriram.ec@gmail.com> wrote:

Hi,

I have installed distributed starlingx 4.0 in "All in one Duplex" mode. There are two nodes in the central cloud and two in the edge cloud.

I have enabled SRIOV in bios settings of edge cloud nodes and set total VFs as 16. 

After that, while installing starlingX I followed the steps to enable SRIOV.

system host-label-assign controller-0 sriovdp=enabled
system host-memory-modify controller-0 0 -1G 100
system host-memory-modify controller-0 1 -1G 100  

and ran these steps for controller-1 as well. 

As I understand the first step would label the node "controller-0 and controller-1" as "sriovdp=enabled" and set the number of 1G huge pages to 200.

Once the installation was complete, I saw that k8s sriov-device plugin was not coming up. It complained that the resource list was empty. 

I had to set  "/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/sriov_numvfs" to 8 (I needed 8 virtual interfaces) and update the resource list in 

/etc/pcidp/config.json

{
  "resourceList": [
            {
                "resourceName": "bcm_sriov_netdevice",
                "selectors": {
                    "vendors": ["14e4"],
                    "devices": ["16dc"],
                    "drivers": ["bnxt_en"],
                    "pfNames": ["enp59s0f0#0-7"]
                }
             }
    ]
}

to see that sriov-dp comes up properly.

Is there any way to pass on the number of VF's( sriov_numvfs )required per node and resourcelist during the time of installation when we label the nodes as  sriovdp=enabled

Regards,

Sriram

 

 

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss