[Starlingx-discuss] [Distributed StarlingX 4.0] Multiple SRIOV pools

[Dharwadkar, Sriram]

Can you please reply to the below question ?
Is it possible to create multiple device pools based on the selectors like below using starlingx CLI,
controller-1:/home/sysadmin# cat /etc/pcidp/config.json
{
  "resourceList": [
    {
      "resourceName": "cuplanepool",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0#0-4"
        ]
      }
    },
    {
      "resourceName": "flupool",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0#5-9"
        ]
      }
    },
    {
      "resourceName": "o1pool",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0#10-15"
        ]
      }
    }
  ]
}

Regards,
Sriram

From: Dharwadkar, Sriram
Sent: Thursday, January 7, 2021 8:50 PM
To: starlingx-discuss at lists.starlingx.io
Subject: [Distributed StarlingX 4.0] Multiple SRIOV pools

Hi,

We have installed Starlingx - Aug 5th release build for central cloud and edge cloud.

controller-1:~$ cat /etc/build.info
###
### StarlingX
###     Release 20.06
###

OS="centos"
SW_VERSION="20.06"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="r/stx.4.0"

JOB="STX_4.0_build_layer_flock"
BUILD_BY="starlingx.build at cengn.ca<mailto:starlingx.build at cengn.ca>"
BUILD_NUMBER="22"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-08-05 12:25:52 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_4.0_build_layer_flock"
FLOCK_BUILD_BY="starlingx.build at cengn.ca<mailto:starlingx.build at cengn.ca>"
FLOCK_BUILD_NUMBER="22"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2020-08-05 12:25:52 +0000"

Hardware on which edge cloud is installed has Mellanox nic which is SRIOV capable.

Commands that I m using to enable SRIOV functionality are,
DATA0IF=enp101s0f0
export NODE=controller-1
PHYSNET0='physnet0'
SPL=/tmp/tmp-system-port-list
SPIL=/tmp/tmp-system-host-if-list
system host-port-list ${NODE} --nowrap > ${SPL}
cat  /tmp/tmp-system-port-list
system host-if-list -a ${NODE} --nowrap > ${SPIL}
cat /tmp/tmp-system-host-if-list
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
echo $DATA0IFUUID
system datanetwork-add ${PHYSNET0} vlan
system host-if-modify $NODE -n sriov0 -c pci-sriov -N 16 --vf-driver=netdevice  $DATA0IFUUID
system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}

With these commands I m able to see that /etc/pcidp/config.json file is getting updated with the below contents

controller-1:/home/sysadmin# cat /etc/pcidp/config.json
{
  "resourceList": [
    {
      "resourceName": "pci_sriov_net_physnet0",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0"
        ]
      }
    }
  ]
}

One device pool is created with 16 vfs.
controller-1:/home/sysadmin# kubectl get node controller-1 -o json | jq '.status.allocatable'
{
  "cpu": "22",
  "ephemeral-storage": "9391196145",
  "hugepages-1Gi": "20Gi",
  "hugepages-2Mi": "0",
  "intel.com/pci_sriov_net_physnet0": "16",
  "memory": "70207428Ki",
  "pods": "110"
}

Our usecase requires us to create multiple device pools like below.
controller-1:/home/sysadmin# cat /etc/pcidp/config.json
{
  "resourceList": [
    {
      "resourceName": "cuplanepool",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0#0-4"
        ]
      }
    },
    {
      "resourceName": "flupool",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0#5-9"
        ]
      }
    },
    {
      "resourceName": "o1pool",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "isRdma": true,
        "drivers": [
          "mlx5_core"
        ],
        "devices": [
          "1014"
        ],
        "pfNames": [
          "enp101s0f0#10-15"
        ]
      }
    }
  ]
}

Please let me know if it is possible to create multiple device pools from the same nic (with the filters as above) using StarlingX system CLI.

Regards,
Sriram

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210129/0394b076/attachment-0001.html>