Can you please reply to the below question ?

Is it possible to create multiple device pools based on the selectors like below using starlingx CLI,

controller-1:/home/sysadmin# cat /etc/pcidp/config.json

{

  "resourceList": [

    {

      "resourceName": "cuplanepool",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0#0-4"

        ]

      }

    },

    {

      "resourceName": "flupool",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0#5-9"

        ]

      }

    },

    {

      "resourceName": "o1pool",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0#10-15"

        ]

      }

    }

  ]

}

 

Regards,
Sriram

 

From: Dharwadkar, Sriram
Sent: Thursday, January 7, 2021 8:50 PM
To: starlingx-discuss@lists.starlingx.io
Subject: [Distributed StarlingX 4.0] Multiple SRIOV pools

 

Hi,

 

We have installed Starlingx – Aug 5th release build for central cloud and edge cloud.

 

controller-1:~$ cat /etc/build.info

###

### StarlingX

###     Release 20.06

###

 

OS="centos"

SW_VERSION="20.06"

BUILD_TARGET="Host Installer"

BUILD_TYPE="Formal"

BUILD_ID="r/stx.4.0"

 

JOB="STX_4.0_build_layer_flock"

BUILD_BY="starlingx.build@cengn.ca"

BUILD_NUMBER="22"

BUILD_HOST="starlingx_mirror"

BUILD_DATE="2020-08-05 12:25:52 +0000"

 

FLOCK_OS="centos"

FLOCK_JOB="STX_4.0_build_layer_flock"

FLOCK_BUILD_BY="starlingx.build@cengn.ca"

FLOCK_BUILD_NUMBER="22"

FLOCK_BUILD_HOST="starlingx_mirror"

FLOCK_BUILD_DATE="2020-08-05 12:25:52 +0000"

 

Hardware on which edge cloud is installed has Mellanox nic which is SRIOV capable.

 

Commands that I m using to enable SRIOV functionality are,

DATA0IF=enp101s0f0

export NODE=controller-1

PHYSNET0='physnet0'

SPL=/tmp/tmp-system-port-list

SPIL=/tmp/tmp-system-host-if-list

system host-port-list ${NODE} --nowrap > ${SPL}

cat  /tmp/tmp-system-port-list

system host-if-list -a ${NODE} --nowrap > ${SPIL}

cat /tmp/tmp-system-host-if-list

DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')

DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')

DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')

DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')

echo $DATA0IFUUID

system datanetwork-add ${PHYSNET0} vlan

system host-if-modify $NODE -n sriov0 -c pci-sriov -N 16 --vf-driver=netdevice  $DATA0IFUUID

system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}

 

With these commands I m able to see that /etc/pcidp/config.json file is getting updated with the below contents

 

controller-1:/home/sysadmin# cat /etc/pcidp/config.json

{

  "resourceList": [

    {

      "resourceName": "pci_sriov_net_physnet0",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0"

        ]

      }

    }

  ]

}

 

One device pool is created with 16 vfs.

controller-1:/home/sysadmin# kubectl get node controller-1 -o json | jq '.status.allocatable'

{

  "cpu": "22",

  "ephemeral-storage": "9391196145",

  "hugepages-1Gi": "20Gi",

  "hugepages-2Mi": "0",

  "intel.com/pci_sriov_net_physnet0": "16",

  "memory": "70207428Ki",

  "pods": "110"

}

 

Our usecase requires us to create multiple device pools like below.

controller-1:/home/sysadmin# cat /etc/pcidp/config.json

{

  "resourceList": [

    {

      "resourceName": "cuplanepool",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0#0-4"

        ]

      }

    },

    {

      "resourceName": "flupool",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0#5-9"

        ]

      }

    },

    {

      "resourceName": "o1pool",

      "selectors": {

        "vendors": [

          "15b3"

        ],

        "isRdma": true,

        "drivers": [

          "mlx5_core"

        ],

        "devices": [

          "1014"

        ],

        "pfNames": [

          "enp101s0f0#10-15"

        ]

      }

    }

  ]

}

 

Please let me know if it is possible to create multiple device pools from the same nic (with the filters as above) using StarlingX system CLI.

 

Regards,

Sriram