<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin:0cm;
margin-bottom:.0001pt;
text-indent:21.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1930234491;
mso-list-type:hybrid;
mso-list-template-ids:2088125494 2011487786 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">The following 2 bugs have been reported to track the limitations 1&2:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://bugs.launchpad.net/starlingx/+bug/1835018">https://bugs.launchpad.net/starlingx/+bug/1835018</a><span style="color:#1F497D"><o:p></o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="https://bugs.launchpad.net/starlingx/+bug/1835020">https://bugs.launchpad.net/starlingx/+bug/1835020</a></span><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">Limitation 3 should be posted on StarlingX guide on how to use MULTUS/SR-IOV CNI which is still missing. Just like below:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">To run SRIOV+DPDK, the pod need to request memory and mount hugepage-volume on correct host path as following:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> resources:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">requests:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> memory: 2Gi<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> intel.com/pci_sriov_net_physnet0: 2<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:5.25pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">limits:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> memory: 2Gi<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> intel.com/pci_sriov_net_physnet0: 2<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:5.25pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">volumeMounts:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:5.25pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">- name: hugepage-volume<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> mountPath: /dev/hugepages<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:5.25pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">volumes:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:5.25pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">- name: hugepage-volume<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:10.5pt"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">hostPath:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"> path: /dev/hugepages<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">Best Regards,<o:p></o:p></span></a></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;color:#1F497D">Xu, Chenjie<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Xu, Chenjie
<br>
<b>Sent:</b> Thursday, June 27, 2019 4:32 PM<br>
<b>To:</b> Webster, Steven <Steven.Webster@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com><br>
<b>Cc:</b> Zhao, Forrest <forrest.zhao@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com><br>
<b>Subject:</b> Limitations on Multus/SRIOV CNI Plugins<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">Hi Steven,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">During my testing on Multus/SRIOV CNI Plugins, I have following findings:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt">Need sysadmin to set MAC address manually for VF.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt">The configuration for SR-IOV network device plugin has changed as following:<br>
</span><span lang="EN-US"><a href="https://github.com/intel/sriov-network-device-plugin#configurations">https://github.com/intel/sriov-network-device-plugin#configurations</a></span><span lang="EN-US" style="font-size:10.5pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:18.0pt"><span lang="EN-US" style="font-size:10.5pt">For now StarlingX uses its own docker image and doesn’t need to change the configuration. But in the future, this will be a bug when StarlingX updates the docker image
to newer version.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt">This is not a bug but should be noticed:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt">Normally, huge pages should be supported by kubernetes to run SRIOV+DPDK. And the pod needs to request huge pages as following:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">resources:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt">requests:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> memory: 2Gi<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> intel.com/pci_sriov_net_physnet0: 2<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt">limits:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> hugepages-1Gi: 2Gi<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> memory: 2Gi<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"> intel.com/pci_sriov_net_physnet0: 2<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">volumeMounts:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:36.75pt"><span lang="EN-US" style="font-size:10.5pt">- name: hugepage-volume<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"> mountPath: /dev/hugepages<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">volumes:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:36.75pt"><span lang="EN-US" style="font-size:10.5pt">- name: hugepage-volume<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:26.25pt"><span lang="EN-US" style="font-size:10.5pt">emptyDir:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"> medium: HugePages<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt">However the kubernetes provided by StarlingX will:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">enable huge pages for non-openstack based worker node<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">disable huge pages for openstack based worker node:<br>
</span><span lang="EN-US"><a href="https://opendev.org/starlingx/config/src/branch/master/puppet-manifests/src/modules/platform/manifests/kubernetes.pp#L118">https://opendev.org/starlingx/config/src/branch/master/puppet-manifests/src/modules/platform/manifests/kubernetes.pp#L118</a><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:16.5pt"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US">But by my testing, the pod can still get huge pages in openstack based worker node on which kubernetes doesn’t provide support for huge pages. And the pod need to request
memory and mount hugepage-volume on correct host path as following:<br>
</span><span lang="EN-US" style="font-size:10.5pt"> resources:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt">requests:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> memory: 2Gi<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> intel.com/pci_sriov_net_physnet0: 2<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt">limits:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt"> memory: 2Gi<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"> intel.com/pci_sriov_net_physnet0: 2<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">volumeMounts:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:36.75pt"><span lang="EN-US" style="font-size:10.5pt">- name: hugepage-volume<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"> mountPath: /dev/hugepages<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:15.75pt"><span lang="EN-US" style="font-size:10.5pt">volumes:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:36.75pt"><span lang="EN-US" style="font-size:10.5pt">- name: hugepage-volume<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt">hostPath:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt"> path: /dev/hugepages<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">Xu, Chenjie<o:p></o:p></span></p>
</div>
</body>
</html>