<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Marvin :<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:9.6pt"><span style="color:#1F497D">The simplest way to check huge page feature is ‘kubectl describe node compute-1 | grep hugepage’ if compute is enabled hugepage and has pre-allocated 2M hugepage.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">If compute did not enable hugepage , then the command should return empty.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">For pod configuration you can refer to [1], but I did not tried trigger pod with hugepage.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[1] <a href="https://stupefied-goodall-e282f7.netlify.com/contributors/design-proposals/resource-management/hugepages/">
https://stupefied-goodall-e282f7.netlify.com/contributors/design-proposals/resource-management/hugepages/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">BR<br>
Austin Sun. <o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:9.6pt"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-indent:9.6pt"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> Liu, Tao [mailto:Tao.Liu@windriver.com]
<br>
<b>Sent:</b> Thursday, May 2, 2019 8:48 AM<br>
<b>To:</b> Huang, Marvin <Marvin.Huang@windriver.com>; Sun, Austin <austin.sun@intel.com><br>
<b>Cc:</b> Eslimi, Dariush <Dariush.Eslimi@windriver.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> RE: questions about '[Feature] Huge page management' https://storyboard.openstack.org/#!/story/2004763<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Marvin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">See inline comments below for the questions with regards to platform huge pages management, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I will let Austin to address your queries related to k8s huge page feature testing.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Tao<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> Huang, Marvin
<br>
<b>Sent:</b> Wednesday, May 1, 2019 5:20 PM<br>
<b>To:</b> Liu, Tao; Sun, Austin<br>
<b>Cc:</b> Eslimi, Dariush; Waheed, Numan; Xie, Cindy; <a href="mailto:starlingx-discuss@lists.starlingx.io">
starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> RE: questions about '[Feature] Huge page management' <a href="https://storyboard.openstack.org/#!/story/2004763">
https://storyboard.openstack.org/#!/story/2004763</a><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Tao,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regarding to Huge Page management rework, there’s a US
<a href="https://storyboard.openstack.org/#!/story/2004763">https://storyboard.openstack.org/#!/story/2004763</a>, which has the following are the requirements:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Remove automatic allocation of the 1G huge pages from the kick starts for vswitch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Enable k8s huge page feature for worker nodes that do not have the openstack compute label. It should be disabled otherwise.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Automatically defaults for worker nodes with openstack compute label. Changes will be applied on the unlock.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> - Current 2M huge page default settings<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> - 1-1G huge page per numa node for vswitch
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Do not allocate any huge pages by default for worker nodes without the openstack compute label. Up to the customer to configure.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Here’re some questions, please comment on them:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">REQUIRMENT 1:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Remove automatic allocation of the 1G huge pages from the kick starts for vswitch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> Is that meaning that, by default, no 1G huge pages for vswitch, no 'openstack compute label', and 'k8s huge page feature' IS ENABLED?<o:p></o:p></span></p>
<p class="MsoNormal">TL> Yes<span style="color:#1F497D">, </span>1G per NUMA node for vswitch will be allocated when the vswitch type is set. 'k8s huge page feature' will be disabled when 'openstack compute label' is set.<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">REQUIRMENT 2:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Enable k8s huge page feature for worker nodes that do not have the openstack compute label. It should be disabled otherwise.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> q1: What is it meaning by 'Enable k8s huge page feature'? How to verify it?<o:p></o:p></span></p>
<p class="MsoNormal">TL>Perhaps Austin could share how he tested it.<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 1 Will any commands show the feature is enabled/disabled?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> e.g. Will it be shown in the options apiserver been started with, i.e. in the outputs of command<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 'sudo ps aux | grep apiserver | grep feature-gates'?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> And/or will any configuration files indicating it's enabled/disabled?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> I found currently, /var/lib/kubelet/config.yaml contains lines having 'featureGates'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> q2: Will adding/removing the label 'openstack-compute-node' after initial provision/installation, enable/disable the 'k8s huge page feature'?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> And those the states of enabled/disabled can be verified using the any methods for q1.<o:p></o:p></span></p>
<p class="MsoNormal">TL> It should (after the host is unlocked).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">REQUIRMENT 3:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Automatically defaults for worker nodes with openstack compute label. Changes will be applied on the unlock.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> - Current 2M huge page default settings<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> - 1-1G huge page per numa node for vswitch
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Do not allocate any huge pages by default for worker nodes without the openstack compute label. Up to the customer to configure.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> q1: Does it mean if the label 'openstack-compute-node' is added to a node (hence 'k8s huge pages feature' is DISABLED), <br>
after been unlocked (it must be locked in order to add the label), certain numbers of 2M-pages and 1 1G-page will be allocated on the node,
<br>
and can be verified with above mentioned methods?<o:p></o:p></span></p>
<p class="MsoNormal">TL> Only 2M huge pages will be allocated for applications by default, and they are shown in CLI and Horizon. The remaining available memory is calculated as 4K when the host is unlocked.<o:p></o:p></p>
<p class="MsoNormal">You can also cat /sys/devices/system/node/nodex/hugepages/hugepages-2048kB/nr_hugepages to see the allocation in Linux.<o:p></o:p></p>
<p class="MsoNormal">The users can configure 1G pages for applications via CLI/Horizon.<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Marvin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
</body>
</html>