<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:848255422;
mso-list-type:hybrid;
mso-list-template-ids:-537496612 1399096402 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:%1;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.5in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Austin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks for your information!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">So given ‘</span><span style="color:#1F497D">This feature is enable K8S feature-gate as describe in [1]’:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.75in;text-indent:-.5in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">1<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">How to tell if this ‘hugepage feature’ is (currently) enabled or disabled? Any user visible signs in Horizon, CLIs outputs? Or it’s transparent to users? Or:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Is it only reflected in arguments to ‘kubeadm init --feature-gates=’…,hugepage=enable/disable’, which is called to provision a node/master?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">And/or (also) is shown in /var/lib/kubelet/config.yaml, e.g.:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="color:#1F497D">controller-0:~$ grep 'featureGate' /var/lib/kubelet/config.yaml -A2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="color:#1F497D">featureGates:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="color:#1F497D"> HugePages: false<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">c.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Any OS level options changes, like options inside /etc/default/grub?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.75in;text-indent:-.5in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">2<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Can changing label enable or/and disable the ‘hugepage feature’? For example, now assuming worker compute-0 has label ‘openstack-compute-node’, hence hugepage is disabled:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">We can remove the label ‘openstack-compute-node ‘ using CLI<br>
system host-label-remove compute-0 openstack-compute-node<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">What the expected the system behavior after the label is removed? The ‘hugepage feature’ will be enabled after been unlocked, which can be verified using methods in 1?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> 3 Any user aware difference between the features enabled/disabled?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Marvin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Sun, Austin [mailto:austin.sun@intel.com]
<br>
<b>Sent:</b> Wednesday, April 24, 2019 9:55 PM<br>
<b>To:</b> Huang, Marvin<br>
<b>Cc:</b> starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> RE: questions about '[Feature] Huge page management' https://storyboard.openstack.org/#!/story/2004763<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D">Hi, Marvin:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"> This feature is not directly related with “system host-memory-modify” function, user still should be able to modify mem config as before.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"> This feature is enable K8S feature-gate as describe in [1]. And K8S enabling hugepage feature is opposite to compute label<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"> ( means compute label is tagged, then k8s hugepage feature is disabled (false), if compute label is not tagged , then k8s hugepage is enabled )
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D">About your question about VM hugepage decrease, I did not dig into VM mem , so I cannot give more comments. <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"> <o:p>
</o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in">[1] <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/">
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Thanks.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">BR<br>
Austin Sun. <span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><a name="_MailEndCompose"></a><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><a name="_____replyseparator"></a><b>From:</b> Huang, Marvin [mailto:Marvin.Huang@windriver.com]
<br>
<b>Sent:</b> Thursday, April 25, 2019 4:33 AM<br>
<b>To:</b> Sun, Austin <austin.sun@intel.com><br>
<b>Cc:</b> starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> questions about '[Feature] Huge page management' https://storyboard.openstack.org/#!/story/2004763<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Hi Austin,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">According to (Storyboard) <a href="https://storyboard.openstack.org/%23!/story/2004763">
2004763</a>, I know that you’re working on the feature “Huge page management”.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I’ve some questions about the feature.<br>
Though one of its 2 tasks is still in Review (the other is shown merged), you might now have the answers already.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">In the description of the Story, there are requirement:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">“- Enable k8s huge page feature for
<span style="background:yellow;mso-highlight:yellow">worker nodes that <b><span style="color:red">do not</span></b><span style="color:red">
</span>have the openstack compute label</span>. It should be disabled otherwise.”<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><b>Questions: what is this meaning to users?<o:p></o:p></b></p>
<p class="MsoListParagraph" style="margin-left:1.25in">By ‘Enable’, is it meaning users can modify memory allocation on the node? (via the following):<br>
system host-modify <worker-name> [-2M <2M hugepages number>] [-1G <1G hugepages number>] [-f <function>] …<br>
or Horizon: Admin -> Platform -> Host Inventory …<br>
<br>
Otherwise (‘disabled’), the CLIs (system host-memory-xxx) will reject any requests?
<br>
Or the corresponding Horizon pages do not have any items to update the memory application? Or those were disabled?<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.25in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">“- Automatically defaults for <span style="background:yellow;mso-highlight:yellow">
worker nodes <span style="color:red">with </span>openstack compute label</span>. Changes will be applied on the unlock.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> - Current 2M huge page default settings<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> - 1-1G huge page per numa node for vswitch “<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><b>Questions: in this situation, is the k8s huge page feature disabled (according to the above requirement)?<br>
</b>And the (host-memory) CLIs will reject any requests?<b><o:p></o:p></b></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">And a question related with VMs:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">If a VM using huge page (with flavor having ‘hw:mem_page_size=large’ or ‘hw:mem_page_size=1048576’) is launched, will the free memory pages decreased accordingly on the worker it’s running on?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">That is, if the VM is consuming 1G huge-page, the number of free page of 1G size on the hosting worker should be reduced by 1. Is this still the expected behavior?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">This is the assumption in <a href="https://bugs.launchpad.net/starlingx/+bug/1813325">
https://bugs.launchpad.net/starlingx/+bug/1813325</a>.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Regards,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Marvin<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</body>
</html>