<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1062097369;
mso-list-type:hybrid;
mso-list-template-ids:854628720 761188092 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:DengXian;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1304697705;
mso-list-type:hybrid;
mso-list-template-ids:525134954 -1169685308 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:23.25pt;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:DengXian;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:59.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:95.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:131.25pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:167.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:203.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:239.25pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:275.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:311.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:2128574792;
mso-list-type:hybrid;
mso-list-template-ids:1008261440 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>Hi:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>The main code of NUMA aware live migration was merged. I’m testing it recently.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>If only set NUMA property(‘hw:numa_nodes’, ‘hw:numa_cpus’, ‘hw:numa_mem’), it works well. But if add the property “hw:cpu_policy='dedicated'”, it will not correct after serval live migrations.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Which means the live migrate can be success, but the vCPU pin are not correct(two instance have serval same vCPU pin on same host).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Below is my test steps.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>env:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'> code: master branch (build on 16 September 2019, include the patches of NUMA aware live migration)<o:p></o:p></span></p><p class=MsoNormal style='text-indent:18.0pt'><span style='font-size:11.0pt'>three compute node:<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:54.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo3'><![if !supportLists]><span style='font-size:11.0pt'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt'>s1: 24C, 48G (2 NUMA nodes)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:54.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo3'><![if !supportLists]><span style='font-size:11.0pt'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt'>stein-2: 12C, 24G (2 NUMA nodes)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:54.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo3'><![if !supportLists]><span style='font-size:11.0pt'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt'>stein-3: 8C, 16G (2 NUMA nodes)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:18.0pt'><span style='font-size:11.0pt'>flavor1 (2c2g): hw:cpu_policy='dedicated', hw:numa_cpus.0='0', hw:numa_cpus.1='1', hw:numa_mem.0='1024', hw:numa_mem.1='1024', hw:numa_nodes='2'<o:p></o:p></span></p><p class=MsoNormal style='margin-left:18.0pt'><span style='font-size:11.0pt'>flavor2 (4c4g): hw:cpu_policy='dedicated', hw:numa_cpus.0='0,1,2', hw:numa_cpus.1='3', hw:numa_mem.0='1024', hw:numa_mem.1='3072', hw:numa_nodes='2'<o:p></o:p></span></p><p class=MsoNormal style='margin-left:18.0pt'><span style='font-size:11.0pt'>image has no property.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>I create four instances(2*flavor1, 2* flavor2), then begin live migration on by one(one instance live migrate done, next instance begin live migrate) and check the vCPU pin whether is correct.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>After serval live migrations, the vCPU pin will not correct. (You can find full migration list in attached file). The last live migrate is:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><i><span style='font-size:11.0pt'>+-----+--------------------------------------+-------------+-----------+----------------+--------------+----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>| Id | UUID | Source Node | Dest Node | Source Compute | Dest Compute | Dest Host | Status | Instance UUID | Old Flavor | New Flavor | Created At | Updated At | Type |<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>+-----+--------------------------------------+-------------+-----------+----------------+--------------+----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>| 470 | 2a9ba183-4f91-4fbf-93cf-6f0e55cc085a | s1 | stein-3 | s1 | stein-3 | 172.16.130.153 | completed | bf0466f6-4815-4824-8586-899817207564 | 1 | 1 | 2019-09-17T10:28:46.000000 | 2019-09-17T10:29:09.000000 | live-migration |<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>| 469 | c05ea0e8-f040-463e-8957-a59f70ed8bf6 | s1 | stein-3 | s1 | stein-3 | 172.16.130.153 | completed | a3ec7a29-80de-4541-989d-4b9f4377f0bd | 1 | 1 | 2019-09-17T10:28:21.000000 | 2019-09-17T10:28:45.000000 | live-migration |<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>| 468 | cef4c609-157e-4b39-b6cc-f5528d49c75a | s1 | stein-2 | s1 | stein-2 | 172.16.130.152 | completed | 83dab721-3343-436d-bee7-f5ffc0d0d38d | 4 | 4 | 2019-09-17T10:27:57.000000 | 2019-09-17T10:28:21.000000 | live-migration |<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>| 467 | 5471e441-2a50-465a-bb63-3fe1bb2e81b9 | s1 | stein-2 | s1 | stein-2 | 172.16.130.152 | completed | e3c19fbe-7b94-4a65-a803-51daa9934378 | 4 | 4 | 2019-09-17T10:27:32.000000 | 2019-09-17T10:27:57.000000 | live-migration |<o:p></o:p></span></i></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>There are two instances land on stein-3, and the two instances have same vCPU pin:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><i><span style='font-size:11.0pt'>(nova-libvirt)[root@stein-3 /]# virsh list --all<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> Id Name State<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>----------------------------------------------------<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> 32 instance-00000025 running<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> 33 instance-00000024 running<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'><o:p> </o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>(nova-libvirt)[root@stein-3 /]# virsh vcpupin 32<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>VCPU: CPU Affinity<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>----------------------------------<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> 0: 2<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> 1: 7<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'><o:p> </o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>(nova-libvirt)[root@stein-3 /]# virsh vcpupin 33<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>VCPU: CPU Affinity<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'>----------------------------------<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> 0: 2<o:p></o:p></span></i></p><p class=MsoNormal><i><span style='font-size:11.0pt'> 1: 7<o:p></o:p></span></i></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>I checked the nova compute’s log on stein-3(you can find the log in attached log), then I found ‘host_topology’ isn’t updated when ‘hardware.numa_fit_instance_to_host’ be called in claims. ‘host_topology’ is the property of ‘objects.ComputeNode’ and it’s cached in ‘ResourceTracker’, it will use cached ‘cn’ to build ‘claim’ when ‘check_can_live_migrate_destination’ called. Therefore, I guess the cache was not updated or updated too late or some other reason.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>I also checked the database, the NUMA topology of <i>the two instances</i> have same vCPU pin: “[0,2], [1,7]”, and the <i>compute node: stein-</i>3 only has vCPU pin: “[2], [7]”.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Please correct me if there is something wrong :)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;color:black'>Best Regards</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>