[Starlingx-discuss] Question - Using 1Gb mem pages

Michel Thebeau michel.thebeau at windriver.com
Mon Oct 22 17:09:31 UTC 2018


Hi Ricardo,

That's cool.

I had added cpu flag 'pdpe1gb' to compute under-virtualization, 
otherwise as persented by the virtualization wiki, to see if it would work:
    <cpu match='exact'>
      <model fallback='forbid'>Nehalem</model>
      <topology sockets='1' cores='4' threads='1'/>
      <feature policy='require' name='vmx'/>
+    <feature policy='require' name='pdpe1gb'/>
      <feature policy='optional' name='svm'/>
    </cpu>
    <clock offset='utc'/>

And I was able to configure the compute and launch VMs with 1G pages 
(per the attached example).  ... I found the memory accounting or 
display to be a bit wonky though.  I'm not sure what to make of that.  
I'll mention it to my bosses, but I'm sure my method is not a supported. ;)

M

On 2018-10-22 12:54 p.m., Perez, Ricardo O wrote:
> It's working now:
>
> wrsroot at compute-0 ~(keystone_admin)]$ openstack flavor set m1.small --property hw:mem_page_size=1GB
>   [wrsroot at compute-0 ~(keystone_admin)]$ openstack server create --image cirros --flavor m1.small --nic net-id=net3 testvm
> [wrsroot at compute-0 ~(keystone_admin)]$ openstack server show testvm
> +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Field                               | Value                                                                                                                                                                                         |
> +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | wrs-res:pci_devices                 |                                                                                                                                                                                               |
> | wrs-res:topology                    | node:0,  1024MB, pgsize:1G, vcpus:0-3, pol:sha                                                                                                                                                |
> | wrs-res:vcpus                       | [4, 4, 4]                                                                                                                                                                                     |
> | wrs-sg:server_group                 |                                                                                                                                                                                               |
> +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>
>
> Thanks !
> -Richo
>
>> -----Original Message-----
>> From: Michel Thebeau [mailto:michel.thebeau at windriver.com]
>> Sent: Monday, October 22, 2018 11:18 AM
>> To: Perez, Ricardo O <ricardo.o.perez at intel.com>; starlingx-
>> discuss at lists.starlingx.io
>> Subject: Re: [Starlingx-discuss] Question - Using 1Gb mem pages
>>
>> Hi Ricardo,
>>
>> I was able to select 4K and 2M pages in the following way (attached txt file).  I'll
>> try to see if I can get a system up with 1G pages, and I'll follow up with you.
>>
>>
>>
>> M
>>
>>
>> On 2018-10-22 10:59 a.m., Michel Thebeau wrote:
>>> Hi Ricardo,
>>>
>>> The documentation indicates to configure the the huge pages per numa
>>> node on the compute, and to configure the flavour extra specs for the
>>> VM.  I am not aware of anything else in particular. There's some tips
>>> and caveats in the document, but I don't see anything in your message
>>> to think of those.
>>>
>>> If there's any chance you are trying to do this under virtualization,
>>> I am not aware that that is currently supported.
>>>
>>> M
>>>
>>>
>>> On 2018-10-19 6:14 p.m., Perez, Ricardo O wrote:
>>>> Hi Michael,
>>>>
>>>> Thanks for your answer, I just forgot to put the computes information
>>>> about memory configuration, here is the info:
>>>>
>>>> Compute-0
>>>>
>>>>      VM Pages
>>>> 0   Reserved for Platform: 8000 MiB
>>>> Usable Total: 86387 MiB
>>>> Available: 84787 MiB
>>>>       4K Pages:
>>>>       Total: 230656
>>>> 2M Hugepages:
>>>>       Total: 1271
>>>>       Available: 983
>>>> 1G Hugepages:
>>>>       Total: 80
>>>>       Available: 80
>>>>
>>>> 1   Reserved for Platform: 2000 MiB
>>>> Usable Total: 93698 MiB
>>>> Available: 92674 MiB
>>>>       4K Pages:
>>>>       Total: 29696
>>>> 2M Hugepages:
>>>>       Total: 5319
>>>>       Available: 5319
>>>> 1G Hugepages:
>>>>       Total: 80
>>>>       Available: 80
>>>>
>>>> ***********************************
>>>> Compute-1
>>>> Processor
>>>>       Memory
>>>>       VM Pages
>>>> 0   Reserved for Platform: 8000 MiB
>>>> Usable Total: 86504 MiB
>>>> Available: 84968 MiB
>>>>       4K Pages:
>>>>       Total: 0
>>>> 2M Hugepages:
>>>>       Total: 1780
>>>>       Available: 1524
>>>> 1G Hugepages:
>>>>       Total: 80
>>>>       Available: 80
>>>>
>>>> 1   Reserved for Platform: 2000 MiB
>>>> Usable Total: 93552 MiB
>>>> Available: 92528 MiB
>>>>       4K Pages:
>>>>       Total: 0
>>>> 2M Hugepages:
>>>>       Total: 5304
>>>>       Available: 5304
>>>> 1G Hugepages:
>>>>       Total: 80
>>>>       Available: 80
>>>>
>>>> Am I missing something else to made use of 1G memory pages ?
>>>>
>>>> Thanks
>>>> -Richo
>>>>
>>>>> -----Original Message-----
>>>>> From: Michel Thebeau [mailto:michel.thebeau at windriver.com]
>>>>> Sent: Friday, October 19, 2018 1:39 PM
>>>>> To: Perez, Ricardo O <ricardo.o.perez at intel.com>; starlingx-
>>>>> discuss at lists.starlingx.io
>>>>> Subject: Re: [Starlingx-discuss] Question - Using 1Gb mem pages
>>>>>
>>>>> On 2018-10-19 1:45 p.m., Perez, Ricardo O wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to execute a test, which his goal is to made use of huge
>>>>>> pages (2M /
>>>>> 1G) launching a VM.
>>>>>> I have already verified the configuration of the controllers GRUB
>>>>>> options and
>>>>> we have defined the variable hugepagesz=1G, however, inside the same
>>>>> parameters list, there is a "default_hugepagesz=2M". The default
>>>>> hugepagesz, has been modified to 1G, and the compute has been
>>>>> restarted with no issues.
>>>>>
>>>>>
>>>>> Hi Ricardo,
>>>>>
>>>>> You wrote, "controllers GRUB options" then "the compute has been
>>>>> restarted with no issues".  Is there a GRUB option one can configure
>>>>> on the controller that is for the computes?  I don't remember seeing
>>>>> one. If you're actually changing /etc/default/grub, I doubt that's
>>>>> supported.
>>>>> I did not find a procedure for changing default_hugepagesz  on
>>>>> controllers or computes grub configuration.
>>>>>
>>>>> It's supposed to be an option, for example through in horizon
>>>>> interface or 'system host-memory-modify'... but you didn't quote
>>>>> either of those.
>>>>>
>>>>>
>>>>>> The flavor has been modified, with the following option: "
>>>>> hw:mem_page_size=104875"  (1G) page.
>>>>>
>>>>>
>>>>> Is "104875" a typo?  It looks a lot like 1024 * 1024 = 1048576, but
>>>>> not quite.
>>>>>
>>>>>
>>>>> M
>>>>>
>>>>>
>>>>>> However when I launch a VM using the modified flavor it creates the
>>>>>> VM, but if
>>>>> I check the field values of the VM using " openstack server show
>>>>> <vm_name>".
>>>>> I'm getting the following value for pgsize:
>>>>>> wrs-res:topology                    | node:0,   512MB, pgsize:2M,
>>>>>> vcpus:0, pol:sha |
>>>>>>
>>>>>> So, seems like it's still taking the 2M pagesize instead of the 1G.
>>>>>>
>>>>>> What I have already tried is to:
>>>>>>
>>>>>> *        Change also the image metadata: hw:mem_page_size=large and
>>>>>> also
>>>>> 104875. Result, VM launched successfully, but pgsize value still 2M.
>>>>>
>>>>> There's the interesting "104875" again.
>>>>>
>>>>>
>>>>>> *        Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=104875, launch the VM with ERROR status.
>>>>>
>>>>> 104875
>>>>>
>>>>>> *        Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=2048, launch the VM with success, , pgsize value is
>>>>> 2M, which is expected, ok.
>>>>>> *        Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=small launch the VM with success, , pgsize value is
>>>>> 4k, which is  expected, ok.
>>>>>> *        Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=any launch the VM with success, , pgsize value is
>>>>> 2M, which is  expected, ok.
>>>>>> Could be this an issue trying to use 1G mem pages or do I'm missing
>>>>>> some
>>>>> other configuration parameter?
>>>>>> Thanks in advance for your help
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Starlingx-discuss mailing list
>>>>>> Starlingx-discuss at lists.starlingx.io
>>>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
>>>>>> s

-------------- next part --------------
# nova flavor-key s.f1     set hw:mem_page_size=2048
# nova flavor-key s.f1.1G     set hw:mem_page_size=1048576

# openstack flavor show s.f1 | grep extra
| properties                 | aggregate_instance_extra_specs:storage='local_image', hw:mem_page_size='2048' |

# nova flavor-show s.f1.1G | grep "extra"
| extra_specs                | {"hw:mem_page_size": "1048576", "aggregate_instance_extra_specs:storage": "local_image"} |


# system host-memory-modify -m 2048 -2M 2048 -1G 4 compute-1 0

# system host-memory-show compute-1 -0
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB)     | 13747                                |
|         Platform     (MiB)     | 2048                                 |
|         Available    (MiB)     | 12723                                |
| Huge Pages Configured          | True                                 |
| vSwitch Huge Pages: Size (MiB) | 1024                                 |
|                     Total      | 1                                    |
|                     Available  | 0                                    |
| VM  Pages (4K): Total          | 1158656                              |
| VM  Huge Pages (2M): Total     | 2048                                 |
|                 Total Pending  | 2048                                 |
|                 Available      | 1536                                 |
| VM  Huge Pages (1G): Total     | 4                                    |
|                 Total Pending  | 4                                    |
|                 Available      | 5                                    |

# fm alarm-list | grep compute-0


# launch 2 vms with either flavor
# ...

# nova show compute1-2M | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1 |
| wrs-res:topology                     | node:0,   500MB, pgsize:2M, vcpus:0, unallocated, pol:sha

# nova show shouldbe1G | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1 |
| wrs-res:topology                     | node:0,  1024MB, pgsize:1G, vcpus:0, pol:sha

# system host-memory-show compute-1 0
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB)     | 13742                                |
|         Platform     (MiB)     | 2048                                 |
|         Available    (MiB)     | 11694                                |
| Huge Pages Configured          | True                                 |
| vSwitch Huge Pages: Size (MiB) | 1024                                 |
|                     Total      | 1                                    |
|                     Available  | 0                                    |
| VM  Pages (4K): Total          | 1158656                              |
| VM  Huge Pages (2M): Total     | 2048                                 |
|                 Available      | 1536                                 |
| VM  Huge Pages (1G): Total     | 4                                    |
|                 Available      | 4                                    |


####
# test memory accounting/display
#
# destroy the other VMs and start two more, both with flavor s.f1.1G

# nova show asdf | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1 |
| wrs-res:topology                     | node:0,  1024MB, pgsize:1G, vcpus:0, pol:sha

# nova show qwer | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1 |
| wrs-res:topology                     | node:0,  1024MB, pgsize:1G, vcpus:0, pol:sha

# system host-memory-show compute-1 0
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB)     | 13742                                |
|         Platform     (MiB)     | 2048                                 |
|         Available    (MiB)     | 10670                                |
| Huge Pages Configured          | True                                 |
| vSwitch Huge Pages: Size (MiB) | 1024                                 |
|                     Total      | 1                                    |
|                     Available  | 0                                    |
| VM  Pages (4K): Total          | 1158656                              |
| VM  Huge Pages (2M): Total     | 2048                                 |
|                 Available      | 1536                                 |
| VM  Huge Pages (1G): Total     | 4                                    |
|                 Available      | 3                                    |




More information about the Starlingx-discuss mailing list