[Starlingx-discuss] Question - Using 1Gb mem pages
Michel Thebeau
michel.thebeau at windriver.com
Mon Oct 22 17:09:31 UTC 2018
Hi Ricardo,
That's cool.
I had added cpu flag 'pdpe1gb' to compute under-virtualization,
otherwise as persented by the virtualization wiki, to see if it would work:
<cpu match='exact'>
<model fallback='forbid'>Nehalem</model>
<topology sockets='1' cores='4' threads='1'/>
<feature policy='require' name='vmx'/>
+ <feature policy='require' name='pdpe1gb'/>
<feature policy='optional' name='svm'/>
</cpu>
<clock offset='utc'/>
And I was able to configure the compute and launch VMs with 1G pages
(per the attached example). ... I found the memory accounting or
display to be a bit wonky though. I'm not sure what to make of that.
I'll mention it to my bosses, but I'm sure my method is not a supported. ;)
M
On 2018-10-22 12:54 p.m., Perez, Ricardo O wrote:
> It's working now:
>
> wrsroot at compute-0 ~(keystone_admin)]$ openstack flavor set m1.small --property hw:mem_page_size=1GB
> [wrsroot at compute-0 ~(keystone_admin)]$ openstack server create --image cirros --flavor m1.small --nic net-id=net3 testvm
> [wrsroot at compute-0 ~(keystone_admin)]$ openstack server show testvm
> +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Field | Value |
> +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | wrs-res:pci_devices | |
> | wrs-res:topology | node:0, 1024MB, pgsize:1G, vcpus:0-3, pol:sha |
> | wrs-res:vcpus | [4, 4, 4] |
> | wrs-sg:server_group | |
> +-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
>
>
> Thanks !
> -Richo
>
>> -----Original Message-----
>> From: Michel Thebeau [mailto:michel.thebeau at windriver.com]
>> Sent: Monday, October 22, 2018 11:18 AM
>> To: Perez, Ricardo O <ricardo.o.perez at intel.com>; starlingx-
>> discuss at lists.starlingx.io
>> Subject: Re: [Starlingx-discuss] Question - Using 1Gb mem pages
>>
>> Hi Ricardo,
>>
>> I was able to select 4K and 2M pages in the following way (attached txt file). I'll
>> try to see if I can get a system up with 1G pages, and I'll follow up with you.
>>
>>
>>
>> M
>>
>>
>> On 2018-10-22 10:59 a.m., Michel Thebeau wrote:
>>> Hi Ricardo,
>>>
>>> The documentation indicates to configure the the huge pages per numa
>>> node on the compute, and to configure the flavour extra specs for the
>>> VM. I am not aware of anything else in particular. There's some tips
>>> and caveats in the document, but I don't see anything in your message
>>> to think of those.
>>>
>>> If there's any chance you are trying to do this under virtualization,
>>> I am not aware that that is currently supported.
>>>
>>> M
>>>
>>>
>>> On 2018-10-19 6:14 p.m., Perez, Ricardo O wrote:
>>>> Hi Michael,
>>>>
>>>> Thanks for your answer, I just forgot to put the computes information
>>>> about memory configuration, here is the info:
>>>>
>>>> Compute-0
>>>>
>>>> VM Pages
>>>> 0 Reserved for Platform: 8000 MiB
>>>> Usable Total: 86387 MiB
>>>> Available: 84787 MiB
>>>> 4K Pages:
>>>> Total: 230656
>>>> 2M Hugepages:
>>>> Total: 1271
>>>> Available: 983
>>>> 1G Hugepages:
>>>> Total: 80
>>>> Available: 80
>>>>
>>>> 1 Reserved for Platform: 2000 MiB
>>>> Usable Total: 93698 MiB
>>>> Available: 92674 MiB
>>>> 4K Pages:
>>>> Total: 29696
>>>> 2M Hugepages:
>>>> Total: 5319
>>>> Available: 5319
>>>> 1G Hugepages:
>>>> Total: 80
>>>> Available: 80
>>>>
>>>> ***********************************
>>>> Compute-1
>>>> Processor
>>>> Memory
>>>> VM Pages
>>>> 0 Reserved for Platform: 8000 MiB
>>>> Usable Total: 86504 MiB
>>>> Available: 84968 MiB
>>>> 4K Pages:
>>>> Total: 0
>>>> 2M Hugepages:
>>>> Total: 1780
>>>> Available: 1524
>>>> 1G Hugepages:
>>>> Total: 80
>>>> Available: 80
>>>>
>>>> 1 Reserved for Platform: 2000 MiB
>>>> Usable Total: 93552 MiB
>>>> Available: 92528 MiB
>>>> 4K Pages:
>>>> Total: 0
>>>> 2M Hugepages:
>>>> Total: 5304
>>>> Available: 5304
>>>> 1G Hugepages:
>>>> Total: 80
>>>> Available: 80
>>>>
>>>> Am I missing something else to made use of 1G memory pages ?
>>>>
>>>> Thanks
>>>> -Richo
>>>>
>>>>> -----Original Message-----
>>>>> From: Michel Thebeau [mailto:michel.thebeau at windriver.com]
>>>>> Sent: Friday, October 19, 2018 1:39 PM
>>>>> To: Perez, Ricardo O <ricardo.o.perez at intel.com>; starlingx-
>>>>> discuss at lists.starlingx.io
>>>>> Subject: Re: [Starlingx-discuss] Question - Using 1Gb mem pages
>>>>>
>>>>> On 2018-10-19 1:45 p.m., Perez, Ricardo O wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to execute a test, which his goal is to made use of huge
>>>>>> pages (2M /
>>>>> 1G) launching a VM.
>>>>>> I have already verified the configuration of the controllers GRUB
>>>>>> options and
>>>>> we have defined the variable hugepagesz=1G, however, inside the same
>>>>> parameters list, there is a "default_hugepagesz=2M". The default
>>>>> hugepagesz, has been modified to 1G, and the compute has been
>>>>> restarted with no issues.
>>>>>
>>>>>
>>>>> Hi Ricardo,
>>>>>
>>>>> You wrote, "controllers GRUB options" then "the compute has been
>>>>> restarted with no issues". Is there a GRUB option one can configure
>>>>> on the controller that is for the computes? I don't remember seeing
>>>>> one. If you're actually changing /etc/default/grub, I doubt that's
>>>>> supported.
>>>>> I did not find a procedure for changing default_hugepagesz on
>>>>> controllers or computes grub configuration.
>>>>>
>>>>> It's supposed to be an option, for example through in horizon
>>>>> interface or 'system host-memory-modify'... but you didn't quote
>>>>> either of those.
>>>>>
>>>>>
>>>>>> The flavor has been modified, with the following option: "
>>>>> hw:mem_page_size=104875" (1G) page.
>>>>>
>>>>>
>>>>> Is "104875" a typo? It looks a lot like 1024 * 1024 = 1048576, but
>>>>> not quite.
>>>>>
>>>>>
>>>>> M
>>>>>
>>>>>
>>>>>> However when I launch a VM using the modified flavor it creates the
>>>>>> VM, but if
>>>>> I check the field values of the VM using " openstack server show
>>>>> <vm_name>".
>>>>> I'm getting the following value for pgsize:
>>>>>> wrs-res:topology | node:0, 512MB, pgsize:2M,
>>>>>> vcpus:0, pol:sha |
>>>>>>
>>>>>> So, seems like it's still taking the 2M pagesize instead of the 1G.
>>>>>>
>>>>>> What I have already tried is to:
>>>>>>
>>>>>> * Change also the image metadata: hw:mem_page_size=large and
>>>>>> also
>>>>> 104875. Result, VM launched successfully, but pgsize value still 2M.
>>>>>
>>>>> There's the interesting "104875" again.
>>>>>
>>>>>
>>>>>> * Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=104875, launch the VM with ERROR status.
>>>>>
>>>>> 104875
>>>>>
>>>>>> * Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=2048, launch the VM with success, , pgsize value is
>>>>> 2M, which is expected, ok.
>>>>>> * Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=small launch the VM with success, , pgsize value is
>>>>> 4k, which is expected, ok.
>>>>>> * Leave the GRUB default values, and use modified flavor to
>>>>> hw:mem_page_size=any launch the VM with success, , pgsize value is
>>>>> 2M, which is expected, ok.
>>>>>> Could be this an issue trying to use 1G mem pages or do I'm missing
>>>>>> some
>>>>> other configuration parameter?
>>>>>> Thanks in advance for your help
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Starlingx-discuss mailing list
>>>>>> Starlingx-discuss at lists.starlingx.io
>>>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discus
>>>>>> s
-------------- next part --------------
# nova flavor-key s.f1 set hw:mem_page_size=2048
# nova flavor-key s.f1.1G set hw:mem_page_size=1048576
# openstack flavor show s.f1 | grep extra
| properties | aggregate_instance_extra_specs:storage='local_image', hw:mem_page_size='2048' |
# nova flavor-show s.f1.1G | grep "extra"
| extra_specs | {"hw:mem_page_size": "1048576", "aggregate_instance_extra_specs:storage": "local_image"} |
# system host-memory-modify -m 2048 -2M 2048 -1G 4 compute-1 0
# system host-memory-show compute-1 -0
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB) | 13747 |
| Platform (MiB) | 2048 |
| Available (MiB) | 12723 |
| Huge Pages Configured | True |
| vSwitch Huge Pages: Size (MiB) | 1024 |
| Total | 1 |
| Available | 0 |
| VM Pages (4K): Total | 1158656 |
| VM Huge Pages (2M): Total | 2048 |
| Total Pending | 2048 |
| Available | 1536 |
| VM Huge Pages (1G): Total | 4 |
| Total Pending | 4 |
| Available | 5 |
# fm alarm-list | grep compute-0
# launch 2 vms with either flavor
# ...
# nova show compute1-2M | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |
| wrs-res:topology | node:0, 500MB, pgsize:2M, vcpus:0, unallocated, pol:sha
# nova show shouldbe1G | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |
| wrs-res:topology | node:0, 1024MB, pgsize:1G, vcpus:0, pol:sha
# system host-memory-show compute-1 0
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB) | 13742 |
| Platform (MiB) | 2048 |
| Available (MiB) | 11694 |
| Huge Pages Configured | True |
| vSwitch Huge Pages: Size (MiB) | 1024 |
| Total | 1 |
| Available | 0 |
| VM Pages (4K): Total | 1158656 |
| VM Huge Pages (2M): Total | 2048 |
| Available | 1536 |
| VM Huge Pages (1G): Total | 4 |
| Available | 4 |
####
# test memory accounting/display
#
# destroy the other VMs and start two more, both with flavor s.f1.1G
# nova show asdf | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |
| wrs-res:topology | node:0, 1024MB, pgsize:1G, vcpus:0, pol:sha
# nova show qwer | grep -i "pgsize\|hyper"
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |
| wrs-res:topology | node:0, 1024MB, pgsize:1G, vcpus:0, pol:sha
# system host-memory-show compute-1 0
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| Memory: Usable Total (MiB) | 13742 |
| Platform (MiB) | 2048 |
| Available (MiB) | 10670 |
| Huge Pages Configured | True |
| vSwitch Huge Pages: Size (MiB) | 1024 |
| Total | 1 |
| Available | 0 |
| VM Pages (4K): Total | 1158656 |
| VM Huge Pages (2M): Total | 2048 |
| Available | 1536 |
| VM Huge Pages (1G): Total | 4 |
| Available | 3 |
More information about the Starlingx-discuss
mailing list