[Starlingx-discuss] Alarm "Memory threshold"

Thu Feb 18 15:55:42 UTC 2021

After increasing to 6400MiB I see intermittent outage in the stx dashboard
and also noticed multiple warnings of 'Multi-Node Recovery Mode' in events.
I did lock the host before set reserved memory for the platform and then
unlocked the host.

On Thu, Feb 18, 2021 at 9:10 PM open infra <openinfradn at gmail.com> wrote:

> Thank Liu.
>
>
>
>
>
> On Thu, Feb 18, 2021 at 8:22 PM Liu, Tao <Tao.Liu at windriver.com> wrote:
>
>> Yes, you can modify the host memory reservation via system command.
>>
>>
>>
>> # system host-memory-modify -m <Platform Reserved MiB> <hostname or id>
>> <processor>
>>
>> system host-memory-modify -m 6400 1 0
>>
>>
>>
>> # show current memory configuration
>>
>> system host-memory-show <hostname or id> <processor>
>>
>> system host-memory-list <hostname or id>
>>
>>
>>
>> Tao
>>
>>
>>
>> *From:* open infra <openinfradn at gmail.com>
>> *Sent:* Wednesday, February 17, 2021 9:55 PM
>> *To:* Ildiko Vancsa <ildiko.vancsa at gmail.com>
>> *Cc:* Rai, Ankush <Ankush.Rai at commscope.com>;
>> starlingx-discuss at lists.starlingx.io
>> *Subject:* Re: [Starlingx-discuss] Alarm "Memory threshold"
>>
>>
>>
>> [Please note: This e-mail is from an EXTERNAL e-mail address]
>>
>> Hi,
>>
>>
>>
>> Is there a way we can adjust memory and CPU of controller-0 before we
>> deploy starlingx?
>>
>> In my case without having a single VM under OpenStack has following
>> memory usage though host machine (I use AIO simplex on the virtualized
>> environment) has enough memory.
>>
>>
>>
>> controller-0:~$ free -m
>>               total        used        free      shared  buff/cache
>> available
>> Mem:          17894       11131         406          65        6356
>>  5967
>> Swap:             0           0           0
>>
>>
>>
>> On Sun, Feb 14, 2021 at 7:55 PM Ildiko Vancsa <ildiko.vancsa at gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I saw that you have a ‘reserved for platform’ memory entry with the value
>> of 4600 MiB.
>>
>> I’ve found entries below that report platform memory usage having an over
>> 100% value:
>>
>> daemon.log:2021-02-14T07:14:55.000 controller-0 collectd[130887]: info
>> platform memory usage: Usage: 114.7%; Reserved: 4600.0 MiB, Platform:
>> 5274.3 MiB (Base: 4684.1, k8s-system: 590.2), k8s-addon: 0.0
>>
>> So while the overall memory usage in the system isn’t over the threshold
>> I assume the usage of that reserved amount of memory still exceeds it. Have
>> you seen any configuration option to increase the amount of platform memory?
>>
>> You can also look into collectd if that leads you closer to what it is
>> reading to get those values.
>>
>> I don’t have access to a StarlingX install to look into this, so I cannot
>> tell where to look for that.
>>
>> I found some documentation for Kubernetes to set memory limits, but I’m
>> not sure that applies here. (
>> https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/
>> )
>>
>> Have you looked into the above already?
>>
>> Thanks,
>> Ildikó
>>
>>
>> > On Feb 14, 2021, at 10:51, Rai, Ankush <Ankush.Rai at commscope.com>
>> wrote:
>> >
>> > Not sure exactly which log file to check. Captured some data here,
>> please check if this can help.
>> >
>> > Software Version: 20.06
>> > Memory:
>> > Reserved for Platform: 4600 MiB
>> > Usable Total: 13293 MiB
>> > Available: 13293 MiB
>> >
>> > The fm has logged these events.
>> >
>> > fm-event.log:2021-02-10T08:57:25.000 controller-0 fmManager: info {
>> "event_log_id" : "100.103", "reason_text" : "Platform Memory threshold
>> exceeded ; threshold 80.00%, actual 88.83%", "entity_instance_id" :
>> "region=RegionOne.system=6e232236-df0a-4bc5-9006-76f782b5493f.host=controller-0",
>> "severity" : "major", "state" : "set", "timestamp" : "2021-02-10
>> 08:57:25.484131" }
>> > fm-event.log:2021-02-10T09:17:55.000 controller-0 fmManager: info {
>> "event_log_id" : "100.103", "reason_text" : "Platform Memory threshold
>> exceeded ; threshold 90.00%, actual 95.19%", "entity_instance_id" :
>> "region=RegionOne.system=6e232236-df0a-4bc5-9006-76f782b5493f.host=controller-0",
>> "severity" : "critical", "state" : "set", "timestamp" : "2021-02-10
>> 09:17:55.482154" }
>> >
>>
>> [snip]
>>
>> >
>> > From: Ildiko Vancsa <ildiko.vancsa at gmail.com>
>> > Sent: Sunday, February 14, 2021 2:42 PM
>> > To: Rai, Ankush <Ankush.Rai at commscope.com>
>> > Cc: starlingx-discuss at lists.starlingx.io
>> > Subject: Re: [Starlingx-discuss] Alarm "Memory threshold"
>> >
>> >
>> > Hi Ankush,
>> >
>> > Do you have any log entries on the system you could share here that
>> show the memory readings the alarm might be triggered by?
>> >
>> > Thanks,
>> > Ildikó
>> >
>> >
>> > > On Feb 14, 2021, at 09:26, Rai, Ankush <Ankush.Rai at commscope.com>
>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > Below alarm is getting raised for every node of the central and edge
>> cloud.
>> > >
>> > > “Platform Memory threshold exceeded ; threshold 90.00%, actual 95.19%”
>> > >
>> > > It looks to be the false alarm as nodes are having enough available
>> memory. Please config the root cause of this alarm.
>> > >
>> > > Thanks,
>> > > Ankush
>> > >
>> > >
>> > > _______________________________________________
>> > > Starlingx-discuss mailing list
>> > > Starlingx-discuss at lists.starlingx.io
>> > >
>> http://secure-web.cisco.com/1PV8HDMvVRW0RqVu-SKTRWk6WWzv0LMknZ5lZPnq_3AC44IJMRoTkCmBzbDzfSRKy1sR51Ro2VO06UwWt-4FdJVml-0HlHOjpHq9J5nmMvsPseu4RcqzNaQvw5haLJsQqt0HLmSNJtbid7y2kxIipLb1hZwBP5gx-ZIdH71ha2sHnU9iy8kbtB51Y5tHpLdGIcUYnJfot6KLcOe6xS2sPGBEOGVhleDZ83q7d7l9kxhO6HkHdmUPURKbTELhCqw8pf-r7fiE8usjs79rMvOl3im3n1pT0kPRXHzHN_DY7SMgFhzOa2MXDTiDCKn-33Ump/http%3A%2F%2Flists.starlingx.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fstarlingx-discuss
>> >
>> >
>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210218/d985ade4/attachment-0001.html>