After increasing to 6400MiB I see intermittent outage in the stx dashboard and also noticed multiple warnings of 'Multi-Node Recovery Mode' in events. I did lock the host before set reserved memory for the platform and then unlocked the host. On Thu, Feb 18, 2021 at 9:10 PM open infra <openinfradn@gmail.com> wrote:
Thank Liu.
On Thu, Feb 18, 2021 at 8:22 PM Liu, Tao <Tao.Liu@windriver.com> wrote:
Yes, you can modify the host memory reservation via system command.
# system host-memory-modify -m <Platform Reserved MiB> <hostname or id> <processor>
system host-memory-modify -m 6400 1 0
# show current memory configuration
system host-memory-show <hostname or id> <processor>
system host-memory-list <hostname or id>
Tao
*From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, February 17, 2021 9:55 PM *To:* Ildiko Vancsa <ildiko.vancsa@gmail.com> *Cc:* Rai, Ankush <Ankush.Rai@commscope.com>; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Alarm "Memory threshold"
[Please note: This e-mail is from an EXTERNAL e-mail address]
Hi,
Is there a way we can adjust memory and CPU of controller-0 before we deploy starlingx?
In my case without having a single VM under OpenStack has following memory usage though host machine (I use AIO simplex on the virtualized environment) has enough memory.
controller-0:~$ free -m total used free shared buff/cache available Mem: 17894 11131 406 65 6356 5967 Swap: 0 0 0
On Sun, Feb 14, 2021 at 7:55 PM Ildiko Vancsa <ildiko.vancsa@gmail.com> wrote:
Hi,
I saw that you have a ‘reserved for platform’ memory entry with the value of 4600 MiB.
I’ve found entries below that report platform memory usage having an over 100% value:
daemon.log:2021-02-14T07:14:55.000 controller-0 collectd[130887]: info platform memory usage: Usage: 114.7%; Reserved: 4600.0 MiB, Platform: 5274.3 MiB (Base: 4684.1, k8s-system: 590.2), k8s-addon: 0.0
So while the overall memory usage in the system isn’t over the threshold I assume the usage of that reserved amount of memory still exceeds it. Have you seen any configuration option to increase the amount of platform memory?
You can also look into collectd if that leads you closer to what it is reading to get those values.
I don’t have access to a StarlingX install to look into this, so I cannot tell where to look for that.
I found some documentation for Kubernetes to set memory limits, but I’m not sure that applies here. ( https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-... )
Have you looked into the above already?
Thanks, Ildikó
On Feb 14, 2021, at 10:51, Rai, Ankush <Ankush.Rai@commscope.com> wrote:
Not sure exactly which log file to check. Captured some data here, please check if this can help.
Software Version: 20.06 Memory: Reserved for Platform: 4600 MiB Usable Total: 13293 MiB Available: 13293 MiB
The fm has logged these events.
fm-event.log:2021-02-10T08:57:25.000 controller-0 fmManager: info { "event_log_id" : "100.103", "reason_text" : "Platform Memory threshold exceeded ; threshold 80.00%, actual 88.83%", "entity_instance_id" : "region=RegionOne.system=6e232236-df0a-4bc5-9006-76f782b5493f.host=controller-0", "severity" : "major", "state" : "set", "timestamp" : "2021-02-10 08:57:25.484131" } fm-event.log:2021-02-10T09:17:55.000 controller-0 fmManager: info { "event_log_id" : "100.103", "reason_text" : "Platform Memory threshold exceeded ; threshold 90.00%, actual 95.19%", "entity_instance_id" : "region=RegionOne.system=6e232236-df0a-4bc5-9006-76f782b5493f.host=controller-0", "severity" : "critical", "state" : "set", "timestamp" : "2021-02-10 09:17:55.482154" }
[snip]
From: Ildiko Vancsa <ildiko.vancsa@gmail.com> Sent: Sunday, February 14, 2021 2:42 PM To: Rai, Ankush <Ankush.Rai@commscope.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Alarm "Memory threshold"
Hi Ankush,
Do you have any log entries on the system you could share here that
show the memory readings the alarm might be triggered by?
Thanks, Ildikó
On Feb 14, 2021, at 09:26, Rai, Ankush <Ankush.Rai@commscope.com>
wrote:
Hi,
Below alarm is getting raised for every node of the central and edge
cloud.
“Platform Memory threshold exceeded ; threshold 90.00%, actual 95.19%”
It looks to be the false alarm as nodes are having enough available
memory. Please config the root cause of this alarm.
Thanks, Ankush
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io
http://secure-web.cisco.com/1PV8HDMvVRW0RqVu-SKTRWk6WWzv0LMknZ5lZPnq_3AC44IJ...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss