[Starlingx-discuss] stx-openstack apply failed after enabling cpu_dedicated_set
Hi, I have enabled cpu on a worker node as per the documentation [1]. There were couple of VMs already created and vGPUs were attached to each VM. After configuring the worker, I just unlocked the worker and noticed that stx-openstack apply failed. Here is the armada log: https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/ Couple of pods were failing. NAME READY STATUS RESTARTS AGE nova-compute-worker-ov-01-cdc7009a-n95w9 1/2 CrashLoopBackOff 18 (94s ago) 72m pci-irq-affinity-agent-2kgds 0/1 Init:0/1 0 24h Collect Logs available at [2]. pci-irq-affinity-agent-2kgds pod log: Error from server (BadRequest): container "pci-irq-affinity-agent" in pod "pci-irq-affinity-agent-2kgds" is waiting to start: PodInitializing [1] https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-share... [2] https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp... Regards, Danishka
container logs of nova-compute-worker-ov-01-cdc7009a pod: https://paste.opendev.org/show/bGrxgZTRBp4SDnG8HHmI/ https://paste.opendev.org/show/b7TNroMVFQRibpPwt5Rc/ On Wed, Jan 4, 2023 at 9:43 PM open infra <openinfradn@gmail.com> wrote:
Hi,
I have enabled cpu on a worker node as per the documentation [1]. There were couple of VMs already created and vGPUs were attached to each VM.
After configuring the worker, I just unlocked the worker and noticed that stx-openstack apply failed. Here is the armada log: https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/
Couple of pods were failing. NAME READY STATUS RESTARTS AGE nova-compute-worker-ov-01-cdc7009a-n95w9 1/2 CrashLoopBackOff 18 (94s ago) 72m pci-irq-affinity-agent-2kgds 0/1 Init:0/1 0 24h
Collect Logs available at [2].
pci-irq-affinity-agent-2kgds pod log: Error from server (BadRequest): container "pci-irq-affinity-agent" in pod "pci-irq-affinity-agent-2kgds" is waiting to start: PodInitializing
[1] https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-share...
[2] https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp...
Regards, Danishka
Hi Danishka, Thanks for raising this issue. Do you mind creating a Launchpad for it though? This way we can centralize any discussion/investigation topics there and it can go through the appropriate triage process so the team can prioritize this work. --- Regards, Thales ________________________________ From: open infra <openinfradn@gmail.com> Sent: Wednesday, January 4, 2023 1:13 PM To: StarlingX ML <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] stx-openstack apply failed after enabling cpu_dedicated_set CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi, I have enabled cpu on a worker node as per the documentation [1]. There were couple of VMs already created and vGPUs were attached to each VM. After configuring the worker, I just unlocked the worker and noticed that stx-openstack apply failed. Here is the armada log: https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/<https://urldefense.com/v3/__https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdDksFgwL$> Couple of pods were failing. NAME READY STATUS RESTARTS AGE nova-compute-worker-ov-01-cdc7009a-n95w9 1/2 CrashLoopBackOff 18 (94s ago) 72m pci-irq-affinity-agent-2kgds 0/1 Init:0/1 0 24h Collect Logs available at [2]. pci-irq-affinity-agent-2kgds pod log: Error from server (BadRequest): container "pci-irq-affinity-agent" in pod "pci-irq-affinity-agent-2kgds" is waiting to start: PodInitializing [1] https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-shared-cpu-pools-on-hosts.html<https://urldefense.com/v3/__https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-shared-cpu-pools-on-hosts.html__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdBg35YA9$> [2] https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp=sharing<https://urldefense.com/v3/__https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp=sharing__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdONVN0BR$> Regards, Danishka
Hi Thales, I have created a LP. https://bugs.launchpad.net/starlingx/+bug/2002157 Regards, Danishka On Fri, Jan 6, 2023 at 6:48 PM Cervi, Thales Elero < ThalesElero.Cervi@windriver.com> wrote:
Hi Danishka,
Thanks for raising this issue. Do you mind creating a Launchpad for it though? This way we can centralize any discussion/investigation topics there and it can go through the appropriate triage process so the team can prioritize this work.
--- Regards, Thales ------------------------------ *From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, January 4, 2023 1:13 PM *To:* StarlingX ML <starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] stx-openstack apply failed after enabling cpu_dedicated_set
* CAUTION: This email comes from a non Wind River email account!* Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi,
I have enabled cpu on a worker node as per the documentation [1]. There were couple of VMs already created and vGPUs were attached to each VM.
After configuring the worker, I just unlocked the worker and noticed that stx-openstack apply failed. Here is the armada log: https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/ <https://urldefense.com/v3/__https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdDksFgwL$>
Couple of pods were failing. NAME READY STATUS RESTARTS AGE nova-compute-worker-ov-01-cdc7009a-n95w9 1/2 CrashLoopBackOff 18 (94s ago) 72m pci-irq-affinity-agent-2kgds 0/1 Init:0/1 0 24h
Collect Logs available at [2].
pci-irq-affinity-agent-2kgds pod log: Error from server (BadRequest): container "pci-irq-affinity-agent" in pod "pci-irq-affinity-agent-2kgds" is waiting to start: PodInitializing
[1] https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-share... <https://urldefense.com/v3/__https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-shared-cpu-pools-on-hosts.html__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdBg35YA9$>
[2] https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp... <https://urldefense.com/v3/__https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp=sharing__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdONVN0BR$>
Regards, Danishka
Brilliant Danishka, thanks for raising it. We will handle it via LP from now on, probably will discuss the priority in the next planning or community call after some triage. --- Regards, Thales ________________________________ From: open infra <openinfradn@gmail.com> Sent: Friday, January 6, 2023 12:17 PM To: Cervi, Thales Elero <ThalesElero.Cervi@windriver.com> Cc: StarlingX ML <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] stx-openstack apply failed after enabling cpu_dedicated_set CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Thales, I have created a LP. https://bugs.launchpad.net/starlingx/+bug/2002157<https://urldefense.com/v3/__https://bugs.launchpad.net/starlingx/*bug/2002157__;Kw!!AjveYdw8EvQ!fTCFk4CBrQ-lUY8N2oUIa2GLGJF3CgBsv2d4C5dYSOnoYYneZUHFnFNUTBzDIcllzV5gmqts59YZ11MVX28cvDDrXOhX$> Regards, Danishka On Fri, Jan 6, 2023 at 6:48 PM Cervi, Thales Elero <ThalesElero.Cervi@windriver.com<mailto:ThalesElero.Cervi@windriver.com>> wrote: Hi Danishka, Thanks for raising this issue. Do you mind creating a Launchpad for it though? This way we can centralize any discussion/investigation topics there and it can go through the appropriate triage process so the team can prioritize this work. --- Regards, Thales ________________________________ From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Wednesday, January 4, 2023 1:13 PM To: StarlingX ML <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] stx-openstack apply failed after enabling cpu_dedicated_set CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi, I have enabled cpu on a worker node as per the documentation [1]. There were couple of VMs already created and vGPUs were attached to each VM. After configuring the worker, I just unlocked the worker and noticed that stx-openstack apply failed. Here is the armada log: https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/<https://urldefense.com/v3/__https://paste.opendev.org/show/b4z1jLSPbeEZDXyvFeBV/__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdDksFgwL$> Couple of pods were failing. NAME READY STATUS RESTARTS AGE nova-compute-worker-ov-01-cdc7009a-n95w9 1/2 CrashLoopBackOff 18 (94s ago) 72m pci-irq-affinity-agent-2kgds 0/1 Init:0/1 0 24h Collect Logs available at [2]. pci-irq-affinity-agent-2kgds pod log: Error from server (BadRequest): container "pci-irq-affinity-agent" in pod "pci-irq-affinity-agent-2kgds" is waiting to start: PodInitializing [1] https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-shared-cpu-pools-on-hosts.html<https://urldefense.com/v3/__https://docs.starlingx.io/admintasks/openstack/configure-dedicated-and-shared-cpu-pools-on-hosts.html__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdBg35YA9$> [2] https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp=sharing<https://urldefense.com/v3/__https://drive.google.com/drive/folders/1YxuWOlkoHBsdg9UUuJ0ozZ2maxXcWzX-?usp=sharing__;!!AjveYdw8EvQ!aIoLD5HbVTJ8KlCewpFQL8xgZHumTW4124Y1ufHn3vKfqyge91zxEsxGkWx1uY7K1YYmLHCnnk4m_464k8_3a9pFdONVN0BR$> Regards, Danishka
participants (2)
-
Cervi, Thales Elero
-
open infra