Hey, any other recommendations?

On 2023-03-23, Thu at 16:25, voipas <voipas@gmail.com> wrote:
Hey Douglas,

  Thanks for your response. I recreated a new VM with 24 GB RAM - again after installation, Kubelet is still not launching... Also, attaching disk layout, just in case we have sufficient space
NAME                     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                        8:0    0   520G  0 disk
|-sda1                     8:1    0     1M  0 part
|-sda2                     8:2    0  29.3G  0 part /var/rootdirs/opt/platform-backup
|-sda3                     8:3    0   300M  0 part /boot/efi
|-sda4                     8:4    0     2G  0 part /boot
`-sda5                     8:5    0 488.4G  0 part
  |-cgts--vg-root--lv    253:0    0    20G  0 lvm  /sysroot
  |-cgts--vg-var--lv     253:1    0    20G  0 lvm  /var
  |-cgts--vg-log--lv     253:2    0   7.8G  0 lvm  /var/log
  `-cgts--vg-scratch--lv 253:3    0  15.6G  0 lvm  /var/rootdirs/scratch
sr0                       11:0    1  1024M  0 rom
nvme0n1                  259:0    0    50G  0 disk


I see these kind of errors in daemon log:

2023-03-23T13:58:51.020 localhost systemd-tmpfiles[454]: notice /etc/tmpfiles.d/kubernetes.conf:1: Line references path below legacy directory /var/run/, updating /var/run/kubernetes <E2><86><92> /run/kubernetes; please update the tmpfiles.d/ drop-in file accordingly.
2023-03-23T13:58:51.020 localhost systemd-modules-load[438]: info Inserted module 'ib_cm'
2023-03-23T13:58:51.020 localhost systemd-modules-load[438]: info Inserted module 'ib_ucm'
2023-03-23T13:58:51.020 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.020 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.020 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.020 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.021 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.021 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.021 localhost systemd-tmpfiles[454]: warning Failed to parse ACL "group:sys_protected:r--,group:wheel:r--": No such file or directory. Ignoring
2023-03-23T13:58:51.021 localhost systemd[1]: info Finished Create Static Device Nodes in /dev.
2023-03-23T13:58:51.021 localhost systemd[1]: info Starting Rule-based Manager for Device Events and Files...
2023-03-23T13:58:51.021 localhost systemd-modules-load[438]: info Inserted module 'ib_uverbs'
2023-03-23T13:58:51.021 localhost systemd-modules-load[438]: info Inserted module 'iw_cm'
2023-03-23T13:58:51.021 localhost systemd-modules-load[438]: info Inserted module 'rdma_cm'
2023-03-23T13:58:51.021 localhost systemd-modules-load[438]: info Inserted module 'rdma_ucm'
2023-03-23T13:58:51.021 localhost systemd-udevd[459]: err /usr/lib/udev/rules.d/95-ceph-osd-lvm.rules:8 Unknown user 'ceph', ignoring
2023-03-23T13:58:51.022 localhost systemd-udevd[459]: err /usr/lib/udev/rules.d/95-ceph-osd-lvm.rules:8 Unknown group 'ceph', ignoring
2023-03-23T13:58:51.022 localhost systemd-udevd[459]: err /usr/lib/udev/rules.d/95-ceph-osd-lvm.rules:13 Unknown user 'ceph', ignoring
2023-03-23T13:58:51.022 localhost systemd-udevd[459]: err /usr/lib/udev/rules.d/95-ceph-osd-lvm.rules:13 Unknown group 'ceph', ignoring
2023-03-23T13:58:51.022 localhost systemd[1]: info Started Rule-based Manager for Device Events and Files.

2023-03-23T13:58:51.022 localhost systemd[1]: info Starting Apply Kernel Variables...
2023-03-23T13:58:51.022 localhost systemd-sysctl[482]: info Couldn't write '20' to 'fs/negative-dentry-limit', ignoring: No such file or directory
2023-03-23T13:58:51.022 localhost systemd[1]: info Finished Apply Kernel Variables.

2023-03-23T13:58:51.022 localhost systemd-udevd[474]: info Using interface naming scheme 'vSTX7_0'.
2023-03-23T13:58:51.022 localhost systemd-udevd[474]: info ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
2023-03-23T13:58:51.022 localhost systemd-udevd[463]: info ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: notice /etc/tmpfiles.d/kubernetes.conf:1: Line references path below legacy directory /var/run/, updating /var/run/kubernetes <E2><86><92> /run/kubernetes; please update the tmpfiles.d/ drop-in file accordingly.

2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd-tmpfiles[760]: warning Failed to parse ACL "group:sys_protected:r--,group:wheel:r--": No such file or directory. Ignoring
2023-03-23T13:58:51.024 localhost systemd[1]: info Finished Create Volatile Files and Directories.

2023-03-23T13:58:51.025 localhost systemd[1]: info Started Kubernetes Kubelet Server.
2023-03-23T13:58:51.025 localhost polkitd[801]: info started daemon version 0.105 using authority implementation `local' version `0.105'
2023-03-23T13:58:51.025 localhost systemd[863]: info kubelet.service: Failed to locate executable /usr/bin/kubelet: No such file or directory
2023-03-23T13:58:51.025 localhost systemd[863]: err kubelet.service: Failed at step EXEC spawning /usr/bin/kubelet: No such file or directory
2023-03-23T13:58:51.025 localhost systemd[1]: info Starting Kubernetes Isolated CPU Plugin Daemon...

2023-03-23T13:59:10.551 localhost controller_config[1459]: info Pausing for 5 seconds...
2023-03-23T13:59:14.605 localhost lldpd[998]: info removal request for address of fe80::a00:27ff:fe85:8445%2, but no knowledge of it
2023-03-23T13:59:14.842 localhost lldpd[998]: info removal request for address of fe80::a00:27ff:fe4f:69bd%3, but no knowledge of it
2023-03-23T13:59:15.565 localhost systemd[1]: notice controllerconfig.service: Main process exited, code=exited, status=1/FAILURE
2023-03-23T13:59:15.565 localhost systemd[1]: warning controllerconfig.service: Failed with result 'exit-code'.
2023-03-23T13:59:15.579 localhost systemd[1]: info Finished General StarlingX config gate.
2023-03-23T13:59:15.581 localhost systemd[1]: info Starting StarlingX Maintenance Filesystem Monitor...
2023-03-23T13:59:15.583 localhost systemd[1]: info Started Getty on tty1.
2023-03-23T13:59:15.584 localhost systemd[1]: info Reached target Login Prompts.
2023-03-23T13:59:15.586 localhost systemd[1]: info Starting StarlingX Maintenance Worker Goenable Ready...
2023-03-23T13:59:15.587 localhost systemd[1]: info Starting StarlingX Maintenance Goenable Ready...
2023-03-23T13:59:15.588 localhost systemd[1]: info Starting StarlingX Maintenance Heartbeat Client...
2023-03-23T13:59:15.589 localhost systemd[1]: info Starting Starling-X Maintenance Link Monitor...
2023-03-23T13:59:15.590 localhost systemd[1]: info Starting StarlingX Maintenance Alarm Handler Client...
2023-03-23T13:59:15.591 localhost systemd[1]: info Starting StarlingX Maintenance Logger...
2023-03-23T13:59:15.593 localhost systemd[1]: info Starting StarlingX Pxeboot Feed Refresh...
2023-03-23T13:59:15.594 localhost systemd[1]: info Starting Service Management Unit...
2023-03-23T13:59:15.597 localhost systemd[1]: info Finished StarlingX Maintenance Worker Goenable Ready.
2023-03-23T13:59:15.610 localhost goenabled[1504]: info Goenabled Ready: [  OK  ]
2023-03-23T13:59:15.610 localhost systemd[1]: info Finished StarlingX Maintenance Goenable Ready.
2023-03-23T13:59:15.630 localhost lmon[1507]: info Starting lmond: OK
2023-03-23T13:59:15.630 localhost systemd[1]: info lmon.service: Can't open PID file /run/lmond.pid (yet?) after start: Operation not permitted
2023-03-23T13:59:15.633 localhost mtclog[1509]: info Starting mtclogd: OK
2023-03-23T13:59:15.634 localhost hbsClient[1506]: info Starting hbsClient: OK
2023-03-23T13:59:15.635 localhost fsmon[1501]: info Starting fsmond: OK
2023-03-23T13:59:15.636 localhost systemd[1]: info mtclog.service: Can't open PID file /run/mtclogd.pid (yet?) after start: Operation not permitted
2023-03-23T13:59:15.636 localhost systemd[1]: info hbsClient.service: Can't open PID file /run/hbsClient.pid (yet?) after start: Operation not permitted
2023-03-23T13:59:15.637 localhost systemd[1]: info fsmon.service: Can't open PID file /run/fsmond.pid (yet?) after start: Operation not permitted
2023-03-23T13:59:15.637 localhost mtcalarm[1508]: info Starting mtcalarmd: OK
2023-03-23T13:59:15.639 localhost systemd[1]: info mtcalarm.service: Can't open PID file /run/mtcalarmd.pid (yet?) after start: Operation not permitted


2023-03-23T14:03:53.832 localhost affine-tasks.sh(1218): info : Recovery wait, elapsed 301 seconds. Reason: k8s-infra not configured
2023-03-23T14:08:00.073 localhost avahi-daemon[796]: info Joining mDNS multicast group on interface enp0s3.IPv4 with address 10.0.1.3.
2023-03-23T14:08:00.074 localhost avahi-daemon[796]: info New relevant interface enp0s3.IPv4 for mDNS.
2023-03-23T14:08:00.074 localhost avahi-daemon[796]: info Registering new address record for 10.0.1.3 on enp0s3.IPv4.
2023-03-23T14:08:01.560 localhost avahi-daemon[796]: info Joining mDNS multicast group on interface enp0s3.IPv6 with address fe80::a00:27ff:fe85:8445.
2023-03-23T14:08:01.561 localhost avahi-daemon[796]: info New relevant interface enp0s3.IPv6 for mDNS.
2023-03-23T14:08:01.561 localhost avahi-daemon[796]: info Registering new address record for fe80::a00:27ff:fe85:8445 on enp0s3.*.
2023-03-23T14:08:54.436 localhost affine-tasks.sh(1218): info : Recovery wait, elapsed 602 seconds. Reason: k8s-infra not configured
2023-03-23T14:11:06.065 localhost lldpd[998]: info removal request for address of fe80::a00:27ff:fe4f:69bd%3, but no knowledge of it
2023-03-23T14:13:37.869 localhost systemd[1]: info Starting Cleanup of Temporary Directories...
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: notice /etc/tmpfiles.d/kubernetes.conf:1: Line references path below legacy directory /var/run/, updating /var/run/kubernetes <E2><86><92> /run/kubernetes; please update the tmpfiles.d/ drop-in file accor
dingly.
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "d:group:sys_protected:r-x,d:group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "group:sys_protected:r-x,group:wheel:r-x": No such file or directory. Ignoring
2023-03-23T14:13:37.874 localhost systemd-tmpfiles[2092]: warning Failed to parse ACL "group:sys_protected:r--,group:wheel:r--": No such file or directory. Ignoring
2023-03-23T14:13:37.912 localhost systemd[1]: info systemd-tmpfiles-clean.service: Succeeded.
2023-03-23T14:13:37.912 localhost systemd[1]: info Finished Cleanup of Temporary Directories.
2023-03-23T14:13:55.158 localhost affine-tasks.sh(1218): info : Recovery wait, elapsed 903 seconds. Reason: k8s-infra not configured
2023-03-23T14:18:55.627 localhost affine-tasks.sh(1218): info : Recovery wait, elapsed 1203 seconds. Reason: k8s-infra not configured
2023-03-23T14:22:57.658 localhost lldpd[998]: info removal request for address of fe80::a00:27ff:fe4f:69bd%3, but no knowledge of it

On Thu, Mar 23, 2023 at 3:06 PM Pereira, Douglas <Douglas.Pereira@windriver.com> wrote:

Hi Giedrius,

 

Have you tried increasing the VM memory? The documentation suggests 20480 MB for the AIO-SX configuration and you are using only 16GB.

 

Regards,

Doug

 

From: voipas <voipas@gmail.com>
Sent: Wednesday, March 22, 2023 2:40 PM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] failing to deploy Simplex Starlingx AIO on VirtualBox

 

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hello colleagues,

 

  I need your support here. Simplex Starlingx AIO installation fails on first steps... 

  • After installation and reboot I see that kubelet.service - Kubernetes Kubelet Server Failed. Not sure if it is normal or not at this phase... See more details below
  • Bootstrapping failed - Failed to provision initial system configuration.

 

I'm trying to install Starlingx on my Intel Nuc box (i5, 64 GB RAM, 2 TB disk) with Ubuntu Desktop OS. VirtualBox version 6.1

 

VM configuration:

  • 8 vCPU (VT-X/AMD-V, Nested Paging, PAE/NX, KVM Paravirtualization)
  • 16 GB RAM
  • Storage:
    • Controller SATA 520 GB
    • Controller NVMe 20 GB
  • Network:
    • Intel Pro/1000 MT Desktop - OAM network (internet accessible)
    •  
    • Intel Pro/1000 MT Desktop - Data network (internet accessible)

 

 

So I wonder what is wrong with this deployment , am I missing something?

 

 

Kubelet failure (/var/log/daemon.log):

2023-03-21T19:35:19.876 localhost systemd[1]: info Started StarlingX Affine Tasks.
2023-03-21T19:35:19.968 localhost iscsid: info iSCSI daemon with pid=912 started!
2023-03-21T19:35:20.057 localhost affine-tasks.sh(1211): info : Starting.
2023-03-21T19:35:20.058 localhost affine-tasks.sh(1211): info : Affine all tasks, CPUS: 0-7; online=0-7 (0xff), isol=, nonisol=0-7 (0xff)
2023-03-21T19:35:20.128 localhost affine-tasks.sh(1211): info : Affined 58 processes to all cores.
2023-03-21T19:35:20.302 localhost systemd[1]: info kubelet.service: Scheduled restart job, restart counter is at 5.
2023-03-21T19:35:20.303 localhost systemd[1]: info Stopping Kubernetes Isolated CPU Plugin Daemon...
2023-03-21T19:35:20.304 localhost systemd[1]: info isolcpu_plugin.service: Succeeded.
2023-03-21T19:35:20.305 localhost systemd[1]: info Stopped Kubernetes Isolated CPU Plugin Daemon.
2023-03-21T19:35:20.306 localhost systemd[1]: info Stopped Kubernetes Kubelet Server.
2023-03-21T19:35:20.306 localhost systemd[1]: warning kubelet.service: Start request repeated too quickly.
2023-03-21T19:35:20.306 localhost systemd[1]: warning kubelet.service: Failed with result 'exit-code'.
2023-03-21T19:35:20.306 localhost systemd[1]: err Failed to start Kubernetes Kubelet Server.
2023-03-21T19:35:20.308 localhost systemd[1]: warning Dependency failed for Kubernetes Isolated CPU Plugin Daemon.
2023-03-21T19:35:20.309 localhost systemd[1]: notice isolcpu_plugin.service: Job isolcpu_plugin.service/start failed with result 'dependency'.
2023-03-21T19:35:20.514 localhost sysinv-agent[1012]: info /etc/init.d/sysinv-agent: line 114: [: =: unary operator expected

 

 

Bootstrap failure:

TASK [bootstrap/persist-config : Fail if populate config script throws an exception] *********************************************************************************************************************************************************************************
Wednesday 22 March 2023  17:29:05 +0000 (0:00:00.024)       0:01:40.002 *******
fatal: [localhost]: FAILED! => changed=false
  msg: Failed to provision initial system configuration.

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=180  changed=45   unreachable=0    failed=1    skipped=235  rescued=0    ignored=0

2023-03-22 17:29:05,960 p=323063 u=sysadmin n=ansible | TASK [bootstrap/persist-config : debug] **********************************************************************************************************************************************************************
********************************************************
2023-03-22 17:29:05,960 p=323063 u=sysadmin n=ansible | Wednesday 22 March 2023  17:29:05 +0000 (0:00:06.932)       0:01:39.978 *******
2023-03-22 17:29:05,981 p=323063 u=sysadmin n=ansible | ok: [localhost] =>
  populate_result:
    changed: true
    failed: false
    failed_when_result: false
    msg: non-zero return code
    rc: 1
    stderr: |-
      Traceback (most recent call last):
        File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1679506139.0395112-327066-74144393708544/populate_initial_config.py", line 1327, in <module>
          populate_service_parameter_config(client)
        File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1679506139.0395112-327066-74144393708544/populate_initial_config.py", line 1046, in populate_service_parameter_config
          populate_docker_kube_config(client)
        File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1679506139.0395112-327066-74144393708544/populate_initial_config.py", line 838, in populate_docker_kube_config
          client.sysinv.service_parameter.delete(parameter.uuid)
        File "/usr/lib/python3/dist-packages/cgtsclient/v1/service_parameter.py", line 45, in delete
          return self._delete(self._path(parameter_id))
        File "/usr/lib/python3/dist-packages/cgtsclient/common/base.py", line 95, in _delete
          self.api.raw_request('DELETE', url)
        File "/usr/lib/python3/dist-packages/cgtsclient/common/http.py", line 224, in raw_request
          return self._http_request(url, method, **kwargs)
        File "/usr/lib/python3/dist-packages/cgtsclient/common/http.py", line 186, in _http_request
          raise exceptions.from_response(
      cgtsclient.exc.HTTPInternalServerError: 'int' object is not callable
    stderr_lines:
    - 'Traceback (most recent call last):'
    - '  File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1679506139.0395112-327066-74144393708544/populate_initial_config.py", line 1327, in <module>'
    - '    populate_service_parameter_config(client)'
    - '  File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1679506139.0395112-327066-74144393708544/populate_initial_config.py", line 1046, in populate_service_parameter_config'
    - '    populate_docker_kube_config(client)'
    - '  File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1679506139.0395112-327066-74144393708544/populate_initial_config.py", line 838, in populate_docker_kube_config'
    - '    client.sysinv.service_parameter.delete(parameter.uuid)'
    - '  File "/usr/lib/python3/dist-packages/cgtsclient/v1/service_parameter.py", line 45, in delete'
    - '    return self._delete(self._path(parameter_id))'
    - '  File "/usr/lib/python3/dist-packages/cgtsclient/common/base.py", line 95, in _delete'
    - '    self.api.raw_request(''DELETE'', url)'
    - '  File "/usr/lib/python3/dist-packages/cgtsclient/common/http.py", line 224, in raw_request'
    - '    return self._http_request(url, method, **kwargs)'
    - '  File "/usr/lib/python3/dist-packages/cgtsclient/common/http.py", line 186, in _http_request'
    - '    raise exceptions.from_response('
    - 'cgtsclient.exc.HTTPInternalServerError: ''int'' object is not callable'
    stdout: |-
      Updating system config...
      System config completed.
      Deleting network, routes, addresses, and address pool for network mgmt...
      Updating management network...
      Deleting network, routes, addresses, and address pool for network pxeboot...
      Updating pxeboot network...
      Deleting network, routes, addresses, and address pool for network oam...
      Updating oam network...
      Deleting network, routes, addresses, and address pool for network multicast...
      Updating multicast network...
      Deleting network, routes, addresses, and address pool for network cluster-host...
      Updating cluster host network...
      Deleting network, routes, addresses, and address pool for network cluster-pod...
      Updating cluster pod network...
      Deleting network, routes, addresses, and address pool for network cluster-service...
      Updating cluster service network...
      Network config completed.
      Populating/Updating DNS config...
      DNS config completed.

 

Thanks in advance

 

--

Best Regards,
Giedrius



--
Best Regards,
Giedrius
--
Best Regards,
Giedrius