Will disabling this clean-up service avoid this issue? systemctl disable systemd-tmpfiles-clean.timer Thanks, Ankush From: Wensley, Barton <Barton.Wensley@windriver.com> Sent: Wednesday, March 24, 2021 9:18 PM To: Rai, Ankush <Ankush.Rai@commscope.com>; starlingx-discuss@lists.starlingx.io Subject: RE: unable to add 3ed edge on central-cloud My mistake - the fix you need did not make it into stx.4.0: https://opendev.org/starlingx/distcloud/commit/e8d916756d439cf63eed52f347a2a... The fix will be available in the upcoming stx.5.0 re External (barton.wensley@windriver.com<mailto:barton.wensley@windriver.com>) Report This Email<https://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL2Fua3VzaC5yYWlAY29tbXNjb3BlLmNvbS9kYzgzMzM2NWU2MjAxMjA1NGI2ZGYzNzcxNjZmNTQ0MC8xNjE2NjAwODc3LjY4#key=f6faecde91fbe192e292102edce0a81f> FAQ<https://www.inky.com/banner-faq/> Protection by INKY<https://www.inky.com> My mistake - the fix you need did not make it into stx.4.0: https://opendev.org/starlingx/distcloud/commit/e8d916756d439cf63eed52f347a2a547fc5124cb<https://secure-web.cisco.com/18Nni-eUAIx-l9wQk3zAnso8hjxCFhb8Q9BlphAA7uJsqvKD8zGOz0MFTa0F16KE7Df3u9LkTe2OU4btLuGKmPF1d_4LHMEyTC4nUK1srBnohRNYssK4ZPfVkdHleADWz0tRfXnM_bFPQlyQZAZi5QLCai3PUILdl7DWBEW7nsXvRc69W3top9dxdT4L50A9geFntYy7GUteevSsQeWgjrJ2RBfbPvmpFUZMgUFNiuRA1bUMp_MLk1ZKrwURDed7SoUeO7qxAldVVNeFhhmH-BgsrRvi8jJ_qSsbnodXmTk54RLjcH8T-POT-k8Gw5dB9/https%3A%2F%2Fopendev.org%2Fstarlingx%2Fdistcloud%2Fcommit%2Fe8d916756d439cf63eed52f347a2a547fc5124cb> The fix will be available in the upcoming stx.5.0 release. Bart From: Rai, Ankush <Ankush.Rai@commscope.com<mailto:Ankush.Rai@commscope.com>> Sent: Wednesday, March 24, 2021 11:41 AM To: Wensley, Barton <Barton.Wensley@windriver.com<mailto:Barton.Wensley@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: unable to add 3ed edge on central-cloud [Please note: This e-mail is from an EXTERNAL e-mail address] No it's 4.0 only. 20.06 Thanks Ankush Get Outlook for iOS<https://secure-web.cisco.com/1X2WSa7uNw-o03_A0bVDolfynbOhgObwmUoyAdOHWwvKf9I7qsvsdPQDrZdFf1d0AM7UE8XDOzBziGkU0T5UD1jHdtK-lXVCOJnZ15nmi3m4aKMJ38mtfZq7Ngo1mtJIcMm2u7m5wCK9_wwUFfIxBiw-4VxWHl4JYVXXqHCwfcJjifX_cubluYcblY_-Do8VUoM2B92XvMVJ0pq879E7CnL7UXu0WIiIvdzruxZCB0YO4G-H_8gWtsCwAJPNCqcXeaYsKb-Nla4iyjocUsKljQxBT6djF0zAL-xN5LWjd2ljTrVWHg6-DZXf7rEnL0gck/https%3A%2F%2Faka.ms%2Fo0ukef> ________________________________ From: Wensley, Barton <Barton.Wensley@windriver.com<mailto:Barton.Wensley@windriver.com>> Sent: Wednesday, March 24, 2021 9:09:04 PM To: Rai, Ankush <Ankush.Rai@commscope.com<mailto:Ankush.Rai@commscope.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: unable to add 3ed edge on central-cloud It looks like you may have hit this issue: https://bugs.launchpad.net/starlingx/+bug/1883599<https://secure-web.cisco.com/1aPpINLnbONoKiUQZQ5RPYFkqL5v5_ksBOFPp0JjlWdSeWfkRO9HNvZS17mcVuP4o7qyxuuThVBwGubwqW8rwdZR2Oo3WSx1zAWVoVpJvHKDHiPQNR29CSG8-qeZBsJS-zeS_M51wDT2yB2Lr6vnaKPZqQ4twxIkC8Uwb2L_-ep_9acPxCoPIJelY3sTWW0Zv4N21SlJaZogtTn1KLqXASNFrR1gqSnTM6LZCh6hYidDSMcU2rnwYgX-0nQ9_mjjIIh7FzyL6nljIPMSN79xKwSu4C9g_wdqiMVzn4KpHsKlWmqu98kpz7L3sNGpmwDDS/https%3A%2F%2Fbugs.launchpad.net%2Fstarlingx%2F%2Bbug%2F1883599> I believe this was fixed in stx.4.0 - are you using an earlier release? Bart From: Rai, Ankush <Ankush.Rai@commscope.com<mailto:Ankush.Rai@commscope.com>> Sent: Wednesday, March 24, 2021 2:04 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] unable to add 3ed edge on central-cloud [Please note: This e-mail is from an EXTERNAL e-mail address] Please reply on this. I am stuck here. Thanks, Ankush From: Rai, Ankush Sent: Tuesday, March 23, 2021 5:00 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: unable to add 3ed edge on central-cloud Hi, Sub-cloud addition is failing for 3ed edge. Not understanding what is going wrong here. Please help me to debug the issue. dcmanager subcloud add --bootstrap-address 10.214.5.1 --bootstrap-values $HOME/edge3/bootstrap.yml Enter the sysadmin password for the subcloud: Re-enter sysadmin password to confirm: +-----------------------------+----------------------------+ | Field | Value | +-----------------------------+----------------------------+ | id | 13 | | name | edge3 | | description | StarlingX edge3 | | location | COMMSCOPE_LAB | | software_version | 20.06 | | management | unmanaged | | availability | offline | | deploy_status | not-deployed | | management_subnet | 10.214.6.0/24 | | management_start_ip | 10.214.6.2 | | management_end_ip | 10.214.6.20 | | management_gateway_ip | 10.214.6.254 | | systemcontroller_gateway_ip | 10.214.2.254 | | group_id | 1 | | created_at | 2021-03-23 11:16:56.943669 | | updated_at | None | +-----------------------------+----------------------------+ subcloud list +----+-------+------------+--------------+--------------------+---------+ | id | name | management | availability | deploy status | sync | +----+-------+------------+--------------+--------------------+---------+ | 3 | edge1 | managed | online | complete | in-sync | | 6 | edge | managed | offline | complete | unknown | | 13 | edge3 | unmanaged | offline | deploy-prep-failed | unknown | +----+-------+------------+--------------+--------------------+---------+ ==> /var/log/dcmanager/dcmanager.log <== 2021-03-23 11:16:56.949 118088 INFO eventlet.wsgi.server [req-4dbde86f-545a-4e34-948a-401bf7795450 8a78866e123740abaa939956db6a949b - - default default] 10.214.2.3 "POST /v1.0/subclouds/ HTTP/1.1" status: 200 len: 770 time: 2.0502820 2021-03-23 11:16:56.950 116829 INFO dcmanager.manager.service [req-4dbde86f-545a-4e34-948a-401bf7795450 8a78866e123740abaa939956db6a949b - - default default] Handling add_subcloud request for: edge3 2021-03-23 11:16:56.951 116829 INFO dcmanager.manager.subcloud_manager [req-4dbde86f-545a-4e34-948a-401bf7795450 8a78866e123740abaa939956db6a949b - - default default] Adding subcloud edge3. 2021-03-23 11:16:58.859 116829 INFO dccommon.drivers.openstack.sysinv_v1 [req-4dbde86f-545a-4e34-948a-401bf7795450 8a78866e123740abaa939956db6a949b - - default default] Creating route: interface: 814ed6e8-d700-4140-9c62-0d3d18535664 dest: 10.214.6.0/24 gateway: 10.214.2.254 metric 1 2021-03-23 11:17:05.067 116829 INFO dccommon.drivers.openstack.sysinv_v1 [req-4dbde86f-545a-4e34-948a-401bf7795450 8a78866e123740abaa939956db6a949b - - default default] Creating route: interface: 14e4aec4-2f41-4bc9-9ec9-d704ac460787 dest: 10.214.6.0/24 gateway: 10.214.2.254 metric 1 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager [req-4dbde86f-545a-4e34-948a-401bf7795450 8a78866e123740abaa939956db6a949b - - default default] Failed to create subcloud edge3: ConfigException: File does not exists: /tmp/tmpADel38 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager Traceback (most recent call last): 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/dcmanager/manager/subcloud_manager.py", line 353, in add_subcloud 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager self._create_intermediate_ca_cert(payload) 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/dcmanager/manager/subcloud_manager.py", line 160, in _create_intermediate_ca_cert 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager kube.apply_cert_manager_certificate(CERT_NAMESPACE, cert_name, cert) 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/dccommon/kubeoperator.py", line 112, in apply_cert_manager_certificate 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager custom_object_api = self._get_kubernetesclient_custom_objects() 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/dccommon/kubeoperator.py", line 52, in _get_kubernetesclient_custom_objects 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager self._load_kube_config() 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/dccommon/kubeoperator.py", line 31, in _load_kube_config 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager config.load_kube_config(KUBE_CONFIG_PATH) 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 531, in load_kube_config 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager loader.load_and_set(config) 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 413, in load_and_set 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager self._load_cluster_info() 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 395, in _load_cluster_info 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager file_base_path=self._config_base_path).as_file() 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 116, in as_file 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager raise ConfigException("File does not exists: %s" % self._file) 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager ConfigException: File does not exists: /tmp/tmpADel38 2021-03-23 11:17:12.880 116829 ERROR dcmanager.manager.subcloud_manager 2021-03-23 11:17:20.778 118089 INFO eventlet.wsgi.server [req-59e03ba0-a0ca-4194-9b04-21b62dffe4bd 8a78866e123740abaa939956db6a949b - - default default] 10.214.2.3,10.214.2.3 "GET /v1.0/alarms/ HTTP/1.1" status: 200 len: 728 time: 0.0075271 2021-03-23 11:17:20.785 118088 INFO eventlet.wsgi.server [req-eeb598dd-38d7-4165-af67-f663cf513828 8a78866e123740abaa939956db6a949b - - default default] 10.214.2.3,10.214.2.3 "GET /v1.0/subclouds/ HTTP/1.1" status: 200 len: 3023 time: 0.0140100 +----+-------+-----------------+---------------+------------------+------------+--------------+--------------------+-------------------+---------------------+-------------------+-----------------------+-----------------------------+----------+----------------------------+----------------------------+ | id | name | description | location | software_version | management | availability | deploy_status | management_subnet | management_start_ip | management_end_ip | management_gateway_ip | systemcontroller_gateway_ip | group_id | created_at | updated_at | +----+-------+-----------------+---------------+------------------+------------+--------------+--------------------+-------------------+---------------------+-------------------+-----------------------+-----------------------------+----------+----------------------------+----------------------------+ | 3 | edge1 | StarlingX edge1 | COMMSCOPE_LAB | 20.06 | managed | online | complete | 10.214.4.0/24 | 10.214.4.2 | 10.214.4.20 | 10.214.4.254 | 10.214.2.254 | 1 | 2021-02-24 08:17:01.698138 | 2021-03-12 11:21:57.684272 | | 6 | edge | StarlingX edge1 | COMMSCOPE_LAB | 20.06 | managed | offline | complete | 10.207.7.0/24 | 10.207.7.2 | 10.207.7.20 | 10.207.7.254 | 10.214.2.254 | 1 | 2021-03-03 18:56:52.779913 | 2021-03-23 11:18:51.674364 | | 13 | edge3 | StarlingX edge3 | COMMSCOPE_LAB | 20.06 | unmanaged | offline | deploy-prep-failed | 10.214.6.0/24 | 10.214.6.2 | 10.214.6.20 | 10.214.6.254 | 10.214.2.254 | 1 | 2021-03-23 11:16:56.943669 | 2021-03-23 11:17:12.899548 | +----+-------+-----------------+---------------+------------------+------------+--------------+--------------------+-------------------+---------------------+-------------------+-----------------------+-----------------------------+----------+----------------------------+----------------------------+ cat /home/sysadmin/edge3/bootstrap.yml system_mode: duplex name: "edge3" description: "StarlingX edge3" location: "COMMSCOPE_LAB" management_subnet: 10.214.6.0/24 management_floating_address: 10.214.6.1 management_start_address: 10.214.6.2 management_end_address: 10.214.6.20 management_gateway_address: 10.214.6.254 external_oam_subnet: 10.214.5.0/24 external_oam_gateway_address: 10.214.5.254 external_oam_floating_address: 10.214.5.1 external_oam_start_address: 10.214.5.2 external_oam_end_address: 10.214.5.7 systemcontroller_gateway_address: 10.214.2.254 docker_registries: k8s.gcr.io: url: 10.214.1.1/starlingx quay.io: url: 10.214.1.1/starlingx docker.io: url: 10.214.1.1/starlingx gcr.io: url: 10.214.1.1/starlingx defaults: url: 10.214.1.1/starlingx username: admin password: XYZ ssl_ca_cert: /home/sysadmin/centralharborcert/ca.crt Thanks, Ankush