October 2020 - Starlingx-discuss

[Starlingx-discuss] [build-report] STX_BUILD_container_setup_layered - Build # 974 - Failure!
by build.starlingx＠gmail.com 07 Oct '20

07 Oct '20

Project: STX_BUILD_container_setup_layered Build #: 974 Status: Failure Timestamp: 20201007T141934Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… -------------------------------------------------------------------------------- Parameters PROJECT: master-containers MY_WORKSPACE: /localdisk/loadbuild/jenkins/master-containers/20201005T230407Z DOCKER_BUILD_ID: jenkins-master-containers-20201005T230407Z-builder PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… DOCKER_BUILD_TAG: master-containers-20201005T230407Z-builder-image PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/containers/20201005T230407Z/logs LAYER: containers MY_REPO_ROOT: /localdisk/designer/jenkins/master-containers

1 0

[Starlingx-discuss] Updated invitation: Weekly StarlingX mtg 7:00am Pacific Timezone @ Weekly from 9am to 10am on Wednesday from Wed Nov 7, 2018 to Tue Oct 6 (CST) (starlingx-discuss@lists.starlingx.io)
by claire＠openstack.org 07 Oct '20

07 Oct '20

This event has been changed. Title: Weekly StarlingX mtg 7:00am Pacific Timezone Agenda & Notes: https://wiki.openstack.org/wiki/StarlingX#Meetings When: Weekly from 9am to 10am on Wednesday from Wed Nov 7, 2018 to Tue Oct 6 Central Time - Chicago (changed) Where: https://zoom.us/j/342730236 Calendar: starlingx-discuss(a)lists.starlingx.io Who: * claire(a)openstack.org - organizer * Ildiko Vancsa * starlingx-discuss(a)lists.starlingx.io Event details: https://www.google.com/calendar/event?action=VIEW&eid=XzZvb2syYzloNjkxMzBiO… Invitation from Google Calendar: https://www.google.com/calendar/ You are receiving this courtesy email at the account starlingx-discuss(a)lists.starlingx.io because you are an attendee of this event. To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar. Forwarding this invitation could allow any recipient to send a response to the organizer and be added to the guest list, or invite others regardless of their own invitation status, or to modify your RSVP. Learn more at https://support.google.com/calendar/answer/37135#forwarding

3 3

Re: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded
by Dharwadkar, Sriram 07 Oct '20

07 Oct '20

Hi Eric, I was able to figure out the issue. Thanks for the pointers. While bringing up data network, I had used the below command for the both the workers as I wanted sriov interfaces. * system host-if-modify $NODE -n sriov0 -c pci-sriov -N 8 --vf-driver=netdevice $DATA0IFUUID I m not sure if this is the reason the why worker nodes were in disabled and offline state. When I changed the class from pci-sriov to “data”, worker nodes are enabled and operational status is available. * system host-if-modify -m 1500 -n data0 -c data worker-0 2ded1706-1cd3-43d3-80d2-c7e0ce2b79c Do you see any problem in first command. Regards, Sriram From: MacDonald, Eric <Eric.MacDonald(a)windriver.com> Sent: Tuesday, October 6, 2020 12:36 AM To: Dharwadkar, Sriram <Sriram.Dharwadkar(a)commscope.com> Subject: Re: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded Sriram, For the worker node configuration failures, look or Error Warn logs in /var/log/puppet/* logs. Those logs should help you understand what config failed. Eric. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ External (eric.macdonald(a)windriver.com<mailto:eric.macdonald@windriver.com>) Report This Email<https://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3NyaXJhbS5kaGFyd2Fka…> FAQ<https://www.inky.com/banner-faq/> Protection by INKY<https://www.inky.com> Sriram, For the worker node configuration failures, look or Error Warn logs in /var/log/puppet/* logs. Those logs should help you understand what config failed. Eric. ________________________________ From: Dharwadkar, Sriram <Sriram.Dharwadkar(a)commscope.com<mailto:Sriram.Dharwadkar@commscope.com>> Sent: Monday, October 5, 2020 2:57 PM To: MacDonald, Eric <Eric.MacDonald(a)windriver.com<mailto:Eric.MacDonald@windriver.com>> Subject: RE: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded Hi Eric, Thanks for the explanation. Initially I was planning to bring up 2 controller (All in one stand alone) + 1 worker node. For this kind of configuration, we would need high capacity h/w for all the nodes which may not be available. That’s the reason I deployed 1 standard controller -with storage + 2 worker nodes configuration. Controller node came up w/o any issues, worker nodes also came up after system host-unlock worker-0 and worker-1. But I could see these alarms * worker-1 experienced a service-affecting failure. Auto-recovery | host=worker-1 | critical | 2020-10-05T | in progress. Manual Lock and Unlock may be required if auto-recovery is unsuccessful * worker-1 experienced a configuration failure. * worker-0 experienced a service-affecting failure. Auto-recovery | host=worker-0 | critical | 2020-10-05T | in progress. Manual Lock and Unlock may be required if auto-recovery is unsuccessful * worker-0 experienced a configuration failure. I did lock and unlock of nodes. After nodes came up, I still see the same issue. How to check for configuration failures? Also there is taint on both the worker nodes “Taints: services=disabled:NoExecute”, which I could see using kubectl describe node worker-0 and kubectl describe node worker-1. So, I don’t see Sriov CNI and Sriov DP being deployed in worker nodes. Regards, Sriram From: MacDonald, Eric <Eric.MacDonald(a)windriver.com<mailto:Eric.MacDonald@windriver.com>> Sent: Monday, October 5, 2020 9:14 PM To: Dharwadkar, Sriram <Sriram.Dharwadkar(a)commscope.com<mailto:Sriram.Dharwadkar@commscope.com>> Subject: Re: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded Sriram, The AIO controller has a lot more work to do during startup as it contains both control and compute functions. As a result, you may temporarily see CPU resource utilization logs during the latter stages of the provisioning. Now that you went to standard config, you might see another degrade for a short period of time following the unlock of the second controller during initial filesystem sync. Again, degrade is general, but if you see a degrade there should be an alarm that represents the reason for the degrade. Eric. ________________________________ From: Dharwadkar, Sriram <Sriram.Dharwadkar(a)commscope.com<mailto:Sriram.Dharwadkar@commscope.com>> Sent: Monday, October 5, 2020 10:44 AM To: MacDonald, Eric <Eric.MacDonald(a)windriver.com<mailto:Eric.MacDonald@windriver.com>> Subject: RE: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded Thanks Eric. I reinstalled the system with iso again. I didn’t see this error again with different configuration (I selected standard controller, instead of all in one duplex). I will check for alarms in case of any issues further. Regards, Sriram From: MacDonald, Eric <Eric.MacDonald(a)windriver.com<mailto:Eric.MacDonald@windriver.com>> Sent: Monday, October 5, 2020 5:55 PM To: Dharwadkar, Sriram <Sriram.Dharwadkar(a)commscope.com<mailto:Sriram.Dharwadkar@commscope.com>> Subject: Re: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded What does 'fm alarm-list' show ? As a general rule ; if a host is degraded there should be an alarm raised for that degraded condition. Eric MacDonald StarlingX Maintenance ________________________________ From: Dharwadkar, Sriram <Sriram.Dharwadkar(a)commscope.com<mailto:Sriram.Dharwadkar@commscope.com>> Sent: Monday, October 5, 2020 4:00 AM To: starlingx-discuss(a)lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss(a)lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded Hi, I have installed distributed starlingx 4.0 in "All in one Duplex" mode. There are two nodes in the central cloud and two in the edge cloud. Central cloud is up and running. For the edge cloud configuration, in the bootstrap override file, I have configured the private registry. From the central cloud, I was able to add the edge cloud. Images required for starlingX installation are downloaded from private registry and installation goes through w/o any issues. [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud list +----+------+------------+--------------+---------------+---------+ | id | name | management | availability | deploy status | sync | +----+------+------------+--------------+---------------+---------+ | 47 | edge | unmanaged | offline | complete | unknown | +----+------+------------+--------------+---------------+---------+ [sysadmin@controller-0 ~(keystone_admin)]$ Then I followed the steps mentioned in the document https://docs.starlingx.io/deploy_install_guides/r4_release/bare_metal/aio_d…<https://secure-web.cisco.com/18J4qBN9kVYaO0J69T77cfP7fEjqwtAkIFtWZbtdqn4HlY…> And finally did unlock of controller-0. System went for reboot and it came up successfully. After I see availability as “degraded” [sysadmin@controller-0 log(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | degraded | +----+--------------+-------------+----------------+-------------+--------------+ Tail -f /var/log/sysinv.log shows – prerequisites not met.. ysinv 2020-10-05 07:57:12.271 96246 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"50634828-68b2-43c4-aaa0-ebf53f6e675a","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":7,"quorum":[0],"quorum_names":["controller"],"monmap":{"epoch":1,"fsid":"50634828-68b2-43c4-aaa0-ebf53f6e675a","modified":"2020-10-05 06:53:11.461060","created":"2020-10-05 06:53:11.461060","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller","addr":"192.168.22.101:6789/0","public_addr":"192.168.22.101:6789/0"}]},"osdmap":{"osdmap":{"epoch":10,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[],"num_pgs":0,"num_pools":0,"num_objects":0,"data_bytes":0,"bytes_used":112181248,"bytes_avail":1197865828352,"bytes_total":1197978009600},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":48,"active_gid":24132,"active_name":"controller-0","active_addr":"192.168.22.102:6804/93283","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":true,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:7999/<https://secure-web.cisco.com/1-3m0_FGRL2G021BGsNLb_f3jpLYppw2gODW7dMb6VvFuo…>"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'140404196232080'} sysinv 2020-10-05 07:57:12.284 96246 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. sysinv 2020-10-05 07:57:12.286 96246 INFO sysinv.conductor.manager [-] Platform managed application oidc-auth-apps: Prerequisites not met. sysinv 2020-10-05 07:57:12.291 96246 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://localhost:30001/nfvi-plugins/v1/sw-update hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:None sysinv 2020-10-05 07:57:12.293 96246 INFO sysinv.ap I’m not sure why availability is shown as degraded. Any help would be appreciated. Let me know if any logs are required. Regards, Sriram

1 0

[Starlingx-discuss] [Distributed StarlingX-4.0] - Controller-0 in edge cloud shows availability as degraded
by Dharwadkar, Sriram 05 Oct '20

05 Oct '20

Hi, I have installed distributed starlingx 4.0 in "All in one Duplex" mode. There are two nodes in the central cloud and two in the edge cloud. Central cloud is up and running. For the edge cloud configuration, in the bootstrap override file, I have configured the private registry. From the central cloud, I was able to add the edge cloud. Images required for starlingX installation are downloaded from private registry and installation goes through w/o any issues. [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud list +----+------+------------+--------------+---------------+---------+ | id | name | management | availability | deploy status | sync | +----+------+------------+--------------+---------------+---------+ | 47 | edge | unmanaged | offline | complete | unknown | +----+------+------------+--------------+---------------+---------+ [sysadmin@controller-0 ~(keystone_admin)]$ Then I followed the steps mentioned in the document https://docs.starlingx.io/deploy_install_guides/r4_release/bare_metal/aio_d… And finally did unlock of controller-0. System went for reboot and it came up successfully. After I see availability as "degraded" [sysadmin@controller-0 log(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | degraded | +----+--------------+-------------+----------------+-------------+--------------+ Tail -f /var/log/sysinv.log shows - prerequisites not met.. ysinv 2020-10-05 07:57:12.271 96246 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"50634828-68b2-43c4-aaa0-ebf53f6e675a","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":7,"quorum":[0],"quorum_names":["controller"],"monmap":{"epoch":1,"fsid":"50634828-68b2-43c4-aaa0-ebf53f6e675a","modified":"2020-10-05 06:53:11.461060","created":"2020-10-05 06:53:11.461060","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller","addr":"192.168.22.101:6789/0","public_addr":"192.168.22.101:6789/0"}]},"osdmap":{"osdmap":{"epoch":10,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[],"num_pgs":0,"num_pools":0,"num_objects":0,"data_bytes":0,"bytes_used":112181248,"bytes_avail":1197865828352,"bytes_total":1197978009600},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":48,"active_gid":24132,"active_name":"controller-0","active_addr":"192.168.22.102:6804/93283","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":true,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:7999/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'140404196232080'} sysinv 2020-10-05 07:57:12.284 96246 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. sysinv 2020-10-05 07:57:12.286 96246 INFO sysinv.conductor.manager [-] Platform managed application oidc-auth-apps: Prerequisites not met. sysinv 2020-10-05 07:57:12.291 96246 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://localhost:30001/nfvi-plugins/v1/sw-update hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:None sysinv 2020-10-05 07:57:12.293 96246 INFO sysinv.ap I'm not sure why availability is shown as degraded. Any help would be appreciated. Let me know if any logs are required. Regards, Sriram

1 0

[Starlingx-discuss] Openstack related question - can I get response on this ticket please
by bnovickovs＠weecodelab.com 04 Oct '20

04 Oct '20

Hi, Can I get response on that ticket please http://lists.starlingx.io/pipermail/starlingx-discuss/2020-September/009679…

1 0

[Starlingx-discuss] [build-report] STX_build_layer_compiler_master_master - Build # 300 - Failure!
by build.starlingx＠gmail.com 03 Oct '20

03 Oct '20

Project: STX_build_layer_compiler_master_master Build #: 300 Status: Failure Timestamp: 20201003T013000Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/20… -------------------------------------------------------------------------------- Parameters FULL_BUILD: false FORCE_BUILD: false

1 0

[Starlingx-discuss] [build-report] STX_publish - Build # 1689 - Failure!
by build.starlingx＠gmail.com 03 Oct '20

03 Oct '20

Project: STX_publish Build #: 1689 Status: Failure Timestamp: 20201003T013108Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/20… -------------------------------------------------------------------------------- Parameters MY_WORKSPACE: /localdisk/loadbuild/jenkins/master-compiler/20201003T013000Z OS: centos PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/20… TIMESTAMP: 20201003T013000Z PUBLISH_INPUTS_BASE: /export/mirror/starlingx/master/centos/compiler/20201003T013000Z/inputs PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/compiler/20201003T013000Z/logs MASTER_JOB_NAME: STX_build_layer_compiler_master_master LAYER: compiler PUBLISH_OUTPUTS_BASE: /export/mirror/starlingx/master/centos/compiler/20201003T013000Z/outputs MY_REPO_ROOT: /localdisk/designer/jenkins/master-compiler PUBLISH_DISTRO_BASE: /export/mirror/starlingx/master/centos/compiler

1 0

[Starlingx-discuss] SRIOV in starlingx
by Sriram 01 Oct '20

01 Oct '20

Hi, I have installed distributed starlingx 4.0 in "All in one Duplex" mode. There are two nodes in the central cloud and two in the edge cloud. I have enabled SRIOV in bios settings of edge cloud nodes and set total VFs as 16. After that, while installing starlingX I followed the steps to enable SRIOV. system host-label-assign controller-0 sriovdp=enabled > system host-memory-modify controller-0 0 -1G 100 > system host-memory-modify controller-0 1 -1G 100 and ran these steps for controller-1 as well. As I understand the first step would label the node "controller-0 and controller-1" as "sriovdp=enabled" and set the number of 1G huge pages to 200. Once the installation was complete, I saw that k8s sriov-device plugin was not coming up. It complained that the resource list was empty. I had to set "/sys/devices/pci0000:3a/0000:3a:00.0/0000:3b:00.0/sriov_numvfs" to 8 (I needed 8 virtual interfaces) and update the resource list in /etc/pcidp/config.json { > "resourceList": [ > { > "resourceName": "bcm_sriov_netdevice", > "selectors": { > "vendors": ["14e4"], > "devices": ["16dc"], > "drivers": ["bnxt_en"], > "pfNames": ["enp59s0f0#0-7"] > } > } > ] > } to see that sriov-dp comes up properly. Is there any way to pass on the number of VF's( sriov_numvfs )required per node and resourcelist during the time of installation when we label the nodes as sriovdp=enabled Regards, Sriram

2 5