[Starlingx-discuss] Daily sanity is GREEN but I observed some errors in logs

Mon Jul 13 15:41:11 UTC 2020

Hello everybody,

Related to layered images, even if the daily sanity is GREEN, I observed some errors in /var/log/sysinv.log file. I can tell that I observed this after ussuri was merged.
This is visible in all configurations, I attached only the sysinv.log from Duplex, image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200711T013416Z/
I don't know if this errors are important or not, probably some of you already know about them, but I decided to ask...

The observed errors are:

  1.
ing again in 1 seconds.: error: [Errno 111] ECONNREFUSED
sysinv 2020-07-10 18:59:34.037 74028 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
sysinv 2020-07-10 18:59:35.044 74050 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED
sysinv 2020-07-10 18:59:35.044 74028 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED

2.
sysinv 2020-07-10 19:19:28.039 85066 WARNING wsme.api [-] Client-side error: No disk with id 348cd8f4-8357-48c5-a986-bce82b5c0b0b: DiskNotFound: No disk with id 348cd8f4-8357-48c5-a986-bce82b5c0b0b
sysinv 2020-07-10 19:19:43.655 120176 WARNING sysinv.api.controllers.v1.rest_api [-] URLError Error e=<urlopen error [Errno 111] ECONNREFUSED>: URLError: <urlopen error [Errno 111] ECONNREFUSED>
2020-07-10 19:19:43.660 74028 ERROR sysinv.openstack.common.rpc.amqp     error=None)
2020-07-10 19:19:43.660 74028 ERROR sysinv.openstack.common.rpc.amqp     error=error))
sysinv 2020-07-10 19:19:51.651 85066 WARNING wsme.api [-] Client-side error: No disk with id 49df7ab4-d633-46c3-a7bb-4eb16c979f78: DiskNotFound: No disk with id 49df7ab4-d633-46c3-a7bb-4eb16c979f78

3.
sysinv 2020-07-10 19:35:43.735 104173 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"25763f55-f3fb-44dd-b3aa-c69fd353ee14","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":3,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"25763f55-f3fb-44dd-b3aa-c69fd353ee14","modified":"2020-07-10 19:27:06.196102","created":"2020-07-10 19:27:06.196102","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":6,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[],"num_pgs":0,"num_pools":0,"num_objects":0,"data_bytes":0,"bytes_used":112189440,"bytes_avail":958541803520,"bytes_total":958653992960},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":58,"active_gid":4135,"active_name":"controller-0","active_addr":"192.168.204.2:6804/103669","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":true,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:7999/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'140490042174928'}

You can check the attached log file, please let me know it there are questions. Are this issues known issues or should we open new bugs?

Kind regards,
Alex

[cid:image002.png at 01D65945.197E6E60]

Dimofte Alexandru
Software Engineer
Transportation Solutions Division
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte at intel.com<mailto:alexandru.dimofte at intel.com>
Intel Romania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200713/3602c06a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 10911 bytes
Desc: image001.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200713/3602c06a/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 20513 bytes
Desc: image002.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200713/3602c06a/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sysinv.log
Type: application/octet-stream
Size: 3704407 bytes
Desc: sysinv.log
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200713/3602c06a/attachment-0001.obj>