[Starlingx-discuss] About execute bash command in sysinv conductor
Hi, I have met an issue when add a new plugin which called by sysinv, The plugin is to execute some bash commands, such as kubectl to get information, but it seems I have tried by using: Subprocess.Popen() and subprocess.check_output() it return failed with non-zero exit status 1. But I tried on my local Ubuntu environment, the result is OK. Is there any audit or privilege or some audit in Sysinv Service? Piece of code: execute_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=sys.stderr) objects = yaml.safe_load(execute_process.stdout) if execute_process.wait() != 0: raise ApiError return objects Following is the output: ----------------------------------------------------- sysinv 2020-01-14 10:31:20.038 1172858 INFO sysinv.common.ceph [-] ceph osd crush rule ls sysinv 2020-01-14 10:31:20.038 1172858 ERROR rook_client_kubeapi [-] ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:20.038 1172858 INFO rook_client_kubeapi [-] Get pod command: ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:45.026 1172858 ERROR sysinv.common.ceph [-] Exception in ls crush rule: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1: CalledProcessError: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 sysinv 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: CephFailure: OSD crush rule ls failed. Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp **args) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4405, in iplatform_update_by_ihost 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp cceph.fix_crushmap(self.dbapi) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 947, in fix_crushmap 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp rule_is_present, __, __ = _operator._crush_rule_status(default_ceph_tier_name) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 67, in _crush_rule_status 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp raise exception.CephFailure("OSD crush rule ls failed. %s" %str(e)) Thanks, Tingjie SSP NST Storage Tel: +86(21)88216699 Mobile: 15901876439
Tingjie, It looks like you are missing the --kubeconfig parameter for kubectl, which should point to the /etc/kubernetes/admin.conf file which contains the authentication credentials. However, it is better to use the python-kubernetes API instead of kubectl - take a look at the KubeOperator class in sysinv/common/kubernetes.py - that is our preferred way to communicate with the kubernetes API. Bart From: Chen, Tingjie [mailto:tingjie.chen@intel.com] Sent: January 14, 2020 5:47 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] About execute bash command in sysinv conductor Hi, I have met an issue when add a new plugin which called by sysinv, The plugin is to execute some bash commands, such as kubectl to get information, but it seems I have tried by using: Subprocess.Popen() and subprocess.check_output() it return failed with non-zero exit status 1. But I tried on my local Ubuntu environment, the result is OK. Is there any audit or privilege or some audit in Sysinv Service? Piece of code: execute_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=sys.stderr) objects = yaml.safe_load(execute_process.stdout) if execute_process.wait() != 0: raise ApiError return objects Following is the output: ----------------------------------------------------- sysinv 2020-01-14 10:31:20.038 1172858 INFO sysinv.common.ceph [-] ceph osd crush rule ls sysinv 2020-01-14 10:31:20.038 1172858 ERROR rook_client_kubeapi [-] ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:20.038 1172858 INFO rook_client_kubeapi [-] Get pod command: ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:45.026 1172858 ERROR sysinv.common.ceph [-] Exception in ls crush rule: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1: CalledProcessError: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 sysinv 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: CephFailure: OSD crush rule ls failed. Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp **args) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4405, in iplatform_update_by_ihost 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp cceph.fix_crushmap(self.dbapi) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 947, in fix_crushmap 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp rule_is_present, __, __ = _operator._crush_rule_status(default_ceph_tier_name) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 67, in _crush_rule_status 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp raise exception.CephFailure("OSD crush rule ls failed. %s" %str(e)) Thanks, Tingjie SSP NST Storage Tel: +86(21)88216699 Mobile: 15901876439
Thanks Barton, I think it is the reason for the failure :) Yes, It is a better way to use Kubernetes.client API to communicate with Kubernetes cluster, but the interface in sysinv/common/Kubernetes.py is not enough to support the Ceph plugin, so I have made a simple kube Api to verify the functionality and it will be better to extend the interface in Kubernetes.py to support Ceph plugin further, for example: https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/Core... Thanks, Tingjie From: Wensley, Barton <Barton.Wensley@windriver.com> Sent: Tuesday, January 14, 2020 8:58 PM To: Chen, Tingjie <tingjie.chen@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: About execute bash command in sysinv conductor Tingjie, It looks like you are missing the --kubeconfig parameter for kubectl, which should point to the /etc/kubernetes/admin.conf file which contains the authentication credentials. However, it is better to use the python-kubernetes API instead of kubectl - take a look at the KubeOperator class in sysinv/common/kubernetes.py - that is our preferred way to communicate with the kubernetes API. Bart From: Chen, Tingjie [mailto:tingjie.chen@intel.com] Sent: January 14, 2020 5:47 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] About execute bash command in sysinv conductor Hi, I have met an issue when add a new plugin which called by sysinv, The plugin is to execute some bash commands, such as kubectl to get information, but it seems I have tried by using: Subprocess.Popen() and subprocess.check_output() it return failed with non-zero exit status 1. But I tried on my local Ubuntu environment, the result is OK. Is there any audit or privilege or some audit in Sysinv Service? Piece of code: execute_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=sys.stderr) objects = yaml.safe_load(execute_process.stdout) if execute_process.wait() != 0: raise ApiError return objects Following is the output: ----------------------------------------------------- sysinv 2020-01-14 10:31:20.038 1172858 INFO sysinv.common.ceph [-] ceph osd crush rule ls sysinv 2020-01-14 10:31:20.038 1172858 ERROR rook_client_kubeapi [-] ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:20.038 1172858 INFO rook_client_kubeapi [-] Get pod command: ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:45.026 1172858 ERROR sysinv.common.ceph [-] Exception in ls crush rule: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1: CalledProcessError: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 sysinv 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: CephFailure: OSD crush rule ls failed. Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp **args) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4405, in iplatform_update_by_ihost 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp cceph.fix_crushmap(self.dbapi) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 947, in fix_crushmap 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp rule_is_present, __, __ = _operator._crush_rule_status(default_ceph_tier_name) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 67, in _crush_rule_status 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp raise exception.CephFailure("OSD crush rule ls failed. %s" %str(e)) Thanks, Tingjie SSP NST Storage Tel: +86(21)88216699 Mobile: 15901876439
Thanks Tingjie. Using kubectl for prototyping is OK, but when you are submitting code for review, please extend the KubeOperator to support the APIs you require. Bart From: Chen, Tingjie [mailto:tingjie.chen@intel.com] Sent: January 14, 2020 8:27 AM To: Wensley, Barton; starlingx-discuss@lists.starlingx.io Subject: RE: About execute bash command in sysinv conductor Thanks Barton, I think it is the reason for the failure :) Yes, It is a better way to use Kubernetes.client API to communicate with Kubernetes cluster, but the interface in sysinv/common/Kubernetes.py is not enough to support the Ceph plugin, so I have made a simple kube Api to verify the functionality and it will be better to extend the interface in Kubernetes.py to support Ceph plugin further, for example: https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/Core... Thanks, Tingjie From: Wensley, Barton <Barton.Wensley@windriver.com> Sent: Tuesday, January 14, 2020 8:58 PM To: Chen, Tingjie <tingjie.chen@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: About execute bash command in sysinv conductor Tingjie, It looks like you are missing the --kubeconfig parameter for kubectl, which should point to the /etc/kubernetes/admin.conf file which contains the authentication credentials. However, it is better to use the python-kubernetes API instead of kubectl - take a look at the KubeOperator class in sysinv/common/kubernetes.py - that is our preferred way to communicate with the kubernetes API. Bart From: Chen, Tingjie [mailto:tingjie.chen@intel.com] Sent: January 14, 2020 5:47 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] About execute bash command in sysinv conductor Hi, I have met an issue when add a new plugin which called by sysinv, The plugin is to execute some bash commands, such as kubectl to get information, but it seems I have tried by using: Subprocess.Popen() and subprocess.check_output() it return failed with non-zero exit status 1. But I tried on my local Ubuntu environment, the result is OK. Is there any audit or privilege or some audit in Sysinv Service? Piece of code: execute_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=sys.stderr) objects = yaml.safe_load(execute_process.stdout) if execute_process.wait() != 0: raise ApiError return objects Following is the output: ----------------------------------------------------- sysinv 2020-01-14 10:31:20.038 1172858 INFO sysinv.common.ceph [-] ceph osd crush rule ls sysinv 2020-01-14 10:31:20.038 1172858 ERROR rook_client_kubeapi [-] ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:20.038 1172858 INFO rook_client_kubeapi [-] Get pod command: ['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"'] sysinv 2020-01-14 10:31:45.026 1172858 ERROR sysinv.common.ceph [-] Exception in ls crush rule: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1: CalledProcessError: Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 sysinv 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: CephFailure: OSD crush rule ls failed. Command '['/usr/bin/kubectl', 'get', '--namespace', 'kube-system', 'pod', '-l', 'app=rook-ceph-tools', '-o', 'jsonpath="{.items[0].metadata.name}"']' returned non-zero exit status 1 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp **args) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4405, in iplatform_update_by_ihost 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp cceph.fix_crushmap(self.dbapi) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 947, in fix_crushmap 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp rule_is_present, __, __ = _operator._crush_rule_status(default_ceph_tier_name) 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 67, in _crush_rule_status 2020-01-14 10:31:45.027 1172858 ERROR sysinv.openstack.common.rpc.amqp raise exception.CephFailure("OSD crush rule ls failed. %s" %str(e)) Thanks, Tingjie SSP NST Storage Tel: +86(21)88216699 Mobile: 15901876439
participants (2)
-
Chen, Tingjie
-
Wensley, Barton