Adding this thread to public mailing list. From: Alonso, Juan Carlos Sent: Tuesday, May 28, 2019 7:08 PM To: 'Ngo, Tee' <Tee.Ngo@windriver.com>; Perez Carranza, Jose <jose.perez.carranza@intel.com> Cc: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; Church, Robert <Robert.Church@windriver.com>; Wensley, Barton <Barton.Wensley@windriver.com> Subject: RE: Live debug session Hi Tee, After make the workarounds you told, platform-integ-apps is still uploading and applying automatically, applying still failing. I did not put docker_no_proxy list on localhost.yml since I saw last email after finish test. As Barton, I got apply-failed, but app uploaded and applied automatically. manager.py: [cid:image003.jpg@01D51588.D5699660] After Ansible: [Service] Environment="HTTPS_PROXY=http://proxy-chain.intel.com:912" Environment="NO_PROXY=localhost,127.0.0.1,192.168.204.2,192.168.204.3,10.10.10.3,10.10.10.4" I did not add docker_no_proxy list on localhost.yml. Before Ansible NO_PROXY was empty, after Ansible it contains a list of IPs. After Unlock: [wrsroot@controller-0 ~(keystone_admin)]$ system application-list +---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+ | application | version | manifest name | manifest file | status | progress | +---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+ | platform-integ-apps | 1.0-5 | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail | +---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+ Regards. Juan Carlos Alonso From: Ngo, Tee [mailto:Tee.Ngo@windriver.com] Sent: Tuesday, May 28, 2019 6:03 PM To: Perez Carranza, Jose <jose.perez.carranza@intel.com<mailto:jose.perez.carranza@intel.com>>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Church, Robert <Robert.Church@windriver.com<mailto:Robert.Church@windriver.com>>; Wensley, Barton <Barton.Wensley@windriver.com<mailto:Barton.Wensley@windriver.com>> Subject: RE: Live debug session Just an update on docker_proxy test front: Bart was able to get passed the upload of platform-integ-apps which entailed successful download of images using May 28th code (I think using May 27th official build is good enough for you guys) and the following: a) Fixed the flakiness of the proxy server b) Provided docker_no_proxy list in the localhost.yml before running Ansible bootstrap Note: Once this commit https://review.opendev.org/#/c/661825/ gets merged, you would not need to provide docker_no_proxy list unless you want to add more IPs to the derived list Bart is now having platform-integ-apps apply failure in his environment. Bob is investigating... So if you encounter apply failure, stay tuned. Thanks, Tee P.S. Cristopher, this is regarding docker_proxy test. Please let me know how far you get with private registry test in the other thread with Ada. Based on the screenshot you showed us today, you also passed upload stage in that config. From: Ngo, Tee Sent: May-28-19 4:23 PM To: 'Perez Carranza, Jose'; Alonso, Juan Carlos Cc: Lemus Contreras, Cristopher J; Church, Robert (Robert.Church@windriver.com<mailto:Robert.Church@windriver.com>) Subject: RE: Live debug session As discussed, JC and Jose, let's switch to May 27th build. Here are the instructions to disable automatic upload and apply of platform-integ-apps Step 1: Switch to "fresh install" snapshot Step 2: As root, back up the original file cp /usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py /usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py.orig Step 3: As root, update the manager.py file (see diff below) vim /usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py --- a/sysinv/sysinv/sysinv/sysinv/conductor/manager.py +++ b/sysinv/sysinv/sysinv/sysinv/conductor/manager.py @@ -5223,6 +5223,10 @@ class ConductorManager(service.PeriodicService): LOG.debug("Platform managed application %s: %s" % (app_name, status)) if status == constants.APP_NOT_PRESENT: + if not _met_app_apply_prerequisites(app_name): + LOG.info("Platform managed application %s: Prerequisites " + "not met." % app_name) + continue if _patching_operation_is_occurring(): continue Step 4: Take a snapshot, name it (fresh install + integ app disabled) Step 5: Run Ansible and bring up the simplex system Step 6: Verify that app is not installed after the unlock system application-list Step 7: Take a snapshot, name it (host unlocked, integ app disabled) Step 8: Upload the app manually and check if there are errors system application-upload /usr/local/share/applications/helm/platform-integ-apps-1.0-5.tgz Step 9: Apply the app manually system application-apply platform-integ-apps Tee From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: May-28-19 2:54 PM To: Ngo, Tee; Alonso, Juan Carlos Cc: Lemus Contreras, Cristopher J Subject: RE: Live debug session I'm using last Friday code BUILD_ID="20190524T013000Z" Regards, José From: Ngo, Tee [mailto:Tee.Ngo@windriver.com] Sent: Tuesday, May 28, 2019 1:51 PM To: Perez Carranza, Jose <jose.perez.carranza@intel.com<mailto:jose.perez.carranza@intel.com>>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Subject: RE: Live debug session Are you running the latest code? From: Perez Carranza, Jose [mailto:jose.perez.carranza@intel.com] Sent: May-28-19 2:46 PM To: Ngo, Tee; Alonso, Juan Carlos Cc: Lemus Contreras, Cristopher J Subject: RE: Live debug session Hi Tee I another machine I used both docker_http_proxy and docker_https_proxy, but I'm also facing same issue. The only difference that I noted is that using config_controller ips 192.168.204.4 and 10.10.10.5 listed as part of no_proxy whilst using a ANSIBLE that IP is not listed in no_proxy. CONFIG_CONTROLLER [Service] Environment="HTTP_PROXY=http://proxy-chain.intel.com:911" Environment="HTTPS_PROXY=http://proxy-chain.intel.com:912" Environment="NO_PROXY=localhost,127.0.0.1,192.168.204.2,192.168.204.3,192.168.204.4,10.10.10.3,10.10.10.4,10.10.10.5" ANSIBLE [Service] Environment="HTTP_PROXY=http://proxy-chain.intel.com:911" Environment="HTTPS_PROXY=http://proxy-chain.intel.com:912" Environment="NO_PROXY=localhost,127.0.0.1,192.168.204.2,192.168.204.3,10.10.10.3,10.10.10.4" Regards, José From: Ngo, Tee [mailto:Tee.Ngo@windriver.com] Sent: Tuesday, May 28, 2019 1:05 PM To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com<mailto:jose.perez.carranza@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Subject: RE: Live debug session A couple of findings: 1. Your localhost.yml does not have http_proxy specified. Hence, it is missing in both http-proxy.conf and service-parameter-list output. 2. There is a bug which I have resolved, for now please specify the list of docker_no_proxy in localhost.yml as you would have specified in .ini file for config_controller. I will explain in our meeting this afternoon. Tee From: Alonso, Juan Carlos [mailto:juan.carlos.alonso@intel.com] Sent: May-28-19 1:37 PM To: Ngo, Tee Cc: Perez Carranza, Jose; Lemus Contreras, Cristopher J Subject: RE: Live debug session Find the information requested attached. System host-parameter-list command does not exists. I think you are referring system service-parameter-list. I only use https proxy for Ansible and for config controller both http and https proxy was set. We did not get any ansible nor docker error during installation. Regards. Juan Carlos Alonso From: Ngo, Tee [mailto:Tee.Ngo@windriver.com] Sent: Tuesday, May 28, 2019 12:14 PM To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: Perez Carranza, Jose <jose.perez.carranza@intel.com<mailto:jose.perez.carranza@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Subject: RE: Live debug session By the way, could you send me the following info prior to debug meeting today from both Ansible setup and config_controller setup? Content of localhost.yml Content of /etc/systemd/system/docker.service.d/http-proxy.conf Output of system host-parameter-list Ansible/docker daemon error logs Thank you. Tee From: Alonso, Juan Carlos [mailto:juan.carlos.alonso@intel.com] Sent: May-28-19 12:44 PM To: Ngo, Tee Subject: Live debug session Hi Tee, What time works for you to have the live debug session? Regards. Juan Carlos Alonso