[Starlingx-discuss] So many packages are missing
Hi,all I am referring to the link https://wiki.openstack.org/wiki/StarlingX/Installation_Guide/Simplex for StarlingX environment deployment. And when I went to setp "Download Packages". There are more than 100 packages missing in the end. It's hard for me to download them manually. So I want to ask if there is an easy way to solve the problem. Or, can I drop this step and download the iso file directly? Thanks! Chang
Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang 刘 畅 云计算实验室 中国联合网络通信有限公司研究院 移动电话:18610741986 地址:北京市亦庄经济开发区北环东路1号 From: Cordoba Malibran, Erich Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang ________________________________ 刘 畅 云计算实验室 中国联合网络通信有限公司研究院 移动电话:18610741986 地址:北京市亦庄经济开发区北环东路1号 From: Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com> Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang 刘 畅 云计算实验室 中国联合网络通信有限公司研究院 移动电话:18610741986 地址:北京市亦庄经济开发区北环东路1号 From: Cordoba Malibran, Erich Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Chang, Yumdownloader will stop download the package if speed is lower than 1K/S. To work around it, I think you could use ��sudo �CE yumdownloader �CC --url <PACKAGE>�� to get the url of the package. Then use wget to download it directly. To setup the mirror will take some time depends on the network. But the good news is you just need do it one time. And do incremental downloader later which will be much easier. Best Regards Shuicheng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Friday, August 10, 2018 10:16 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng<mailto:shuicheng.lin@intel.com> Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com<mailto:erich.cordoba.malibran@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Thanks��Erich�� According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang ________________________________ �� �� �Ƽ���ʵ���� �й���������ͨ������˾�о�Ժ �ƶ��绰:18610741986 ��ַ:��������ׯ���ÿ�����������·1�� From: Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com> Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It��s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run ��yum makecache�� to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Shuicheng According to your advice. I found it is really a problem with my network speed. But in the process of my repeated attempts. I found that some packages that have already been downloaded will be re-downloaded. This takes up bandwidth and increases deployment time. So I think we should judge whether we have downloaded the package before downloading it. And I made a change to implement this function. Hope this will help. https://review.openstack.org/#/c/591544/ Chang 发件人: liuchang77@chinaunicom.cn 发送时间: 2018-08-10 10:16 收件人: Lin, Shuicheng; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io 主题: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang From: Cordoba Malibran, Erich Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Thanks Liu Chang with the patch to optimize the mirror download scripts. Question: with this patch, have you successfully setup your local mirror already? Or anything particular you expect WR or Intel to do further to accelerate it? Thx. - cindy From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Tuesday, August 14, 2018 10:01 AM To: Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Shuicheng According to your advice. I found it is really a problem with my network speed. But in the process of my repeated attempts. I found that some packages that have already been downloaded will be re-downloaded. This takes up bandwidth and increases deployment time. So I think we should judge whether we have downloaded the package before downloading it. And I made a change to implement this function. Hope this will help. https://review.openstack.org/#/c/591544/< https:/review.openstack.org/#/c/591544/> Chang 发件人: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> 发送时间: 2018-08-10 10:16 收件人: Lin, Shuicheng<mailto:shuicheng.lin@intel.com>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> 主题: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng<mailto:shuicheng.lin@intel.com> Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com<mailto:erich.cordoba.malibran@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang From: Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com> Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Cindy, I have tested this script in my local environment. It works as I expected. And I hope my patch iwill help. But I haven't successfully downloaded all the packages. When I manually use the yumdownloader to download these packages. There shows the error "Cannot find repomd.xml file for base-source/7". I'm looking for ways to solve this problem. Chang 发件人: Xie, Cindy 发送时间: 2018-08-14 11:25 收件人: liuchang77@chinaunicom.cn; Lin, Shuicheng 抄送: starlingx-discuss@lists.starlingx.io 主题: RE: [Starlingx-discuss] 回复: Re: So many packages are missing Thanks Liu Chang with the patch to optimize the mirror download scripts. Question: with this patch, have you successfully setup your local mirror already? Or anything particular you expect WR or Intel to do further to accelerate it? Thx. - cindy From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Tuesday, August 14, 2018 10:01 AM To: Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Shuicheng According to your advice. I found it is really a problem with my network speed. But in the process of my repeated attempts. I found that some packages that have already been downloaded will be re-downloaded. This takes up bandwidth and increases deployment time. So I think we should judge whether we have downloaded the package before downloading it. And I made a change to implement this function. Hope this will help. https://review.openstack.org/#/c/591544/ Chang 发件人: liuchang77@chinaunicom.cn 发送时间: 2018-08-10 10:16 收件人: Lin, Shuicheng; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io 主题: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang From: Cordoba Malibran, Erich Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Chang, Good to see your patch! We actually have a similar patch ongoing https://review.openstack.org/#/c/589333/ @Abraham add features like sum check, existed package check as you did. You can also review it and discuss with him, and finalized the patch. Thanks zhipeng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: 2018年8月14日 10:01 To: Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Shuicheng According to your advice. I found it is really a problem with my network speed. But in the process of my repeated attempts. I found that some packages that have already been downloaded will be re-downloaded. This takes up bandwidth and increases deployment time. So I think we should judge whether we have downloaded the package before downloading it. And I made a change to implement this function. Hope this will help. https://review.openstack.org/#/c/591544/< https:/review.openstack.org/#/c/591544/> Chang 发件人: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> 发送时间: 2018-08-10 10:16 收件人: Lin, Shuicheng<mailto:shuicheng.lin@intel.com>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> 主题: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng<mailto:shuicheng.lin@intel.com> Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com<mailto:erich.cordoba.malibran@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang From: Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com> Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi zhipeng, Ok, the patch @Abraham made adds checksum verification work, it is better than my method. And I think maybe I can help with the packages in 'other_downloads.lst' in my review. Chang 发件人: Liu, ZhipengS 发送时间: 2018-08-14 11:27 收件人: liuchang77@chinaunicom.cn; Lin, Shuicheng; Arce Moreno, Abraham 抄送: starlingx-discuss@lists.starlingx.io 主题: RE: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Chang, Good to see your patch! We actually have a similar patch ongoing https://review.openstack.org/#/c/589333/ @Abraham add features like sum check, existed package check as you did. You can also review it and discuss with him, and finalized the patch. Thanks zhipeng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: 2018年8月14日 10:01 To: Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Shuicheng According to your advice. I found it is really a problem with my network speed. But in the process of my repeated attempts. I found that some packages that have already been downloaded will be re-downloaded. This takes up bandwidth and increases deployment time. So I think we should judge whether we have downloaded the package before downloading it. And I made a change to implement this function. Hope this will help. https://review.openstack.org/#/c/591544/ Chang 发件人: liuchang77@chinaunicom.cn 发送时间: 2018-08-10 10:16 收件人: Lin, Shuicheng; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io 主题: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang From: Cordoba Malibran, Erich Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Hi Chang, Sure, welcome to join us to work together! There is a ticket related to mirror optimization, FYI https://storyboard.openstack.org/#!/story/2002736 Zhipeng From: liuchang77@chinaunicom.cn [mailto:liuchang77@chinaunicom.cn] Sent: 2018年8月14日 15:03 To: Liu, ZhipengS <zhipengs.liu@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Arce Moreno, Abraham <abraham.arce.moreno@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: RE: [Starlingx-discuss] 回复: Re: So many packages are missing Hi zhipeng, Ok, the patch @Abraham made adds checksum verification work, it is better than my method. And I think maybe I can help with the packages in 'other_downloads.lst' in my review. Chang 发件人: Liu, ZhipengS<mailto:zhipengs.liu@intel.com> 发送时间: 2018-08-14 11:27 收件人: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; Lin, Shuicheng<mailto:shuicheng.lin@intel.com>; Arce Moreno, Abraham<mailto:abraham.arce.moreno@intel.com> 抄送: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> 主题: RE: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Chang, Good to see your patch! We actually have a similar patch ongoing https://review.openstack.org/#/c/589333/ @Abraham add features like sum check, existed package check as you did. You can also review it and discuss with him, and finalized the patch. Thanks zhipeng From: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> [mailto:liuchang77@chinaunicom.cn] Sent: 2018年8月14日 10:01 To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] 回复: Re: So many packages are missing Hi Shuicheng According to your advice. I found it is really a problem with my network speed. But in the process of my repeated attempts. I found that some packages that have already been downloaded will be re-downloaded. This takes up bandwidth and increases deployment time. So I think we should judge whether we have downloaded the package before downloading it. And I made a change to implement this function. Hope this will help. https://review.openstack.org/#/c/591544/< https:/review.openstack.org/#/c/591544/> Chang 发件人: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> 发送时间: 2018-08-10 10:16 收件人: Lin, Shuicheng<mailto:shuicheng.lin@intel.com>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> 主题: Re: RE: [Starlingx-discuss] So many packages are missing Hi Shuicheng, Thanks for your advice. I can access the links you provided. But the speed is very slow. So I think this is really a problem with my network speed. I work at the carrier, so our network is definitely fine. Maybe the real problem is Chinese firewall. I spent a few hours to re-run the download package script, and the missing list has not changed. So the only way I can think of now is to use vpn inside the container. But after I tried if, I found that the kernel of the system inside the container does not support pptp. I will keep trying to find another way to solve it. Thank you very much! Best Regards Chang From: Lin, Shuicheng<mailto:shuicheng.lin@intel.com> Date: 2018-08-09 16:05 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] So many packages are missing Hi Chang, I try to check some package in the missing list. Can you access below link for the package? http://vault.centos.org/7.5.1804/os/Source/SPackages/bash-4.2.46-30.el7.src.... https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm If yes, I think it maybe network speed issue. You could re-run the download package script. And check whether the missing list be shorter or not. If not, you may need check the networking setting. Best Regards Shuicheng From: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn> [mailto:liuchang77@chinaunicom.cn] Sent: Thursday, August 9, 2018 3:28 PM To: Cordoba Malibran, Erich <erich.cordoba.malibran@intel.com<mailto:erich.cordoba.malibran@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Thanks,Erich, According to your advice, I ran "yum makecache" inside the container. Although it was slow, the task was complete with no error. Attachment 1 is my execution log. And attachment 2 is my missing packages' list. I also think that there are so many missing packages that are not normal. I suspect this is related to Chinese firewall. But I couldn't connect my vpn server in the container. I sincerely hope that you can give me some advice. Chang From: Cordoba Malibran, Erich<mailto:erich.cordoba.malibran@intel.com> Date: 2018-08-09 11:28 To: liuchang77@chinaunicom.cn<mailto:liuchang77@chinaunicom.cn>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] So many packages are missing Hi Chang, You are right, download manually that amount of packages is a painful task. It’s not normal to have so many failures, my guess is that some of the repositories is down or blocked in the network. You can try to identify from where those packages are being downloaded to root cause the problem. Inside the mirror container you can run “yum makecache” to sync the repositories and try to see which repo failed. Also if you can share the list of failed packages we can help to identify what is the problematic repository. -Erich
Chang, Zhipeng,
Ok, the patch @Abraham made adds checksum verification work, it is better than my method. And I think maybe I can help with the packages in 'other_downloads.lst' in my review.
Please let me know if this change [0] works for you as intended. I am now taking the content generated to build the ISO. [0] https://review.openstack.org/#/c/589333
participants (6)
-
Arce Moreno, Abraham
-
Cordoba Malibran, Erich
-
Lin, Shuicheng
-
Liu, ZhipengS
-
liuchang77@chinaunicom.cn
-
Xie, Cindy