[Starlingx-discuss] download_mirrors.sh issues

Thu Jul 26 15:23:50 UTC 2018

Proposal 2

Can we explore pulling old rpms from kojipkgs.fedoraproject.org. It 
seems to retain a history of old rpms far longer than the main repos.  
This may allow use to remove many repos from StarlingX_3rd.repo

--- snip ---
cat koji_url.py
#!/usr/bin/python

import sys
from rpmUtils.miscutils import splitFilename

for filename in sys.argv[1:]:
    (n, v, r, e, a) = splitFilename(filename)
    print 
"https://kojipkgs.fedoraproject.org/packages/%s/%s/%s/%s/%s-%s-%s.%s.rpm" 
% (n,v,r,a,n,v,r,a)

--- end snip ---

e.g.
./koji_url.py epel-rpm-macros-7-18.noarch.rpm
https://kojipkgs.fedoraproject.org/packages/epel-rpm-macros/7/18/noarch/epel-rpm-macros-7-18.noarch.rpm

So a loop like this might catch a lot of missing rpms ... add some 
logging and error handling ...
for missing_rpm in $(cat output/*missing*.txt); do
     url=$(./koji_url.py $missing_rpm)
     wget $url
done

Anything downloaded in this manner should be flagged for upgrade. We 
don't want to rely on it as a matter of routine.  Just a way to avoid 
people getting blocked on missing downloads.

I suspect there is a lot of repos in StarlingX_3rd.repo that we don't 
need.  Also a lot of rpms currently listed under 
rpms_from_centos_3rd_parties.lst or rpms_from_3rd_parties.lst that could 
move back to rpms_from_centos_repo.lst.  i.e. Ideally they come from 
CentOS/EPEL (they aren't patches or customized), they just aged out.

Scott

On 18-07-25 03:33 PM, Scott Little wrote:
> The http://epel.blizoo.mk/epel/ site is blocked by the security 
> gateway internal to Windriver.  Reason is 'potentially unwanted 
> programs'.  Is there an alternative to this site?
>
> What packages are being supplied by this site?
>
> Scott
>
>
> On 18-07-25 02:32 PM, Arce Moreno, Abraham wrote:
>>> I am following the instructions here:
>>>
>>> https://wiki.openstack.org/wiki/StarlingX/Developer_Guide#Software_Requir 
>>>
>>> ements
>> Thanks Al!
>>
>> A request for everybody, please continuously refresh the webpage due 
>> to the
>> daily changes. The word "Temporal!" is used across this guide to 
>> highlight those patches
>> needed to have a successful build.
>>
>> For branch Master, we are identifying those patches not merged as:
>>      Temporal! Only Branch Master
>>
>> For branch m/2018.07, we are identifying those patches not merged as:
>>      Temporal! Only Branch m/2018.07
>>
>>> I have been un-successful in running the first setup actions for 
>>> doing an
>>> official build, which is getting my mirror setup.
>>>
>>> I have all the latest commits in master for stx-tools.
>> Our commit ids for today's build are:
>>
>> stx-tools branch master
>>
>>      $ git log --pretty=oneline -2
>>      f31f99ab91435c32c24a86efd2e697177074f4b7 Merge 
>> "policycoreutils-newrole was missing"
>>      c3000c2936f56ad0257142d5c5bad3afc08c46cc Merge "upgrade 
>> linux-firmware RPM version"```
>>
>> stx-tools branch m/2018.07
>>
>>      $ git log --pretty=oneline -5
>>      88aadc543085e24e3c5df10ec63996b9f5551ff6 Merge "Package updates 
>> for new version of selinux-policy" into m/2018.07
>>     9c44470a2d6ff86a40f42c2b608e723c355ae38f Merge "Copy 
>> mock.cfg.proto and comps.xml files into cgcs-centos-repo" into 
>> m/2018.07```
>>
>>> The first issue encountered was the macros.   This has already been 
>>> reported
>>> here
>>> https://storyboard.openstack.org/#!/story/2003072
>>>
>>> I only had 4 entries listed as missing, until after I updated these 
>>> to the
>>> following newer versions
>>>
>>> epel-rpm-macros-7-19.noarch.rpm
>>> python2-rpm-macros-3-20.el7.noarch.rpm
>>> python-rpm-macros-3-20.el7.noarch.rpm
>>> python-srpm-macros-3-20.el7.noarch.rpm
>> Also in this setup...
>>
>>> I don't know if that is the appropriate solution, since all this 
>>> allowed me to do
>>> was fail to download some other things.
>> We have 2 options:
>>
>> 1. a manual download of the packages
>> 2. use of a tool, in review status [0]
>>
>>> Running download_mirror.sh again showed 140+ rpms as missing
>>>
>>> As an aside, sometimes the "missing"  log files are blank, even 
>>> though not all
>>> components were downloaded.   I assume this is due to something 
>>> aborting.
>>
>>> Lighttpd  for example  is never being found.
>> Same here:
>>
>>      Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
>>      warning: 
>> /var/cache/yum/x86_64/7/epel/packages/lighttpd-1.4.49-1.el7.x86_64.rpm: 
>> \
>>          Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
>>      Public key for lighttpd-1.4.49-1.el7.x86_64.rpm is not installed
>>
>>> By redirecting the script to log everything to a file, it shows this 
>>> particular
>>> error message which may be a potential culprit
>>>
>>> "Loaded plugins: fastestmirror, ovl
>>> http://epel.blizoo.mk/epel/7Server/x86_64/repodata/repomd.xml: 
>>> [Errno 14]
>>> HTTP Error 403 - Forbidden
>>> Trying other mirror.
>>> To address this issue please refer to the below knowledge base article
>>> https://access.redhat.com/solutions/69319
>>> If above article doesn't help to resolve this issue please create a 
>>> bug on
>>> https://bugs.centos.org/
>>> One of the configured repositories failed (Starlingx-
>>> epel.blizoo.mk_epel_7Server_x86_64 -
>>> epel.blizoo.mk_epel_7Server_x86_64),
>>> and yum doesn't have enough cached data to continue. At this point 
>>> the only
>>> safe thing yum can do is fail."
>>>
>>> Does anyone know anything about this blizoo  repo?  Is it causing 
>>> anyone else
>>> to not be able to download all the rpms,  or  is it unrelated to the 
>>> build setup
>>> issues?
>> Not seeing the blizoo repo failure in our side... internet connection 
>> is slow so will come back to
>> you with more updates some time later.
>>
>> [0] https://review.openstack.org/#/c/585009/
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss