[Starlingx-discuss] Cengn Mirror server in down?!
Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania
CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> <mailto:Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On 1/26/21 1:28 PM, Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
OUCH...!!!
StarlingX Community: As you are all likely aware there has been an extended outage at our CENGN facility. Our build team members and CENGN folks have been working extra hours through the recovery. This likely will require another 2 days before normal builds can occur. In the meantime please note the following impact: 1. If you have a local copy of the mirror and did a repo sync on Jan 20 or earlier you likely can continue to make some progress for your development. 2. Do NOT perform a download mirror step at this time as the mirror is corrupted and being repaired. Also a repo sync may not be successful especially if a commit was merged since Jan 20 that required a .lst file change. 3. Do NOT merge or approve any commits that require a .lst file change. 4. Scott will provide daily email updates on the recovery of the CENGN mirror and will let you know when you can resume doing downloads of the mirror. Thanks for your patience as the recovery is being worked. Frank From: Scott Little <scott.little@windriver.com> Sent: Tuesday, January 26, 2021 1:28 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote: Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don't have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Frank, my thanks to you and Scott for your diligence on this issue. The community really appreciates your support! brucej From: Miller, Frank <Frank.Miller@windriver.com> Sent: Tuesday, January 26, 2021 8:07 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! StarlingX Community: As you are all likely aware there has been an extended outage at our CENGN facility. Our build team members and CENGN folks have been working extra hours through the recovery. This likely will require another 2 days before normal builds can occur. In the meantime please note the following impact: 1. If you have a local copy of the mirror and did a repo sync on Jan 20 or earlier you likely can continue to make some progress for your development. 2. Do NOT perform a download mirror step at this time as the mirror is corrupted and being repaired. Also a repo sync may not be successful especially if a commit was merged since Jan 20 that required a .lst file change. 3. Do NOT merge or approve any commits that require a .lst file change. 4. Scott will provide daily email updates on the recovery of the CENGN mirror and will let you know when you can resume doing downloads of the mirror. Thanks for your patience as the recovery is being worked. Frank From: Scott Little <scott.little@windriver.com<mailto:scott.little@windriver.com>> Sent: Tuesday, January 26, 2021 1:28 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote: Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don't have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
The audit of repos identified ~13000 corrupted rpms So far ~5000 have been restored from redundant copies within cengn. Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution. That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined. Scott On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> <mailto:Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Update Of the original ~13000 corrupted rpms, we now have ... 5700 have been restored 2400 have yet to be assessed 1800 having no backup within cengn. Upstream availability has yet to be determined. 3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving. Scott On 2021-01-27 1:37 a.m., Scott Little wrote:
The audit of repos identified ~13000 corrupted rpms
So far ~5000 have been restored from redundant copies within cengn.
Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution.
That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined.
Scott
On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> <mailto:Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Update All known firewall issues have been fixed. Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running Of the original ~13000 corrupted rpms, we now have ... ~9250 have been restored, including all rpms that are knowm inputs to supported builds ~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported. ~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run. I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow. Scott On 2021-01-27 9:56 a.m., Scott Little wrote:
Update
Of the original ~13000 corrupted rpms, we now have ...
5700 have been restored
2400 have yet to be assessed
1800 having no backup within cengn. Upstream availability has yet to be determined.
3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving.
Scott
On 2021-01-27 1:37 a.m., Scott Little wrote:
The audit of repos identified ~13000 corrupted rpms
So far ~5000 have been restored from redundant copies within cengn.
Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution.
That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined.
Scott
On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote: > The hosting company had an issue with their servers, they are > working on it. They hope to be back online mid afternoon (Jan > 21, Eastern Standard Time). > ------------------------------------------------------------------------ > *From:* Panech, Davlet <Davlet.Panech@windriver.com> > <mailto:Davlet.Panech@windriver.com> > *Sent:* January 21, 2021 9:56 AM > *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> > <mailto:alexandru.dimofte@intel.com>; > starlingx-discuss@lists.starlingx.io > <mailto:starlingx-discuss@lists.starlingx.io> > <starlingx-discuss@lists.starlingx.io> > <mailto:starlingx-discuss@lists.starlingx.io> > *Subject:* Re: Cengn Mirror server in down?! > Thanks Alexandru, we are looking into it. > > ------------------------------------------------------------------------ > *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> > <mailto:alexandru.dimofte@intel.com> > *Sent:* January 21, 2021 3:24 AM > *To:* starlingx-discuss@lists.starlingx.io > <mailto:starlingx-discuss@lists.starlingx.io> > <starlingx-discuss@lists.starlingx.io> > <mailto:starlingx-discuss@lists.starlingx.io> > *Subject:* [Starlingx-discuss] Cengn Mirror server in down?! > > Hello guys, > > I observed, trying to connect to > http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ > , that cengn mirror server is down. > > Best regards, > > Alex > > Logo Description automatically generated > > > > Dimofte Alexandru > > Software Engineer > > STARLINGX TEAM > > Skype no: +40 336403734 > > Personal Mobile: +40 743167456 > > alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com> > > Intel Romania > > > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Thanks Scott for grinding through this. From: Scott Little <scott.little@windriver.com> Sent: Thursday, January 28, 2021 12:28 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! Update All known firewall issues have been fixed. Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running Of the original ~13000 corrupted rpms, we now have ... ~9250 have been restored, including all rpms that are knowm inputs to supported builds ~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported. ~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run. I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow. Scott On 2021-01-27 9:56 a.m., Scott Little wrote: Update Of the original ~13000 corrupted rpms, we now have ... 5700 have been restored 2400 have yet to be assessed 1800 having no backup within cengn. Upstream availability has yet to be determined. 3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving. Scott On 2021-01-27 1:37 a.m., Scott Little wrote: The audit of repos identified ~13000 corrupted rpms So far ~5000 have been restored from redundant copies within cengn. Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution. That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined. Scott On 2021-01-26 1:28 a.m., Scott Little wrote: The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote: Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don't have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
We just merged a fix for tb.sh/Dockerfile for CENGN-related build problems that may affect some builds, depending on the environment: ________________________________ From: Zvonar, Bill <Bill.Zvonar@windriver.com> Sent: January 28, 2021 6:31 AM To: Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! Thanks Scott for grinding through this. From: Scott Little <scott.little@windriver.com> Sent: Thursday, January 28, 2021 12:28 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! Update All known firewall issues have been fixed. Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running Of the original ~13000 corrupted rpms, we now have ... ~9250 have been restored, including all rpms that are knowm inputs to supported builds ~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported. ~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run. I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow. Scott On 2021-01-27 9:56 a.m., Scott Little wrote: Update Of the original ~13000 corrupted rpms, we now have ... 5700 have been restored 2400 have yet to be assessed 1800 having no backup within cengn. Upstream availability has yet to be determined. 3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving. Scott On 2021-01-27 1:37 a.m., Scott Little wrote: The audit of repos identified ~13000 corrupted rpms So far ~5000 have been restored from redundant copies within cengn. Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution. That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined. Scott On 2021-01-26 1:28 a.m., Scott Little wrote: The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote: Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
We just merged a fix for tb.sh/Dockerfile for CENGN-related build problems that may affect some builds, depending on the environment: https://review.opendev.org/c/starlingx/tools/+/772790 If you compiled a builder image during the outage (during the last week or so) for the first time, it may affect your builds. The symptoms include this error message from "docker build": rmdir: failed to remove '/var/lib/mock': No such file or directory You may need to refresh your starlingx/tools checkout or cherry pick that commit into any private branches you are interested in building. This patch is only in the master branch. Another workaround is to delete any cached builder images that came out of starlingx/tools/Dockerfile using "docker image rm" etc. Hope this helps, D. ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com> Sent: January 28, 2021 12:31 PM To: Zvonar, Bill <Bill.Zvonar@windriver.com>; Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! We just merged a fix for tb.sh/Dockerfile for CENGN-related build problems that may affect some builds, depending on the environment: ________________________________ From: Zvonar, Bill <Bill.Zvonar@windriver.com> Sent: January 28, 2021 6:31 AM To: Little, Scott <Scott.Little@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! Thanks Scott for grinding through this. From: Scott Little <scott.little@windriver.com> Sent: Thursday, January 28, 2021 12:28 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! Update All known firewall issues have been fixed. Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running Of the original ~13000 corrupted rpms, we now have ... ~9250 have been restored, including all rpms that are knowm inputs to supported builds ~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported. ~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run. I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow. Scott On 2021-01-27 9:56 a.m., Scott Little wrote: Update Of the original ~13000 corrupted rpms, we now have ... 5700 have been restored 2400 have yet to be assessed 1800 having no backup within cengn. Upstream availability has yet to be determined. 3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving. Scott On 2021-01-27 1:37 a.m., Scott Little wrote: The audit of repos identified ~13000 corrupted rpms So far ~5000 have been restored from redundant copies within cengn. Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution. That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined. Scott On 2021-01-26 1:28 a.m., Scott Little wrote: The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote: Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Update Well a few more networking issues have been solved. thanks to Davlet, and Moh at CENGN. We have had a successful build of the 'compiler' and 'distro' layers. The 'flock' layer built the rpms, but failed to update the installer. It appears to relate to the loopback device. I'll look into that tomorrow. ~350 RPMS failed to restore from upstream. None are critical to any of our current builds. I don't plan to persu them any further The mirroring job is throwing a lot more error messages than I'm used to seeing. Many rpm download attempts are failing, and left 2000 new corrupt rpms in its wake. I am cleaning up the new mess and adjusting the mirroring script to prevent download failures from leaving us with corrupt repos. The download failures seem to come in batches, so I'll reach out to CENGN to see if they had any network issues that might explain the problem. Scott On 2021-01-28 12:27 a.m., Scott Little wrote:
Update
All known firewall issues have been fixed.
Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running
Of the original ~13000 corrupted rpms, we now have ...
~9250 have been restored, including all rpms that are knowm inputs to supported builds
~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported.
~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run.
I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow.
Scott
On 2021-01-27 9:56 a.m., Scott Little wrote:
Update
Of the original ~13000 corrupted rpms, we now have ...
5700 have been restored
2400 have yet to be assessed
1800 having no backup within cengn. Upstream availability has yet to be determined.
3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving.
Scott
On 2021-01-27 1:37 a.m., Scott Little wrote:
The audit of repos identified ~13000 corrupted rpms
So far ~5000 have been restored from redundant copies within cengn.
Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution.
That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined.
Scott
On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote: > > CENGN remains out of service > > > AC to the server room has been restored. They are now working > some firewall issues. > > > They hope to have it restored in a few hours. > > > Scott > > > > On 2021-01-21 10:07 a.m., Panech, Davlet wrote: >> The hosting company had an issue with their servers, they are >> working on it. They hope to be back online mid afternoon (Jan >> 21, Eastern Standard Time). >> ------------------------------------------------------------------------ >> *From:* Panech, Davlet <Davlet.Panech@windriver.com> >> <mailto:Davlet.Panech@windriver.com> >> *Sent:* January 21, 2021 9:56 AM >> *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> >> <mailto:alexandru.dimofte@intel.com>; >> starlingx-discuss@lists.starlingx.io >> <mailto:starlingx-discuss@lists.starlingx.io> >> <starlingx-discuss@lists.starlingx.io> >> <mailto:starlingx-discuss@lists.starlingx.io> >> *Subject:* Re: Cengn Mirror server in down?! >> Thanks Alexandru, we are looking into it. >> >> ------------------------------------------------------------------------ >> *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> >> <mailto:alexandru.dimofte@intel.com> >> *Sent:* January 21, 2021 3:24 AM >> *To:* starlingx-discuss@lists.starlingx.io >> <mailto:starlingx-discuss@lists.starlingx.io> >> <starlingx-discuss@lists.starlingx.io> >> <mailto:starlingx-discuss@lists.starlingx.io> >> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?! >> >> Hello guys, >> >> I observed, trying to connect to >> http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ >> , that cengn mirror server is down. >> >> Best regards, >> >> Alex >> >> Logo Description automatically generated >> >> >> >> Dimofte Alexandru >> >> Software Engineer >> >> STARLINGX TEAM >> >> Skype no: +40 336403734 >> >> Personal Mobile: +40 743167456 >> >> alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com> >> >> Intel Romania >> >> >> >> >> _______________________________________________ >> Starlingx-discuss mailing list >> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> >> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Update CENGN hasn't found any specific network issues as of yet. They have acknowledged that the current network configuration implemented in the wake of power outage and the switch to the new server is sub-optimal and will revisit it soon. We have worked over the last several days to get the firewall configuration right to allow all the required services on the build server to run. One particular quirk of the current network setup is that the DNS reported address of the mirror is inaccessible. An /etc/hosts entry is required to force the mirror address to resolve to an alternate IP address. I've spent the last several sleepless nights learning how to push that change not only into our build container, but into all the STX service containers that we build. I can finally report successful builds of the entire layered build stack. On the mirror update side, I have adjusted the mirroring algorithm to do more validation steps as it downloads. This should prevent further corruptions, but will slow the frequency with with we can refresh our repos from upstream. In stead of updating all repos daily, some will be on a ~1/week schedule. Scott On 2021-01-29 1:02 a.m., Scott Little wrote:
Update
Well a few more networking issues have been solved. thanks to Davlet, and Moh at CENGN.
We have had a successful build of the 'compiler' and 'distro' layers. The 'flock' layer built the rpms, but failed to update the installer. It appears to relate to the loopback device. I'll look into that tomorrow.
~350 RPMS failed to restore from upstream. None are critical to any of our current builds. I don't plan to persu them any further
The mirroring job is throwing a lot more error messages than I'm used to seeing. Many rpm download attempts are failing, and left 2000 new corrupt rpms in its wake. I am cleaning up the new mess and adjusting the mirroring script to prevent download failures from leaving us with corrupt repos. The download failures seem to come in batches, so I'll reach out to CENGN to see if they had any network issues that might explain the problem.
Scott
On 2021-01-28 12:27 a.m., Scott Little wrote:
Update
All known firewall issues have been fixed.
Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running
Of the original ~13000 corrupted rpms, we now have ...
~9250 have been restored, including all rpms that are knowm inputs to supported builds
~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported.
~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run.
I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow.
Scott
On 2021-01-27 9:56 a.m., Scott Little wrote:
Update
Of the original ~13000 corrupted rpms, we now have ...
5700 have been restored
2400 have yet to be assessed
1800 having no backup within cengn. Upstream availability has yet to be determined.
3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving.
Scott
On 2021-01-27 1:37 a.m., Scott Little wrote:
The audit of repos identified ~13000 corrupted rpms
So far ~5000 have been restored from redundant copies within cengn.
Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution.
That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined.
Scott
On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote: > > CENGN remains out of service. > > > A new problem has been identified with ceph storage. > > > They continue to work the outage. > > > Scott > > > > > On 2021-01-22 9:55 a.m., Scott Little wrote: >> >> CENGN remains out of service >> >> >> AC to the server room has been restored. They are now working >> some firewall issues. >> >> >> They hope to have it restored in a few hours. >> >> >> Scott >> >> >> >> On 2021-01-21 10:07 a.m., Panech, Davlet wrote: >>> The hosting company had an issue with their servers, they are >>> working on it. They hope to be back online mid afternoon (Jan >>> 21, Eastern Standard Time). >>> ------------------------------------------------------------------------ >>> *From:* Panech, Davlet <Davlet.Panech@windriver.com> >>> <mailto:Davlet.Panech@windriver.com> >>> *Sent:* January 21, 2021 9:56 AM >>> *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> >>> <mailto:alexandru.dimofte@intel.com>; >>> starlingx-discuss@lists.starlingx.io >>> <mailto:starlingx-discuss@lists.starlingx.io> >>> <starlingx-discuss@lists.starlingx.io> >>> <mailto:starlingx-discuss@lists.starlingx.io> >>> *Subject:* Re: Cengn Mirror server in down?! >>> Thanks Alexandru, we are looking into it. >>> >>> ------------------------------------------------------------------------ >>> *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> >>> <mailto:alexandru.dimofte@intel.com> >>> *Sent:* January 21, 2021 3:24 AM >>> *To:* starlingx-discuss@lists.starlingx.io >>> <mailto:starlingx-discuss@lists.starlingx.io> >>> <starlingx-discuss@lists.starlingx.io> >>> <mailto:starlingx-discuss@lists.starlingx.io> >>> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?! >>> >>> Hello guys, >>> >>> I observed, trying to connect to >>> http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ >>> , that cengn mirror server is down. >>> >>> Best regards, >>> >>> Alex >>> >>> Logo Description automatically generated >>> >>> >>> >>> Dimofte Alexandru >>> >>> Software Engineer >>> >>> STARLINGX TEAM >>> >>> Skype no: +40 336403734 >>> >>> Personal Mobile: +40 743167456 >>> >>> alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com> >>> >>> Intel Romania >>> >>> >>> >>> >>> _______________________________________________ >>> Starlingx-discuss mailing list >>> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> >>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss >> >> _______________________________________________ >> Starlingx-discuss mailing list >> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> >> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Update The CENGN builds will now return to the regular daily schedule. None of the new builds have been sanitized as yet. Until those builds are sanitized, it might be advisable to keep software updates to a minimum. I'll defer to the TSC on that one. However from a purely CENGN/mirror/build perspective, I think we are ready to allow updates to .lst files again. Scott On 2021-01-31 4:11 p.m., Scott Little wrote:
Update
CENGN hasn't found any specific network issues as of yet. They have acknowledged that the current network configuration implemented in the wake of power outage and the switch to the new server is sub-optimal and will revisit it soon.
We have worked over the last several days to get the firewall configuration right to allow all the required services on the build server to run.
One particular quirk of the current network setup is that the DNS reported address of the mirror is inaccessible. An /etc/hosts entry is required to force the mirror address to resolve to an alternate IP address. I've spent the last several sleepless nights learning how to push that change not only into our build container, but into all the STX service containers that we build. I can finally report successful builds of the entire layered build stack.
On the mirror update side, I have adjusted the mirroring algorithm to do more validation steps as it downloads. This should prevent further corruptions, but will slow the frequency with with we can refresh our repos from upstream. In stead of updating all repos daily, some will be on a ~1/week schedule.
Scott
On 2021-01-29 1:02 a.m., Scott Little wrote:
Update
Well a few more networking issues have been solved. thanks to Davlet, and Moh at CENGN.
We have had a successful build of the 'compiler' and 'distro' layers. The 'flock' layer built the rpms, but failed to update the installer. It appears to relate to the loopback device. I'll look into that tomorrow.
~350 RPMS failed to restore from upstream. None are critical to any of our current builds. I don't plan to persu them any further
The mirroring job is throwing a lot more error messages than I'm used to seeing. Many rpm download attempts are failing, and left 2000 new corrupt rpms in its wake. I am cleaning up the new mess and adjusting the mirroring script to prevent download failures from leaving us with corrupt repos. The download failures seem to come in batches, so I'll reach out to CENGN to see if they had any network issues that might explain the problem.
Scott
On 2021-01-28 12:27 a.m., Scott Little wrote:
Update
All known firewall issues have been fixed.
Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running
Of the original ~13000 corrupted rpms, we now have ...
~9250 have been restored, including all rpms that are knowm inputs to supported builds
~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported.
~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run.
I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow.
Scott
On 2021-01-27 9:56 a.m., Scott Little wrote:
Update
Of the original ~13000 corrupted rpms, we now have ...
5700 have been restored
2400 have yet to be assessed
1800 having no backup within cengn. Upstream availability has yet to be determined.
3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving.
Scott
On 2021-01-27 1:37 a.m., Scott Little wrote:
The audit of repos identified ~13000 corrupted rpms
So far ~5000 have been restored from redundant copies within cengn.
Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution.
That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined.
Scott
On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote: > Work continues at CENGN repair damaged repos and to move us to a > new backend infrastructure with better redundancy. > > I still recommend that you avoid using download_mirrir.sh until > the repos are fully repaired. I don’t have a firm ETA for that, > perhaps two before we have full confidence in the repairs. > > Scott > > > ------------------------------------------------------------------------ > *From:* Scott Little <scott.little@windriver.com> > *Sent:* January 23, 2021 2:24 AM > *To:* starlingx-discuss@lists.starlingx.io > <starlingx-discuss@lists.starlingx.io> > *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?! > > CENGN is up, but there seems to be significant damage to the > repositories. > > > We are investigate options to restore the repositories to health. > > > I do NOT recommend running download_mirror.sh at this time. > > > Scott > > > > > On 2021-01-22 3:53 p.m., Scott Little wrote: >> >> CENGN remains out of service. >> >> >> A new problem has been identified with ceph storage. >> >> >> They continue to work the outage. >> >> >> Scott >> >> >> >> >> On 2021-01-22 9:55 a.m., Scott Little wrote: >>> >>> CENGN remains out of service >>> >>> >>> AC to the server room has been restored. They are now working >>> some firewall issues. >>> >>> >>> They hope to have it restored in a few hours. >>> >>> >>> Scott >>> >>> >>> >>> On 2021-01-21 10:07 a.m., Panech, Davlet wrote: >>>> The hosting company had an issue with their servers, they are >>>> working on it. They hope to be back online mid afternoon (Jan >>>> 21, Eastern Standard Time). >>>> ------------------------------------------------------------------------ >>>> *From:* Panech, Davlet <Davlet.Panech@windriver.com> >>>> <mailto:Davlet.Panech@windriver.com> >>>> *Sent:* January 21, 2021 9:56 AM >>>> *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> >>>> <mailto:alexandru.dimofte@intel.com>; >>>> starlingx-discuss@lists.starlingx.io >>>> <mailto:starlingx-discuss@lists.starlingx.io> >>>> <starlingx-discuss@lists.starlingx.io> >>>> <mailto:starlingx-discuss@lists.starlingx.io> >>>> *Subject:* Re: Cengn Mirror server in down?! >>>> Thanks Alexandru, we are looking into it. >>>> >>>> ------------------------------------------------------------------------ >>>> *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> >>>> <mailto:alexandru.dimofte@intel.com> >>>> *Sent:* January 21, 2021 3:24 AM >>>> *To:* starlingx-discuss@lists.starlingx.io >>>> <mailto:starlingx-discuss@lists.starlingx.io> >>>> <starlingx-discuss@lists.starlingx.io> >>>> <mailto:starlingx-discuss@lists.starlingx.io> >>>> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?! >>>> >>>> Hello guys, >>>> >>>> I observed, trying to connect to >>>> http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ >>>> , that cengn mirror server is down. >>>> >>>> Best regards, >>>> >>>> Alex >>>> >>>> Logo Description automatically generated >>>> >>>> >>>> >>>> Dimofte Alexandru >>>> >>>> Software Engineer >>>> >>>> STARLINGX TEAM >>>> >>>> Skype no: +40 336403734 >>>> >>>> Personal Mobile: +40 743167456 >>>> >>>> alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com> >>>> >>>> Intel Romania >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Starlingx-discuss mailing list >>>> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> >>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss >>> >>> _______________________________________________ >>> Starlingx-discuss mailing list >>> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> >>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss >> >> _______________________________________________ >> Starlingx-discuss mailing list >> Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> >> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Thanks again for all the work on this Scott. From: Scott Little <scott.little@windriver.com> Sent: Sunday, January 31, 2021 4:26 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! Update The CENGN builds will now return to the regular daily schedule. None of the new builds have been sanitized as yet. Until those builds are sanitized, it might be advisable to keep software updates to a minimum. I'll defer to the TSC on that one. However from a purely CENGN/mirror/build perspective, I think we are ready to allow updates to .lst files again. Scott On 2021-01-31 4:11 p.m., Scott Little wrote: Update CENGN hasn't found any specific network issues as of yet. They have acknowledged that the current network configuration implemented in the wake of power outage and the switch to the new server is sub-optimal and will revisit it soon. We have worked over the last several days to get the firewall configuration right to allow all the required services on the build server to run. One particular quirk of the current network setup is that the DNS reported address of the mirror is inaccessible. An /etc/hosts entry is required to force the mirror address to resolve to an alternate IP address. I've spent the last several sleepless nights learning how to push that change not only into our build container, but into all the STX service containers that we build. I can finally report successful builds of the entire layered build stack. On the mirror update side, I have adjusted the mirroring algorithm to do more validation steps as it downloads. This should prevent further corruptions, but will slow the frequency with with we can refresh our repos from upstream. In stead of updating all repos daily, some will be on a ~1/week schedule. Scott On 2021-01-29 1:02 a.m., Scott Little wrote: Update Well a few more networking issues have been solved. thanks to Davlet, and Moh at CENGN. We have had a successful build of the 'compiler' and 'distro' layers. The 'flock' layer built the rpms, but failed to update the installer. It appears to relate to the loopback device. I'll look into that tomorrow. ~350 RPMS failed to restore from upstream. None are critical to any of our current builds. I don't plan to persu them any further The mirroring job is throwing a lot more error messages than I'm used to seeing. Many rpm download attempts are failing, and left 2000 new corrupt rpms in its wake. I am cleaning up the new mess and adjusting the mirroring script to prevent download failures from leaving us with corrupt repos. The download failures seem to come in batches, so I'll reach out to CENGN to see if they had any network issues that might explain the problem. Scott On 2021-01-28 12:27 a.m., Scott Little wrote: Update All known firewall issues have been fixed. Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running Of the original ~13000 corrupted rpms, we now have ... ~9250 have been restored, including all rpms that are knowm inputs to supported builds ~3100 damaged rpms from the 2.0.0 release will not restored. TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported. ~650 Have yet to be restored from Upstream. The jenkins job that pulls from upstream sources continues to run. I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests. Please wait a little longer. We should be back in business sometime tomorrow. Scott On 2021-01-27 9:56 a.m., Scott Little wrote: Update Of the original ~13000 corrupted rpms, we now have ... 5700 have been restored 2400 have yet to be assessed 1800 having no backup within cengn. Upstream availability has yet to be determined. 3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving. Scott On 2021-01-27 1:37 a.m., Scott Little wrote: The audit of repos identified ~13000 corrupted rpms So far ~5000 have been restored from redundant copies within cengn. Aprox 7500 have yet to be assessed. Those remaining rpms are inputs or outputs of actual builds. The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes. Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas. I'm still pondering the implications of such a substitution. That leaves ~500 having no backup within cengn. Upstream availability has yet to be determined. Scott On 2021-01-26 1:28 a.m., Scott Little wrote: The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired. The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one. Restoration of rpm repos will proceed in several phases: 1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS. 2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available, 3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror) 4) Resume building new loads All of the release iso's on cengn were corrupt. I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts. Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps. Scott On 2021-01-25 11:08 a.m., Little, Scott wrote: Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy. I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don't have a firm ETA for that, perhaps two before we have full confidence in the repairs. Scott ________________________________ From: Scott Little <scott.little@windriver.com><mailto:scott.little@windriver.com> Sent: January 23, 2021 2:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?! CENGN is up, but there seems to be significant damage to the repositories. We are investigate options to restore the repositories to health. I do NOT recommend running download_mirror.sh at this time. Scott On 2021-01-22 3:53 p.m., Scott Little wrote: CENGN remains out of service. A new problem has been identified with ceph storage. They continue to work the outage. Scott On 2021-01-22 9:55 a.m., Scott Little wrote: CENGN remains out of service AC to the server room has been restored. They are now working some firewall issues. They hope to have it restored in a few hours. Scott On 2021-01-21 10:07 a.m., Panech, Davlet wrote: The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ________________________________ From: Panech, Davlet <Davlet.Panech@windriver.com><mailto:Davlet.Panech@windriver.com> Sent: January 21, 2021 9:56 AM To: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it. ________________________________ From: Dimofte, Alexandru <alexandru.dimofte@intel.com><mailto:alexandru.dimofte@intel.com> Sent: January 21, 2021 3:24 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Cengn Mirror server in down?! Hello guys, I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down. Best regards, Alex [Logo Description automatically generated] Dimofte Alexandru Software Engineer STARLINGX TEAM Skype no: +40 336403734 Personal Mobile: +40 743167456 alexandru.dimofte@intel.com<mailto:alexandru.dimofte@intel.com> Intel Romania _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
One other item The CENGN firewall is currently servicing browser/curl/wget requests, but is refusing some if not all requests originating from yum, possibly depending on the vintage of yum being used. It seems to hinge on the 'agent' field of the http request. Thanks to Davlet for figuring that one out. CENGN is working the problem, and think they have a fix. We'll test it in the morning. Scott On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------------------------------------------------ *From:* Scott Little <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------------------------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> <mailto:Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?! Thanks Alexandru, we are looking into it.
------------------------------------------------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <mailto:alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
Logo Description automatically generated
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com <mailto:alexandru.dimofte@intel.com>
Intel Romania
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io <mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Guys, Based on this issue, do you have any plan to make mirror list or supoort rsync so we can sync startingx packages. Thanks! On Wed, Jan 27, 2021 at 2:44 PM Scott Little <scott.little@windriver.com> wrote:
One other item
The CENGN firewall is currently servicing browser/curl/wget requests, but is refusing some if not all requests originating from yum, possibly depending on the vintage of yum being used.
It seems to hinge on the 'agent' field of the http request. Thanks to Davlet for figuring that one out.
CENGN is working the problem, and think they have a fix. We'll test it in the morning.
Scott
On 2021-01-26 1:28 a.m., Scott Little wrote:
The jenkins in the cengn build is offline while we repair the file system on the mirror. The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.
The cengn mirror has been cut over to the new storage back end. However any corrupt files from the old storage back end will remain corrupted on the new one.
Restoration of rpm repos will proceed in several phases:
1) An audit of all rpm repos is underway. Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.
2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,
3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places. e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)
4) Resume building new loads
All of the release iso's on cengn were corrupt.
I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.
Original build artifacts for 3.0.0 are missing. I do have build artifacts for builds that were intended to lead up to a 3.0.1 release. If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1. Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.
Scott
On 2021-01-25 11:08 a.m., Little, Scott wrote:
Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.
I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired. I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.
Scott
------------------------------ *From:* Scott Little <scott.little@windriver.com> <scott.little@windriver.com> *Sent:* January 23, 2021 2:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> *Subject:* Re: [Starlingx-discuss] Cengn Mirror server in down?!
CENGN is up, but there seems to be significant damage to the repositories.
We are investigate options to restore the repositories to health.
I do NOT recommend running download_mirror.sh at this time.
Scott
On 2021-01-22 3:53 p.m., Scott Little wrote:
CENGN remains out of service.
A new problem has been identified with ceph storage.
They continue to work the outage.
Scott
On 2021-01-22 9:55 a.m., Scott Little wrote:
CENGN remains out of service
AC to the server room has been restored. They are now working some firewall issues.
They hope to have it restored in a few hours.
Scott
On 2021-01-21 10:07 a.m., Panech, Davlet wrote:
The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time). ------------------------------ *From:* Panech, Davlet <Davlet.Panech@windriver.com> <Davlet.Panech@windriver.com> *Sent:* January 21, 2021 9:56 AM *To:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <alexandru.dimofte@intel.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> *Subject:* Re: Cengn Mirror server in down?!
Thanks Alexandru, we are looking into it.
------------------------------ *From:* Dimofte, Alexandru <alexandru.dimofte@intel.com> <alexandru.dimofte@intel.com> *Sent:* January 21, 2021 3:24 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Cengn Mirror server in down?!
Hello guys,
I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.
Best regards,
Alex
[image: Logo Description automatically generated]
Dimofte Alexandru
Software Engineer
STARLINGX TEAM
Skype no: +40 336403734
Personal Mobile: +40 743167456
alexandru.dimofte@intel.com
Intel Romania
_______________________________________________ Starlingx-discuss mailing listStarlingx-discuss@lists.starlingx.iohttp://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing listStarlingx-discuss@lists.starlingx.iohttp://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing listStarlingx-discuss@lists.starlingx.iohttp://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing listStarlingx-discuss@lists.starlingx.iohttp://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing listStarlingx-discuss@lists.starlingx.iohttp://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (9)
-
Dimofte, Alexandru
-
Jones, Bruce E
-
Lists
-
Little, Scott
-
Miller, Frank
-
Panech, Davlet
-
Qi Zhang
-
Scott Little
-
Zvonar, Bill