[Starlingx-discuss] Cengn Mirror server in down?!

Thu Jan 28 17:36:47 UTC 2021

We just merged a fix for tb.sh/Dockerfile for CENGN-related build problems that may affect some builds, depending on the environment:

https://review.opendev.org/c/starlingx/tools/+/772790

If you compiled a builder image during the outage (during the last week or so) for the first time, it may affect your builds. The symptoms include this error message from "docker build":

  rmdir: failed to remove '/var/lib/mock': No such file or directory

You may need to refresh your starlingx/tools checkout or cherry pick that commit into any private branches you are interested in building. This patch is only in the master branch.

Another workaround is to delete any cached builder images that came out of starlingx/tools/Dockerfile using "docker image rm" etc.

Hope this helps,
D.

________________________________
From: Panech, Davlet <Davlet.Panech at windriver.com>
Sent: January 28, 2021 12:31 PM
To: Zvonar, Bill <Bill.Zvonar at windriver.com>; Little, Scott <Scott.Little at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?!

We just merged a fix for tb.sh/Dockerfile for CENGN-related build problems that may affect some builds, depending on the environment:

________________________________
From: Zvonar, Bill <Bill.Zvonar at windriver.com>
Sent: January 28, 2021 6:31 AM
To: Little, Scott <Scott.Little at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?!

Thanks Scott for grinding through this.

From: Scott Little <scott.little at windriver.com>
Sent: Thursday, January 28, 2021 12:28 AM
To: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?!

Update

All known firewall issues have been fixed.

Jenkins is now up and running on the build server once again. Our first manually triggered build attempt of the compiler layer is now running

Of the original ~13000 corrupted rpms, we now have ...

~9250 have been restored, including all rpms that are knowm inputs to supported builds

~3100 damaged rpms from the 2.0.0 release will not restored.  TSC and community meetings agreed that 2.0.0, being 2 releases out of date, is no longer supported.

~650 Have yet to be restored from Upstream.  The jenkins job that pulls from upstream sources continues to run.

I would like to have successful CENGN builds of all layers before you all hammer the new server with your pent-up download_mirror.sh requests.  Please wait a little longer.  We should be back in business sometime tomorrow.

Scott

On 2021-01-27 9:56 a.m., Scott Little wrote:

Update

Of the original ~13000 corrupted rpms, we now have ...

5700 have been restored

2400 have yet to be assessed

1800 having no backup within cengn. Upstream availability has yet to be determined.

3100 I propose to drop as they fall within the 2.0.0 release which is in very bad shape, whereas 2.0.1 is in fairly good shape and worth saving.

Scott

On 2021-01-27 1:37 a.m., Scott Little wrote:

The audit of repos identified ~13000 corrupted rpms

So far ~5000 have been restored from redundant copies within cengn.

Aprox 7500 have yet to be assessed.  Those remaining rpms are inputs or outputs of actual builds.  The rpms produced by builds might carry the same files name, but may contain different content due to embedded file dates if not real changes.  Extra care is required to restore from the correct source. If an exact match is unavailable, I may be forced to use a source that is 'near' in time and therefore less likely to contain anything other than trivial file timestamp deltas.  I'm still pondering the implications of such a substitution.

That leaves ~500 having no backup within cengn.  Upstream availability has yet to be determined.

Scott

On 2021-01-26 1:28 a.m., Scott Little wrote:

The jenkins in the cengn build is offline while we repair the file system on the mirror.  The build machine both read from, and writes to the mirror, and I don't want it to cause further corruptions while the mirror is being repaired.

The cengn mirror has been cut over to the new storage back end.  However any corrupt files from the old storage back end will remain corrupted on the new one.

Restoration of rpm repos will proceed in several phases:

1) An audit of all rpm repos is underway.  Rpm's that fail their checksum are being removed and repodata will be updated to reflect the missing RPMS.

2) The mirror update job (downloading rpms from upstream rpm repos) will be run to restore as many missing files as possible. Many rpms may no longer be available as we kept old versions of packages that upstream drops when a new version is available,

3) If rpms that are not available from upstream, but are required for our build, we can attempt to restore them from one of several places.   e.g. the inputs directory of various builds, or the downloads directory on the cengn build server (independent storage from the mirror)

4) Resume building new loads

All of the release iso's on cengn were corrupt.

I've been able to restore 4.0.1 and 2.0.1 from the original build artifacts.

Original build artifacts for 3.0.0 are missing.  I do have build artifacts for builds that were intended to lead up to a 3.0.1 release.  If a verifiable copy of 3.0.0 can't be found, we may want to consider issuing a 3.0.1.  Alternatively I can rebuild 3.0.0, but the file checksums wont match the original build due to embedded timestamps.

Scott

On 2021-01-25 11:08 a.m., Little, Scott wrote:

Work continues at CENGN repair damaged repos and to move us to a new backend infrastructure with better redundancy.

I still recommend that you avoid using download_mirrir.sh until the repos are fully repaired.  I don’t have a firm ETA for that, perhaps two before we have full confidence in the repairs.

Scott

________________________________

From: Scott Little <scott.little at windriver.com><mailto:scott.little at windriver.com>
Sent: January 23, 2021 2:24 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io><mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Cengn Mirror server in down?!

CENGN is up, but there seems to be significant damage to the repositories.

We are investigate options to restore the repositories to health.

I do NOT recommend running download_mirror.sh at this time.

Scott

On 2021-01-22 3:53 p.m., Scott Little wrote:

CENGN remains out of service.

A new problem has been identified with ceph storage.

They continue to work the outage.

Scott

On 2021-01-22 9:55 a.m., Scott Little wrote:

CENGN remains out of service

AC to the server room has been restored. They are now working some firewall issues.

They hope to have it restored in a few hours.

Scott

On 2021-01-21 10:07 a.m., Panech, Davlet wrote:

The hosting company had an issue with their servers, they are working on it. They hope to be back online mid afternoon (Jan 21, Eastern Standard Time).

________________________________

From: Panech, Davlet <Davlet.Panech at windriver.com><mailto:Davlet.Panech at windriver.com>
Sent: January 21, 2021 9:56 AM
To: Dimofte, Alexandru <alexandru.dimofte at intel.com><mailto:alexandru.dimofte at intel.com>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io><mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: Cengn Mirror server in down?!

Thanks Alexandru, we are looking into it.

________________________________

From: Dimofte, Alexandru <alexandru.dimofte at intel.com><mailto:alexandru.dimofte at intel.com>
Sent: January 21, 2021 3:24 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io><mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Cengn Mirror server in down?!

Hello guys,

I observed, trying to connect to http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ , that cengn mirror server is down.

Best regards,

Alex

  [Logo Description                                          automatically generated]

Dimofte Alexandru

Software Engineer

STARLINGX TEAM

Skype no: +40 336403734

Personal Mobile: +40 743167456

alexandru.dimofte at intel.com<mailto:alexandru.dimofte at intel.com>

Intel Romania

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________

Starlingx-discuss mailing list

Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>

http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210128/ae9e6846/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 5187 bytes
Desc: image002.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210128/ae9e6846/attachment-0001.png>