<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Update</div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">Of the original ~13000 corrupted rpms,
      we now have ...<br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">5700 have been restored</div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">2400 have yet to be assessed</div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">1800 having no backup within cengn.
      Upstream availability has yet to be determined.</div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">3100 I propose to drop as they fall
      within the 2.0.0 release which is in very bad shape, whereas 2.0.1
      is in fairly good shape and worth saving.<br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">Scott</div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">On 2021-01-27 1:37 a.m., Scott Little
      wrote:<br>
    </div>
    <blockquote type="cite" cite="mid:34128b7c-d78d-13a5-8682-99b8615487a5@windriver.com">
      
      <p>The audit of repos identified ~13000 corrupted rpms</p>
      <p><br>
      </p>
      <p>So far ~5000 have been restored from redundant copies within
        cengn.  <br>
      </p>
      <p><br>
      </p>
      <p>Aprox 7500 have yet to be assessed.  Those remaining rpms are
        inputs or outputs of actual builds.  The rpms produced by builds
        might carry the same files name, but may contain different
        content due to embedded file dates if not real changes.  Extra
        care is required to restore from the correct source. If an exact
        match is unavailable, I may be forced to use a source that is
        'near' in time and therefore less likely to contain anything
        other than trivial file timestamp deltas.  I'm still pondering
        the implications of such a substitution. <br>
      </p>
      <p><br>
      </p>
      <p> That leaves ~500 having no backup within cengn.  Upstream
        availability has yet to be determined.</p>
      <p><br>
      </p>
      <p>Scott<br>
      </p>
      <p><br>
      </p>
      <p><br>
      </p>
      <div class="moz-cite-prefix">On 2021-01-26 1:28 a.m., Scott Little
        wrote:<br>
      </div>
      <blockquote type="cite" cite="mid:eb52e4c9-f06d-4683-e068-5aebb51ce7e4@windriver.com">
        <div class="moz-cite-prefix">The jenkins in the cengn build is
          offline while we repair the file system on the mirror.  The
          build machine both read from, and writes to the mirror, and I
          don't want it to cause further corruptions while the mirror is
          being repaired.<br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">The cengn mirror has been cut over
          to the new storage back end.  However any corrupt files from
          the old storage back end will remain corrupted on the new one.<br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">Restoration of rpm repos will
          proceed in several phases:</div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">1) An audit of all rpm repos is
          underway.  Rpm's that fail their checksum are being removed
          and repodata will be updated to reflect the missing RPMS.</div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">2) The mirror update job
          (downloading rpms from upstream rpm repos) will be run to
          restore as many missing files as possible. Many rpms may no
          longer be available as we kept old versions of packages that
          upstream drops when a new version is available,<br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">3) If rpms that are not available
          from upstream, but are required for our build, we can attempt
          to restore them from one of several places.   e.g. the inputs
          directory of various builds, or the downloads directory on the
          cengn build server (independent storage from the mirror)<br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">4) Resume building new loads<br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <p>All of the release iso's on cengn were corrupt.</p>
        <p><br>
        </p>
        <p>I've been able to restore 4.0.1 and 2.0.1 from the original
          build artifacts.</p>
        <p><br>
        </p>
        <p>Original build artifacts for 3.0.0 are missing.  I do have
          build artifacts for builds that were intended to lead up to a
          3.0.1 release.  If a verifiable copy of 3.0.0 can't be found,
          we may want to consider issuing a 3.0.1.  Alternatively I can
          rebuild 3.0.0, but the file checksums wont match the original
          build due to embedded timestamps.<br>
        </p>
        <p><br>
        </p>
        <p><br>
        </p>
        <div class="moz-cite-prefix">Scott</div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix"><br>
        </div>
        <div class="moz-cite-prefix">On 2021-01-25 11:08 a.m., Little,
          Scott wrote:<br>
        </div>
        <blockquote type="cite" cite="mid:BY5PR11MB441835A7B47040913D2228B186BD9@BY5PR11MB4418.namprd11.prod.outlook.com">
          <div>Work continues at CENGN repair damaged repos and to move
            us to a new backend infrastructure with better redundancy.</div>
          <div><br>
          </div>
          <div>I still recommend that you avoid using download_mirrir.sh
            until the repos are fully repaired.  I don’t have a firm ETA
            for that, perhaps two before we have full confidence in the
            repairs.</div>
          <div><br>
          </div>
          <div>Scott </div>
          <div><br>
          </div>
          <div><br>
          </div>
          <hr style="display:inline-block;width:98%" tabindex="-1">
          <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
              Scott Little <a class="moz-txt-link-rfc2396E" href="mailto:scott.little@windriver.com" moz-do-not-send="true"><scott.little@windriver.com></a><br>
              <b>Sent:</b> January 23, 2021 2:24 AM<br>
              <b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">starlingx-discuss@lists.starlingx.io</a>
              <a class="moz-txt-link-rfc2396E" href="mailto:starlingx-discuss@lists.starlingx.io" moz-do-not-send="true"><starlingx-discuss@lists.starlingx.io></a><br>
              <b>Subject:</b> Re: [Starlingx-discuss] Cengn Mirror
              server in down?!</font>
            <div> </div>
          </div>
          <div>
            <p>CENGN is up, but there seems to be significant damage to
              the repositories.  <br>
            </p>
            <p><br>
            </p>
            <p>We are investigate options to restore the repositories to
              health.</p>
            <p><br>
            </p>
            <p>I do NOT recommend running download_mirror.sh at this
              time.</p>
            <p><br>
            </p>
            <p>Scott</p>
            <p><br>
            </p>
            <p><br>
            </p>
            <p><br>
            </p>
            <div class="x_moz-cite-prefix">On 2021-01-22 3:53 p.m.,
              Scott Little wrote:<br>
            </div>
            <blockquote type="cite">
              <p>CENGN remains out of service.</p>
              <p><br>
              </p>
              <p>A new problem has been identified with ceph storage.</p>
              <p><br>
              </p>
              <p>They continue to work the outage.  <br>
              </p>
              <p><br>
              </p>
              <p>Scott</p>
              <p><br>
              </p>
              <p><br>
              </p>
              <p><br>
              </p>
              <div class="x_moz-cite-prefix">On 2021-01-22 9:55 a.m.,
                Scott Little wrote:<br>
              </div>
              <blockquote type="cite">
                <p>CENGN remains out of service</p>
                <p><br>
                </p>
                <p>AC to the server room has been restored. They are now
                  working some firewall issues.</p>
                <p><br>
                </p>
                <p>They hope to have it restored in a few hours.</p>
                <p><br>
                </p>
                <p>Scott</p>
                <p><br>
                </p>
                <p><br>
                </p>
                <div class="x_moz-cite-prefix">On 2021-01-21 10:07 a.m.,
                  Panech, Davlet wrote:<br>
                </div>
                <blockquote type="cite">
                  <style type="text/css" style="display:none">
<!--
p
        {margin-top:0;
        margin-bottom:0}
-->
</style>
                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;
                    font-size:12pt; color:rgb(0,0,0)"> The hosting
                    company had an issue with their servers, they are
                    working on it. They hope to be back online mid
                    afternoon (Jan 21, Eastern Standard Time).<br>
                  </div>
                  <hr tabindex="-1" style="display:inline-block;
                    width:98%">
                  <div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Panech, Davlet <a class="x_moz-txt-link-rfc2396E" href="mailto:Davlet.Panech@windriver.com" moz-do-not-send="true"><Davlet.Panech@windriver.com></a><br>
                      <b>Sent:</b> January 21, 2021 9:56 AM<br>
                      <b>To:</b> Dimofte, Alexandru <a class="x_moz-txt-link-rfc2396E" href="mailto:alexandru.dimofte@intel.com" moz-do-not-send="true">
                        <alexandru.dimofte@intel.com></a>; <a class="x_moz-txt-link-abbreviated" href="mailto:starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">
                        starlingx-discuss@lists.starlingx.io</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">
                        <starlingx-discuss@lists.starlingx.io></a><br>
                      <b>Subject:</b> Re: Cengn Mirror server in down?!</font>
                    <div> </div>
                  </div>
                  <style type="text/css" style="display:none">
<!--
p
        {margin-top:0;
        margin-bottom:0}
-->
</style>
                  <div dir="ltr">
                    <div style="font-family:Calibri,Arial,Helvetica,sans-serif;
                      font-size:12pt; color:rgb(0,0,0)"> Thanks
                      Alexandru, we are looking into it.<br>
                    </div>
                    <div style="font-family:Calibri,Arial,Helvetica,sans-serif;
                      font-size:12pt; color:rgb(0,0,0)"> <br>
                    </div>
                    <hr tabindex="-1" style="display:inline-block;
                      width:98%">
                    <div id="x_x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri,
                        sans-serif" color="#000000"><b>From:</b>
                        Dimofte, Alexandru <a class="x_moz-txt-link-rfc2396E" href="mailto:alexandru.dimofte@intel.com" moz-do-not-send="true"><alexandru.dimofte@intel.com></a><br>
                        <b>Sent:</b> January 21, 2021 3:24 AM<br>
                        <b>To:</b> <a class="x_moz-txt-link-abbreviated" href="mailto:starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">
                          starlingx-discuss@lists.starlingx.io</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">
                          <starlingx-discuss@lists.starlingx.io></a><br>
                        <b>Subject:</b> [Starlingx-discuss] Cengn Mirror
                        server in down?!</font>
                      <div> </div>
                    </div>
                    <style>
<!--
@font-face
        {font-family:"Cambria Math"}
@font-face
        {font-family:Calibri}
@font-face
        {font-family:"Intel Clear"}
@font-face
        {font-family:"Intel Clear Pro"}
@font-face
        {font-family:"Lucida Handwriting"}
p.x_x_x_MsoNormal, li.x_x_x_MsoNormal, div.x_x_x_MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
a:link, span.x_x_x_MsoHyperlink
        {color:#0563C1;
        text-decoration:underline}
span.x_x_x_EmailStyle17
        {font-family:"Calibri",sans-serif;
        color:windowtext}
.x_x_x_MsoChpDefault
        {font-family:"Calibri",sans-serif}
@page WordSection1
        {margin:1.0in 1.0in 1.0in 1.0in}
-->
</style>
                    <div lang="EN-US">
                      <div class="x_x_x_WordSection1">
                        <p class="x_x_x_MsoNormal">Hello guys,</p>
                        <p class="x_x_x_MsoNormal"> </p>
                        <p class="x_x_x_MsoNormal">I observed, trying to
                          connect to <a href="http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/" moz-do-not-send="true">
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/</a> ,
                          that cengn mirror server is down.</p>
                        <p class="x_x_x_MsoNormal"> </p>
                        <p class="x_x_x_MsoNormal">Best regards,</p>
                        <p class="x_x_x_MsoNormal">Alex<span style="color:#1F497D"></span></p>
                        <p class="x_x_x_MsoNormal"><span style="color:#1F497D"> </span></p>
                        <table class="x_x_x_MsoNormalTable" style="margin-left:.15in;
                          border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
                          <tbody>
                            <tr>
                              <td style="width:99.3pt; border:none;
                                border-right:solid #0071C5 1.5pt;
                                padding:0in 5.4pt 0in 5.4pt" width="132" valign="top">
                                <p class="x_x_x_MsoNormal"><span style="color:#00B0F0">  </span><span style="color:#00B0F0" lang="PL"><img id="x_x_x_Picture_x0020_1" alt="Logo Description
                                      automatically generated" class="" data-outlook-trace="F:3|T:3" src="cid:part12.0EB194EB.26CD1174@windriver.com" width="127" height="55" border="0"></span></p>
                              </td>
                              <td style="width:365.75pt; padding:0in
                                5.4pt 0in 5.4pt" width="488" valign="top">
                                <p class="x_x_x_MsoNormal"><span style="">Dimofte Alexandru</span></p>
                                <p class="x_x_x_MsoNormal"><span style="">Software Engineer</span></p>
                                <p class="x_x_x_MsoNormal"><span style="">STARLINGX TEAM</span></p>
                                <p class="x_x_x_MsoNormal"><span style="">Skype no: +40 336403734</span></p>
                                <p class="x_x_x_MsoNormal"><span style="">Personal Mobile: +40
                                    743167456</span></p>
                                <p class="x_x_x_MsoNormal"><span style="color:#00B0F0"><a href="mailto:alexandru.dimofte@intel.com" moz-do-not-send="true"><span style="color:#00B0F0">alexandru.dimofte</span><span style="">@intel.com</span></a></span><span style=""></span></p>
                                <p class="x_x_x_MsoNormal"><span style="" lang="DE">Intel Romania</span></p>
                              </td>
                            </tr>
                          </tbody>
                        </table>
                        <p class="x_x_x_MsoNormal"><span style="font-size:9.0pt;
                            font-family:"Lucida Handwriting";
                            color:black"><br>
                            <br>
                          </span></p>
                        <p class="x_x_x_MsoNormal"> </p>
                      </div>
                    </div>
                  </div>
                  <br>
                  <fieldset class="x_mimeAttachmentHeader"></fieldset>
                  <pre class="x_moz-quote-pre">_______________________________________________
Starlingx-discuss mailing list
<a class="x_moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">Starlingx-discuss@lists.starlingx.io</a>
<a class="x_moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" moz-do-not-send="true">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
                </blockquote>
                <br>
                <fieldset class="x_mimeAttachmentHeader"></fieldset>
                <pre class="x_moz-quote-pre">_______________________________________________
Starlingx-discuss mailing list
<a class="x_moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">Starlingx-discuss@lists.starlingx.io</a>
<a class="x_moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" moz-do-not-send="true">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
              </blockquote>
              <br>
              <fieldset class="x_mimeAttachmentHeader"></fieldset>
              <pre class="x_moz-quote-pre">_______________________________________________
Starlingx-discuss mailing list
<a class="x_moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">Starlingx-discuss@lists.starlingx.io</a>
<a class="x_moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" moz-do-not-send="true">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
            </blockquote>
          </div>
          <br>
          <fieldset class="mimeAttachmentHeader"></fieldset>
          <pre class="moz-quote-pre" wrap="">_______________________________________________
Starlingx-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">Starlingx-discuss@lists.starlingx.io</a>
<a class="moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" moz-do-not-send="true">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
        </blockquote>
        <p><br>
        </p>
        <br>
        <fieldset class="mimeAttachmentHeader"></fieldset>
        <pre class="moz-quote-pre" wrap="">_______________________________________________
Starlingx-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io" moz-do-not-send="true">Starlingx-discuss@lists.starlingx.io</a>
<a class="moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" moz-do-not-send="true">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
      </blockquote>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
Starlingx-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Starlingx-discuss@lists.starlingx.io">Starlingx-discuss@lists.starlingx.io</a>
<a class="moz-txt-link-freetext" href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>