<div dir="ltr">Hi Guys,<div><br></div><div>Based on this issue, do you have any plan to make mirror list or supoort rsync so we can sync startingx packages.</div><div><br></div><div>Thanks!</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 27, 2021 at 2:44 PM Scott Little <<a href="mailto:scott.little@windriver.com">scott.little@windriver.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">

  
  <div>
    <p>One other item</p>
    <p><br>
    </p>
    <p>The CENGN firewall is currently servicing browser/curl/wget
      requests, but is refusing some if not all requests originating
      from yum, possibly depending on the vintage of yum being used. <br>
    </p>
    <p><br>
    </p>
    <p>It seems to hinge on the 'agent' field of the http request. 
      Thanks to Davlet for figuring that one out.</p>
    <p><br>
    </p>
    <p>CENGN is working the problem, and think they have a fix.  We'll
      test it in the morning.<br>
    </p>
    <p><br>
    </p>
    <p>Scott</p>
    <p><br>
    </p>
    <p><br>
    </p>
    <div>On 2021-01-26 1:28 a.m., Scott Little
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div>The jenkins in the cengn build is
        offline while we repair the file system on the mirror.  The
        build machine both read from, and writes to the mirror, and I
        don't want it to cause further corruptions while the mirror is
        being repaired.<br>
      </div>
      <div><br>
      </div>
      <div>The cengn mirror has been cut over to
        the new storage back end.  However any corrupt files from the
        old storage back end will remain corrupted on the new one.<br>
      </div>
      <div><br>
      </div>
      <div>Restoration of rpm repos will proceed
        in several phases:</div>
      <div><br>
      </div>
      <div>1) An audit of all rpm repos is
        underway.  Rpm's that fail their checksum are being removed and
        repodata will be updated to reflect the missing RPMS.</div>
      <div><br>
      </div>
      <div>2) The mirror update job (downloading
        rpms from upstream rpm repos) will be run to restore as many
        missing files as possible. Many rpms may no longer be available
        as we kept old versions of packages that upstream drops when a
        new version is available,<br>
      </div>
      <div><br>
      </div>
      <div>3) If rpms that are not available
        from upstream, but are required for our build, we can attempt to
        restore them from one of several places.   e.g. the inputs
        directory of various builds, or the downloads directory on the
        cengn build server (independent storage from the mirror)<br>
      </div>
      <div><br>
      </div>
      <div>4) Resume building new loads<br>
      </div>
      <div><br>
      </div>
      <div><br>
      </div>
      <div><br>
      </div>
      <p>All of the release iso's on cengn were corrupt.</p>
      <p><br>
      </p>
      <p>I've been able to restore 4.0.1 and 2.0.1 from the original
        build artifacts.</p>
      <p><br>
      </p>
      <p>Original build artifacts for 3.0.0 are missing.  I do have
        build artifacts for builds that were intended to lead up to a
        3.0.1 release.  If a verifiable copy of 3.0.0 can't be found, we
        may want to consider issuing a 3.0.1.  Alternatively I can
        rebuild 3.0.0, but the file checksums wont match the original
        build due to embedded timestamps.<br>
      </p>
      <p><br>
      </p>
      <p><br>
      </p>
      <div>Scott</div>
      <div><br>
      </div>
      <div><br>
      </div>
      <div><br>
      </div>
      <div>On 2021-01-25 11:08 a.m., Little,
        Scott wrote:<br>
      </div>
      <blockquote type="cite">
        <div>Work continues at CENGN repair damaged repos and to move us
          to a new backend infrastructure with better redundancy.</div>
        <div><br>
        </div>
        <div>I still recommend that you avoid using download_mirrir.sh
          until the repos are fully repaired.  I don’t have a firm ETA
          for that, perhaps two before we have full confidence in the
          repairs.</div>
        <div><br>
        </div>
        <div>Scott </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <hr style="display:inline-block;width:98%">
        <div id="gmail-m_5661243939917570784divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
            Scott Little <a href="mailto:scott.little@windriver.com" target="_blank"><scott.little@windriver.com></a><br>
            <b>Sent:</b> January 23, 2021 2:24 AM<br>
            <b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a>
            <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank"><starlingx-discuss@lists.starlingx.io></a><br>
            <b>Subject:</b> Re: [Starlingx-discuss] Cengn Mirror server
            in down?!</font>
          <div> </div>
        </div>
        <div>
          <p>CENGN is up, but there seems to be significant damage to
            the repositories.  <br>
          </p>
          <p><br>
          </p>
          <p>We are investigate options to restore the repositories to
            health.</p>
          <p><br>
          </p>
          <p>I do NOT recommend running download_mirror.sh at this time.</p>
          <p><br>
          </p>
          <p>Scott</p>
          <p><br>
          </p>
          <p><br>
          </p>
          <p><br>
          </p>
          <div>On 2021-01-22 3:53 p.m., Scott
            Little wrote:<br>
          </div>
          <blockquote type="cite">
            <p>CENGN remains out of service.</p>
            <p><br>
            </p>
            <p>A new problem has been identified with ceph storage.</p>
            <p><br>
            </p>
            <p>They continue to work the outage.  <br>
            </p>
            <p><br>
            </p>
            <p>Scott</p>
            <p><br>
            </p>
            <p><br>
            </p>
            <p><br>
            </p>
            <div>On 2021-01-22 9:55 a.m.,
              Scott Little wrote:<br>
            </div>
            <blockquote type="cite">
              <p>CENGN remains out of service</p>
              <p><br>
              </p>
              <p>AC to the server room has been restored. They are now
                working some firewall issues.</p>
              <p><br>
              </p>
              <p>They hope to have it restored in a few hours.</p>
              <p><br>
              </p>
              <p>Scott</p>
              <p><br>
              </p>
              <p><br>
              </p>
              <div>On 2021-01-21 10:07 a.m.,
                Panech, Davlet wrote:<br>
              </div>
              <blockquote type="cite">
                
                <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> The hosting company
                  had an issue with their servers, they are working on
                  it. They hope to be back online mid afternoon (Jan 21,
                  Eastern Standard Time).<br>
                </div>
                <hr style="display:inline-block;width:98%">
                <div id="gmail-m_5661243939917570784x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Panech, Davlet <a href="mailto:Davlet.Panech@windriver.com" target="_blank"><Davlet.Panech@windriver.com></a><br>
                    <b>Sent:</b> January 21, 2021 9:56 AM<br>
                    <b>To:</b> Dimofte, Alexandru <a href="mailto:alexandru.dimofte@intel.com" target="_blank">
                      <alexandru.dimofte@intel.com></a>; <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">
                      starlingx-discuss@lists.starlingx.io</a> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">
                      <starlingx-discuss@lists.starlingx.io></a><br>
                    <b>Subject:</b> Re: Cengn Mirror server in down?!</font>
                  <div> </div>
                </div>
                
                <div dir="ltr">
                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Thanks Alexandru,
                    we are looking into it.<br>
                  </div>
                  <div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br>
                  </div>
                  <hr style="display:inline-block;width:98%">
                  <div id="gmail-m_5661243939917570784x_x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Dimofte, Alexandru <a href="mailto:alexandru.dimofte@intel.com" target="_blank"><alexandru.dimofte@intel.com></a><br>
                      <b>Sent:</b> January 21, 2021 3:24 AM<br>
                      <b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">
                        starlingx-discuss@lists.starlingx.io</a> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">
                        <starlingx-discuss@lists.starlingx.io></a><br>
                      <b>Subject:</b> [Starlingx-discuss] Cengn Mirror
                      server in down?!</font>
                    <div> </div>
                  </div>
                  
                  <div lang="EN-US">
                    <div>
                      <p>Hello guys,</p>
                      <p> </p>
                      <p>I observed, trying to
                        connect to <a href="http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/" target="_blank">
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/</a> ,
                        that cengn mirror server is down.</p>
                      <p> </p>
                      <p>Best regards,</p>
                      <p>Alex<span style="color:rgb(31,73,125)"></span></p>
                      <p><span style="color:rgb(31,73,125)"> </span></p>
                      <table style="margin-left:0.15in;border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
                        <tbody>
                          <tr>
                            <td style="width:99.3pt;border-top:none;border-bottom:none;border-left:none;border-right:1.5pt solid rgb(0,113,197);padding:0in 5.4pt" width="132" valign="top">
                              <p><span style="color:rgb(0,176,240)">  </span><span style="color:rgb(0,176,240)" lang="PL"><img id="gmail-m_5661243939917570784x_x_x_Picture_x0020_1" alt="Logo
                                    Description automatically generated" src="cid:17742d3c2e4692e331" width="127" height="55" border="0"></span></p>
                            </td>
                            <td style="width:365.75pt;padding:0in 5.4pt" width="488" valign="top">
                              <p><span>Dimofte
                                  Alexandru</span></p>
                              <p><span>Software
                                  Engineer</span></p>
                              <p><span>STARLINGX
                                  TEAM</span></p>
                              <p><span>Skype
                                  no: +40 336403734</span></p>
                              <p><span>Personal
                                  Mobile: +40 743167456</span></p>
                              <p><span style="color:rgb(0,176,240)"><a href="mailto:alexandru.dimofte@intel.com" target="_blank"><span style="color:rgb(0,176,240)">alexandru.dimofte</span><span>@intel.com</span></a></span><span></span></p>
                              <p><span lang="DE">Intel Romania</span></p>
                            </td>
                          </tr>
                        </tbody>
                      </table>
                      <p><span style="font-size:9pt;font-family:"Lucida Handwriting";color:black"><br>
                          <br>
                        </span></p>
                      <p> </p>
                    </div>
                  </div>
                </div>
                <br>
                <fieldset></fieldset>
                <pre>_______________________________________________
Starlingx-discuss mailing list
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
              </blockquote>
              <br>
              <fieldset></fieldset>
              <pre>_______________________________________________
Starlingx-discuss mailing list
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
            </blockquote>
            <br>
            <fieldset></fieldset>
            <pre>_______________________________________________
Starlingx-discuss mailing list
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
          </blockquote>
        </div>
        <br>
        <fieldset></fieldset>
        <pre>_______________________________________________
Starlingx-discuss mailing list
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
      </blockquote>
      <p><br>
      </p>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
Starlingx-discuss mailing list
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a>
</pre>
    </blockquote>
  </div>

_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" rel="noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote></div>