<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{margin-right:0in;
margin-left:0in;
font-size:10.0pt;
font-family:"Times New Roman",serif}
span.emailstyle17
{font-family:"Calibri",sans-serif;
color:#1F497D}
span.EmailStyle20
{font-family:"Calibri",sans-serif;
color:#1F497D}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
-->
</style><style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0" vlink="#954F72" link="#0563C1" lang="EN-US">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:black"></span><span style="color:black"></span>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<div>
<div class="WordSection1">
<div>
<div>
<div>
<div><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"></span><span style="color:black"></span>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"><br>
</span></p>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Hi Folks,</span><span style="color:black"></span>
</p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"><br>
Last piece of the Backup & Restore StarlingX feature has been merged on Friday (also known as B&R).
</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt; font-family:"Tahoma",sans-serif; color:#1F497D"> </span><span style="font-family:"Tahoma",sans-serif; color:black">
</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">This feature provides us with a last resort disaster recovery option in cases where the StarlingX software and/or data
</span><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:red"></span><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">are compromised. The feature provides a backup utility to create a tarball containing a snapshot
with the deployment state. This tarball contains all that's needed to restore the deployment to a previously good working state.
<br>
<br>
This feature provides two B&R options:<br>
I. Platform B&R - backup and restores the platform configuration and data. This step takes care of the platform data: nodes, their configuration and user data.<br>
II. Openstack Application - backup and restores the stx-openstack application<br>
<br>
The Platform B&R supports two modes: </span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">I. Keep all Ceph cluster data (to recover from platform software or data corruption)</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">II. Wipe all Ceph cluster data (to perform a full system restore for software and/or application corruption)</span><span style="color:black"></span></p>
<p class="MsoNormal"><font color="000000"><span style="font-size: 10pt; font-family: "Tahoma", sans-serif;">In both cases, the restore will re-build all platform data required. Applications will require additional steps.
</span><span style="font-size: 10pt; font-family: "Tahoma", sans-serif;">For the case where the user decides to keep the Ceph data (eg: application data is preserved)</span><span style="font-size: 10pt; font-family: "Tahoma", sans-serif;"> some effort may be
required by the applications to re-use the preserved data</span><i><span style="font-size: 11pt; font-family: "Tahoma", sans-serif;">.
</span></i></font><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"><font color="000000">For the case where the user decides to start with a clean Ceph, the user will need to recover application data through other means (e.g. snapshot
and use rbd export commands for block devices).</font><br>
<br>
</span><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:#1F497D"></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">The backup and restore is done through Ansible and the restore process involves reinstalling each node in the system. The backup is run from the active controller
and the restore is run from controller-0 (local play) or remotely by running Ansible and pointing it to controller-0 (remote play).
<br>
<br>
This email describe the Platform B&R procedure. The Openstack procedure will be described in a later mail.<br>
<br>
</span><font color="000000"><span style="font-size: 10pt; font-family: "Courier New", serif;"> Backing up the platform<br>
==========================<br>
<br>
Local play</span><span style="font-size: 10pt; font-family: "Courier New", serif;"> method</span><span style="font-size: 10pt; font-family: "Courier New", serif;"><br>
~~~~~~~~~~~~</span><span style="font-size: 10pt; font-family: "Courier New", serif;">~~~~~~~</span></font></p>
<font color="000000"></font>
<p class="MsoNormal"><font color="000000"><span style="font-family: "Courier New", serif;">R</span><span style="font-size: 10pt; font-family: "Courier New", serif;">un:<br>
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml -e "ansible_become_pass=<sysadmin password> admin_password=<sysadmin password>"<br>
<br>
The <admin_password> and <ansible_become_pass> need to be set correctly via the “-e” option on the command line or an override file or in the ansible secret file.<br>
This will </span><span style="font-size: 10pt; font-family: "Courier New", serif;">o</span><span style="font-size: 10pt; font-family: "Courier New", serif;">utput a file named in this format: <inventory_hostname>_platform_backup_<timestamp>.tgz. The prefix
<platform_backup_filename_prefix> and <openstack_backup_filename_prefix> can be overridden via the “-e” option on the command line or an override file.<br>
<br>
The generated backup tar files will look like this: localhost_platform_backup_2019_08_08_15_25_36.tgz and localhost_openstack_backup_2019_08_08_15_25_36.tgz. They are located in /opt/backups directory on controller-0.<br>
<br>
Remote play</span><span style="font-size: 10pt; font-family: "Courier New", serif;"> method</span><span style="font-size: 10pt; font-family: "Courier New", serif;"><br>
~~~~~~~~~~~~~</span><span style="font-size: 10pt; font-family: "Courier New", serif;">~~~~~~~</span></font><span style="font-size:10.0pt; font-family:"Courier New",serif; color:black"><br>
1. Login to the host where ansible is installed and clone the playbook code from opendev at
<a href="https://opendev.org/starlingx/ansible-playbooks.git" target="_blank" rel="noopener noreferrer">
https://opendev.org/starlingx/ansible-playbooks.git</a><br>
2. Provide an inventory file, either a customized one that is specified via the ‘-i’ option or the default one which resides in Ansible configuration directory (i.e. /etc/ansible/hosts), must specify the IP of the controller host. For
<font color="000000">example, if the host-name is my_vbox, the inventory-file should have an entry called my_vbox :<br>
---<br>
all:<br>
hosts:<br>
wc68:<br>
ansible_host: 128.222.100.02<br>
my_vbox:<br>
ansible_host: 128.224.141.74<br>
<br>
3. Run ansible:<br>
ansible-playbook <path-to-backup-playbook-entry-file> --limit host-name -i <inventory-file> -e <optional-extra-vars><br>
<br>
The generated backup tar files can be found in <host_backup_dir> which is $HOME by default. It can be overridden by “-e” option on the command line or in an override file.<br>
<br>
The generated backup tar have naming convention as in a local play.<br>
<br>
Example:<br>
ansible-playbook /localdisk/designer/repo/cgcs-root/stx/stx-ansible-playbooks/playbookconfig/src/playbooks/backup-restore/backup.yml --limit my_vbox -i $HOME/br_test/hosts -e "host_backup_dir=$HOME/br_test ansible_become_pass=Li69nux* admin_password=Li69nux*
ansible_ssh_pass=Li69nux* ansible_ssh_pass=Li69nux*"<br>
<br>
Detailed information of the contents of the backup<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
- Postgresql config: Backup roles, table spaces and schemas for databases<br>
- Postgresql data:<br>
o template1, sysinv, barbican db data, fm db data, <br>
o keystone db for primary region, <br>
o dcmanager db for dc controller, <br>
o dcorch db for dc controller<br>
- LDAP db<br>
- Ceph crushmap<br>
- DNS server list<br>
- System Inventory network overrides. These are needed at restore to correctly set up the OS configuration:<br>
o addrpool<br>
o pxeboot_subnet<br>
o management_subnet<br>
o management_start_address<br>
o cluster_host_subnet<br>
o cluster_pod_subnet<br>
o cluster_service_subnet<br>
o external_oam_subnet<br>
o external_oam_gateway_address<br>
o external_oam_floating_address<br>
- Docker registries on controller<br>
- Docker no=proxy<br>
- Backup up data:<br>
o OS configuration<br>
ok: [localhost] => (item=/etc) - note although everything here is backed up, not all of the content will be restored.<br>
o Home directory ‘sysadmin’ user and all LDAP user accounts<br>
ok: [localhost] => (item=/home)<br>
o Geberated platform configuration<br>
ok: [localhost] => (item=/opt/platform/config/19.09)<br>
ok: [localhost] => (item=/opt/platform/puppet/19.09/hieradata) - All the hieradata under is backed-up. However only the static hieradata (static.yaml and secure_static.yaml) will be restored to bootstrap controller-0.<br>
o Keyring<br>
ok: [localhost] => (item=/opt/platform/.keyring/19.09)<br>
o Patching and package repositories<br>
ok: [localhost] => (item=/opt/patching)<br>
ok: [localhost] => (item=/www/pages/updates)<br>
o Extension filesystem<br>
ok: [localhost] => (item=/opt/extension)<br>
o atch-vault filesystem for distributed cloud system-controller<br>
ok: [localhost] => (item=/opt/patch-vault)<br>
o Armada manifests<br>
ok: [localhost] => (item=/opt/platform/armada/19.09)<br>
o Helm charts<br>
ok: [localhost] => (item=/opt/platform/helm_charts)<br>
<br>
<br>
Restoring<br>
=============<br>
<br>
I. Platform Restore<br>
----------------------<br>
Platform restore can be done in two modes:<br>
A. By keeping the ceph cluster data intact - after restore previous Ceph data will remain in place.
</font></span><font color="000000"><span style="font-size: 10pt; font-family: "Courier New", serif;"></span></font></p>
<font color="000000"><span style="font-size: 10pt; font-family: "Courier New", serif;">B. By wiping Ceph cluster entireley - Ceph cluster has to be recreated</span><span style="font-size: 10pt; font-family: "Courier New", serif;">. </span></font>
<p class="MsoNormal"><font color="000000"><span style="font-size: 10pt; font-family: "Courier New", serif;"><br>
Warning:<br>
- The StarlingX system data backup can only be used to restore the system from which the backup was made. You cannot use the system data backup to restore the system on different hardware.<br>
- Restore has to use the exact same version of boot image (iso) that </span><span style="font-size: 10pt; font-family: "Courier New", serif;">was</span><span style="font-size: 10pt; font-family: "Courier New", serif;"> used
</span><span style="font-size: 10pt; font-family: "Courier New", serif;">at the time of</span><span style="font-size: 10pt; font-family: "Courier New", serif;"> the original installation.<br>
<br>
Prerequisite: a backup file stored externally and StarlingX iso<br>
<br>
Prepare for the procedure:<br>
- Power down all nodes. If you have a storage setup and want to keep Ceph data </span>
<span style="font-size: 10pt; font-family: "Courier New", serif;">then </span><span style="font-size: 10pt; font-family: "Courier New", serif;">power down all the nodes except the storage ones; cluster has to be functional during restore.<br>
- Install iso on controller-0 (boot from ISO and reinstall it)<br>
<br>
The restore procedure has multiple steps, dependent on the system type. The first step of the procedure is identical for all: run the ansible playbook with the backup file as input. For this step, similar to the backup procedure, we have two options: l</span><span style="font-size: 10pt; font-family: "Courier New", serif;">o</span></font><span style="font-size:10.0pt; font-family:"Courier New",serif; color:black"><font color="000000">cal
and remote play.<br>
<br>
Local play<br>
~~~~~~~~~~~~<br>
First download the backup to the controller (you can also use an external storage device, e.g. an USB drive).<br>
<br>
Then run the command:<br>
<br>
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "initial_backup_dir=<location_of_tarball> ansible_become_pass=<admin_password> admin_password=<admin_password> backup_filename=<backup_filename>"<br>
<br>
Optional:<br>
o <wipe_ceph_osds> set to wipe_ceph_osds=true or wipe_ceph_osds=false (default to false). This will select one of the two restore modes: A. keep Ceph data intact (false) or B. start with an empty ceph cluster (true).<br>
<br>
Example for a backup file in /home/sysadmin:<br>
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "initial_backup_dir=/home/sysadmin ansible_become_pass=Li69nux* admin_password=Li69nux* backup_filename=localhost_platform_backup_2019_09_27_07_48_48.tgz"<br>
<br>
Remote play<br>
~~~~~~~~~~~~~~</font><br>
<font color="000000">1. Login to the host where ansible is installed and clone the playbook code from OpenDev at</font>
<a href="https://opendev.org/starlingx/ansible-playbooks.git" target="_blank" rel="noopener noreferrer">
https://opendev.org/starlingx/ansible-playbooks.git</a><br>
<font color="000000">2. Provide an inventory file, either a customized one that is specified via the ‘-i’ option or the default one which resides in Ansible configuration directory (i.e. /etc/ansible/hosts), must specify the IP of the controller host. For example,
if the host-name is my_vbox, the inventory-file should have an entry called my_vbox :<br>
---<br>
all:<br>
hosts:<br>
wc68:<br>
ansible_host: 128.222.100.02<br>
my_vbox:<br>
ansible_host: 128.224.141.74<br>
<br>
3. Run ansible:<br>
ansible-playbook <path-to-backup-playbook-entry-file> --limit host-name -i <inventory-file> -e <optional-extra-vars><br>
<br>
Where optional-extra-vars can be:<br>
o <wipe_ceph_osds> set to wipe_ceph_osds=true or wipe_ceph_osds=false (default to false). This will select if the user wants to keep Ceph data intact (false) or start with an empty ceph cluster (true)<br>
o The <backup_filename> is the platform backup tar file. It must be provided via the “-e” option on the command line, e.g. -e “backup_filename= localhost_platform_backup_2019_07_15_14_46_37.tgz”<br>
o The <initial_backup_dir> is the location on the Ansible control machine where the platform backup tar file is placed to restore the platform. It must be provided via “-e” option on the command line.<br>
o The <admin_password> , <ansible_become_pass> and <ansible_ssh_pass> need to be set correctly via the “-e” option on the command line or in the ansible secret file. <ansible_ssh_pass> is the password to the sysadmin user on controller-0.<br>
o The <ansible_remote_tmp> should be set to a new directory (no need to create it ahead of time) under /home/sysadmin on controller-0 via the “-e” option on the command line<br>
<br>
e.g. <br>
ansible-playbook /localdisk/designer/jenkins/tis-stx-dev/cgcs-root/stx/ansible-playbooks/playbookconfig/src/playbooks/restore_platform.yml --limit my_vbox -i $HOME/br_test/hosts -e " ansible_become_pass=Li69nux* admin_password=Li69nux* ansible_ssh_pass=Li69nux*
initial_backup_dir=$HOME/br_test backup_filename=my_vbox_system_backup_2019_08_08_15_25_36.tgz ansible_remote_tmp=/home/sysadmin/ansible-restore"<br>
<br>
After ansible is exectued then the following steps are based on the deployment mode:<br>
<br>
AIO-SX<br>
~~~~~~~~~<br>
1. Unlock controller-0 & wait for it to boot<br>
<br>
AIO-DX<br>
~~~~~~~~~<br>
1. Unlock controller-0 & wait for it to boot<br>
2. Reinstall controller-1 (boot it from PXE, wait for it to become 'online')<br>
3. Unlock controller-1<br>
<br>
Standard with controller storage<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
1. Unlock controller-0 & wait for it to boot<br>
2. Reinstall controller-1 and all computes (boot them from PXE, wait for them to become 'online')<br>
3. Unlock controller-1 and wait for it to be available<br>
4. Unlock compute nodes<br>
<br>
Standard with storage nodes<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
This depends on wipe_ceph_osds configuration.<br>
<br>
A. If wipe_ceph_osd is false (default option):<br>
1. Unlock controller-0 & wait for it to boot. Afer unlock you will see storage nodes as available and ceph operational.<br>
2. Reinstall controller-1 and all computes (boot them from PXE, wait for them to become 'online')<br>
3. Unlock controller-1 and wait for it to be available<br>
4. Unlock compute nodes<br>
<br>
Storage nodes do not need reinstall.<br>
<br>
B. If wipe_ceph_osd is false (default option):<br>
1. Unlock controller-0 & wait for it to boot. Afer unlock you will see all nodes, including storage nodes as offline.<br>
2. Reinstall controller-1, all storage and compute nodes (boot them from PXE, wait for them to become 'online')<br>
3. Unlock controller-1 and wait for it to be available<br>
4. Unlock storage nodes and wait for them to be available</font></span></p>
<font color="000000"></font>
<div><font color="000000"></font>
<p class="MsoNormal"><font color="000000"><span style="font-size: 10pt; font-family: "Courier New", serif;">5</span><span style="font-size: 10pt; font-family: "Courier New", serif;">. Unlock compute nodes and wait for them to be available</span></font><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:black">Regards,</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:black">Ovidiu Poncea</span><span style="color:black"></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>