Category: DBaaS

12 Days of #Cloud: Day 7

There are three types of storage expansion possible on Oracle #DBaaS instance.

1. Increase Default File Systems

Increasing the default file systems that hold data files (/u02), FRA and backup (/u03) is easy.  These file systems can be increased by using the menu item “Scale Up / Down Service” and choosing the appropriate file system to extend. Below example shows extending the backup file system.

No other file systems can be increased easily. To increase a storage volume, you need to create a new volume, copy contents from old to new, and remove the old.

2. Creating New Storage Volume

In the above screen (Scale up / down Service), you can choose the first option of “Create New Storage Volume”. Specify a size in GB and click the confirm (Yes, Scale Up/Down Service) button.

The DBaaS Monitor Console (OS Storage) shows the storage is available on the node.

Also, view details in the Oracle Compute Cloud Services, Storage section.

#DBaaS service assigns the next available device and volume number.

File system is updated in /etc/fstab as well, so when VM is rebooted, the storage you just added is automatically mounted.

Notice that, though we added a new storage volume that has nothing to do with the running database, DBaaS service did take an outage on the environment and rebooted the node.

A note from Oracle documentation: Looks like an unnecessary limitation.

3. Attaching New Storage Volume from Compute Cloud

This is similar to #2, but more manual steps. Used Oracle documentation as a reference to do these steps.

Choose Oracle Compute Cloud Service and click on the “Create Storage Volume” button.

Pick a name and size for the storage volume, and the type of storage.

Once the storage volume is created, attach the storage to a DBaaS VM.

Choose the instance where the volume need to be attached to.

Storage is attached to the DBaaS host.

But the storage is not visible on the Linux VM as it is not mounted. Before mounting the storage, there are few steps to complete. The new storage added is /dev/xvdg.

A note from Oracle Documentation.

 

12 Days of #Cloud: Day 6

When I created the #DBaaS instance, did not create a backup policy. I chose the backups as “None”. Now that some work has been done on the database, would like to create an adhoc backup as well as want to schedule regular local disk backups.

On-demand backup can be performed by running script “/var/opt/oracle/bkup_api/bkup_api bkup_start” as the root user.

[opc@CLDB4ACED ~]$ sudo -s
[root@CLDB4ACED opc]#
[root@CLDB4ACED opc]# /var/opt/oracle/bkup_api/bkup_api bkup_start
DBaaS Backup API V1.5 @2015 Multi-Oracle home
-> Action : bkup_start
-> logfile: /var/opt/oracle/bkup_api/log/bkup_api.log
API INFO : Backups are not configured on this DB Instance. Nothing to do.
*
* RETURN CODE:0
##################################################
[root@CLDB4ACED opc]#

Oops, no backup configured! So, for the first time, we need to configure backup before we can perform a backup.

Reference: Oracle Documentation

Backup configuration is saved in file /var/opt/oracle/ocde/assistants/bkup/bkup.cfg. Let’s edit the file and add required parameters.

[opc@CLDB4ACED ~]$ sudo -s
[root@CLDB4ACED opc]#
[root@CLDB4ACED opc]# cd /var/opt/oracle/ocde/assistants/bkup/
[root@CLDB4ACED bkup]# ls *cfg
bkup.cfg
[root@CLDB4ACED bkup]# cat bkup.cfg
# The bkup.cfg file will hold all the config info for BKUP
[root@CLDB4ACED bkup]# cp -p bkup.cfg bkup.cfg.1
[root@CLDB4ACED bkup]# vi bkup.cfg 

[root@CLDB4ACED bkup]# cat bkup.cfg
# The bkup.cfg file will hold all the config info for BKUP
bkup_cfg_files=yes
bkup_disk=yes
bkup_disk_recovery_window=2
bkup_oss=no
bkup_oss_url=oss_url
bkup_oss_user=username
bkup_oss_passwd=password
bkup_oss_recovery_window=30
[root@CLDB4ACED bkup]#

[root@CLDB4ACED bkup]# chown root bkup.cfg
[root@CLDB4ACED bkup]# chmod 0600 bkup.cfg

[root@CLDB4ACED bkup]# ./bkup -cfg bkup.cfg
Starting BKUP
Logfile is /var/opt/oracle/log/bkup/bkup_2016-12-06_17:01:47.log
Config file is bkup.cfg
dbname: CDDB01
Dataguard configuration
Enabled: 0
Mode: PRIMARY
Looking for previous configuration:
Directory : /home/oracle/bkup/CDDB01
  -> obkup
  -> dbcfg.spec
  -> log
 -> Found: 3 files
 -> Moving previous configutarion to /home/oracle/bkup/CDDB01_20161206170154
No initial bkup of PFILE needed.
Configuring Backup to disk
Common RMAN Config
Instantiating obkup
Instantiating dbcfg.spec
Configuring backup of Config File
Updating Control File Record Keep Time
Enabling block change tracking
Updating RMAN defaults
Adding entry to crontab
INFO: Archivelog management enabled.
Warning: there are already 1 entries for crontab
  -> 0,30 * * * * root /home/oracle/bkup/CDDB01/obkup -dbname=CDDB01 -archivelog
Accessing to your Database ID ..
The DBaaS instance database id is: 4135691349
Deleting unencrypted autobackups.
#### Completed Execution.
[root@CLDB4ACED bkup]#

BKUP_DISK is for local disk backup, and enabled local backup by setting this parameter to YES. Local disk backups are written to the Flash Recovery Area, which is /u03/app/oracle/fast_recovery_area directory. The parameter BKUP_OSS is to write backups to Oracle Cloud Storage.

The bkup script, backup assistant, created few cron jobs to do the backup periodically.

[root@CLDB4ACED etc]# cat /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=""
HOME=/

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

0,30 * * * * root /home/oracle/bkup/CDDB01/obkup -dbname=CDDB01 -archivelog
15 03 * * 6 oracle /var/opt/oracle/cleandb/cleandblogs.pl
54 2 * * * root /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=CDDB01
[root@CLDB4ACED etc]#

Let’s kick off the backup again.

[root@CLDB4ACED etc]# /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=CDDB01
DBaaS Backup API V1.5 @2015 Multi-Oracle home
-> Action : bkup_start
-> logfile: /var/opt/oracle/bkup_api/log/bkup_api.log
** process started with PID: 23297
** see log file for monitor progress
-------------------------------------
[root@CLDB4ACED etc]#

[root@CLDB4ACED etc]# cat /var/opt/oracle/bkup_api/log/bkup_api.log
... ... ...
Tue, 06 Dec 2016 17:17:25 DBaaS Backup API V1.5 @2015 Multi-Oracle home
Tue, 06 Dec 2016 17:17:25 -> Action : bkup_start
Tue, 06 Dec 2016 17:17:25 -> logfile: /var/opt/oracle/bkup_api/log/bkup_api.log
Tue, 06 Dec 2016 17:17:25 ** process started with PID: 23297
Tue, 06 Dec 2016 17:17:25 db35376e-bbd7-11e6-9159-c6b07b253285 -> Starting execution of backup log in background
Tue, 06 Dec 2016 17:17:25 ** see log file for monitor progress
Tue, 06 Dec 2016 17:17:25 -------------------------------------
Tue, 06 Dec 2016 17:17:25 db35376e-bbd7-11e6-9159-c6b07b253285 STARTING BACKUP REQUEST
Tue, 06 Dec 2016 17:17:25 Creating Backup Api registry datatabase
Tue, 06 Dec 2016 17:17:25 UUID db35376e-bbd7-11e6-9159-c6b07b253285 written with PID 23297
Tue, 06 Dec 2016 17:17:33 db35376e-bbd7-11e6-9159-c6b07b253285 Checking if CDDB01 resource is available
Tue, 06 Dec 2016 17:17:33 db35376e-bbd7-11e6-9159-c6b07b253285 registering request into the database
Tue, 06 Dec 2016 17:17:37 db35376e-bbd7-11e6-9159-c6b07b253285 current backups 0
Tue, 06 Dec 2016 17:17:37 db35376e-bbd7-11e6-9159-c6b07b253285 command /home/oracle/bkup/CDDB01/obkup -dbname=CDDB01
Tue, 06 Dec 2016 17:19:54 db35376e-bbd7-11e6-9159-c6b07b253285 The backup process failed

Backup failed! This time, it was due to insufficient local storage to hold the backup. Since I chose no backup option, Oracle Cloud Service did not allocate storage to perform backups to /u03 file system. The database was created in ARCHIVELOG mode, and the Cloud Service setup a cron job to remove the archivelog files hourly.

It took a while to figure out how to add storage to /u03 file system. More storage is added to the standard file systems by using the “Scale up or down Resources” menu item.

Added 50G to “Backup Storage Volume”, which is /u03. 6G volume is now 56G.

Adding disk space alone is not sufficient, need to increase the Fast Recovery Area as well.

Though backup is scheduled to run periodically using the cronjobs, lets kick off a backup to verify if the added storage in /u03 resolved the backup issue.

Voila, successful backup!

DBaaS monitor console also shows the status.

The /home/oracle/bkup/CDDB01/obkup cron job performs operating system file backups. The file backups are based on two configuration files, and their contents are:

[root@CLDB4ACED CDDB01]# pwd
/home/oracle/bkup/CDDB01

[root@CLDB4ACED CDDB01]# cat dbcfg.spec
### Oracle_Home configuration files.
#
# Doc Spec
dbcfg.spec
# DB id
dbid
#
# Directories
/u01/app/oracle/product/12.2.0/dbhome_1/admin/CDDB01/xdb_wallet
/u01/app/oracle/admin/CDDB01/xdb_wallet
/u01/app/oracle/admin/CDDB01/db_wallet
/u01/app/oracle/admin/CDDB01/opc_wallet
# Note: tde_wallet must be backed up in a different location than DATA bkup.
/u01/app/oracle/admin/CDDB01/tde_wallet
/u01/app/oracle/admin/CDDB01/cat_wallet
#/u01/app/oraInventory
#
# Single files
/u01/app/oracle/admin/CDDB01/opcCDDB01.ora
/u01/app/oracle/product/12.2.0/dbhome_1/dbs/orapwCDDB01
/u01/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
/u01/app/oracle/product/12.2.0/dbhome_1/network/admin/sqlnet.ora
/u01/app/oracle/product/12.2.0/dbhome_1/network/admin/tnsnames.ora
/u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/env_rdbms.mk
/u01/app/oracle/product/12.2.0/dbhome_1/rdbms/lib/ins_rdbms.mk
#
# Creg
/var/opt/oracle/creg/CDDB01.ini
#
[root@CLDB4ACED CDDB01]#
[root@CLDB4ACED CDDB01]# cat oscfg.spec
## OS Configuration Files
#
# Doc Spec
oscfg.spec
#
# Directories
/etc/rc.d
/home/oracle/bkup
#
# Single files
/home/oracle/.bashrc
/etc/crontab
/etc/sysctl.conf
/etc/passwd
/etc/group
/etc/oraInst.loc
/etc/oratab
/etc/fstab
[root@CLDB4ACED CDDB01]#

As you can see, all important files from the Operating System are backed up. If you have directories to be backed up, add to the dbcfg.spec file.

Enabling and performing cloud backups is also using similar steps. Update these parameters accordingly in the bkup.cfg file.

bkup_oss=yes
bkup_oss_url=oss_url
bkup_oss_user=username
bkup_oss_passwd=password
bkup_oss_recovery_window=30