Merge pull request #30 from oracle-devrel/block.storage.01.00.28.06.2023

initial block storage version 1.0
oracle-devrel · Jun 30, 2023 · db02137 · db02137
2 parents fef1635 + 4983b29
commit db02137
Show file tree

Hide file tree

Showing 7 changed files with 819 additions and 0 deletions.
diff --git a/cloud-infrastructure/storage/block-storage/README.md b/cloud-infrastructure/storage/block-storage/README.md
@@ -0,0 +1,71 @@
+# Block Storage
+
+High-performance block storage at any scale.
+
+We've designed our storage platform as an ideal complement to Oracle compute and networking services to support the highest performance requirements. OCI Block Volume uses the latest NVMe SSDs and provides nonblocking network connectivity to every host. Oracle delivers a consistent, low-latency performance of up to 225 IOPS/GB to a maximum of 300,000 IOPS and 2,680 MB/sec of throughput per volume.
+
+
+# Table of Contents
+
+1. [Team Publications](#team-publications)
+2. [Useful Links](#useful-uinks)
+3. [Reusable Assets Overview](#reusable-assets-overview)
+
+## Team Publications
+
+- [OCI provides native high availability and data resilience](https://blogs.oracle.com/cloud-infrastructure/post/oci-provide-cloud-resilience-by-default)
+    - Blog: Oracle Cloud Infrastructure (OCI) provides high data durability and availability as a core feature, enabling businesses to concentrate on their customer experience and economic success.
+- [Make your cloud resilient against regional outages](https://www.youtube.com/watch?v=IVqLe_XH_AE)
+    - 6-minute Video about "Make your cloud resilient against regional outages".
+- [Effortless Cloud Resilience – Automate your Disaster Recovery with Oracle Cloud Infrastructure](https://www.youtube.com/watch?v=P3qWyjE9HMQ)
+    - 1h Video about "Effortless Cloud Resilience – Automate your Disaster Recovery with Oracle Cloud Infrastructure".
+
+## Useful Links
+
+- [OCI Block Storage documentation](https://docs.oracle.com/en-us/iaas/Content/Block/home.htm)
+  - Oracle Block Storage documentation.
+- Introduction to Oracle Cloud Infrastructure Block Volume [Part 01](https://www.youtube.com/watch?v=rNrBxdDC8vc) and [Part 02](https://www.youtube.com/watch?v=ldZDySWv8sw)
+  - Short Videos to introduce Oracle Cloud Infrastructure Block Volume.
+- [OCI Block Storage on oracle.com](https://www.oracle.com/cloud/storage/block-volumes/)
+  - Oracle Cloud Infrastructure (OCI) Block Volumes provide reliable, high-performance, low-cost block storage. OCI Block Volumes persist beyond the lifespan of a virtual machine, have built-in redundancy, and can scale to 1 PB per compute instance.
+- [Blogs on Block Storage](https://blogs.oracle.com/authors/max-verun)
+  - See all Block Storage Blogs from Oracle's Product Management.
+- [Block Storage Release Notes](https://docs.oracle.com/en-us/iaas/releasenotes/services/blockvolume/)
+- [Block Volumes FAQ](https://www.oracle.com/cloud/storage/block-volumes/faq)
+- [Oracle Cloud Infrastructure Service Level Agreement (SLA)](https://www.oracle.com/cloud/sla/)
+  - Enterprises demand more than just availability from their cloud infrastructure. Mission-critical workloads also require consistent performance, and the ability to manage, monitor, and modify resources running in the cloud at any time. Only Oracle offers end-to-end SLAs covering the performance, availability, and manageability of services.
+- [Oracle PaaS and IaaS Public Cloud Services Pillar Document Block Storage](https://www.oracle.com/assets/paas-iaas-pub-cld-srvs-pillar-4021422.pdf#page=28)
+  - Oracle offers end-to-end SLAs covering performance, availability, and manageability. Here you find details regarding Block Storage.
+- [Oracle PaaS and IaaS Universal Credits Service Descriptions - Storage section](https://www.oracle.com/us/corporate/contracts/paas-iaas-universal-credits-3940775.pdf#page=178)
+  - OCI Block Storage Service Description.
+- [Resilience and Continuous Availability of Oracle Cloud Infrastructure Services and Platform FAQ](https://www.oracle.com/cloud/iaas/faq.html)
+  - Resilience and Continuous Availability FAQs.
+- [Oracle Cloud Infrastructure Vault: Block Volume Encryption](https://www.youtube.com/watch?v=3GBPIx4hlRU)
+ - This video helps you to encrypt a block volume with the KMS Vault customer-managed encryption key.
+
+## Reusable Assets Overview
+
+- [Cloud Resilience](https://gitlab.com/hmielimo/cloud-resilience/-/blob/main/doc/cloud.resilience/README.md)
+  - All details regarding storage-based resiliency incl. example backup and recovery automation script.
+- [Cloud Resilience by default](https://gitlab.com/hmielimo/cloud-resilience-by-default/)
+  - All details regarding storage-based Server resiliency including
+    - Set up your Server with Resilience by default Using the Console
+    - Set up your Server with Resilience by default using CLI
+    - Performance-Based Auto-tune Use Case script
+- [OCI Storage Health-Check](https://gitlab.com/hmielimo/oci-storage-health-check/)
+  - Storage Health-Check example scripts
+    - validate all boot and block volumes regarding backup policies
+    - showcase boot and block volume security e.g. using Customer managed key, Key-Rotation, Backup
+- [TRIM showcase - example script](asset/trim-showcase.md)
+- [boot/block volume security best practice - example script](asset/secure.storage.sh)
+- [boot/block volume individual (to a customer-managed bucket) backup and restore  - example script](asset/block-volume-backup-and-restore.md)
+
+
+
+# License
+
+Copyright (c) 2023 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/folder-structure/LICENSE) for more details.
diff --git a/cloud-infrastructure/storage/block-storage/asset/README.md b/cloud-infrastructure/storage/block-storage/asset/README.md
@@ -0,0 +1,12 @@
+# Asset
+
+OCI Block Storage assets.
+
+
+# License
+
+Copyright (c) 2023 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/folder-structure/LICENSE) for more details.
diff --git a/...d-infrastructure/storage/block-storage/asset/block-volume-backup-and-restore.md b/...d-infrastructure/storage/block-storage/asset/block-volume-backup-and-restore.md
@@ -0,0 +1,148 @@
+# License
+
+Copyright (c) 2023 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/folder-structure/LICENSE) for more details.
+
+
+# Showcase manual block volume backup and restore
+
+## Information
+- [Mounting an Object Storage Bucket as File System on Oracle Linux](https://blogs.oracle.com/cloud-infrastructure/post/mounting-an-object-storage-bucket-as-file-system-on-oracle-linux)
+- [OCI Compute - How to Install s3fs-fuse on Oracle Linux 8 (Doc ID 2938554.1)](https://support.oracle.com/epmos/faces/DocumentDisplay?id=2938554.1)
+
+## Prepare your environment
+- set up your individual "operate" instance
+	- An Oracle enterprise Linux server with Command Line Interface (*see [Set up your Server with Resilience by default using CLI](https://gitlab.com/hmielimo/cloud-resilience-by-default/#set-up-your-server-with-resilience-by-default-using-cli)  for details*)
+	- create and attach block volume: e.g. backup-test (*50GB*)
+	- create a bucket e.g. backup-test
+
+## Set needed variables (*please adjust variables to your needs before past into your terminal*)
+~~~
+DEVICEoPATH1=/dev/oracleoci/oraclevdc
+DEVICEoPATH2=/dev/oracleoci/oraclevdc1
+MYoSOURCE1=/dev/sdb
+MYoSOURCE2=/dev/sdb1
+DATAoDIR=/mnt/MyBlockVolume.backup-test
+OBJECTDATAoDIR=/mnt/MyObjectStorage.backup-test
+BUCKEToNAME=backup-test
+MYoREGION=<region>
+MYoNAMESPACE=<your_namespace>
+MYoSECREToKEY=<your_secret_key>
+MYoACCESSoKEY=<your_access_key>
+~~~
+
+## Format and mount block volume
+~~~
+sudo umount ${DATAoDIR}
+sudo mkdir ${DATAoDIR}
+sudo fdisk -l
+sudo fdisk ${MYoSOURCE1}
+
+ n   add a new partition
+ p   primary (0 primary, 0 extended, 4 free)
+ Partition number (1-4, default 1):
+ First sector (2048-419430399, default 2048):
+ Last sector, +sectors or +size{K,M,G,T,P} (2048-419430399, default 419430399): +1G
+ w   write table to disk and exit
+
+sudo mkfs -t ext4 ${DEVICEoPATH2}
+sudo mount ${DEVICEoPATH2} ${DATAoDIR}
+sudo chown -R opc:opc ${DATAoDIR}
+ls -lah ${DATAoDIR}
+sudo mount | grep ${DATAoDIR}
+lsblk
+~~~
+
+## Update fstab
+~~~
+sudo vi /etc/fstab
+/dev/oracleoci/oraclevdc1  /mnt/MyBlockVolume.backup-test    ext4    defaults,_netdev,noatime  0  2
+~~~
+
+## Mounting an Object Storage Bucket as File System
+~~~
+# Generate Secret Key: <your_secret_key>
+
+sudo yum update
+sudo dnf upgrade
+sudo dnf search epel
+
+sudo dnf clean all
+sudo dnf repolist all
+sudo dnf install oracle-epel-release-el8
+
+sudo tee /etc/yum.repos.d/ol8-epel.repo<<EOF
+[ol8_developer_EPEL]
+name= Oracle Linux \$releasever EPEL (\$basearch)
+baseurl=https://yum.oracle.com/repo/OracleLinux/OL8/developer/EPEL/\$basearch/
+gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
+gpgcheck=1
+enabled=1
+EOF
+sudo dnf makecache
+sudo dnf repolist all
+
+sudo yum install s3fs-fuse
+
+echo ${MYoACCESSoKEY}:${MYoSECREToKEY} > ${HOME}/.passwd-s3fs
+chmod 600 ${HOME}/.passwd-s3fs
+cat ${HOME}/.passwd-s3fs
+sudo rm -fr ${OBJECTDATAoDIR}
+sudo mkdir ${OBJECTDATAoDIR}
+sudo chown -R opc:opc ${OBJECTDATAoDIR}
+sudo umount ${OBJECTDATAoDIR}
+s3fs ${BUCKEToNAME} ${OBJECTDATAoDIR} -o endpoint=${MYoREGION} -o passwd_file=${HOME}/.passwd-s3fs -o url=https://${MYoNAMESPACE}.compat.objectstorage.${MYoREGION}.oraclecloud.com/ -o nomultipart -o use_path_request_style
+ls -lah ${OBJECTDATAoDIR}
+sudo mount | grep ${OBJECTDATAoDIR}
+sudo mount | grep ${DATAoDIR}
+~~~
+
+## Create a Set of test data
+~~~
+touch ${DATAoDIR}/test.tmp
+touch ${OBJECTDATAoDIR}/test.tmp
+
+tee ${DATAoDIR}/test.txt<<EOF
+This is a testfile located in a block volume.
+EOF
+
+tee ${OBJECTDATAoDIR}/test.txt<<EOF
+This is a testfile located in object storage.
+EOF
+
+cat ${DATAoDIR}/test.txt
+cat ${OBJECTDATAoDIR}/test.txt
+
+ls -lah ${DATAoDIR}
+ls -lah ${OBJECTDATAoDIR}
+~~~
+
+## Backup block volume (*here only a partition for testing purposes*)
+~~~
+sudo umount ${DATAoDIR}
+sudo dd if=${MYoSOURCE} conv=sync,noerror bs=128M | gzip -c > ${OBJECTDATAoDIR}/my-disk.image.gz
+ls -lah ${OBJECTDATAoDIR}/my-disk.image.gz
+~~~
+
+## Delete ${DATAoDIR}/test.txt
+~~~
+sudo mount ${DEVICEoPATH} ${DATAoDIR}
+ls -lah ${DATAoDIR}
+rm ${DATAoDIR}/test.txt
+ls -lah ${DATAoDIR}
+~~~
+
+## Restore
+~~~
+sudo umount ${DATAoDIR}
+gunzip -c ${OBJECTDATAoDIR}/my-disk.image.gz | sudo dd of=${MYoSOURCE}
+~~~
+
+## Test if restore was successful
+~~~
+sudo mount ${DEVICEoPATH} ${DATAoDIR}
+ls -lah ${DATAoDIR}
+~~~
diff --git a/cloud-infrastructure/storage/block-storage/asset/functions0.sh b/cloud-infrastructure/storage/block-storage/asset/functions0.sh
@@ -0,0 +1,155 @@
+#!/bin/bash
+# Version: @(#).functions0.sh 1.0.0 
+# License
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License (UPL), Version 1.0.
+# See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/folder-structure/LICENSE) for more details.
+#
+#@  functions needed to run healthcheck.storage.sh
+#@
+#
+# Update history:
+#
+# V 1.0.0 28.06.2023 initial version
+#
+
+
+if [ 1 -eq 1 ] ; then # define colors
+Color_Off='\e[0m'       # Text Reset
+
+# Regular Colors
+Black='\e[0;30m'        # Black
+Red='\e[0;31m'          # Red
+Green='\e[0;32m'        # Green
+Yellow='\e[0;33m'       # Yellow
+Blue='\e[0;34m'         # Blue
+Purple='\e[0;35m'       # Purple
+Cyan='\e[0;36m'         # Cyan
+White='\e[0;37m'        # White
+
+# Bold
+BBlack='\e[1;30m'       # Black
+BRed='\e[1;31m'         # Red
+BGreen='\e[1;32m'       # Green
+BYellow='\e[1;33m'      # Yellow
+BBlue='\e[1;34m'        # Blue
+BPurple='\e[1;35m'      # Purple
+BCyan='\e[1;36m'        # Cyan
+BWhite='\e[1;37m'       # White
+
+# Underline
+UBlack='\e[4;30m'       # Black
+URed='\e[4;31m'         # Red
+UGreen='\e[4;32m'       # Green
+UYellow='\e[4;33m'      # Yellow
+UBlue='\e[4;34m'        # Blue
+UPurple='\e[4;35m'      # Purple
+UCyan='\e[4;36m'        # Cyan
+UWhite='\e[4;37m'       # White
+
+# Background
+On_Black='\e[40m'       # Black
+On_Red='\e[41m'         # Red
+On_Green='\e[42m'       # Green
+On_Yellow='\e[43m'      # Yellow
+On_Blue='\e[44m'        # Blue
+On_Purple='\e[45m'      # Purple
+On_Cyan='\e[46m'        # Cyan
+On_White='\e[47m'       # White
+
+# High Intensity
+IBlack='\e[0;90m'       # Black
+IRed='\e[0;91m'         # Red
+IGreen='\e[0;92m'       # Green
+IYellow='\e[0;93m'      # Yellow
+IBlue='\e[0;94m'        # Blue
+IPurple='\e[0;95m'      # Purple
+ICyan='\e[0;96m'        # Cyan
+IWhite='\e[0;97m'       # White
+
+# Bold High Intensity
+BIBlack='\e[1;90m'      # Black
+BIRed='\e[1;91m'        # Red
+BIGreen='\e[1;92m'      # Green
+BIYellow='\e[1;93m'     # Yellow
+BIBlue='\e[1;94m'       # Blue
+BIPurple='\e[1;95m'     # Purple
+BICyan='\e[1;96m'       # Cyan
+BIWhite='\e[1;97m'      # White
+
+# High Intensity backgrounds
+On_IBlack='\e[0;100m'   # Black
+On_IRed='\e[0;101m'     # Red
+On_IGreen='\e[0;102m'   # Green
+On_IYellow='\e[0;103m'  # Yellow
+On_IBlue='\e[0;104m'    # Blue
+On_IPurple='\e[0;105m'  # Purple
+On_ICyan='\e[0;106m'    # Cyan
+On_IWhite='\e[0;107m'   # White
+fi
+
+if [ 1 -eq 1 ] ; then # set environement
+. parameter1.sh
+fi
+
+if [ 1 -eq 1 ] ; then # level 0 functions (this functions do not need/call other then level 0 functions)
+
+function color_print() {
+if [ ${DEBUG_LEVEL} -eq 0 ] ; then echo -e "$1: $2 ${Color_Off}"     ; fi
+if [ ${DEBUG_LEVEL} -eq 1 ] ; then echo -e "$1(${DEBUG_LEVEL}) $(date "+%d.%m.%Y %H:%M:%S") : $2 ${Color_Off}"     ; fi
+if [ ${DEBUG_LEVEL} -eq 2 ] ; then echo -e "$1(${DEBUG_LEVEL}) $(date "+%d.%m.%Y %H:%M:%S") : $2 ${Color_Off}"     ; fi
+if [ ${DEBUG_LEVEL} -gt 2 ] ; then echo -e "$1(${DEBUG_LEVEL}) $(date "+%d.%m.%Y %H:%M:%S") : $2 ${Color_Off}"     ; fi
+}
+
+TIMESTAMP=$(date "+%Y%m%d%H%M%S")			# timestamp
+UNIQUE_ID="k4JgHrt${TIMESTAMP}"				# generate a unique ID to tag resources created by this script
+if [ -e /dev/urandom ];then
+ UNIQUE_ID=$(cat /dev/urandom|LC_CTYPE=C tr -dc "[:alnum:]"|fold -w 32|head -n 1)
+fi
+TMP_FILE_LIST=()							# array keeps a list of temporary files to cleanup
+THIS_SCRIPT="$(basename ${BASH_SOURCE})"	# sciptname
+
+# Do cleanup
+function Cleanup() {
+color_print "${MYcolor}" "$PF1 function: $FUNCNAME"
+  for i in "${!TMP_FILE_LIST[@]}"; do
+    if [ -f "${TMP_FILE_LIST[$i]}" ]; then
+      #echo -e "deleting ${TMP_FILE_LIST[$i]}"
+      rm -f "${TMP_FILE_LIST[$i]}"
+    fi
+  done
+}
+
+# Do cleanup, display error message and exit
+function Interrupt() {
+  Cleanup
+  exitcode=99
+  echo -e "\nScript '${THIS_SCRIPT}' aborted by user. $Color_Off"
+  exit $exitcode
+}
+
+# trap ctrl-c and call interrupt()
+trap Interrupt INT
+# trap exit and call cleanup()
+trap Cleanup   EXIT
+
+function tempfile()
+{
+  local __resultvar=$1
+  local __tmp_file=$(mktemp -t ${THIS_SCRIPT}_tmp_file.XXXXXXXXXXX) || {
+    echo -e "$Cyan ......#*** Creation of ${__tmp_file} failed $Color_Off";
+    exit 1;
+  }
+  TMP_FILE_LIST+=("${__tmp_file}")
+  if [[ "$__resultvar" ]]; then
+    eval $__resultvar="'$__tmp_file'"
+  else
+    echo -e "$Cyan ......#$__tmp_file"
+  fi
+}
+
+fi
+
+# ---------------------------------------------------------------------------------------------------------------------------------------------
+# end of file
+# ---------------------------------------------------------------------------------------------------------------------------------------------