copyright

lastupdated

keywords

subcollection

years
2014, 2021

2021-04-28

kubernetes, iks

containers

{:DomainName: data-hd-keyref="APPDomain"} {:DomainName: data-hd-keyref="DomainName"} {:android: data-hd-operatingsystem="android"} {:api: .ph data-hd-interface='api'} {:apikey: data-credential-placeholder='apikey'} {:app_key: data-hd-keyref="app_key"} {:app_name: data-hd-keyref="app_name"} {:app_secret: data-hd-keyref="app_secret"} {:app_url: data-hd-keyref="app_url"} {:authenticated-content: .authenticated-content} {:beta: .beta} {:c#: data-hd-programlang="c#"} {:cli: .ph data-hd-interface='cli'} {:codeblock: .codeblock} {:curl: .ph data-hd-programlang='curl'} {:deprecated: .deprecated} {:dotnet-standard: .ph data-hd-programlang='dotnet-standard'} {:download: .download} {:external: target="_blank" .external} {:faq: data-hd-content-type='faq'} {:fuzzybunny: .ph data-hd-programlang='fuzzybunny'} {:generic: data-hd-operatingsystem="generic"} {:generic: data-hd-programlang="generic"} {:gif: data-image-type='gif'} {:go: .ph data-hd-programlang='go'} {:help: data-hd-content-type='help'} {:hide-dashboard: .hide-dashboard} {:hide-in-docs: .hide-in-docs} {:important: .important} {:ios: data-hd-operatingsystem="ios"} {:java: .ph data-hd-programlang='java'} {:java: data-hd-programlang="java"} {:javascript: .ph data-hd-programlang='javascript'} {:javascript: data-hd-programlang="javascript"} {:new_window: target="_blank"} {:note .note} {:note: .note} {:objectc data-hd-programlang="objectc"} {:org_name: data-hd-keyref="org_name"} {:php: data-hd-programlang="php"} {:pre: .pre} {:preview: .preview} {:python: .ph data-hd-programlang='python'} {:python: data-hd-programlang="python"} {:route: data-hd-keyref="route"} {:row-headers: .row-headers} {:ruby: .ph data-hd-programlang='ruby'} {:ruby: data-hd-programlang="ruby"} {:runtime: architecture="runtime"} {:runtimeIcon: .runtimeIcon} {:runtimeIconList: .runtimeIconList} {:runtimeLink: .runtimeLink} {:runtimeTitle: .runtimeTitle} {:screen: .screen} {:script: data-hd-video='script'} {:service: architecture="service"} {:service_instance_name: data-hd-keyref="service_instance_name"} {:service_name: data-hd-keyref="service_name"} {:shortdesc: .shortdesc} {:space_name: data-hd-keyref="space_name"} {:step: data-tutorial-type='step'} {:subsection: outputclass="subsection"} {:support: data-reuse='support'} {:swift: .ph data-hd-programlang='swift'} {:swift: data-hd-programlang="swift"} {:table: .aria-labeledby="caption"} {:term: .term} {:tip: .tip} {:tooling-url: data-tooling-url-placeholder='tooling-url'} {:troubleshoot: data-hd-content-type='troubleshoot'} {:tsCauses: .tsCauses} {:tsResolve: .tsResolve} {:tsSymptoms: .tsSymptoms} {:tutorial: data-hd-content-type='tutorial'} {:ui: .ph data-hd-interface='ui'} {:unity: .ph data-hd-programlang='unity'} {:url: data-credential-placeholder='url'} {:user_ID: data-hd-keyref="user_ID"} {:vbnet: .ph data-hd-programlang='vb.net'} {:video: .video}

Storing data on classic IBM Cloud {{site.data.keyword.blockstorageshort}}

{: #block_storage}

{{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} is persistent, high-performance iSCSI storage that you can add to your apps by using Kubernetes persistent volumes (PVs). You can choose between predefined storage tiers with GB sizes and IOPS that meet the requirements of your workloads. To find out whether {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} is the right storage option for you, see Choosing a storage solution. For more information about pricing, see Pricing. {: shortdesc}

Keep in mind the following requirements when you use the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in. {: important}

{{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in is available only for standard {{site.data.keyword.containerlong_notm}} clusters that are provisioned on classic infrastructure. If you have a VPC cluster, see Storing data on {{site.data.keyword.blockstorageshort}}.
If your cluster cannot access the public network, such as a private cluster behind a firewall or a cluster with only the private cloud service endpoint enabled, make sure that you installed the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in version 1.3.0 or later to connect to your {{site.data.keyword.blockstorageshort}} instance over the private network.
{{site.data.keyword.blockstorageshort}} instances are specific to a single zone. If you have a multizone cluster, consider multizone persistent storage options.

Quickstart for {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}}

{: #block_qs}

In this quickstart guide, you create a 24Gi silver tier {{site.data.keyword.blockstorageshort}} volume in your cluster by creating a PVC to dynamically provision the volume. Then, you create an app deployment that mounts your PVC. {: shortdesc}

First time using {{site.data.keyword.blockstorageshort}} in your cluster? Come back here after you have the installed the {{site.data.keyword.blockstorageshort}} plug-in. {: tip}

Create a file for your PVC and name it pvc.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  labels:
    billingType: "hourly"
    region: # Example: us-south
    zone: # Example: dal13
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 24Gi
  storageClassName: ibmc-block-silver

{: codeblock}

Create the PVC in your cluster.

kubectl apply -f pvc.yaml

{: pre}

After your PVC is bound, create an app deployment that uses your PVC. Create a file for your deployment and name it deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app:
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: # Your contanerized app image.
        name: my-container
        volumeMounts:
        - name: my-volume
          mountPath: /mount-path
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: my-pvc

{: codeblock}

Create the deployment in your cluster.

kubectl apply -f deployment.yaml

{: pre}

For more information, see:

Installing the {{site.data.keyword.blockstorageshort}} plug-in
Adding {{site.data.keyword.blockstorageshort}} to apps.
Storage class reference.
Customizing storage classes.

Installing the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in in your cluster

{: #install_block}

Install the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in with a Helm chart to set up pre-defined storage classes for {{site.data.keyword.blockstorageshort}}. You can use these storage classes to create a PVC to provision {{site.data.keyword.blockstorageshort}} for your apps. {: shortdesc}

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

Make sure that your worker node applies the latest patch for your minor version to run your worker node with the latest security settings. The patch version also ensures that the root password on the worker node is renewed.

If you did not apply updates or reload your worker node within the last 90 days, your root password on the worker node expires and the installation of the storage plug-in might fail. {: note}
1. List the current patch version of your worker nodes.
```
ibmcloud ks worker ls --cluster <cluster_name_or_ID>
```
  {: pre}
  
  Example output:
```
OK
ID                                                  Public IP        Private IP     Machine Type           State    Status   Zone    Version
kube-dal10-crb1a23b456789ac1b20b2nc1e12b345ab-w26   169.xx.xxx.xxx    10.xxx.xx.xxx   b3c.4x16.encrypted     normal   Ready    dal10   1.20.6_1523*
```
  {: screen}
  
  If your worker node does not apply the latest patch version, you see an asterisk (*) in the Version column of your CLI output.
2. Review the version changelog to find the changes that are included in the latest patch version.
3. Apply the latest patch version by reloading your worker node. Follow the instructions in the ibmcloud ks worker reload command to gracefully reschedule any running pods on your worker node before you reload your worker node. Note that during the reload, your worker node machine is updated with the latest image and data is deleted if not stored outside the worker node.
Follow the instructions to install the Helm version 3 client on your local machine.
Add the {{site.data.keyword.cloud_notm}} Helm chart repository to the cluster where you want to use the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in.

If you enabled VRF and service endpoints in your {{site.data.keyword.cloud_notm}} account, you can use the private {{site.data.keyword.cloud_notm}} Helm repository to keep your image pull traffic on the private network. If you cannot enable VRF or service endpoints in your account, use the public registry domain: helm repo add iks-charts https://icr.io/helm/iks-charts. {: note}
```
helm repo add iks-charts https://icr.io/helm/iks-charts
```
{: pre}
Update the Helm repo to retrieve the latest version of all Helm charts in this repo.
```
helm repo update
```
{: pre}

Install the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in and give your installation a name, for example: block-storage-plugin. When you install the plug-in, pre-defined block storage classes are added to your cluster.

helm install <name> iks-charts/ibmcloud-block-storage-plugin -n <namespace>

{: pre}

Example output:

NAME:   <name>
LAST DEPLOYED: Wed Apr 18 10:02:55 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/DaemonSet
NAME                           DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
ibmcloud-block-storage-driver  0        0        0      0           0          <none>         0s

==> v1beta1/Deployment
NAME                           DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ibmcloud-block-storage-plugin  1        0        0           0          0s

==> v1/StorageClass
NAME                      PROVISIONER        AGE
ibmc-block-bronze         ibm.io/ibmc-block  0s
ibmc-block-custom         ibm.io/ibmc-block  0s
ibmc-block-gold           ibm.io/ibmc-block  0s
ibmc-block-retain-bronze  ibm.io/ibmc-block  0s
ibmc-block-retain-custom  ibm.io/ibmc-block  0s
ibmc-block-retain-gold    ibm.io/ibmc-block  0s
ibmc-block-retain-silver  ibm.io/ibmc-block  0s
ibmc-block-silver         ibm.io/ibmc-block  0s

==> v1/ServiceAccount
NAME                           SECRETS  AGE
ibmcloud-block-storage-plugin  1        0s

==> v1beta1/ClusterRole
NAME                           AGE
ibmcloud-block-storage-plugin  0s

==> v1beta1/ClusterRoleBinding
NAME                           AGE
ibmcloud-block-storage-plugin  0s

NOTES:
Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: <name>

{: screen}

Verify that the installation was successful.
```
kubectl get pod -n <namespace> | grep block
```
{: pre}

Example output:
```
ibmcloud-block-storage-driver-kh4mt                              1/1       Running   0          27d       10.118.98.19   10.118.98.19
ibmcloud-block-storage-plugin-58c5f9dc86-pbl4t                   1/1       Running   0          14d       172.21.0.204   10.118.98.19
```
{: screen}

The installation is successful when you see one ibmcloud-block-storage-plugin pod and one or more ibmcloud-block-storage-driver pods. The number of ibmcloud-block-storage-driver pods equals the number of worker nodes in your cluster. All pods must be in a Running state.

Verify that the storage classes for {{site.data.keyword.blockstorageshort}} were added to your cluster.

kubectl get storageclasses | grep block

{: pre}

Example output:

ibmc-block-bronze            ibm.io/ibmc-block
ibmc-block-custom            ibm.io/ibmc-block
ibmc-block-gold              ibm.io/ibmc-block
ibmc-block-retain-bronze     ibm.io/ibmc-block
ibmc-block-retain-custom     ibm.io/ibmc-block
ibmc-block-retain-gold       ibm.io/ibmc-block
ibmc-block-retain-silver     ibm.io/ibmc-block
ibmc-block-silver            ibm.io/ibmc-block

{: screen}

Repeat these steps for every cluster where you want to provision block storage.

You can now continue to create a PVC to provision block storage for your app.

Updating the {{site.data.keyword.cloud_notm}} Block Storage plug-in

{: #update_block}

You can upgrade the existing {{site.data.keyword.cloud_notm}} Block Storage plug-in to the latest version. {: shortdesc}

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

Update the Helm repo to retrieve the latest version of all Helm charts in this repo.

helm repo update

{: pre}

Optional: Download the latest Helm chart to your local machine. Then, extract the package and review the release.md file to find the latest release information.

helm pull iks-charts/ibmcloud-block-storage-plugin --untar

{: pre}

Find the release name and namespace of the block storage Helm chart that you installed in your cluster.

helm ls -A

{: pre}

Example output:

NAME        	NAMESPACE  	REVISION	    UPDATED                             	STATUS  	CHART                              	APP VERSION
<name>	  <namespace>    	1       	2020-01-27 09:18:33.046018 -0500 EST	deployed	ibmcloud-block-storage-plugin-1.6.0

{: screen}

Upgrade the {{site.data.keyword.cloud_notm}} Block Storage plug-in to the latest version.

helm upgrade <name> iks-charts/ibmcloud-block-storage-plugin -n <namespace>

{: pre}

Optional: When you update the plug-in, the default storage class is unset. If you want to set the default storage class to a storage class of your choice, run the following command.

kubectl patch storageclass <storageclass> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

{: pre}

Removing the {{site.data.keyword.cloud_notm}} Block Storage plug-in

{: #rm_block}

If you do not want to provision and use {{site.data.keyword.cloud_notm}} Block Storage in your cluster, you can uninstall the Helm chart. {: shortdesc}

Removing the plug-in does not remove existing PVCs, PVs, or data. When you remove the plug-in, all the related pods and daemon sets are removed from your cluster. You cannot provision new block storage for your cluster or use existing block storage PVCs and PVs after you remove the plug-in. {: important}

Before you begin:

Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
Make sure that you do not have any PVCs or PVs in your cluster that use block storage.

To remove the plug-in:

Find the release name and the namespace of the block storage Helm chart that you installed in your cluster.

helm ls -A

{: pre}

Example output:

NAME        	 NAMESPACE  	REVISION	    UPDATED                             	STATUS  	CHART                              	APP VERSION
<name> <namespace>    	1       	2020-01-27 09:18:33.046018 -0500 EST	deployed	ibmcloud-block-storage-plugin-1.6.0

{: screen}

Delete the {{site.data.keyword.cloud_notm}} Block Storage plug-in.

helm uninstall <name> -n <namespace>

{: pre}

Verify that the block storage pods are removed.

kubectl get pods -n <namespace> | grep block

{: pre} The removal of the pods is successful if no pods are displayed in your CLI output.

Verify that the block storage classes are removed.

kubectl get storageclasses | grep block

{: pre} The removal of the storage classes is successful if no storage classes are displayed in your CLI output.

Deciding on the block storage configuration

{: #block_predefined_storageclass}

{{site.data.keyword.containerlong_notm}} provides pre-defined storage classes for block storage that you can use to provision block storage with a specific configuration. {: shortdesc}

Every storage class specifies the type of block storage that you provision, including available size, IOPS, file system, and the retention policy.

Make sure to choose your storage configuration carefully to have enough capacity to store your data. After you provision a specific type of storage by using a storage class, you cannot change the type or retention policy for the storage device. However, you can change the size and the IOPS if you want to increase your storage capacity and performance. To change the type and retention policy for your storage, you must create a new storage instance and copy the data from the old storage instance to your new one. {: important}

List available storage classes in {{site.data.keyword.containerlong}}.

kubectl get storageclasses | grep block

{: pre}

Example output:

$ kubectl get storageclasses
NAME                         TYPE
ibmc-block-custom            ibm.io/ibmc-block
ibmc-block-bronze            ibm.io/ibmc-block
ibmc-block-gold              ibm.io/ibmc-block
ibmc-block-silver            ibm.io/ibmc-block
ibmc-block-retain-bronze     ibm.io/ibmc-block
ibmc-block-retain-silver     ibm.io/ibmc-block
ibmc-block-retain-gold       ibm.io/ibmc-block
ibmc-block-retain-custom     ibm.io/ibmc-block

{: screen}

Review the configuration of a storage class.
```
kubectl describe storageclass <storageclass_name>
```
{: pre}

For more information about each storage class, see the storage class reference. If you do not find what you are looking for, consider creating your own customized storage class. To get started, check out the customized storage class samples. {: tip}
Choose the type of block storage that you want to provision.
- Bronze, silver, and gold storage classes: These storage classes provision Endurance storage. With Endurance storage, you can choose the size of the storage in gigabytes at predefined IOPS tiers.
- Custom storage class: This storage class provisions Performance storage. With performance storage, you have more control over the size of the storage and the IOPS.

Choose the size and IOPS for your block storage. The size and the number of IOPS define the total number of IOPS (input/ output operations per second) that serves as an indicator for how fast your storage is. The more total IOPS your storage has, the faster it processes read and write operations.

Bronze, silver, and gold storage classes: These storage classes come with a fixed number of IOPS per gigabyte and are provisioned on SSD hard disks. The total number of IOPS depends on the size of the storage that you choose. You can select any whole number of gigabyte within the allowed size range, such as 20 Gi, 256 Gi, or 11854 Gi. To determine the total number of IOPS, you must multiply the IOPS with the selected size. For example, if you select a 1000Gi block storage size in the silver storage class that comes with 4 IOPS per GB, your storage has a total of 4000 IOPS. Table of storage class size ranges and IOPS per gigabyte

Storage class	IOPS per gigabyte	Size range in gigabytes
Bronze	2 IOPS/GB	20-12000 Gi
Silver	4 IOPS/GB	20-12000 Gi
Gold	10 IOPS/GB	20-4000 Gi

Custom storage class: When you choose this storage class, you have more control over the size and IOPS that you want. For the size, you can select any whole number of gigabyte within the allowed size range. The size that you choose determines the IOPS range that is available to you. You can choose an IOPS that is a multiple of 100 that is in the specified range. The IOPS that you choose is static and does not scale with the size of the storage. For example, if you choose 40Gi with 100 IOPS, your total IOPS remains 100.

The IOPS to gigabyte ratio also determines the type of hard disk that is provisioned for you. For example, if you have 500Gi at 100 IOPS, your IOPS to gigabyte ratio is 0.2. Storage with a ratio of less than or equal to 0.3 is provisioned on SATA hard disks. If your ratio is greater than 0.3, then your storage is provisioned on SSD hard disks. Table of custom storage class size ranges and IOPS

Size range in gigabytes	IOPS range in multiples of 100
20-39 Gi	100-1000 IOPS
40-79 Gi	100-2000 IOPS
80-99 Gi	100-4000 IOPS
100-499 Gi	100-6000 IOPS
500-999 Gi	100-10000 IOPS
1000-1999 Gi	100-20000 IOPS
2000-2999 Gi	200-40000 IOPS
3000-3999 Gi	200-48000 IOPS
4000-7999 Gi	300-48000 IOPS
8000-9999 Gi	500-48000 IOPS
10000-12000 Gi	1000-48000 IOPS

Choose if you want to keep your data after the cluster or the persistent volume claim (PVC) is deleted.
- If you want to keep your data, then choose a retain storage class. When you delete the PVC, only the PVC is deleted. The PV, the physical storage device in your IBM Cloud infrastructure account, and your data still exist. To reclaim the storage and use it in your cluster again, you must remove the PV and follow the steps for using existing block storage.
- If you want the PV, the data, and your physical block storage device to be deleted when you delete the PVC, choose a storage class without retain.
Choose if you want to be billed hourly or monthly. Check the pricing{: external} for more information. By default, all block storage devices are provisioned with an hourly billing type.

Setting up encryption for {{site.data.keyword.blockstorageshort}}

{: #block_encryption_setup}

You can set up encryption for {{site.data.keyword.blockstorageshort}} by using {{site.data.keyword.keymanagementservicelong_notm}}. {: shortdesc}

The following example explains how to create a service ID with the required access roles for {{site.data.keyword.keymanagementserviceshort}} and your cluster. The credentials of this service ID are used to enable encryption for your {{site.data.keyword.blockstorageshort}} volumes.

You can enable encryption by creating a Kubernetes secret that uses your personal API key as long as you have the Reader service access role for your {{site.data.keyword.keymanagementserviceshort}} instance as well as the Viewer platform access role and the Writer service access role for your cluster. {: tip}

Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

Make sure that you are assigned the Editor platform access role and the Writer service access role for {{site.data.keyword.keymanagementserviceshort}} so that you can create your own root key that you use to encrypt your {{site.data.keyword.blockstorageshort}} instance. You can review your IAM access roles in the IAM console{: external}. For more information about IAM roles, see IAM access.
If you do not have a {{site.data.keyword.keymanagementserviceshort}} instance, provision one.
Create a root key. By default, the root key is created without an expiration date.
Create an IAM service ID. Replace <service_ID_name> with the name that you want to assign to your service ID. This service ID is used to access your {{site.data.keyword.keymanagementserviceshort}} instance from your {{site.data.keyword.blockstorageshort}} volume. ibmcloud iam service-id-create <service_ID_name> {: pre}

Example output:
```
OK
Service ID test-id is created successfully

ID            ServiceId-a1a11111-bb11-1111-a11b-1111111a11ba   
Name          test-id   
Description      
CRN           crn:v1:bluemix:public:iam-identity::a/1a1111aa2b11111aaa1a1111aa2aa111::serviceid:ServiceId-a1a11111-bb11-1111-a11b-1111111a11bb   
Version       1-bb11aa11a0aa1a11a011a1aaaa11a1bb   
Locked        false
```
{: screen}

Create an API key for your service ID. Replace <api-key-name> with a name for your API key and replace <service_ID_name> with the name of the service ID that you created. Be sure to save your API key as it cannot be retrieved later. This API key is stored in a Kubernetes secret in your cluster in a later step. ibmcloud iam service-api-key-create <api_key_name> <service_ID_name> {: pre}
Retrieve a list of IAM-enabled services in your account and note the name of the {{site.data.keyword.keymanagementserviceshort}} instance that you created. ibmcloud resource service-instances {: pre}
Retrieve the GUID of your {{site.data.keyword.keymanagementserviceshort}} instance. The ID is used to create an IAM service policy for your service ID. ibmcloud resource service-instance "<instance_name>" | grep GUID {: pre}
Create an IAM service policy to grant your service ID access to your {{site.data.keyword.keymanagementserviceshort}} instance. The following command grants your service ID Reader access to your {{site.data.keyword.keymanagementserviceshort}} instance. The Reader access role is the minimum service access role that your service ID must have to retrieve {{site.data.keyword.keymanagementserviceshort}} keys. For more information, see Managing user access for {{site.data.keyword.keymanagementserviceshort}}. ibmcloud iam service-policy-create <service_ID_name> --roles Reader --service-name kms --service-instance <service_instance_GUID> {: pre}

<table summary="The columns are read from left to right. The first column has the parameter of the command. The second column describes the parameter.">
<caption>Understanding the <code>iam service-policy-create</code> command</caption>
<col width="25%">
<thead>
<th>Component</th>
<th>Description</th>
</thead>
<tbody>
<tr>
<td><code>service_ID_name</code></td>
<td>Enter the name of the service ID that you created earlier.</td>
</tr>	    
<tr>
<td><code>service_instance_GUID</code></td>
<td>The GUID of your {{site.data.keyword.keymanagementserviceshort}} service instance that you retrieved earlier.</td>
</tr>
</tbody>
</table>

Create another IAM service access policy to give your service ID access to your cluster. The following command grants the Viewer platform access role and the Writer service access role to your service ID for your cluster. ibmcloud iam service-policy-create <service_ID_name> --roles Writer,Viewer --service-name containers-kubernetes --service-instance <cluster_ID> {: pre}

<table summary="The columns are read from left to right. The first column has the parameter of the command. The second column describes the parameter.">
<caption>Understanding the <code>iam service-policy-create</code> command</caption>
<col width="25%">
<thead>
<th>Component</th>
<th>Description</th>
</thead>
<tbody>
<tr>
<td><code>service_ID_name</code></td>
<td>Enter the name of the service ID that you created earlier.</td>
</tr>	    
<tr>
<td><code>cluster_ID</code></td>
<td>The ID of your cluster. You can retrieve your cluster ID by running <code>ibmcloud ks cluster get <cluster_name></code>.</td>
</tr>
</tbody>
</table>

If you already have the ibmcloud-block-storage-plugin Helm chart installed, you must remove the Helm chart and install a new version.

If you installed the plug-in without using Helm, you must manually remove the block storage plug-in deployment and all associated resources before installing a new version.
{: note}
```
helm uninstall <name> <namespace>
```
{: pre}

Install the ibmcloud-block-storage-plugin Helm chart. helm install <name> iks-charts/ibmcloud-block-storage-plugin {: pre}
Create an ibm-block-secrets namespace. kubectl create ns ibm-block-secrets {: pre}

Create a role binding in the ibm-block-secrets namespace for the block storage plug-in.

kubectl create rolebinding ibmcloud-block-storage-plugin-byok --clusterrole=ibmcloud-block-storage-plugin-byok --serviceaccount=kube-system:ibmcloud-block-storage-plugin --group system:nodes --namespace=ibm-block-secrets

{: pre}

Create a Kubernetes secret that is named secret.yaml and that includes the credentials to access your root key in your {{site.data.keyword.keymanagementserviceshort}} service instance.

Create a configuration file for the secret.

apiVersion: v1
kind: Secret
metadata:
  labels:
    kmsConfig: kpc-secretLabel
  name: <secret_name> # Example: my_secret
  namespace: <namespace> # Example: default
stringData:
  config: |-
      {
        "api_key":"<service_id_api_key>", # Example: "AA1aAAaA1a21AAaA1aAAaAa-AA-1AAaaA1aA1aAaaaAA"
        "iam_endpoint":"https://iam.cloud.ibm.com",
        "key_protect_endpoint":"https://<region>.kms.cloud.ibm.com", # Example: "https://us-east.kms.cloud.ibm.com"
        "root_key_crn":"<rook_key_crn>", # Example: "crn:v1:bluemix:public:kms:<region>:a/1ab011ab2b11111aaa1a1111aa1aa111:11aa111a-1111-11a1-a111-a11a111aa111:key:11a11111-1a1a-111a-111a-11111a1a1aa1",
        "version":""
      }
type: ibm.io/kms-config

{: codeblock}

Understanding the YAML file components

Component	Description
`metadata.name`	Enter a name for your secret.
`metadata.namespace`	Enter the name of the namespace where you want to create the secret. The secret must be in same namespace where your app is deployed.
`stringData.config.api_key`	Enter the API key for the service ID that you created.
`stringData.config.key_protect_endpoint`	Enter the regional endpoint of your {{site.data.keyword.keymanagementserviceshort}} instance. For a list of Key Protect endpoints, see [Regions and endpoints](/docs/key-protect?topic=key-protect-regions).
`stringData.config.root_key_crn`	Enter the CRN of the root key that you created. To retrieve your root key CRN: Navigate to the resource list in the [{{site.data.keyword.cloud_notm}} console](https://cloud.ibm.com/resources){:external}. Click Services, then click your {{site.data.keyword.keymanagementserviceshort}} instance. Find your root key on the Actions Menu, then click View CRN. Click the Copy button to copy the CRN.

Create the secret in your cluster.
```
kubectl apply -f secret.yaml
```
{: pre}
Verify that your secret was created.
```
kubectl get secrets
```
{: pre}

Next steps Choose between the following options to create a {{site.data.keyword.blockstorageshort}} instance that encrypts data with your root key:

Create a custom storage class that references your {{site.data.keyword.keymanagementserviceshort}} secret.
Define the secret in a PVC and use one of the provided storage classes.

Encrypting volume data by using a custom storage class

{: #encrypt_custom_sc}

You can deploy apps that use encrypted volumes by first creating a custom storage class. {: shortdesc}

The following steps explain how to create a custom, encrypted storage class that you can use to create multiple encrypted block storage instances with the same configuration. If you want to create an encrypted PVC by using one of the IBM-provided storage classes, you can do this by referencing the {{site.data.keyword.keymanagementserviceshort}} credentials directly in your PVC.

Decide on a storage configuration.
Create a custom storage class that provisions an encrypted block storage instance by using one of the {{site.data.keyword.IBM_notm}}-provided storage classes as the basis. You can retrieve the details a storage class by running kubectl get storageclass <storageclass_name> -o yaml. The following example is based on the ibmc-block-retain-bronze storage class. yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: <name> # Example: my_custom_storageclass parameters: billingType: hourly classVersion: "2" fsType: ext4 iopsPerGB: "2" sizeRange: '[20-12000]Gi' type: Endurance encrypted: "true" encryptionKeySecret: <secret_name> # Example: my_secret encryptionKeyNamespace: <namespace> # Example: default provisioner: ibm.io/ibmc-block reclaimPolicy: Delete allowVolumeExpansion: true {: codeblock}

<table summary="The columns are read from left to right. The first column has the parameter of the YAML file. The second column describes the parameter.">
<caption>Understanding the YAML file components</caption>
<thead>
<th>Component</th>
<th>Description</th>
</thead>
<tbody>
<tr>
<td><code>name</code></td>
<td>Enter the name of the storage class.</td>
</tr>
<tr>
<td><code>encrypted</code></td>
<td>In the parameters, enter "true" to enable encryption.</td>
</tr>
<tr>
<td><code>encryptionKeyNamespace</code></td>
<td>In the parameters, enter the namespace where you created your secret. Example: <code>default</code>.</td>
</tr>
<tr>
<td><code>encryptionKeySecret</code></td>
<td>In the parameters, enter the name of the secret that you created earlier. Example: <code>my_secret</code>.</td>
</tr>
</tbody>
</table>

Create the storage class in your cluster. kubectl apply -f storageclass.yaml {: pre}
Add {{site.data.keyword.blockstorageshort}} to your app by using your custom storage class to create a PVC.
Verify the encryption of your {{site.data.keyword.blockstorageshort}} volumes.

Create a PVC that references your {{site.data.keyword.blockstorageshort}} secret

{: #pvc_encrypt_label}

You can provision encrypted {{site.data.keyword.blockstorageshort}} by creating a PVC that specifies the Kubernetes secret that holds your {{site.data.keyword.keymanagementserviceshort}} credentials. {: shortdesc}

The following steps show how you can reference your {{site.data.keyword.keymanagementserviceshort}} credentials in your PVC to create an encrypted {{site.data.keyword.blockstorageshort}} instance. To create multiple encrypted volumes without specifying the {{site.data.keyword.keymanagementserviceshort}} credentials in each PVC, you can create a custom, encrypted storage class.

Review the provided {{site.data.keyword.blockstorageshort}} storage classes to determine which storage class best meets your app requirements. If the provided storage classes do not meet your app requirements, you can create your own customized storage class.
Create a PVC configuration file that is named pvc.yaml and that references the Kubernetes secret where you stored the {{site.data.keyword.keymanagementserviceshort}} service credentials. To create this secret, see Setting up encryption for {{site.data.keyword.blockstorageshort}}. ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: <pvc_name> # Enter a name for your PVC. annotations: volume.beta.kubernetes.io/storage-class: "<storage_class>" # Enter a storage class. To see a list of storageclasses run kubectl get storageclasses. labels: encrypted: "true" encryptionKeyNamespace: # Enter the namespace where your secret was created. encryptionKeySecret: <secret_name> # Enter the name of the secret you created. spec: accessModes:
- ReadWriteOnce resources: requests: storage: 20Gi ``` {: codeblock}

<table summary="The columns are read from left to right. The first column has the parameter of the YAML file. The second column describes the parameter.">
<caption>Understanding the YAML file components</caption>
<thead>
<th>Component</th>
<th>Description</th>
</thead>
<tbody>
<tr>
<td><code>name</code></td>
<td>Enter the name of the PVC.</td>
</tr>
<tr>
<td><code>encrypted</code></td>
<td>In the metadata labels section, enter "true" to enable encryption.</td>
</tr>
<tr>
<td><code>encryptionKeyNamespace</code></td>
<td>In the metadata labels section, enter the namespace where you created your secret. Example: <code>default</code>.</td>
</tr>
<tr>
<td><code>encryptionKeySecret</code></td>
<td>In the metadata labels section, enter the name of the secret that you created earlier. Example: <code>my_secret</code>.</td>
</tr>
</tbody>
</table>

Create the PVC in your cluster. kubectl apply -f pvc.yaml {: pre}
Check the status of your PVC. kubectl get pvc {: pre}
Wait for your PVC to bind, then create a deployment that uses your PVC.
Verify the encryption of your {{site.data.keyword.blockstorageshort}} volumes.

Verifying the encryption of your {{site.data.keyword.blockstorageshort}} volumes

{: #block_encrypt}

You can verify the encryption of your volumes by checking the volume mount path. {: shortdesc}

Log in to your app pod. Replace <pod_name> with the name of the pod that mounts your encrypted {{site.data.keyword.blockstorageshort}} volume.

kubectl exec <pod_name> -it bash

{: pre}

List the file system of your pod.

df -h

{: pre}

Review the file system path for your encrypted {{site.data.keyword.blockstorageshort}} volume.

Encrypted volumes have a path structure of /dev/mapper/<pvc-ID_encrypted>. In this example, the encrypted volume is mounted to the /test file path in the pod.

Filesystem                                            Size  Used Avail Use% Mounted on
overlay                                                98G  8.2G   85G   9% /
tmpfs                                                  64M     0   64M   0% /dev
tmpfs                                                 2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/pvc-a011a111-1111-1111-111a-aaa1a1111a11_encrypted   20G   45M   20G   1% /test

{: screen}

Unencrypted volumes have a path structure of dev/mapper/<random_string>.

Filesystem                                     Size  Used Avail Use% Mounted on
overlay                                         98G   16G   78G  17% /
tmpfs                                           64M     0   64M   0% /dev
tmpfs                                          7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/mapper/3600a09803830476e733f4e477370716e   24G   45M   24G   1% /test

{: screen}

Removing your Kubernetes secret does not revoke access to the volume data. If you created a pod-only deployment, you must delete the pod. If you created a deployment, you must delete the deployment. {: note}

Adding block storage to apps

{: #add_block}

Create a persistent volume claim (PVC) to dynamically provision block storage for your cluster. Dynamic provisioning automatically creates the matching persistent volume (PV) and orders the actual storage device in your IBM Cloud infrastructure account. {: shortdesc}

Block storage comes with a ReadWriteOnce access mode. You can mount it to only one pod on one worker node in the cluster at a time. {: note}

Before you begin:

If you have a firewall, allow egress access for the IBM Cloud infrastructure IP ranges of the zones that your clusters are in so that you can create PVCs.
Install the {{site.data.keyword.cloud_notm}} block storage plug-in.
Decide on a pre-defined storage class or create a customized storage class.

Looking to deploy block storage in a stateful set? For more information, see Using block storage in a stateful set. {: tip}

To add block storage:

Create a configuration file to define your persistent volume claim (PVC) and save the configuration as a .yaml file.

Example for bronze, silver, gold storage classes: The following .yaml file creates a claim that is named mypvc of the "ibmc-block-silver" storage class, billed "hourly", with a gigabyte size of 24Gi.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
  labels:
    billingType: "hourly"
    region: us-south
    zone: dal13
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 24Gi
  storageClassName: ibmc-block-silver

{: codeblock}

Example for using the custom storage class: The following .yaml file creates a claim that is named mypvc of the storage class ibmc-block-retain-custom, billed "hourly", with a gigabyte size of 45Gi and IOPS of "300".

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
  labels:
    billingType: "hourly"
    region: us-south
    zone: dal13
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 45Gi
      iops: "300"
  storageClassName: ibmc-block-retain-custom

{: codeblock}

Understanding the YAML file components

Component	Description
`name`	Enter the name of the PVC.
`billingType`	In the metadata labels section, specify the frequency for which your storage bill is calculated, "monthly" or "hourly". The default is "hourly".
`region`	In the metadata labels section, specify the region where you want to provision your block storage. If you specify the region, you must also specify a zone. If you do not specify a region, or the specified region is not found, the storage is created in the same region as your cluster. This option is supported only with the IBM Cloud Block Storage plug-in version 1.0.1 or higher. For older plug-in versions, if you have a multizone cluster, the zone in which your storage is provisioned is selected on a round-robin basis to balance volume requests evenly across all zones. To specify the zone for your storage, you can create a [customized storage class](#block_multizone_yaml) first. Then, create a PVC with your customized storage class.
`zone`	In the metadata labels section, specify the zone where you want to provision your block storage. If you specify the zone, you must also specify a region. If you do not specify a zone or the specified zone is not found in a multizone cluster, the zone is selected on a round-robin basis. This option is supported only with the IBM Cloud Block Storage plug-in version 1.0.1 or higher. For older plug-in versions, if you have a multizone cluster, the zone in which your storage is provisioned is selected on a round-robin basis to balance volume requests evenly across all zones. To specify the zone for your storage, you can create a [customized storage class](#block_multizone_yaml) first. Then, create a PVC with your customized storage class.
`storage`	In the spec resources requests section, enter the size of the block storage, in gigabytes (Gi). After your storage is provisioned, you cannot change the size of your block storage. Make sure to specify a size that matches the amount of data that you want to store.
`iops`	This option is available for the custom storage classes only (`ibmc-block-custom / ibmc-block-retain-custom`). In the spec resources requests section, specify the total IOPS for the storage, selecting a multiple of 100 within the allowable range. If you choose an IOPS other than one that is listed, the IOPS is rounded up.
`storageClassName`	In the spec section, enter the name of the storage class that you want to use to provision block storage. You can choose to use one of the [IBM-provided storage classes](#block_storageclass_reference) or [create your own storage class](#block_custom_storageclass). If you do not specify a storage class, the PV is created with the default storage class `ibmc-file-bronze` Tip: Want to set your own default? See [Changing the default storage class](/docs/containers?topic=containers-kube_concepts#default_storageclass).

If you want to use a customized storage class, create your PVC with the corresponding storage class name, a valid IOPS, and size.
{: tip}

Create the PVC.
```
kubectl apply -f mypvc.yaml
```
{: pre}

Verify that your PVC is created and bound to the PV. This process can take a few minutes.

kubectl describe pvc mypvc

{: pre}

Example output:

Name:		mypvc
Namespace:	default
StorageClass:	""
Status:		Bound
Volume:		pvc-0d787071-3a67-11e7-aafc-eef80dd2dea2
Labels:		<none>
Capacity:	20Gi
Access Modes:	RWO
Events:
  FirstSeen	LastSeen	Count	From								SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----								-------------	--------	------			-------
  3m		3m		1	{ibm.io/ibmc-block 31898035-3011-11e7-a6a4-7a08779efd33 }			Normal		Provisioning		External provisioner is provisioning volume for claim "default/my-persistent-volume-claim"
  3m		1m		10	{persistentvolume-controller }							Normal		ExternalProvisioning	cannot find provisioner "ibm.io/ibmc-block", expecting that a volume for the claim is provisioned either manually or via external software
  1m		1m		1	{ibm.io/ibmc-block 31898035-3011-11e7-a6a4-7a08779efd33 }			Normal		ProvisioningSucceeded	Successfully provisioned volume pvc-0d787071-3a67-11e7-aafc-eef80dd2dea2

{: screen}

{: #block_app_volume_mount}To mount the PV to your deployment, create a configuration .yaml file and specify the PVC that binds the PV.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <deployment_name>
  labels:
    app: <deployment_label>
spec:
  selector:
    matchLabels:
      app: <app_name>
  template:
    metadata:
      labels:
        app: <app_name>
    spec:
      containers:
      - image: <image_name>
        name: <container_name>
        volumeMounts:
        - name: <volume_name>
          mountPath: /<file_path>
      volumes:
      - name: <volume_name>
        persistentVolumeClaim:
          claimName: <pvc_name>

{: codeblock}

Understanding the YAML file components

Component	Description
`app`	In the metadata, enter a label for the deployment.
`matchLabels.app` `labels.app`	In the spec selector and in the template metadata, enter a label for your app.
`image`	The name of the container image that you want to use. To list available images in your {{site.data.keyword.registrylong_notm}} account, run `ibmcloud cr image-list`.
`name`	The name of the container that you want to deploy to your cluster.
`mountPath`	In the container volume mounts section, enter the absolute path of the directory to where the volume is mounted inside the container. Data that is written to the mount path is stored under the root directory in your physical block storage instance. If you want to share a volume between different apps, you can specify [volume sub paths ![External link icon](../icons/launch-glyph.svg "External link icon")](https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath) for each of your apps.
`name`	In the container volume mounts section, enter the name of the volume to mount to your pod.
`name`	In the volumes section, enter the name of the volume to mount to your pod. Typically this name is the same as `volumeMounts/name`.
`claimName`	In the volumes persistent volume claim section, enter the name of the PVC that binds the PV that you want to use.

Create the deployment.
```
kubectl apply -f <local_yaml_path>
```
{: pre}

Verify that the PV is successfully mounted.

kubectl describe deployment <deployment_name>

{: pre}

The mount point is in the Volume Mounts field and the volume is in the Volumes field.

 Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tqp61 (ro)
      /volumemount from myvol (rw)
...
Volumes:
  myvol:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	mypvc
    ReadOnly:	false

{: screen}

Using existing block storage in your cluster

{: #existing_block}

If you have an existing physical storage device that you want to use in your cluster, you can manually create the PV and PVC to statically provision the storage. {: shortdesc}

Before you can start to mount your existing storage to an app, you must retrieve all necessary information for your PV.

Step 1: Retrieving the information of your existing block storage

{: #existing-block-1}

Retrieve or generate an API key for your IBM Cloud infrastructure account.
1. Log in to the IBM Cloud infrastructure portal{: external}.
2. Select Account, then Users, and then User List.
3. Find your user ID.
4. In the API KEY column, click Generate to generate an API key or View to view your existing API key.
Retrieve the API username for your IBM Cloud infrastructure account.
1. From the User List menu, select your user ID.
2. In the API Access Information section, find your API Username.
Log in to the IBM Cloud infrastructure CLI plug-in.
```
ibmcloud sl init
```
{: pre}
Choose to authenticate by using the username and API key for your IBM Cloud infrastructure account.
Enter the username and API key that you retrieved in the previous steps.

List available block storage devices.

ibmcloud sl block volume-list

{: pre}

Example output:

id         username            datacenter   storage_type              capacity_gb   bytes_used   ip_addr         lunId   active_transactions
38642141   IBM02SEL1543159-1   dal10        endurance_block_storage   20            -            169.xx.xxx.xxx   170     0

{: screen}

Retrieve the volume details. Replace <volume_ID> with the ID of the Block storage volume that you retrieved in step 6.

ibmcloud sl block volume-detail <volume_ID>

{: pre}

Example output:

Name                       Value   
ID                         111111111   
User name                  IBM02SEL1111111-111   
Type                       endurance_block_storage   
Capacity (GB)              20   
LUN Id                     1   
Endurance Tier             10_IOPS_PER_GB   
Endurance Tier Per IOPS    10   
Datacenter                 wdc06   
Target IP                  161.XX.XX.XX
# of Active Transactions   0   
Replicant Count            0

{: screen}

Note the ID, Target IP, Capacity GB, the Datacenter, User Name, and LUN Id of the block storage device that you want to mount to your cluster. Note: To mount existing storage to a cluster, you must have a worker node in the same zone as your storage. To verify the zone of your worker node, run ibmcloud ks worker ls --cluster <cluster_name_or_ID>.

Step 2: Creating a persistent volume (PV) and a matching persistent volume claim (PVC)

{: #existing-block-2}

Optional: If you have storage that you provisioned with a retain storage class, when you remove the PVC, the PV and the physical storage device are not removed. To reuse the storage in your cluster, you must remove the PV first.
1. List existing PVs.
```
kubectl get pv
```
  {: pre}
  
  Look for the PV that belongs to your persistent storage. The PV is in a released state.
2. Remove the PV.
```
kubectl delete pv <pv_name>
```
  {: pre}
3. Verify that the PV is removed.
```
kubectl get pv
```
  {: pre}

Create a configuration file for your PV. Include the block storage id, ip_addr, capacity_gb, the datacenter, username, and lunIdID that you retrieved earlier.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "<volume_ID>"
  labels:
     failure-domain.beta.kubernetes.io/region: <region>
     failure-domain.beta.kubernetes.io/zone: <zone>
spec:
  capacity:
    storage: "<storage_size>"
  accessModes:
    - ReadWriteOnce
  flexVolume:
    driver: "ibm/ibmc-block"
    fsType: "<fs_type>"
    options:
      "Lun": "<lun_ID>"
      "TargetPortal": "<IP_address>"
      "VolumeID": "<volume_ID>"
      "volumeName": "<volume_username>"

{: codeblock}

Understanding the YAML file components

Component	Description
`name`	Enter the name of the PV that you want to create.
`labels`	Enter the region and the zone that you retrieved earlier. You must have at least one worker node in the same region and zone as your persistent storage to mount the storage in your cluster.
`storage`	In the spec capacity section, enter the storage size of the existing block storage that you retrieved in the previous step as `capacity-gb`. The storage size must be written in gigabytes, for example, 20Gi (20 GB) or 1000Gi (1 TB).
`fsType`	In the spec flex volume section, enter the file system type that is configured for your existing block storage. Choose between `ext4` or `xfs`. If you do not specify this option, the PV defaults to `ext4`. When the wrong `fsType` is defined, then the PV creation succeeds, but the mounting of the PV to a pod fails.
`Lun`	In the spec flex volume options section, enter the lun ID for your block storage that you retrieved earlier as `lunId`.
`TargetPortal`	In the spec flex volume options section, enter the IP address of your block storage that you retrieved earlier as `ip_addr`.
`VolumeId`	In the spec flex volume options section, enter the ID of your block storage that you retrieved earlier as `id`. Example: "11111111".
`volumeName`	In the spec flex volume options section, enter the `username` of your {{site.data.keyword.blockstorageshort}} volume. Example: "IBM02SEL1111111-111"

Create the PV in your cluster.
```
kubectl apply -f mypv.yaml
```
{: pre}
Verify that the PV is created.
```
kubectl get pv
```
{: pre}
Create another configuration file to create your PVC. In order for the PVC to match the PV that you created earlier, you must choose the same value for storage and accessMode. The storage-class field must be an empty string. If any of these fields do not match the PV, then a new PV is created automatically instead.
```
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: mypvc
spec:
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: "<storage_size>"
 storageClassName: ""
```
{: codeblock}
Create your PVC.
```
kubectl apply -f mypvc.yaml
```
{: pre}

Verify that your PVC is created and bound to the PV that you created earlier. This process can take a few minutes.

kubectl describe pvc mypvc

{: pre}

Example output:

 Name:          mypvc
 Namespace:     default
 StorageClass:  
 Status:        Bound
 Volume:        111111111
 Labels:        <none>
 Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"classic-block","namespace":"default"},"spec":{"acce...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
 Finalizers:    [kubernetes.io/pvc-protection]
 Capacity:      20Gi
 Access Modes:  RWO
 VolumeMode:    Filesystem
 Mounted By:    <none>
 Events:        <none>

{: screen}

You successfully created a PV and bound it to a PVC. Cluster users can now mount the PVC to their deployments and start reading from and writing to the PV.

Using block storage in a stateful set

{: #block_statefulset}

If you have a stateful app such as a database, you can create stateful sets that use block storage to store your app's data. Alternatively, you can use an {{site.data.keyword.cloud_notm}} database-as-a-service and store your data in the cloud. {: shortdesc}

What do I need to be aware of when adding block storage to a stateful set?

To add storage to a stateful set, you specify your storage configuration in the volumeClaimTemplates section of your stateful set YAML. The volumeClaimTemplates is the basis for your PVC and can include the storage class and the size or IOPS of your block storage that you want to provision. However, if you want to include labels in your volumeClaimTemplates, Kubernetes does not include these labels when creating the PVC. Instead, you must add the labels directly to your stateful set.

You cannot deploy two stateful sets at the same time. If you try to create a stateful set before a different one is fully deployed, then the deployment of your stateful set might lead to unexpected results. {: important}

How can I create my stateful set in a specific zone?

In a multizone cluster, you can specify the zone and region where you want to create your stateful set in the spec.selector.matchLabels and spec.template.metadata.labels section of your stateful set YAML. Alternatively, you can add those labels to a customized storage class and use this storage class in the volumeClaimTemplates section of your stateful set.

Can I delay binding of a PV to my stateful pod until the pod is ready?

Yes, you can create a custom storage class for your PVC that includes the volumeBindingMode: WaitForFirstConsumer{: external} field.

What options do I have to add block storage to a stateful set?

If you want to automatically create your PVC when you create the stateful set, use dynamic provisioning. You can also choose to pre-provision your PVCs or use existing PVCs with your stateful set.

Dynamic provisioning: Creating the PVC when you create a stateful set

{: #block_dynamic_statefulset}

Use this option if you want to automatically create the PVC when you create the stateful set. {: shortdesc}

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

Verify that all existing stateful sets in your cluster are fully deployed. If a stateful set is still being deployed, you cannot start creating your stateful set. You must wait until all stateful sets in your cluster are fully deployed to avoid unexpected results.

List existing stateful sets in your cluster.

kubectl get statefulset --all-namespaces

{: pre}

Example output:

NAME              DESIRED   CURRENT   AGE
mystatefulset     3         3         6s

{: screen}

View the Pods Status of each stateful set to ensure that the deployment of the stateful set is finished.

kubectl describe statefulset <statefulset_name>

{: pre}

Example output:

Name:               nginx
Namespace:          default
CreationTimestamp:  Fri, 05 Oct 2018 13:22:41 -0400
Selector:           app=nginx,billingType=hourly,region=us-south,zone=dal10
Labels:             app=nginx
                    billingType=hourly
                    region=us-south
                    zone=dal10
Annotations:        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"nginx","namespace":"default"},"spec":{"podManagementPolicy":"Par...
Replicas:           3 desired | 3 total
Pods Status:        0 Running / 3 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=nginx
           billingType=hourly
           region=us-south
           zone=dal10
...

{: screen}

A stateful set is fully deployed when the number of replicas that you find in the Replicas section of your CLI output equals the number of Running pods in the Pods Status section. If a stateful set is not fully deployed yet, wait until the deployment is finished before you proceed.

Create a configuration file for your stateful set and the service that you use to expose the stateful set.

Example stateful set that specifies a zone:

The following example shows how to deploy NGINX as a stateful set with three replicas. For each replica, a 20 gigabyte block storage device is provisioned based on the specifications that are defined in the ibmc-block-retain-bronze storage class. All storage devices are provisioned in the dal10 zone. Because block storage cannot be accessed from other zones, all replicas of the stateful set are also deployed onto worker nodes that are located in dal10.

apiVersion: v1
kind: Service
metadata:
 name: nginx
 labels:
   app: nginx
spec:
 ports:
 - port: 80
   name: web
 clusterIP: None
 selector:
   app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: nginx
spec:
 serviceName: "nginx"
 replicas: 3
 podManagementPolicy: Parallel
 selector:
   matchLabels:
     app: nginx
     billingType: "hourly"
     region: "us-south"
     zone: "dal10"
 template:
   metadata:
     labels:
       app: nginx
       billingType: "hourly"
       region: "us-south"
       zone: "dal10"
   spec:
     containers:
     - name: nginx
       image: k8s.gcr.io/nginx-slim:0.8
       ports:
       - containerPort: 80
         name: web
       volumeMounts:
       - name: myvol
         mountPath: /usr/share/nginx/html
 volumeClaimTemplates:
 - metadata:
     name: myvol
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
         iops: "300" #required only for performance storage
    storageClassName: ibmc-block-retain-bronze

{: codeblock}

Example stateful set with anti-affinity rule and delayed block storage creation:

The following example shows how to deploy NGINX as a stateful set with three replicas. The stateful set does not specify the region and zone where the block storage is created. Instead, the stateful set uses an anti-affinity rule to ensure that the pods are spread across worker nodes and zones. By defining topologykey: failure-domain.beta.kubernetes.io/zone, the Kubernetes scheduler cannot schedule a pod on a worker node if the worker node is in the same zone as a pod that has the app: nginx label. For each stateful set pod, two PVCs are created as defined in the volumeClaimTemplates section, but the creation of the block storage instances is delayed until a stateful set pod that uses the storage is scheduled. This setup is referred to as topology-aware volume scheduling.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-bronze-delayed
parameters:
  billingType: hourly
  classVersion: "2"
  fsType: ext4
  iopsPerGB: "2"
  sizeRange: '[20-12000]Gi'
  type: Endurance
provisioner: ibm.io/ibmc-block
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 3
  podManagementPolicy: "Parallel"
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - nginx
              topologyKey: failure-domain.beta.kubernetes.io/zone
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: myvol1
          mountPath: /usr/share/nginx/html
        - name: myvol2
          mountPath: /tmp1
  volumeClaimTemplates:
  - metadata:
      name: myvol1
    spec:
      accessModes:
      - ReadWriteOnce # access mode
      resources:
        requests:
          storage: 20Gi
      storageClassName: ibmc-block-bronze-delayed
  - metadata:
      name: myvol2
    spec:
      accessModes:
      - ReadWriteOnce # access mode
      resources:
        requests:
          storage: 20Gi
      storageClassName: ibmc-block-bronze-delayed

{: codeblock}

Understanding the stateful set YAML file components

Component	Description
`name`	Enter a name for your stateful set. The name that you enter is used to create the name for your PVC in the format: `<volume_name>-<statefulset_name>-<replica_number>`.
`serviceName`	Enter the name of the service that you want to use to expose your stateful set.
`replicas`	Enter the number of replicas for your stateful set.
`podManagementPolicy`	Enter the pod management policy that you want to use for your stateful set. Choose between the following options: `OrderedReady`: With this option, stateful set replicas are deployed one after another. For example, if you specified three replicas, then Kubernetes creates the PVC for your first replica, waits until the PVC is bound, deploys the stateful set replica, and mounts the PVC to the replica. After the deployment is finished, the second replica is deployed. For more information about this option, see [`OrderedReady` Pod Management ![External link icon](../icons/launch-glyph.svg "External link icon")](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#orderedready-pod-management). Parallel: With this option, the deployment of all stateful set replicas is started at the same time. If your app supports parallel deployment of replicas, then use this option to save deployment time for your PVCs and stateful set replicas.
`matchLabels`	In the spec selector section, enter all labels that you want to include in your stateful set and your PVC. Labels that you include in the `volumeClaimTemplates` of your stateful set are not recognized by Kubernetes. Sample labels that you might want to include are: `region` and `zone`: If you want all your stateful set replicas and PVCs to be created in one specific zone, add both labels. You can also specify the zone and region in the storage class that you use. If you do not specify a zone and region and you have a multizone cluster, the zone in which your storage is provisioned is selected on a round-robin basis to balance volume requests evenly across all zones. `billingType`: Enter the billing type that you want to use for your PVCs. Choose between `hourly` or `monthly`. If you do not specify this label, all PVCs are created with an hourly billing type.
`labels`	In the spec template metadata section, enter the same labels that you added to the `spec.selector.matchLabels` section.
`affinity`	In the spec template spec section, specify your anti-affinity rule to ensure that your stateful set pods are distributed across worker nodes and zones. The example shows an anti-affinity rule where the stateful set pod prefers not to be scheduled on a worker node where a pod runs that has the `app: nginx` label. The `topologykey: failure-domain.beta.kubernetes.io/zone` restricts this anti-affinity rule even more and prevents the pod to be scheduled on a worker node if the worker node is in the same zone as a pod that has the `app: nginx` label. By using this anti-affinity rule, you can achieve anti-affinity across worker nodes and zones.
`name`	In the spec volume claim templates metadata section, enter a name for your volume. Use the same name that you defined in the `spec.containers.volumeMount.name` section. The name that you enter here is used to create the name for your PVC in the format: `<volume_name>-<statefulset_name>-<replica_number>`.
`storage`	In the spec volume claim templates spec resources requests section, enter the size of the block storage in gigabytes (Gi).
`iops`	In the spec volume claim templates spec resources requests section, if you want to provision [performance storage](#block_predefined_storageclass), enter the number of IOPS. If you use an endurance storage class and specify a number of IOPS, the number of IOPS is ignored. Instead, the IOPS that is specified in your storage class is used.
`storageClassName`	In the spec volume claim templates spec section, enter the storage class that you want to use. To list existing storage classes, run `kubectl get storageclasses \| grep block`. If you do not specify a storage class, the PVC is created with the default storage class that is set in your cluster. Make sure that the default storage class uses the `ibm.io/ibmc-block` provisioner so that your stateful set is provisioned with block storage.

Create your stateful set.
```
kubectl apply -f statefulset.yaml
```
{: pre}
Wait for your stateful set to be deployed.
```
kubectl describe statefulset <statefulset_name>
```
{: pre}

To see the current status of your PVCs, run kubectl get pvc. The name of your PVC is formatted as <volume_name>-<statefulset_name>-<replica_number>. {: tip}

Static provisioning: Using existing PVCs with a stateful set

{: #block_static_statefulset}

You can pre-provision your PVCs before creating your stateful set or use existing PVCs with your stateful set. {: shortdesc}

When you dynamically provision your PVCs when creating the stateful set, the name of the PVC is assigned based on the values that you used in the stateful set YAML file. In order for the stateful set to use existing PVCs, the name of your PVCs must match the name that would automatically be created when using dynamic provisioning.

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

If you want to pre-provision the PVC for your stateful set before you create the stateful set, follow steps 1-3 in Adding block storage to apps to create a PVC for each stateful set replica. Make sure that you create your PVC with a name that follows the following format: <volume_name>-<statefulset_name>-<replica_number>.
- <volume_name>: Use the name that you want to specify in the spec.volumeClaimTemplates.metadata.name section of your stateful set, such as nginxvol.
- <statefulset_name>: Use the name that you want to specify in the metadata.name section of your stateful set, such as nginx_statefulset.
- <replica_number>: Enter the number of your replica, starting with 0.
For example, if you must create three stateful set replicas, create three PVCs with the following names: nginxvol-nginx_statefulset-0, nginxvol-nginx_statefulset-1, and nginxvol-nginx_statefulset-2.

Looking to create a PVC and PV for an existing storage device? Create your PVC and PV by using static provisioning.
Follow the steps in Dynamic provisioning: Creating the PVC when you create a stateful set to create your stateful set. The name of your PVC follows the format <volume_name>-<statefulset_name>-<replica_number>. Make sure to use the following values from your PVC name in the stateful set specification:
- spec.volumeClaimTemplates.metadata.name: Enter the <volume_name> of your PVC name.
- metadata.name: Enter the <statefulset_name> of your PVC name.
- spec.replicas: Enter the number of replicas that you want to create for your stateful set. The number of replicas must equal the number of PVCs that you created earlier.
If your PVCs are in different zones, do not include a region or zone label in your stateful set. {: note}

Verify that the PVCs are used in your stateful set replica pods.

List the pods in your cluster. Identify the pods that belong to your stateful set.
```
kubectl get pods
```
{: pre}

Verify that your existing PVC is mounted to your stateful set replica. Review the ClaimName in the Volumes section of your CLI output.

kubectl describe pod <pod_name>

{: pre}

Example output:

Name:           nginx-0
Namespace:      default
Node:           10.xxx.xx.xxx/10.xxx.xx.xxx
Start Time:     Fri, 05 Oct 2018 13:24:59 -0400
...
Volumes:
  myvol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  myvol-nginx-0

...

{: screen}

Changing the size and IOPS of your existing storage device

{: #block_change_storage_configuration}

If you want to increase storage capacity or performance, you can modify your existing volume. {: shortdesc}

For questions about billing and to find the steps for how to use the {{site.data.keyword.cloud_notm}} console to modify your storage, see Expanding Block Storage capacity and Adjusting IOPS. Updates that you make from the console are not reflected in the persistent volume (PV). To add this information to the PV, run kubectl patch pv <pv_name> and manually update the size and IOPS in the Labels and Annotation section of your PV. {: tip}

List the PVCs in your cluster and note the name of the associated PV from the VOLUME column.

kubectl get pvc

{: pre}

Example output:

NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
myvol            Bound     pvc-01ac123a-123b-12c3-abcd-0a1234cb12d3   20Gi       RWO            ibmc-block-bronze    147d

{: screen}

If you want to change the IOPS and the size for your block storage, edit the IOPS in the metadata.labels.IOPS section of your PV first. You can change to a lower or greater IOPS value. Make sure that you enter an IOPS that is supported for the storage type that you have. For example, if you have endurance block storage with 4 IOPS, you can change the IOPS to either 2 or 10. For more supported IOPS values, see Deciding on your block storage configuration.
```
kubectl edit pv <pv_name>
```
{: pre}

To change the IOPS from the CLI, you must also change the size of your block storage. If you want to change only the IOPS, but not the size, you must request the IOPS change from the console. {: note}
Edit the PVC and add the new size in the spec.resources.requests.storage section of your PVC. You can change to a greater size only up to the maximum capacity that is set by your storage class. You cannot downsize your existing storage. To see available sizes for your storage class, see Deciding on the block storage configuration.
```
kubectl edit pvc <pvc_name>
```
{: pre}

Verify that the volume expansion is requested. The volume expansion is successfully requested when you see a FileSystemResizePending message in the Conditions section of your CLI output.

kubectl describe pvc <pvc_name>

{: pre}

Example output:

...
Conditions:
Type                      Status  LastProbeTime                     LastTransitionTime                Reason  Message
----                      ------  -----------------                 ------------------                ------  -------
FileSystemResizePending   True    Mon, 01 Jan 0001 00:00:00 +0000   Thu, 25 Apr 2019 15:52:49 -0400           Waiting for user to (re-)start a pod to finish file system resize of volume on node.

{: screen}

List all the pods that mount the PVC. If your PVC is mounted by a pod, the volume expansion is automatically processed. If your PVC is not mounted by a pod, you must mount the PVC to a pod so that the volume expansion can be processed.
```
kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
```
{: pre}

Mounted pods are returned in the format: <pod_name>: <pvc_name>.
If your PVC is not mounted by a pod, create a pod or deployment and mount the PVC. If your PVC is mounted by a pod, continue with the next step.
Verify that the size and IOPS are changed in the Labels section of your CLI output. This process might take a few minutes to complete.
```
kubectl describe pv <pv_name>
```
{: pre}

Example output:
```
...
Labels:       CapacityGb=50
              Datacenter=dal10
              IOPS=500
```
{: screen}

Backing up and restoring data

{: #block_backup_restore}

Block storage is provisioned into the same location as the worker nodes in your cluster. The storage is hosted on clustered servers by IBM to provide availability in case a server goes down. However, block storage is not backed up automatically and might be inaccessible if the entire location fails. To protect your data from being lost or damaged, you can set up periodic backups that you can use to restore your data when needed. {: shortdesc}

Review the following backup and restore options for your block storage:

Set up periodic snapshots

You can [set up periodic snapshots for your block storage](/docs/BlockStorage?topic=BlockStorage-snapshots#snapshots), which is a read-only image that captures the state of the instance at a point in time. To store the snapshot, you must request snapshot space on your block storage. Snapshots are stored on the existing storage instance within the same zone. You can restore data from a snapshot if a user accidentally removes important data from the volume.

To create a snapshot for your volume:

[Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](/docs/containers?topic=containers-cs_cli_install#cs_cli_configure)
Log in to the `ibmcloud sl` CLI.
```
ibmcloud sl init
```
List existing PVs in your cluster.
```
kubectl get pv
```
Get the details for the PV for which you want to create snapshot space and note the volume ID, the size, and the IOPS.
```
kubectl describe pv <pv_name>
```
The size and IOPS are shown in the Labels section of your CLI output. To find the volume ID, review the ibm.io/network-storage-id annotation of your CLI output.
Create the snapshot size for your existing volume with the parameters that you retrieved in the previous step.
```
ibmcloud sl block snapshot-order <volume_ID> --size <size> --tier <iops>
```
Wait for the snapshot size to create.
```
ibmcloud sl block volume-detail <volume_ID>
```
The snapshot size is successfully provisioned when the Snapshot Size (GB) in your CLI output changes from 0 to the size that you ordered.
Create the snapshot for your volume and note the ID of the snapshot that is created for you.
```
ibmcloud sl block snapshot-create <volume_ID>
```
Verify that the snapshot is created successfully.
```
ibmcloud sl block snapshot-list <volume_ID>
```

To restore data from a snapshot to an existing volume:

ibmcloud sl block snapshot-restore <volume_ID> <snapshot_ID>

Replicate snapshots to another zone

To protect your data from a zone failure, you can [replicate snapshots](/docs/BlockStorage?topic=BlockStorage-replication#replication) to a block storage instance that is set up in another zone. Data can be replicated from the primary storage to the backup storage only. You cannot mount a replicated block storage instance to a cluster. When your primary storage fails, you can manually set your replicated backup storage to be the primary one. Then, you can mount it to your cluster. After your primary storage is restored, you can restore the data from the backup storage.

Duplicate storage

You can [duplicate your block storage instance](/docs/BlockStorage?topic=BlockStorage-duplicatevolume#duplicatevolume) in the same zone as the original storage instance. A duplicate has the same data as the original storage instance at the point in time that you create the duplicate. Unlike replicas, use the duplicate as an independent storage instance from the original. To duplicate, first set up snapshots for the volume.

Back up data to {{site.data.keyword.cos_full}}

You can use the [**ibm-backup-restore**](/docs/containers?topic=containers-utilities#backup-restore-pvc) Helm chart to spin up a backup and restore pod in your cluster. This pod contains a script to run a one-time or periodic backup for any persistent volume claim (PVC) in your cluster. Data is stored in your {{site.data.keyword.cos_full}} instance that you set up in a zone.

Block storage is mounted with an RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up your data, you must unmount the app pod from the storage, mount it to your backup pod, back up the data, and remount the storage to your app pod.

To make your data even more highly available and protect your app from a zone failure, set up a second {{site.data.keyword.cos_short}} instance and replicate data across zones. If you need to restore data from your {{site.data.keyword.cos_short}} instance, use the restore pod that is provided with the Helm chart.

Copy data to and from pods and containers

You can use the `kubectl cp` [command![External link icon](../icons/launch-glyph.svg "External link icon")](https://kubernetes.io/docs/reference/kubectl/overview/#cp) to copy files and directories to and from pods or specific containers in your cluster.

Before you begin: [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](/docs/containers?topic=containers-cs_cli_install#cs_cli_configure) If you do not specify a container with -c, the command uses to the first available container in the pod.

You can use the command in various ways:

Copy data from your local machine to a pod in your cluster:

kubectl cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath>

Copy data from a pod in your cluster to your local machine:

kubectl cp <namespace>/<pod>:<pod_filepath>/<filename> <local_filepath>/<filename>

Copy data from your local machine to a specific container that runs in a pod in your cluster:
```
kubectl cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath> -c <container>
```

Storage class reference

{: #block_storageclass_reference}

Storage classes that have retain in the title have a reclaim policy of Retain. Example: ibmc-file-retain-bronze. Storage classes that do not have retain in the title have a reclaim policy of Delete. Example: ibmc-file-bronze. {: tip}

Characteristics	Setting
Name	`ibmc-block-bronze` `ibmc-block-retain-bronze`
Type	Endurance storage
File system	`ext4`
IOPS per gigabyte	2
Size range in gigabytes	20-12000 Gi
Hard disk	SSD
Reclaim policy	`ibmc-block-bronze`: Delete `ibmc-block-retain-bronze`: Retain
Billing	The default billing type depends on the version of your {{site.data.keyword.cloud_notm}} Block Storage plug-in: Version 1.0.1 and higher: Hourly Version 1.0.0 and lower: Monthly
Pricing	Pricing information
{: class="simple-tab-table"}
{: caption="Block storage class: bronze" caption-side="top"}
{: #block_bronze}
{: tab-title="Bronze"}
{: tab-group="Block storage class"}

Characteristics	Setting
Name	`ibmc-block-silver` `ibmc-block-retain-silver`
Type	Endurance storage
File system	`ext4`
IOPS per gigabyte	4
Size range in gigabytes	20-12000 Gi
Hard disk	SSD
Reclaim policy	`ibmc-block-silver`: Delete `ibmc-block-retain-silver`: Retain
Billing	The default billing type depends on the version of your {{site.data.keyword.cloud_notm}} Block Storage plug-in: Version 1.0.1 and higher: Hourly Version 1.0.0 and lower: Monthly
Pricing	Pricing information
{: class="simple-tab-table"}
{: caption="Block storage class: silver" caption-side="top"}
{: #block_silver}
{: tab-title="Silver"}
{: tab-group="Block storage class"}

Characteristics	Setting
Name	`ibmc-block-gold` `ibmc-block-retain-gold`
Type	Endurance storage
File system	`ext4`
IOPS per gigabyte	10
Size range in gigabytes	20-4000 Gi
Hard disk	SSD
Reclaim policy	`ibmc-block-gold`: Delete `ibmc-block-retain-gold`: Retain
Billing	The default billing type depends on the version of your {{site.data.keyword.cloud_notm}} Block Storage plug-in: Version 1.0.1 and higher: Hourly Version 1.0.0 and lower: Monthly
Pricing	Pricing information
{: class="simple-tab-table"}
{: caption="Block storage class: gold" caption-side="top"}
{: #block_gold}
{: tab-title="Gold"}
{: tab-group="Block storage class"}

Characteristics	Setting
Name	`ibmc-block-custom` `ibmc-block-retain-custom`
Type	Performance
File system	`ext4`
IOPS and size	Size range in gigabytes / IOPS range in multiples of 100 20-39 Gi / 100-1000 IOPS 40-79 Gi / 100-2000 IOPS 80-99 Gi / 100-4000 IOPS 100-499 Gi / 100-6000 IOPS 500-999 Gi / 100-10000 IOPS 1000-1999 Gi / 100-20000 IOPS 2000-2999 Gi / 200-40000 IOPS 3000-3999 Gi / 200-48000 IOPS 4000-7999 Gi / 300-48000 IOPS 8000-9999 Gi / 500-48000 IOPS 10000-12000 Gi / 1000-48000 IOPS
Hard disk	The IOPS to gigabyte ratio determines the type of hard disk that is provisioned. To determine your IOPS to gigabyte ratio, you divide the IOPS by the size of your storage. Example: You chose 500Gi of storage with 100 IOPS. Your ratio is 0.2 (100 IOPS/500Gi). Overview of hard disk types per ratio: Less than or equal to 0.3: SATA Greater than 0.3: SSD
Reclaim policy	`ibmc-block-custom`: Delete `ibmc-block-retain-custom`: Retain
Billing	The default billing type depends on the version of your {{site.data.keyword.cloud_notm}} Block Storage plug-in: Version 1.0.1 and higher: Hourly Version 1.0.0 and lower: Monthly
Pricing	Pricing information
{: class="simple-tab-table"}
{: caption="Block storage class: custom" caption-side="top"}
{: #block_custom}
{: tab-title="Custom"}
{: tab-group="Block storage class"}

Sample customized storage classes

{: #block_custom_storageclass}

You can create a customized storage class and use the storage class in your PVC. {: shortdesc}

{{site.data.keyword.containerlong_notm}} provides pre-defined storage classes to provision block storage with a particular tier and configuration. In some cases, you might want to provision storage with a different configuration that is not covered in the pre-defined storage classes. You can use the examples in this topic to find sample customized storage classes.

To create your customized storage class, see Customizing a storage class. Then, use your customized storage class in your PVC.

Creating topology-aware storage

{: #topology_yaml}

To use block storage in a multizone cluster, your pod must be scheduled in the same zone as your block storage instance so that you can read and write to the volume. Before topology-aware volume scheduling was introduced by Kubernetes, the dynamic provisioning of your storage automatically created the block storage instance when a PVC was created. Then, when you created your pod, the Kubernetes scheduler tried to deploy the pod to the same data center as your block storage instance. {: shortdesc}

Creating the block storage instance without knowing the constraints of the pod can lead to unwanted results. For example, your pod might not be able to be scheduled to the same worker node as your storage because the worker node has insufficient resources or the worker node is tainted and does not allow the pod to be scheduled. With topology-aware volume scheduling, the block storage instance is delayed until the first pod that uses the storage is created.

To use topology-aware volume scheduling, make sure that you installed the {{site.data.keyword.cloud_notm}} Block Storage plug-in version 1.2.0 or later. {: note}

The following examples show how to create storage classes that delay the creation of the block storage instance until the first pod that uses this storage is ready to be scheduled. To delay the creation, you must include the volumeBindingMode: WaitForFirstConsumer option. If you do not include this option, the volumeBindingMode is automatically set to Immediate and the block storage instance is created when you create the PVC.

Example for Endurance block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-bronze-delayed
parameters:
  billingType: hourly
  classVersion: "2"
  fsType: ext4
  iopsPerGB: "2"
  sizeRange: '[20-12000]Gi'
  type: Endurance
provisioner: ibm.io/ibmc-block
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

{: codeblock}

Example for Performance block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: ibmc-block-performance-storageclass
 labels:
   kubernetes.io/cluster-service: "true"
provisioner: ibm.io/ibmc-block
parameters:
 billingType: "hourly"
 classVersion: "2"
 sizeIOPSRange: |-
   "[20-39]Gi:[100-1000]"
   "[40-79]Gi:[100-2000]"
   "[80-99]Gi:[100-4000]"
   "[100-499]Gi:[100-6000]"
   "[500-999]Gi:[100-10000]"
   "[1000-1999]Gi:[100-20000]"
   "[2000-2999]Gi:[200-40000]"
   "[3000-3999]Gi:[200-48000]"
   "[4000-7999]Gi:[300-48000]"
   "[8000-9999]Gi:[500-48000]"
   "[10000-12000]Gi:[1000-48000]"
 type: "Performance"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

{: codeblock}

Specifying the zone and region

{: #block_multizone_yaml}

If you want to create your block storage in a specific zone, you can specify the zone and region in a customized storage class. {: shortdesc}

Use the customized storage class if you use the {{site.data.keyword.cloud_notm}} Block Storage plug-in version 1.0.0 or if you want to statically provision block storage in a specific zone. In all other cases, specify the zone directly in your PVC. {: note}

The following .yaml file customizes a storage class that is based on the ibm-block-silver non-retaining storage class: the type is "Endurance", the iopsPerGB is 4, the sizeRange is "[20-12000]Gi", and the reclaimPolicy is set to "Delete". The zone is specified as dal12. To use a different storage class as your base, see the storage class reference.

Create the storage class in the same region and zone as your cluster and worker nodes. To get the region of your cluster, run ibmcloud ks cluster get --cluster <cluster_name_or_ID> and look for the region prefix in the Master URL, such as eu-de in https://c2.eu-de.containers.cloud.ibm.com:11111. To get the zone of your worker node, run ibmcloud ks worker ls --cluster <cluster_name_or_ID>.

Example for Endurance block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-silver-mycustom-storageclass
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: ibm.io/ibmc-block
parameters:
  zone: "dal12"
  region: "us-south"
  type: "Endurance"
  iopsPerGB: "4"
  sizeRange: "[20-12000]Gi"
reclaimPolicy: "Delete"

{: codeblock}

Example for Performance block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-performance-storageclass
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: ibm.io/ibmc-block
parameters:
  zone: "dal12"
  region: "us-south"
  type: "Performance"
  sizeIOPSRange: |-
    "[20-39]Gi:[100-1000]"
    "[40-79]Gi:[100-2000]"
    "[80-99]Gi:[100-4000]"
    "[100-499]Gi:[100-6000]"
    "[500-999]Gi:[100-10000]"
    "[1000-1999]Gi:[100-20000]"
    "[2000-2999]Gi:[200-40000]"
    "[3000-3999]Gi:[200-48000]"
    "[4000-7999]Gi:[300-48000]"
    "[8000-9999]Gi:[500-48000]"
    "[10000-12000]Gi:[1000-48000]"
reclaimPolicy: "Delete"

{: codeblock}

Mounting block storage with an `XFS` file system

{: #xfs}

The following examples create a storage class that provisions block storage with an XFS file system. {: shortdesc}

Example for Endurance block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-custom-xfs
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
provisioner: ibm.io/ibmc-block
parameters:
  type: "Endurance"
  iopsPerGB: "4"
  sizeRange: "[20-12000]Gi"
  fsType: "xfs"
reclaimPolicy: "Delete"

{: codeblock}

Example for Performance block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-custom-xfs
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
provisioner: ibm.io/ibmc-block
parameters:
  classVersion: "2"
  type: "Performance"
  sizeIOPSRange: |-
    [20-39]Gi:[100-1000]
    [40-79]Gi:[100-2000]
    [80-99]Gi:[100-4000]
    [100-499]Gi:[100-6000]
    [500-999]Gi:[100-10000]
    [1000-1999]Gi:[100-20000]
    [2000-2999]Gi:[200-40000]
    [3000-3999]Gi:[200-48000]
    [4000-7999]Gi:[300-48000]
    [8000-9999]Gi:[500-48000]
    [10000-12000]Gi:[1000-48000]
  fsType: "xfs"
reclaimPolicy: "Delete"

{: codeblock}

Removing persistent storage from a cluster

{: #cleanup}

When you set up persistent storage in your cluster, you have three main components: the Kubernetes persistent volume claim (PVC) that requests storage, the Kubernetes persistent volume (PV) that is mounted to a pod and described in the PVC, and the IBM Cloud infrastructure instance, such as classic file or block storage. Depending on how you created your storage, you might need to delete all three components separately. {: shortdesc}

Understanding your storage removal options

{: #storage_delete_options}

Removing persistent storage from your {{site.data.keyword.cloud_notm}} account varies depending on how you provisioned the storage and what components you already removed. {: shortdesc}

Is my persistent storage deleted when I delete my cluster?

During cluster deletion, you have the option to remove your persistent storage. However, depending on how your storage was provisioned, the removal of your storage might not include all storage components.

If you dynamically provisioned storage with a storage class that sets reclaimPolicy: Delete, your PVC, PV, and the storage instance are automatically deleted when you delete the cluster. For storage that was statically provisioned, VPC Block Storage, or storage that you provisioned with a storage class that sets reclaimPolicy: Retain, the PVC and the PV are removed when you delete the cluster, but your storage instance and your data remain. You are still charged for your storage instance. Also, if you deleted your cluster in an unhealthy state, the storage might still exist even if you chose to remove it.

How do I delete the storage when I want to keep my cluster?

When you dynamically provisioned the storage with a storage class that sets reclaimPolicy: Delete, you can remove the PVC to start the deletion process of your persistent storage. Your PVC, PV, and storage instance are automatically removed.

For storage that was statically provisioned, VPC Block Storage, or storage that you provisioned with a storage class that sets reclaimPolicy: Retain, you must manually remove the PVC, PV, and the storage instance to avoid further charges.

How does the billing stop after I delete my storage?

Depending on what storage components you delete and when, the billing cycle might not stop immediately. If you delete the PVC and PV, but not the storage instance in your {{site.data.keyword.cloud_notm}} account, that instance still exists and you are charged for it.

If you delete the PVC, PV, and the storage instance, the billing cycle stops depending on the billingType that you chose when you provisioned your storage and how you chose to delete the storage.

When you manually cancel the persistent storage instance from the {{site.data.keyword.cloud_notm}} console or the ibmcloud sl CLI, billing stops as follows:
- Hourly storage: Billing stops immediately. After your storage is canceled, you might still see your storage instance in the console for up to 72 hours.
- Monthly storage: You can choose between immediate cancellation or cancellation on the anniversary date. In both cases, you are billed until the end of the current billing cycle, and billing stops for the next billing cycle. After your storage is canceled, you might still see your storage instance in the console or the CLI for up to 72 hours.
  - Immediate cancellation: Choose this option to immediately remove your storage. Neither you nor your users can use the storage anymore or recover the data.
  - Anniversary date: Choose this option to cancel your storage on the next anniversary date. Your storage instances remain active until the next anniversary date and you can continue to use them until this date, such as to give your team time to make backups of your data.
When you dynamically provisioned the storage with a storage class that sets reclaimPolicy: Delete and you choose to remove the PVC, the PV and the storage instance are immediately removed. For hourly billed storage, billing stops immediately. For monthly billed storage, you are still charged for the remainder of the month. After your storage is removed and billing stops, you might still see your storage instance in the console or the CLI for up to 72 hours.

What do I need to be aware of before I delete persistent storage?

When you clean up persistent storage, you delete all the data that is stored in it. If you need a copy of the data, make a backup for file storage or block storage.

I deleted my storage instance. Why can I still see my instance?

After you remove persistent storage, it can take up to 72 hours for the removal to be fully processed and for the storage to disappear from your {{site.data.keyword.cloud_notm}} console or CLI.

Cleaning up persistent storage

{: #storage_remove}

Remove the PVC, PV, and the storage instance from your {{site.data.keyword.cloud_notm}} account to avoid further charges for your persistent storage. {: shortdesc}

Before you begin:

Make sure that you backed up any data that you want to keep.
Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To clean up persistent data:

List the PVCs in your cluster and note the NAME of the PVC, the STORAGECLASS, and the name of the PV that is bound to the PVC and shown as VOLUME.

kubectl get pvc

{: pre}

Example output:

NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS            AGE
claim1-block-bronze   Bound     pvc-06886b77-102b-11e8-968a-f6612bb731fb   20Gi       RWO           ibmc-block-bronze       78d
claim-file-bronze     Bound     pvc-457a2b96-fafc-11e7-8ff9-b6c8f770356c   4Gi        RWX           ibmc-file-bronze-retain 105d
claim-file-silve      Bound     pvc-1efef0ba-0c48-11e8-968a-f6612bb731fb   24Gi       RWX           ibmc-file-silver        83d

{: screen}

Review the ReclaimPolicy and billingType for the storage class.
```
kubectl describe storageclass <storageclass_name>
```
{: pre}

If the reclaim policy says Delete, your PV and the physical storage are removed when you remove the PVC. Note that VPC Block Storage is not removed automatically, even if you used a Delete storage class to provision the storage. If the reclaim policy says Retain, or if you provisioned your storage without a storage class, then your PV and physical storage are not removed when you remove the PVC. You must remove the PVC, PV, and the physical storage separately.

If your storage is charged monthly, you still get charged for the entire month, even if you remove the storage before the end of the billing cycle. {: important}
Remove any pods that mount the PVC.
1. List the pods that mount the PVC.
```
kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
```
  {: pre}
  
  Example output:
```
blockdepl-12345-prz7b:	claim1-block-bronze  
```
  {: screen}
  
  If no pod is returned in your CLI output, you do not have a pod that uses the PVC.
2. Remove the pod that uses the PVC. If the pod is part of a deployment, remove the deployment.
```
kubectl delete pod <pod_name>
```
  {: pre}
3. Verify that the pod is removed.
```
kubectl get pods
```
  {: pre}
Remove the PVC.
```
kubectl delete pvc <pvc_name>
```
{: pre}
Review the status of your PV. Use the name of the PV that you retrieved earlier as VOLUME.
```
kubectl get pv <pv_name>
```
{: pre}

When you remove the PVC, the PV that is bound to the PVC is released. Depending on how you provisioned your storage, your PV goes into a Deleting state if the PV is deleted automatically, or into a Released state, if you must manually delete the PV. Note: For PVs that are automatically deleted, the status might briefly say Released before it is deleted. Rerun the command after a few minutes to see whether the PV is removed.
If your PV is not deleted, manually remove the PV.
```
kubectl delete pv <pv_name>
```
{: pre}
Verify that the PV is removed.
```
kubectl get pv
```
{: pre}
{: #sl_delete_storage}List the physical storage instance that your PV pointed to and note the id of the physical storage instance.
```
ibmcloud sl block volume-list --columns id --columns notes | grep <pv_name>
```
{: pre}

Example output:
```
12345678   {"plugin":"ibmcloud-block-storage-plugin-689df949d6-4n9qg","region":"us-south","cluster":"aa1a11a1a11b2b2bb22b22222c3c3333","type":"Endurance","ns":"default","pvc":"mypvc","pv":"pvc-d979977d-d79d-77d9-9d7d-d7d97ddd99d7","storageclass":"ibmc-block-silver","reclaim":"Delete"}
```
{: screen}

Understanding the Notes field information:
- "plugin":"ibm-file-plugin-5b55b7b77b-55bb7": The storage plug-in that the cluster uses.
- "region":"us-south": The region that your cluster is in.
- "cluster":"aa1a11a1a11b2b2bb22b22222c3c3333": The cluster ID that is associated with the storage instance.
- "type":"Endurance": The type of file or block storage, either Endurance or Performance.
- "ns":"default": The namespace that the storage instance is deployed to.
- "pvc":"mypvc": The name of the PVC that is associated with the storage instance.
- "pv":"pvc-d979977d-d79d-77d9-9d7d-d7d97ddd99d7": The PV that is associated with the storage instance.
- "storageclass":"ibmc-file-gold": The type of storage class: bronze, silver, gold, or custom.

Remove the physical storage instance.

ibmcloud sl block volume-cancel <classic_block_id>

{: pre}

Verify that the physical storage instance is removed.

The deletion process might take up to 72 hours to complete. {: important}
```
ibmcloud sl block volume-list
```
{: pre}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cs_storage_block.md

cs_storage_block.md

Storing data on classic IBM Cloud {{site.data.keyword.blockstorageshort}}

Quickstart for {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}}

Installing the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in in your cluster

Updating the {{site.data.keyword.cloud_notm}} Block Storage plug-in

Removing the {{site.data.keyword.cloud_notm}} Block Storage plug-in

Deciding on the block storage configuration

Setting up encryption for {{site.data.keyword.blockstorageshort}}

Encrypting volume data by using a custom storage class

Create a PVC that references your {{site.data.keyword.blockstorageshort}} secret

Verifying the encryption of your {{site.data.keyword.blockstorageshort}} volumes

Adding block storage to apps

Using existing block storage in your cluster

Step 1: Retrieving the information of your existing block storage

Step 2: Creating a persistent volume (PV) and a matching persistent volume claim (PVC)

Using block storage in a stateful set

Dynamic provisioning: Creating the PVC when you create a stateful set

Static provisioning: Using existing PVCs with a stateful set

Changing the size and IOPS of your existing storage device

Backing up and restoring data

Storage class reference

Sample customized storage classes

Creating topology-aware storage

Specifying the zone and region

Mounting block storage with an `XFS` file system

Removing persistent storage from a cluster

Understanding your storage removal options

Cleaning up persistent storage

Files

cs_storage_block.md

Latest commit

History

cs_storage_block.md

File metadata and controls

Storing data on classic IBM Cloud {{site.data.keyword.blockstorageshort}}

Quickstart for {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}}

Installing the {{site.data.keyword.cloud_notm}} {{site.data.keyword.blockstorageshort}} plug-in in your cluster

Updating the {{site.data.keyword.cloud_notm}} Block Storage plug-in

Removing the {{site.data.keyword.cloud_notm}} Block Storage plug-in

Deciding on the block storage configuration

Setting up encryption for {{site.data.keyword.blockstorageshort}}

Encrypting volume data by using a custom storage class

Create a PVC that references your {{site.data.keyword.blockstorageshort}} secret

Verifying the encryption of your {{site.data.keyword.blockstorageshort}} volumes

Adding block storage to apps

Using existing block storage in your cluster

Step 1: Retrieving the information of your existing block storage

Step 2: Creating a persistent volume (PV) and a matching persistent volume claim (PVC)

Using block storage in a stateful set

Dynamic provisioning: Creating the PVC when you create a stateful set

Static provisioning: Using existing PVCs with a stateful set

Changing the size and IOPS of your existing storage device

Backing up and restoring data

Storage class reference

Sample customized storage classes

Creating topology-aware storage

Specifying the zone and region

Mounting block storage with an XFS file system

Removing persistent storage from a cluster

Understanding your storage removal options

Cleaning up persistent storage

Mounting block storage with an `XFS` file system