Resource Name Change Validation (#2558)

* kubevirt vendor updates for schedops (#2420) Signed-off-by: Rohit-PX <[email protected]> Co-authored-by: Rohit-PX <[email protected]> * PB-6789:Create an app/vm in specific namespace (#2410) * Fix regex spelling error (#2429) * Fix restore NS name (#2430) * Ingore check for purefile (#2426) * Ingore check for purefile * [TC Automation] PTX-21440 Add test case VolAttachFAPxRestart (#2361) *[TC Automation] PTX-21440 Add test case VolAttachFAPxRestart,VolAttachSameFAPxRestart * Adding check to ignore Volume attaches in case of pure file volume (#2432) * Adding cloud create and s3 bucket deletion function for cloudsnaps (#2406) * Adding cloud create and s3 bucket deletion function for cloudsnaps * Adding bucket cleanup as part of cloudsnap tests --------- Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * Adding ZRS spec for aks (#2393) * Added method for creating volume snapshot class with the given provisioner (#2418) * PB-6789:Create an app/vm in specific namespace * Added snapclass creation * PB-5946: Added logic for PVC cleanup validation (#2282) * Added logic for PVC cleanup validation * Ptx 23576 VolResizeAllVolumes in parallel (#2433) * Trigger vol resize on all volumes at once * PTX-23652: be resilient to failures of GetVolumes() call (#2425) * PTX-23652: be resilient to failures of GetVolumes() call Signed-off-by: pnookala-px <[email protected]> * move total computation to before settting migration Signed-off-by: pnookala-px <[email protected]> * Retry GetVolumes() call Signed-off-by: pnookala-px <[email protected]> --------- Signed-off-by: pnookala-px <[email protected]> * Create and attach 250 + FADA volumes in Longevity or scale testing automation (#2421) Signed-off-by: Smit Jain <[email protected]> * Fixing volume provisoner in case of back up test (#2437) Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * Add a new test case for bootstrapping with local disks (#2428) * Added a new testcase for PTX-22628 * Change to MoveDisks * skip the root disk for detaching * cosmetic changes * Different logic for move disks * Updating node.go * Changes with Detach and AttachDisk * Added print statements * typo * Try with sleep * test detach attachdisk * try with RemoveDevice * moving the comment * testing objects * change to moref.value * change to use the existing finder object * debug statement * newer logic * different way to fetch datastore object * debug statement * old logic * try with replace file op * remove replace * cleanup code * wait for PX to be online * Added logic to remove disks for nonpxnodes * Update node.go * Added log statement for disk operations * temp: commenting out debug statement * ignore master nodes for targetVM * graceful movement of disks, add PXstop to srcVM * add log state for failure * update registry * refreshing driverendpoints * disabling PX on the stopped node * adding another refresh node registry * adding a log statement for debug * update the randNonPxNode object * Added a log statement for successful check of node ID * Test log statement cleanup * move the test to misc_test * correct the import sections * addressed review comments * Added pxbackupversion check to verify api call (#2434) * [PTX-23625] Fix TriggerRebootManyNodes (#2442) * Test automation for VolHAIncreaseAllVolumes (#2441) * Test automation for VolHAIncreaseAllVolumes * Anthos upgrade is failing in anthos reconcile preflight check (#2443) Signed-off-by: Smit Jain <[email protected]> * fixing MultiVolumeMountsForSharedV4 for chos-level (#2446) Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * [PWX-37026] Adding FBDAMultiTenancyBasicTest (#2419) * [PWX-37026] Adding FBDAMultiTenancyBasicTest Signed-off-by: Vinayak Shinde <[email protected]> * [PWX-37026] Updated libopenstorage/openstorage dependancy Signed-off-by: Vinayak Shinde <[email protected]> * [PWX-37026] Added FBDAMultiTenancyBasicTest Signed-off-by: Vinayak Shinde <[email protected]> * [PWX-37026] Addressed review comments Signed-off-by: Vinayak Shinde <[email protected]> --------- Signed-off-by: Vinayak Shinde <[email protected]> * Add test case to bring down network interface of FA Controller (#2455) * Add test case to bring down network interface of FA Controller * Adding senarios for adding storage node and stoageless nodes for clou… (#2449) * Adding senarios for adding storage node and stoageless nodes for cloud platforms * Deleting node scaling reduntant code --------- Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * Adding test for new storkctl flow (#2431) * Ptx 23763 Get FB Cluster Details (#2457) * Add test case to bring down network interface of FB * Calculate node upgrade time for Anthos parallel upgrade (#2387) * Calculate node upgrade time for parallel upgrade * Updating vendoring for latest sched-ops Signed-off-by: Smit Jain <[email protected]> --------- Signed-off-by: Smit Jain <[email protected]> * Fixing the user name length because it is used in namespace creation and exceeding 63 chars (#2459) * Automation scripts for iscsi port drop scenarios [ PTX-23831, PTX-23832,PTX-23833,PTX-23818] (#2460) * Automation scripts for iscsi port drop scenarios * [PWX-33177] Skip pool expansions when app has only FBDA volumes. (#2456) Since FBDA volumes do not get created on PX pool, there is no use of this validation. Signed-off-by: Vinayak Shinde <[email protected]> * PB-6777: Add Backup driver with WaitForBackupPartialCompletion method and its portworx implementation (#2448) * Fix retry for GetPxBackupVersionString (#2471) * Fix for NVME parsing (#2469) * Updating string split logic to split volume serial, extracted from the mount path * New Test case addition (#2466) * New Test case addition deletes Volume Delete on FA Backend * Fix nfs loc validation and pvc cleanup (#2474) * Adding test to recycle all storage driver nodes and validate drive set in cloud platforms (#2473) * adding debug logs for usable address * Adding test to recycle all nodes * Adding test to recycle all nodes --------- Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * [TC Improvement] Take px namespace where it is installed (#2476) * Temporary removeal of pvc cleanup validation (#2478) * Added retry for backup location validation (#2479) * Removed data validation after restore due to PB-6271 (#2480) * [PTX-23856] Increase Timeout In ValidateStorageCluster (#2465) * Ptx 23988 (#2486) function to check and list all the multipath devices present in the node * PB-6917: Add `CUSTOM_REGISTRY` and `CUSTOM_REPO` to support custom registry deployment (#2477) * PB-6917: Add fix to use docker.io in case of cloud * PB-6917: Add fix to use docker.io in case of cloud * Change implementation to `CUSTOM_REGISTRY ` and `CUSTOM_REPO ` * PB-6917: Add fix to use docker.io in case of cloud * Change implementation to `CUSTOM_REGISTRY ` and `CUSTOM_REPO ` * Minor fix * Minor fix * Adding test to increase max nodes per zone after scale up (#2484) Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * Ptx 23988 (#2490) * functions to fetch details on CLoud drives , multipaths , and skip Adddisk if not supported * [PWX-37028] Added FBDAMultiTenancyUpdatePureNFSEnpoint (#2485) * [PWX-37028] Added FBDAMultiTenancyUpdatePureNFSEnpoint * [PWX-37028] Addressed review comments Signed-off-by: Vinayak Shinde <[email protected]> --------- Signed-off-by: Vinayak Shinde <[email protected]> * Run torpedo as a pod in anthos user cluster (#2491) Signed-off-by: Smit Jain <[email protected]> * PTX-24028 Vendor latest libopenstorage/operator package (#2489) Signed-off-by: nikolaypopov <[email protected]> * Kubevirt specs for datavoltemplate , win22 , large datadisk (#2376) * Kubevirt specs for datavoltemplate , win22 , large datadisk * Renamed files * fix nit in vm.yaml to accept .vmID * fix small nit * rename large disk app * remove long names * New test case to Expand Multiple pool when FADA volumes are getting C… (#2495) * New test case to Expand Multiple pool when FADA volumes are getting Created * [PWX-37028] Skip volume validations. (#2497) * [TC Automation] kubevirt test cases (#2483) * Fbda topology creates (#2496) * fbda with top test1 Signed-off-by: pnookala-px <[email protected]> * fbda with top test1 Signed-off-by: pnookala-px <[email protected]> * fbda with top test1 Signed-off-by: pnookala-px <[email protected]> * fbda with top test2 Signed-off-by: pnookala-px <[email protected]> * sleep 30 seconds after deleting pvc Signed-off-by: pnookala-px <[email protected]> * create sc without top Signed-off-by: pnookala-px <[email protected]> * create sc without top Signed-off-by: pnookala-px <[email protected]> * fix spelling Signed-off-by: pnookala-px <[email protected]> * wait 3 mins only Signed-off-by: pnookala-px <[email protected]> * Refactored common code --------- Signed-off-by: pnookala-px <[email protected]> Co-authored-by: Vinayak Shinde <[email protected]> * fixing debian VM spec to have internal mount (#2507) * adding retry with timeout for PDB validation (#2500) * [TC Automation] Add test case RestartPXAndCheckIfVmBindMount (#2510) * add test case RestartPXAndCheckIfVmBindMount * increasing cpu cores for bonnie app (#2488) Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * Vendored the latest px-backup api for 2.7.1 (#2506) * Power off and power on all worker nodes (#2436) Signed-off-by: smarakala_pure <[email protected]> Co-authored-by: smarakala_pure <[email protected]> * [PTX-24172] Fix validateAutoFsTrim (#2512) * Detach clouddrives while IO inprogress and verify pool can be back after maintenance cycle (#2408) * Adding detach drives to longevity Signed-off-by: smarakala_pure <[email protected]> Co-authored-by: smarakala_pure <[email protected]> * minor fix for update node id function (#2515) Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * [PTX-24076] Add Ginkgo Support To Deployment Script (#2509) * [PTX-24076] Add yq Install Command To Dockerfile * [PTX-24076] Add Ginkgo Support To Deployment Script * [PTX-23863] Fix SetupProxyServer (#2524) * PWX-36878: DMthin OCP VSphere pool expand test update (#2503) * DMthin OCP VSphere pool expand test update Signed-off-by: Priyanshu Pandey <[email protected]> --------- Signed-off-by: Priyanshu Pandey <[email protected]> * Adding cloudsnap delete event to longevity (#2444) * Adding cloudsnap delete event to longevity Signed-off-by: smarakala_pure <[email protected]> * Adding params for px-security (#2527) * Fixing OCP zones (#2516) * Fixing OCP zones * Fixing ASG for openshift --------- Co-authored-by: Leela Srinivas Siripurapu <[email protected]> * Removed aks node driver (#2533) * PTX-23958: Vendoring latest operator (#2531) * vendoring latest operator * vendoring after rke2 change * even newer operator code * PTX-24002 Create Volume clone for FADA volume and verify new volume is created in the backend (#2501) * PTX-24002 * PTX-24002 * PTX-24002 changes made according to comments * changes done to exit the loop if clone volume found in FA * changes done according to comments * minor nitpick changes * nitpick * New Libraries for FA RestAPIs 2.x (#2543) New Libraries for FA RestAPIs 2.x * Fixing Json Parsing Issue (#2550) Fixing Json Parsing Issue * Added mysql & postgres spec with restricted PSA (#2521) * Waiting for atleast 2 mongo pods to come up after scaling down to 0 and scaling backup to 3 again (#2552) * name change validation * latest pds api changes * added bronze acc * Fixed names in status * fixing manifest issue --------- Signed-off-by: Rohit-PX <[email protected]> Signed-off-by: pnookala-px <[email protected]> Signed-off-by: Smit Jain <[email protected]> Signed-off-by: Vinayak Shinde <[email protected]> Signed-off-by: nikolaypopov <[email protected]> Signed-off-by: smarakala_pure <[email protected]> Signed-off-by: Priyanshu Pandey <[email protected]> Co-authored-by: Rohit-PX <[email protected]> Co-authored-by: Rohit-PX <[email protected]> Co-authored-by: snigdha-px <[email protected]> Co-authored-by: suhas <[email protected]> Co-authored-by: Kshithij Iyer <[email protected]> Co-authored-by: vprabhakar-px <[email protected]> Co-authored-by: Leela Srinivas <[email protected]> Co-authored-by: Leela Srinivas Siripurapu <[email protected]> Co-authored-by: Adil Rahman <[email protected]> Co-authored-by: sabrarhussaini <[email protected]> Co-authored-by: pnookala-px <[email protected]> Co-authored-by: jainsmit <[email protected]> Co-authored-by: vumesh-px <[email protected]> Co-authored-by: kphalgun-px <[email protected]> Co-authored-by: Vinayak Shinde <[email protected]> Co-authored-by: Leela Srinivas Siripurapu <[email protected]> Co-authored-by: Geet-PX <[email protected]> Co-authored-by: Mithun Koppal <[email protected]> Co-authored-by: sgajawada-px <[email protected]> Co-authored-by: Nikolay Popov <[email protected]> Co-authored-by: px-kannavarapu <[email protected]> Co-authored-by: Dhruv Bhatnagar <[email protected]> Co-authored-by: Swarupa Vijaykumar <[email protected]> Co-authored-by: Santhosh Marakala <[email protected]> Co-authored-by: smarakala_pure <[email protected]> Co-authored-by: Priyanshu Pandey <[email protected]> Co-authored-by: sayalasomayajula-px <[email protected]>
portworx · Jun 4, 2024 · 5eb73d3 · 5eb73d3
1 parent 99a662c
commit 5eb73d3
Show file tree

Hide file tree

Showing 767 changed files with 85,383 additions and 47,607 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -95,6 +95,10 @@ RUN apk add --no-cache openssh sshpass
 # Install dependancy for OCP 4.14 CLI
 RUN apk --update add gcompat
 
+# Install yq
+RUN wget https://github.com/mikefarah/yq/releases/download/v4.25.1/yq_linux_amd64 -O /usr/bin/yq && \
+    chmod +x /usr/bin/yq
+
 # Copy ginkgo & binaries over from previous container
 COPY --from=build /go/bin/ginkgo /bin/ginkgo
 COPY --from=build /go/src/github.com/portworx/torpedo/bin bin

diff --git a/apiServer/taas/apiserver.go b/apiServer/taas/apiserver.go
@@ -33,5 +33,6 @@ func main() {
 	router.POST("taas/stork/upgrade", utils.UpgradeStork)
 	router.DELETE("taas/deletepod", utils.DeletePod)
 	router.GET("taas/getpxbackupnamespace", utils.GetPxBackupNamespace)
+	router.POST("taas/createvolumesnapshotclass", utils.CreateVolumeSnapshotClass)
 	log.Fatal(router.Run(":8080"))
 }
diff --git a/apiServer/taas/utils/k8sutils.go b/apiServer/taas/utils/k8sutils.go
@@ -4,6 +4,8 @@ import (
 	"fmt"
 	"github.com/gin-gonic/gin"
 	"github.com/portworx/torpedo/drivers/pds/lib"
+	"github.com/portworx/torpedo/pkg/log"
+	"github.com/portworx/torpedo/tests"
 	v1 "k8s.io/api/core/v1"
 	"net/http"
 	"os/exec"
@@ -117,3 +119,34 @@ func GetNodeStatus(c *gin.Context) {
 		"DegradedNodes":  degradedNodes,
 	})
 }
+
+// CreateVolumeSnapshotClass creates volume snapshot class
+func CreateVolumeSnapshotClass(c *gin.Context) {
+	log.Infof("Creating volume snapshot class")
+	var createVolumeSnapshotClassRequest struct {
+		VolumeSnapshotClassName      string `json:"volumeSnapshotClassName"`
+		Provisioner                  string `json:"provisioner"`
+		IsDefaultVolumeSnapshotClass bool   `json:"isDefaultVolumeSnapshotClass"`
+		DeletePolicy                 string `json:"deletePolicy"`
+	}
+	if !checkTorpedoInit(c) {
+		c.JSON(http.StatusInternalServerError, gin.H{
+			"error in init": fmt.Errorf("error in InitInstance()"),
+		})
+		return
+	}
+	if err := c.BindJSON(&createVolumeSnapshotClassRequest); err != nil {
+		c.JSON(http.StatusBadRequest, gin.H{"error for request": err.Error()})
+		return
+	}
+	if len(createVolumeSnapshotClassRequest.VolumeSnapshotClassName) == 0 || len(createVolumeSnapshotClassRequest.Provisioner) == 0 {
+		c.JSON(http.StatusBadRequest, gin.H{"error": "volumesnapshotclass name  or provisioner cannot be empty", "vsc name": createVolumeSnapshotClassRequest.VolumeSnapshotClassName, "provisioner": createVolumeSnapshotClassRequest.Provisioner})
+		return
+	}
+	_, err := tests.Inst().S.CreateVolumeSnapshotClasses(createVolumeSnapshotClassRequest.VolumeSnapshotClassName, createVolumeSnapshotClassRequest.Provisioner, createVolumeSnapshotClassRequest.IsDefaultVolumeSnapshotClass, createVolumeSnapshotClassRequest.DeletePolicy)
+	if err != nil {
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	c.JSON(http.StatusOK, gin.H{"message": fmt.Sprintf("VolumeSnapshotClass %v with provisioner %s and setting it as default: %v created successfully", createVolumeSnapshotClassRequest.VolumeSnapshotClassName, createVolumeSnapshotClassRequest.Provisioner, createVolumeSnapshotClassRequest.IsDefaultVolumeSnapshotClass)})
+}
diff --git a/apiServer/taas/utils/torpedoutils.go b/apiServer/taas/utils/torpedoutils.go
@@ -199,7 +199,7 @@ func ScheduleAppsAndValidate(c *gin.Context) {
 	}
 
 	tests.Inst().AppList = requestBody.AppList
-	options := tests.CreateScheduleOptions("")
+	options := tests.CreateScheduleOptions(requestBody.NamespaceSuffix)
 	context, err := tests.Inst().S.Schedule(requestBody.NamespaceSuffix, options)
 	if err != nil {
 		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})

diff --git a/deployments/deploy-pxb-cloud.sh b/deployments/deploy-pxb-cloud.sh
@@ -356,9 +356,6 @@ if [ -z "${NODE_DRIVER}" ]; then
 fi
 if [ -n "${K8S_VENDOR}" ]; then
     case "$K8S_VENDOR" in
-        aks)
-            NODE_DRIVER="aks"
-            ;;
         oracle)
             NODE_DRIVER="oracle"
             ;;
@@ -693,6 +690,10 @@ spec:
       value: "${S3_ENCRYPTION_POLICY}"
     - name: USE_GLOBAL_RULES
       value: "${USE_GLOBAL_RULES}"
+    - name: CUSTOM_REGISTRY
+      value: "${CUSTOM_REGISTRY}"
+    - name: CUSTOM_REPO
+      value: "${CUSTOM_REPO}"
   volumes: [${VOLUMES}]
   restartPolicy: Never
   serviceAccountName: torpedo-account

diff --git a/deployments/deploy-ssh.sh b/deployments/deploy-ssh.sh
@@ -238,6 +238,11 @@ if [ -n "$ANTHOS_INST_PATH" ]; then
     ANTHOS_INST_PATH="${ANTHOS_INST_PATH}"
 fi
 
+if [ -n "$ANTHOS_HOST_PATH" ]; then
+    ANTHOS_HOST_PATH="${ANTHOS_HOST_PATH}"
+fi
+
+
 for i in $@
 do
 case $i in
@@ -249,7 +254,7 @@ case $i in
 esac
 done
 
-echo "checking if we need to override test suite: ${TEST_SUITE}"
+echo "Checking if we need to override test suite: ${TEST_SUITE}"
 
 # TODO: Remove this after all longevity jobs switch to 'bin/longevity.test' for TEST_SUITE.
 case $FOCUS_TESTS in
@@ -317,6 +322,9 @@ TESTRESULTS_MOUNT="{ \"name\": \"testresults\", \"mountPath\": \"/testresults/\"
 AWS_VOLUME="{ \"name\": \"aws-volume\", \"configMap\": { \"name\": \"aws-cm\", \"items\": [{\"key\": \"credentials\", \"path\": \"credentials\"}, {\"key\": \"config\", \"path\": \"config\"}]} }"
 AWS_VOLUME_MOUNT="{ \"name\": \"aws-volume\", \"mountPath\": \"/root/.aws/\" }"
 
+ANTHOS_VOLUME="{ \"name\": \"anthosdir\", \"hostPath\": { \"path\": \"${ANTHOS_HOST_PATH}\", \"type\": \"Directory\" } }"
+ANTHOS_VOLUME_MOUNT="{ \"name\": \"anthosdir\", \"mountPath\": \"/anthosdir\" }"
+
 VOLUMES="${TESTRESULTS_VOLUME}"
 
 if [ "${STORAGE_DRIVER}" == "aws" ]; then
@@ -365,6 +373,11 @@ if [ -n "${TORPEDO_CUSTOM_PARAM_MOUNT}" ]; then
     VOLUME_MOUNTS="${VOLUME_MOUNTS},${TORPEDO_CUSTOM_PARAM_MOUNT}"
 fi
 
+if [ -n "${ANTHOS_HOST_PATH}" ]; then
+  VOLUMES="${VOLUMES},${ANTHOS_VOLUME}"
+  VOLUME_MOUNTS="${VOLUME_MOUNTS},${ANTHOS_VOLUME_MOUNT}"
+fi
+
 BUSYBOX_IMG="busybox"
 if [ -n "${INTERNAL_DOCKER_REGISTRY}" ]; then
     BUSYBOX_IMG="${INTERNAL_DOCKER_REGISTRY}/busybox"
@@ -497,7 +510,7 @@ spec:
     args: [ "--trace",
             "--timeout", "${TIMEOUT}",
             "$FAIL_FAST",
-            "--poll-progress-after", "10m",
+            "--poll-progress-after", "20m",
             --junit-report=$JUNIT_REPORT_PATH,
             "$FOCUS_ARG",
             "$SKIP_ARG",
@@ -568,6 +581,7 @@ spec:
             "--torpedo-job-name=$TORPEDO_JOB_NAME",
             "--torpedo-job-type=$TORPEDO_JOB_TYPE",
             "--torpedo-skip-system-checks=$TORPEDO_SKIP_SYSTEM_CHECKS",
+            "--fa-secret=${FA_SECRET}",
             "$APP_DESTROY_TIMEOUT_ARG",
             "$SCALE_APP_TIMEOUT_ARG",
     ]
@@ -797,6 +811,59 @@ spec:
 
 EOF
 
+if [ "${RUN_GINKGO_COMMAND}" = "true" ]; then
+    torpedo_pod_command="ginkgo"
+    torpedo_pod_args=$(yq e '.spec.containers[] | select(.name == "torpedo") | .args[]' torpedo.yaml | sed 's/,$//')
+
+	# This code removes the comma if the line ends with it.
+	# If the line is an empty string, it is quoted.
+	# Otherwise, the line is printed normally.
+    formatted_torpedo_pod_args=$(echo "$torpedo_pod_args" | awk '{
+        if ($0 ~ /,$/) {
+            gsub(/,$/, "", $0);
+            printf "%s ", $0;
+        } else {
+            if ($0 == "") {
+                printf "\"\" ";
+            } else {
+                printf "%s ", $0;
+            }
+        }
+    }')
+
+    torpedo_pod_ginkgo_command="$torpedo_pod_command $formatted_torpedo_pod_args"
+    echo "Formatted Ginkgo Command: $torpedo_pod_ginkgo_command"
+
+    # This code skips a flag followed by an empty string ("") if the next token is another flag or if it is the end of the command.
+    # This is necessary because Torpedo does not handle empty strings as expected.
+    # Example: In the input ginkgo --trace --timeout "" --fail-fast ... --fa-secret ""
+    # --timeout "" is skipped because it is immediately followed by another flag --fail-fast
+    # --fa-secret "" is also skipped because it is the last token and followed by no other arguments.
+    cleaned_torpedo_pod_ginkgo_command=$(echo "$torpedo_pod_ginkgo_command" | awk '
+	{
+	   output = "";
+	   i = 1;
+	   while (i <= NF) {
+		   if ($(i) ~ /^--/ && $(i+1) == "\"\"") {
+			   if (i+2 <= NF && $(i+2) ~ /^--/) {
+				   i += 2;
+				   continue;
+			   } else if (i+2 > NF) {
+				   i += 2;
+				   continue;
+			   }
+		   }
+		   output = output (output ? " " : "") $(i);
+		   i++;
+	   }
+	   print output;
+	}')
+    echo "Cleaned Ginkgo Command: $cleaned_torpedo_pod_ginkgo_command"
+
+    $cleaned_torpedo_pod_ginkgo_command
+    exit $?
+fi
+
 # If these are passed, we will create a docker config secret to use to pull images
 if [ ! -z $IMAGE_PULL_SERVER ] && [ ! -z $IMAGE_PULL_USERNAME ] && [ ! -z $IMAGE_PULL_PASSWORD ]; then
   echo "Adding Docker registry secret ..."

diff --git a/drivers/backup/backup.go b/drivers/backup/backup.go
@@ -252,6 +252,11 @@ type Backup interface {
 	WaitForBackupCompletion(ctx context.Context, backupName string, orgID string,
 		timeout time.Duration, timeBeforeRetry time.Duration) error
 
+	// WaitForBackupPartialCompletion waits for backup to partial complete successfully
+	// or till timeout is reached. API should poll every `timeBeforeRetry`
+	WaitForBackupPartialCompletion(ctx context.Context, backupName string, orgID string,
+		timeout time.Duration, timeBeforeRetry time.Duration) error
+
 	// WaitForBackupDeletion waits for backup to be deleted successfully
 	// or till timeout is reached. API should poll every `timeBeforeRetry
 	WaitForBackupDeletion(ctx context.Context, backupName string, orgID string,

diff --git a/drivers/backup/portworx/portworx.go b/drivers/backup/portworx/portworx.go
@@ -985,6 +985,61 @@ func (p *portworx) WaitForBackupCompletion(
 	return nil
 }
 
+// WaitForBackupPartialCompletion waits for backup to complete successfully
+// or till timeout is reached. API should poll every `timeBeforeRetry` duration
+func (p *portworx) WaitForBackupPartialCompletion(
+	ctx context.Context,
+	backupName,
+	orgID string,
+	timeout time.Duration,
+	timeBeforeRetry time.Duration,
+) error {
+
+	backupUID, err := p.GetBackupUID(ctx, backupName, orgID)
+	if err != nil {
+		return err
+	}
+	req := &api.BackupInspectRequest{
+		Name:  backupName,
+		OrgId: orgID,
+		Uid:   backupUID,
+	}
+	var backupError error
+	f := func() (interface{}, bool, error) {
+		inspectBkpResp, err := p.backupManager.Inspect(ctx, req)
+		if err != nil {
+			// Error occured, just retry
+			return nil, true, err
+		}
+
+		// Check if backup status is complete
+		currentStatus := inspectBkpResp.GetBackup().GetStatus().GetStatus()
+		if currentStatus == api.BackupInfo_StatusInfo_PartialSuccess {
+			// If backup is complete, dont retry again
+			return nil, false, nil
+		} else if currentStatus == api.BackupInfo_StatusInfo_Failed ||
+			currentStatus == api.BackupInfo_StatusInfo_Aborted ||
+			currentStatus == api.BackupInfo_StatusInfo_Invalid ||
+			currentStatus == api.BackupInfo_StatusInfo_Success {
+			backupError = fmt.Errorf("backup [%v] is in [%s] state. reason: [%v]",
+				req.GetName(), currentStatus,
+				inspectBkpResp.GetBackup().GetStatus().GetReason())
+			return nil, false, backupError
+		}
+		return nil,
+			true,
+			fmt.Errorf("backup [%v] is in [%s] state. Waiting to become Partial Complete",
+				req.GetName(), currentStatus)
+	}
+
+	_, err = task.DoRetryWithTimeout(f, timeout, timeBeforeRetry)
+	if err != nil || backupError != nil {
+		return fmt.Errorf("failed to wait for backup. Error:[%v] Reason:[%v]", err, backupError)
+	}
+
+	return nil
+}
+
 // WaitForBackupDeletion waits for backup to be deleted successfully
 // or till timeout is reached. API should poll every `timeBeforeRetry` duration
 func (p *portworx) WaitForBackupDeletion(

diff --git a/drivers/node/node.go b/drivers/node/node.go
@@ -5,6 +5,7 @@ import (
 	"time"
 
 	"github.com/libopenstorage/openstorage/api"
+	corev1 "github.com/libopenstorage/operator/pkg/apis/core/v1"
 	"github.com/portworx/torpedo/pkg/errors"
 )
 
@@ -201,12 +202,21 @@ type Driver interface {
 	// DestroyVM powers VM
 	DestroyVM(node Node) error
 
+	// MoveDisks moves disks from one node to another
+	MoveDisks(sourceNode Node, targetNode Node) error
+
+	// RemoveNonRootDisks removes non-root disks from the node
+	RemoveNonRootDisks(node Node) error
+
 	// SystemctlUnitExist checks if a given service exists in a node
 	SystemctlUnitExist(n Node, service string, options SystemctlOpts) (bool, error)
 
 	// AddMachine adds the new machine instance to existing map
 	AddMachine(machineName string) error
 
+	// DetachDisk vdisk from node.
+	DetachDrivesFromVM(stc *corev1.StorageCluster, nodeName string) error
+
 	// PowerOnVMByName power on the VM using the vm name
 	PowerOnVMByName(vmName string) error
 
@@ -284,6 +294,13 @@ func (d *notSupportedDriver) RebootNode(node Node, options RebootNodeOpts) error
 	}
 }
 
+func (d *notSupportedDriver) DetachDrivesFromVM(stc *corev1.StorageCluster, nodeName string) error {
+	return &errors.ErrNotSupported{
+		Type:      "Function",
+		Operation: "DetachDrivesFromVM()",
+	}
+}
+
 func (d *notSupportedDriver) RebootNodeAndWait(node Node) error {
 	return &errors.ErrNotSupported{
 		Type:      "Function",
@@ -518,3 +535,17 @@ func (d *notSupportedDriver) GetSupportedDriveTypes() ([]string, error) {
 		Operation: "GetSupportedDriveTypes()",
 	}
 }
+
+func (d *notSupportedDriver) MoveDisks(sourceNode Node, targetNode Node) error {
+	return &errors.ErrNotSupported{
+		Type:      "Function",
+		Operation: "MoveDisks()",
+	}
+}
+
+func (d *notSupportedDriver) RemoveNonRootDisks(node Node) error {
+	return &errors.ErrNotSupported{
+		Type:      "Function",
+		Operation: "RemoveNonRootDisks()",
+	}
+}
diff --git a/drivers/node/node_registry.go b/drivers/node/node_registry.go
@@ -226,3 +226,17 @@ func GetNodeDetailsByNodeID(nodeID string) (Node, error) {
 	}
 	return Node{}, fmt.Errorf("failed to get Node Details by Node ID [%s] ", nodeID)
 }
+
+// GetPXDisabledNodes returns all the nodes where PX is disabled
+func GetPXDisabledNodes() []Node {
+	var nodeList []Node
+	for _, n := range nodeRegistry {
+		if !n.IsStorageDriverInstalled {
+			// Ignore the control plane nodes
+			if !IsMasterNode(n) {
+				nodeList = append(nodeList, n)
+			}
+		}
+	}
+	return nodeList
+}
diff --git a/drivers/node/ssh/ssh.go b/drivers/node/ssh/ssh.go
@@ -820,7 +820,7 @@ func (s *SSH) doCmdSSH(n node.Node, options node.ConnectionOpts, cmd string, ign
 	if ignoreErr == false && err != nil {
 		log.Infof("SSH ERR: %v", err)
 		return out, &node.ErrFailedToRunCommand{
-			Addr:  n.UsableAddr,
+			Addr:  n.Name,
 			Cause: fmt.Sprintf("failed to run command. sterr: %v, err: %v", sterr, err),
 		}
 	}
@@ -839,6 +839,7 @@ func (s *SSH) getConnection(n node.Node, options node.ConnectionOpts) (*ssh_pkg.
 func (s *SSH) getConnectionOnUsableAddr(n node.Node, options node.ConnectionOpts) (*ssh_pkg.Client, error) {
 	var sshErr error
 	var cli interface{}
+	log.Debugf("checking for usable address in: %v for node [%s]", n.Addresses, n.Name)
 	for _, addr := range n.Addresses {
 		t := func() (interface{}, bool, error) {
 			// check if address is responding on port 22
@@ -848,6 +849,7 @@ func (s *SSH) getConnectionOnUsableAddr(n node.Node, options node.ConnectionOpts
 		}
 		if cli, sshErr = task.DoRetryWithTimeout(t, options.Timeout, options.TimeBeforeRetry); sshErr == nil {
 			n.UsableAddr = addr
+			log.Debugf("usable address: [%s] for node [%s]", n.UsableAddr, n.Name)
 			return cli.(*ssh_pkg.Client), nil
 		}
 	}