diff --git a/.gitignore b/.gitignore index 18da50b..b3faa51 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ venv/ -.venv/ \ No newline at end of file +.venv/ +.ignore/ \ No newline at end of file diff --git a/docs/design/motivation.md b/docs/design/motivation.md new file mode 100644 index 0000000..a189ba7 --- /dev/null +++ b/docs/design/motivation.md @@ -0,0 +1,14 @@ +# Motivation for the changes between workspace v1 and v2 + +The initial v1 version of the workspace had several design flaws, making it inflexible and difficult for platform operators to use: + +* Each workspace had a fixed number of components, including exactly one bucket and a predefined list of services that were available to all users. This rigid structure made it difficult to customize workspaces for specific needs. +* The services were deployed into Kubernetes through an explicit call on the workspace-api, using a provisioning pipeline defined in code. Upgrading these services to newer versions for all users was difficult, as it required additional workspace-api calls for redeployment. Extending the list of services for specific users was even more challenging. +* Provisioning of infrastructure components like the object storage bucket(s) and associated IAM policies and user are always operator specific and the customization possibilities with custom webhooks were cumbersome. +* Kubernetes namespaces were used for multi-tenancy, making some operations manual (e.g. installing CRDs) or impossible for an operator (using different versions of a CRD, RBAC with cluster wide permissions,...). + +The v2 version of the workspace mitigates these issues: + +* The workspace controller automatically creates a dedicated space in a Git repository (either a new repository or a separate folder) for each workspace, allowing for declarative descriptions of all deployed services with their current software version and configuration, so this information can be easily inspected. Custom services can be installed via GitOps mechanisms and are declaratively stated the same way. +* The provisioning pipeline is declaratively stated and can be adapted by the operator. Best-practise blueprints are shared in the workspace EOEPCA repository. The provisioning pipeline is continuously reconciled, always trying to match the desired state with the observed state. Updates of services can therefore be selectively or globally triggered via Git. +* The vcluster tooling is automatically deployed per namespace, providing a dedicated Kubernetes API server runtime for each workspace. diff --git a/docs/usage/howtos.md b/docs/usage/howtos.md index f9189e9..743e360 100644 --- a/docs/usage/howtos.md +++ b/docs/usage/howtos.md @@ -1,3 +1,205 @@ # How-Tos How-tos to communicate usage by example. + +## Workspace Building Block + +### UC1: Dedicated Workspace for Users and Projects + +**User Story:** As a user, I want to use an instance of a building block or component (e.g., resource discovery, data access) which is dedicated to my own or my project workspace. + +**Tech Standards:** Kubernetes, GitOps + +**Epic:** E5310 + +**Building Block:** Workspace + +**How-To:** + +1. **Create a Workspace:** Use the Workspace Controller to create a new workspace dedicated to your project or individual use. +2. **Provision Services:** Utilize the Workspace Controller to provision the necessary services within your workspace, such as resource discovery, data access, or visualization tools. +3. **Access Services:** Access the provisioned services within your workspace using the provided APIs or user interfaces. + +### UC2: Workspace Provisioning and Management + +**User Story:** As a platform operator, I want to leverage a SOTA solution to provision and manage workspaces for projects/groups. + +**Tech Standards:** Kubernetes, GitOps + +**Epic:** E5310 + +**Building Block:** Workspace + +**How-To:** + +1. **Configure Workspace Templates:** Define templates for different workspace types (e.g., individual user, project team) using GitOps principles. +2. **Provision Workspaces:** Use the Workspace Controller to provision new workspaces based on the defined templates. +3. **Manage Workspaces:** Monitor and manage workspace resources, including services, storage, and user access, through the Workspace Controller. + +## Storage Building Block + +### UC3: S3 Object Storage for Data Organization + +**User Story:** As a user, I want an S3 object storage to organize and curate data. + +**Tech Standards:** S3 + +**Epic:** E5350 + +**Building Block:** Workspace + +**How-To:** + +1. **Create Storage Buckets:** Use the Storage Controller to create new S3 buckets within your workspace. +2. **Upload Data:** Upload your data files to the created buckets using the provided S3 API or HTTP access. +3. **Organize Data:** Organize your data within the buckets using folders and file naming conventions. + +## Security Building Block + +### UC4: IAM Control for Users + +**User Story:** As a platform operator, I want to keep control on IAM for my users. + +**Tech Standards:** OAuth2 + +**Epic:** E5340 + +**Building Block:** Workspace + +**How-To:** + +1. **Configure IAM Roles:** Define IAM roles with specific permissions for different user groups (e.g., data scientists, project managers). +2. **Assign Roles to Users:** Assign the appropriate IAM roles to users based on their responsibilities and access needs. +3. **Monitor Access:** Monitor user access and activity logs to ensure security and compliance. + +## Application Hub Building Block + +### UC5: Delegated Workspace Service Instantiation + +**User Story:** As a platform operator, I want to delegate Workspace Service instantiation to the end-users without compromising security. + +**Tech Standards:** Kubernetes, Helm + +**Epic:** E5320 + +**Building Block:** Workspace, Application Hub, MLOps + +**How-To:** + +1. **Create Helm Charts:** Package Workspace Services as Helm charts, including dependencies and configuration options. +2. **Publish Charts to Application Hub:** Publish the Helm charts to the Application Hub, making them accessible to users. +3. **User Service Instantiation:** Allow users to install and configure Workspace Services from the Application Hub using Helm. + +## Resource Management Building Block + +### UC6: Runtime Resource Management + +**User Story:** As a platform operator, I want to easily turn on and off projects/groups runtime resources to save costs. + +**Tech Standards:** Kubernetes + +**Epic:** E5320 + +**Building Block:** Workspace + +**How-To:** + +1. **Configure Resource Scaling:** Define resource scaling policies for different workspace types or services. +2. **Automate Resource Management:** Use Kubernetes features like autoscaling and resource quotas to automatically manage resource allocation based on usage patterns. +3. **Monitor Resource Usage:** Monitor resource consumption and adjust scaling policies as needed to optimize costs. + +## Data Management Building Block + +### UC7: Data Integration and Collaboration + +**User Story:** As a user, I want to integrate (copy as well as referencing) data in the project/group space for collaboration. + +**Tech Standards:** S3 + +**Epic:** E5330 + +**Building Block:** Workspace + +**How-To:** + +1. **Upload Data to Workspace Buckets:** Upload data files to the workspace's S3 buckets. +2. **Share Data with Collaborators:** Grant access to the workspace buckets to collaborators, allowing them to view, download, or modify data. +3. **Reference Data:** Use data references (e.g., URLs, S3 paths) to link to data stored in the workspace buckets, enabling collaboration without duplicating data. + +### UC8: Data Discovery and Exploration + +**User Story:** As a user, I want to have an exhaustive view on all available data in the project/group space. + +**Epic:** E5330 + +**Building Block:** Workspace + +**How-To:** + +1. **Use Resource Discovery Service:** Utilize the Workspace's resource discovery service to browse and search for available data within the workspace. +2. **Explore Data Metadata:** Access metadata associated with data files, including file size, format, and creation date. +3. **Filter and Sort Data:** Filter and sort data based on specific criteria to find relevant information. + +### UC9: Data Change Tracking + +**User Story:** As a user, I want to be able to trace changes on data. + +**Epic:** E5330 + +**Building Block:** Workspace + +**How-To:** + +1. **Enable Versioning:** Configure versioning for the workspace's S3 buckets to track changes to data files. +2. **Access Data History:** View previous versions of data files and track changes made over time. +3. **Restore Previous Versions:** Restore previous versions of data files if needed. + +### UC10: Reproducible Data Exploration and Processing + +**User Story:** As a user, I want to be able to explore / process / experiment with stable snapshots of data for reproducibility in a scalable way with common software libraries. + +**Tech Standards:** S3, fsspec + +**Epic:** E5330 + +**Building Block:** Workspace + +**How-To:** + +1. **Create Data Snapshots:** Create snapshots of data within the workspace's S3 buckets to ensure reproducibility. +2. **Use fsspec for Data Access:** Utilize the fsspec library to access data snapshots from within your analysis environment. +3. **Scale Data Processing:** Leverage the scalability of the underlying infrastructure to process large datasets efficiently. + +## Application Hub Building Block + +### UC11: Familiar Data Libraries and Tools + +**User Story:** As a user, I want to use my familiar data library and tool stack for exploration and curation. + +**Tech Standards:** Kubernetes, Helm + +**Epic:** E5350 + +**Building Block:** Workspace + +**How-To:** + +1. **Install Data Libraries:** Install your preferred data libraries and tools within your workspace using Helm charts from the Application Hub. +2. **Configure Libraries:** Configure the installed libraries and tools to access data within the workspace. +3. **Use Familiar Tools:** Utilize your familiar data libraries and tools for exploration, analysis, and curation. + +### UC12: Custom Applications and Tools + +**User Story:** As a user, I want to bring a custom set of existing applications and tools close to the data. + +**Tech Standards:** Kubernetes, Helm + +**Epic:** E5360 + +**Building Block:** Workspace, Application Hub, MLOps + +**How-To:** + +1. **Package Applications as Helm Charts:** Package your custom applications and tools as Helm charts. +2. **Publish Charts to Application Hub:** Publish the Helm charts to the Application Hub. +3. **Deploy Applications to Workspace:** Deploy your custom applications and tools to your workspace using Helm. diff --git a/mkdocs.yml b/mkdocs.yml index 38fb201..e981e3b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -7,21 +7,22 @@ nav: - Home: home.html - Workspace: - index.md - - Getting Started: - - getting-started/quick-start.md - - getting-started/more-getting-started.md + #- Getting Started: + # - getting-started/quick-start.md + # - getting-started/more-getting-started.md - Design: - design/overview.md + - design/motivation.md - design/more-design.md - - Usage: - - usage/tutorials.md - - usage/howtos.md - - Administration: - - admin/configuration.md - - admin/maintenance.md - - API: - - api/endpoint-specification.md - - api/usage.md + #- Usage: + # - usage/tutorials.md + # - usage/howtos.md + #- Administration: + # - admin/configuration.md + # - admin/maintenance.md + #- API: + # - api/endpoint-specification.md + # - api/usage.md theme: # name: mkdocs diff --git a/setup/eoepca-demo/function-patch-and-transform.yaml b/setup/eoepca-demo/function-patch-and-transform.yaml new file mode 100644 index 0000000..f6b0645 --- /dev/null +++ b/setup/eoepca-demo/function-patch-and-transform.yaml @@ -0,0 +1,6 @@ +apiVersion: pkg.crossplane.io/v1beta1 +kind: Function +metadata: + name: function-patch-and-transform +spec: + package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.4 \ No newline at end of file diff --git a/setup/eoepca-demo/provider-kubernetes.yaml b/setup/eoepca-demo/provider-kubernetes.yaml new file mode 100644 index 0000000..a7a4017 --- /dev/null +++ b/setup/eoepca-demo/provider-kubernetes.yaml @@ -0,0 +1,40 @@ +apiVersion: pkg.crossplane.io/v1 +kind: Provider +metadata: + name: provider-kubernetes +spec: + package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.14.0 + runtimeConfigRef: + apiVersion: pkg.crossplane.io/v1beta1 + kind: DeploymentRuntimeConfig + name: provider-kubernetes +--- +apiVersion: pkg.crossplane.io/v1beta1 +kind: DeploymentRuntimeConfig +metadata: + name: provider-kubernetes +spec: + serviceAccountTemplate: + metadata: + name: provider-kubernetes +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: provider-kubernetes-cluster-admin +subjects: + - kind: ServiceAccount + name: provider-kubernetes + namespace: crossplane-system +roleRef: + kind: ClusterRole + name: cluster-admin + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: kubernetes.crossplane.io/v1alpha1 +kind: ProviderConfig +metadata: + name: provider-kubernetes +spec: + credentials: + source: InjectedIdentity \ No newline at end of file diff --git a/setup/eoepca-demo/provider-minio.yaml b/setup/eoepca-demo/provider-minio.yaml new file mode 100644 index 0000000..7bb4603 --- /dev/null +++ b/setup/eoepca-demo/provider-minio.yaml @@ -0,0 +1,18 @@ +apiVersion: pkg.crossplane.io/v1 +kind: Provider +metadata: + name: provider-minio +spec: + package: ghcr.io/vshn/provider-minio/provider:v0.3.0 +--- +apiVersion: minio.crossplane.io/v1 +kind: ProviderConfig +metadata: + name: provider-minio +spec: + credentials: + apiSecretRef: + name: minio-secret + namespace: crossplane-system + source: InjectedIdentity + minioURL: https://minio.develop.eoepca.org \ No newline at end of file diff --git a/setup/eoepca-demo/storage/composition.yaml b/setup/eoepca-demo/storage/composition.yaml new file mode 100644 index 0000000..a13ee4b --- /dev/null +++ b/setup/eoepca-demo/storage/composition.yaml @@ -0,0 +1,55 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: Composition +metadata: + name: storage +spec: + compositeTypeRef: + apiVersion: epca.eo/v1beta1 + kind: XStorage + mode: Pipeline + pipeline: + - step: bucket + functionRef: + name: function-patch-and-transform + input: + apiVersion: pt.fn.crossplane.io/v1beta1 + kind: Resources + resources: + - name: bucket-results + base: + apiVersion: minio.crossplane.io/v1 + kind: Bucket + metadata: {} + spec: + forProvider: + region: eu-central-1 + providerConfigRef: + name: provider-minio + patches: + - type: FromCompositeFieldPath + fromFieldPath: "spec.claimRef.namespace" + toFieldPath: "metadata.name" + transforms: + - type: string + string: + type: Format + fmt: '%s-results' + - name: bucket-stage + base: + apiVersion: minio.crossplane.io/v1 + kind: Bucket + metadata: {} + spec: + forProvider: + region: eu-central-1 + providerConfigRef: + name: provider-minio + patches: + - type: FromCompositeFieldPath + fromFieldPath: "spec.claimRef.namespace" + toFieldPath: "metadata.name" + transforms: + - type: string + string: + type: Format + fmt: '%s-stage' \ No newline at end of file diff --git a/setup/eoepca-demo/storage/definition.yaml b/setup/eoepca-demo/storage/definition.yaml new file mode 100644 index 0000000..e641436 --- /dev/null +++ b/setup/eoepca-demo/storage/definition.yaml @@ -0,0 +1,34 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: CompositeResourceDefinition +metadata: + name: xstorages.epca.eo +spec: + connectionSecretKeys: + - S3_USER_STORAGE_KEY + - S3_USER_STORAGE_SECRET + - S3_USER_STORAGE_BUCKET + group: epca.eo + names: + kind: XStorage + plural: xstorages + claimNames: + kind: Storage + plural: storages + versions: + - name: v1beta1 + served: true + referenceable: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + location: + type: string + acl: + type: string + required: + - location + - acl \ No newline at end of file diff --git a/setup/eoepca-demo/workspace/composition.yaml b/setup/eoepca-demo/workspace/composition.yaml new file mode 100644 index 0000000..a10e5ce --- /dev/null +++ b/setup/eoepca-demo/workspace/composition.yaml @@ -0,0 +1,113 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: Composition +metadata: + name: workspace +spec: + compositeTypeRef: + apiVersion: epca.eo/v1beta1 + kind: XWorkspace + mode: Pipeline + pipeline: + - step: namespace + functionRef: + name: function-patch-and-transform + input: + apiVersion: pt.fn.crossplane.io/v1beta1 + kind: Resources + resources: + - name: namespace + base: + apiVersion: kubernetes.crossplane.io/v1alpha2 + kind: Object + metadata: {} + spec: + forProvider: + manifest: + apiVersion: v1 + kind: Namespace + metadata: + labels: + workspace: "true" + providerConfigRef: + name: provider-kubernetes + patches: + - type: FromCompositeFieldPath + fromFieldPath: "metadata.name" + toFieldPath: "spec.forProvider.manifest.metadata.name" + - name: resourcequota + base: + apiVersion: kubernetes.crossplane.io/v1alpha2 + kind: Object + metadata: {} + spec: + forProvider: + manifest: + apiVersion: v1 + kind: ResourceQuota + metadata: + name: default + spec: + hard: {} + providerConfigRef: + name: provider-kubernetes + patches: + #- type: FromCompositeFieldPath + # fromFieldPath: "metadata.name" + # toFieldPath: "spec.forProvider.manifest.metadata.name" + - type: FromCompositeFieldPath + fromFieldPath: "metadata.name" + toFieldPath: "spec.forProvider.manifest.metadata.namespace" + - type: FromCompositeFieldPath + fromFieldPath: "spec.subscription" + toFieldPath: "spec.forProvider.manifest.spec.hard" + transforms: + - type: map + map: + bronze: + limits.cpu: 1500m + limits.memory: 6Gi + pods: "5" + requests.cpu: "1" + requests.memory: 4Gi + silver: + limits.cpu: "3" + limits.memory: 12Gi + pods: "10" + requests.cpu: "2" + requests.memory: 8Gi + gold: + limits.cpu: "6" + limits.memory: 20Gi + pods: "20" + requests.cpu: "4" + requests.memory: 16Gi + - name: storage + base: + apiVersion: kubernetes.crossplane.io/v1alpha2 + kind: Object + metadata: {} + spec: + forProvider: + manifest: + apiVersion: epca.eo/v1beta1 + kind: Storage + metadata: + name: default + spec: + location: EU + acl: private + providerConfigRef: + name: provider-kubernetes + patches: + #- type: FromCompositeFieldPath + # fromFieldPath: "metadata.name" + # toFieldPath: "spec.forProvider.manifest.metadata.name" + - type: FromCompositeFieldPath + fromFieldPath: "metadata.name" + toFieldPath: "spec.forProvider.manifest.metadata.namespace" + # - name: github TODO + # - name: vcluster TODO + # - name: gitops TODO + # - name: validation webhook TODO + + diff --git a/setup/eoepca-demo/workspace/definition.yaml b/setup/eoepca-demo/workspace/definition.yaml new file mode 100644 index 0000000..5f9f4f2 --- /dev/null +++ b/setup/eoepca-demo/workspace/definition.yaml @@ -0,0 +1,28 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: CompositeResourceDefinition +metadata: + name: xworkspaces.epca.eo +spec: + connectionSecretKeys: [] + group: epca.eo + names: + kind: XWorkspace + plural: xworkspaces + claimNames: + kind: Workspace + plural: workspaces + versions: + - name: v1beta1 + served: true + referenceable: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + subscription: + type: string + required: + - subscription \ No newline at end of file