release 0.61.0 (#3834)

* chore: create release 0.61.0 * feat: jupyter free sessions in Renku v2 (#3712) * feat: consolidate network policies (#3839) * feat: update data service to 0.27.0 and amalthea to 0.14.4 (#3842) * feat: update data service to 0.27.0 * feat: update amalthea to 0.14.4 --------- Co-authored-by: Tasko Olevski <[email protected]> Co-authored-by: Andrea Cordoba <[email protected]> Co-authored-by: Alessandro Degano <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]> Co-authored-by: Rok Roškar <[email protected]> Co-authored-by: Laura <[email protected]> Co-authored-by: eikek <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]>
SwissDataScienceCenter · Nov 26, 2024 · 866dcef · 866dcef
1 parent 075788e
commit 866dcef
Show file tree

Hide file tree

Showing 22 changed files with 1,378 additions and 315 deletions.
diff --git a/.github/workflows/pull-request-test.yml b/.github/workflows/pull-request-test.yml
@@ -58,6 +58,7 @@ jobs:
       renku-ui: ${{ steps.deploy-comment.outputs.renku-ui}}
       renku-data-services: ${{ steps.deploy-comment.outputs.renku-data-services}}
       amalthea: ${{ steps.deploy-comment.outputs.amalthea}}
+      amalthea-sessions: ${{ steps.deploy-comment.outputs.amalthea-sessions}}
       test-enabled: ${{ steps.deploy-comment.outputs.test-enabled}}
       extra-values: ${{ steps.deploy-comment.outputs.extra-values}}
     steps:
@@ -99,6 +100,7 @@ jobs:
           renku_ui: "${{ needs.check-deploy.outputs.renku-ui }}"
           renku_data_services: "${{ needs.check-deploy.outputs.renku-data-services }}"
           amalthea: "${{ needs.check-deploy.outputs.amalthea }}"
+          amalthea_sessions: "${{ needs.check-deploy.outputs.amalthea-sessions }}"
           extra_values: "${{ needs.check-deploy.outputs.extra-values }}"
       - name: Check existing renkubot comment
         if: needs.check-deploy.outputs.pr-contains-string == 'true'

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -1,5 +1,80 @@
 .. _changelog:
 
+0.61.0
+------
+
+Renku 0.61.0 introduces a new version of Amalthea that supports running sessions with Docker images
+that do not contain Jupyter server.
+
+NOTES to administrators: 
+
+- This upgrade introduces a brand new CRD for sessions. All services that support
+  sessions for Renku v2 will switch to this new CRD. Renku v1 sessions remain unchanged.
+  Therefore any old sessions for Renku v2 will not be visible to users after this upgrade. The sessions
+  themselves will not be immediately deleted and as long as users have saved links to their old sessions they
+  should be able to access their sessions and save data. However we recommend that administrators
+  notify users of the change and allow for enough time so that existing Renku v2 sessions can be saved and
+  cleaned up, rather than asking users to save the url to their sessions. In addition to users not being able
+  to see old Renku v2 sessions, they will also not be able to pause, resume or delete old Renku v2 sessions.
+  Therefore it's best if most sessions are properly saved and cleaned up before this update is rolled out. In order 
+  to support the new CRD we have also created a new operator that will manage the new `amaltheasession` resources.
+
+- The network policies for Renku have been consolidated and revamped. The most notable change here is the 
+  removal of the egress policy that prevented egress to internal IP addresses from sessions. Now we disallow
+  all ingress in the Renku release namespace by default and explicitly grant permissions to any pods that need
+  to access other pods inside the Renku release namespace. Two properties relevant to this have been added to the
+  Helm chart values file that allows administrators to grant access to all Renku services from a specific namespace
+  or to do the same for specific pods within the Renku namespace. These are not needed for Renku to function and the 
+  default network policies should be sufficient, they have been added so that administrators can allow ingress for
+  other services that may not come with the Renku Helm chart such as logging or monitoring. This change will result in
+  the removal of some network policies and the creation of several new policies.
+
+User-Facing Changes
+~~~~~~~~~~~~~~~~~~~
+
+**✨ Improvements**
+
+- **UI**: Enable the use of custom images that don’t contain Jupyter, streamlining the image-building process and allowing for the use of “off-the-shelf” images (`#3341 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3341>`__).
+- **Sessions**: Enable running session images that do not contain Jupyter in them.
+
+Internal Changes
+~~~~~~~~~~~~~~~~
+
+**Improvements**
+
+- **Data services**: Add support for OAuth storage providers
+- **Data services**: Move notebooks code to data services
+- **Helm chart**: Consolidate and revamp network policies
+- **Data services**: Add support for project documentation
+- **Data services**: Add support for cloning projects
+
+**Bug Fixes**
+
+- **Gateway**: Pass on session cookie to data services for anonymous session authentication
+- **Data services**: Correct pagination for namespaces
+- **Data services**: Add creation date and created_by for namespaces
+- **Data services**: Pin RClone version in data services image
+- **Data services**: Properly handle multi-architecture docker images when getting working directory
+- **Data services**: Make environment working directory and mount directory optional
+- **Amalthea**: Add readiness and health checks to sessions.
+- **Amalthea**: Do not authenticate the authentication proxy health check
+- **Amalthea**: Do not mount the Kubernetes service account in sessions
+- **Amalthea**: Do not add Kubernetes specific environment variables in sessions
+
+Individual Components
+~~~~~~~~~~~~~~~~~~~~~
+
+- `renku-gateway 1.3.1 <https://github.com/SwissDataScienceCenter/renku-gateway/releases/tag/1.3.1>`_
+- `renku-ui 3.42.0 <https://github.com/SwissDataScienceCenter/renku-ui/releases/tag/3.42.0>`_
+- `renku-data-services 0.26.0 <https://github.com/SwissDataScienceCenter/renku-data-services/releases/tag/v0.26.0>`_
+- `renku-data-services 0.27.0 <https://github.com/SwissDataScienceCenter/renku-data-services/releases/tag/v0.27.0>`_
+- `amalthea 0.13.0 <https://github.com/SwissDataScienceCenter/amalthea/releases/tag/0.13.0>`_
+- `amalthea 0.14.0 <https://github.com/SwissDataScienceCenter/amalthea/releases/tag/0.14.0>`_
+- `amalthea 0.14.1 <https://github.com/SwissDataScienceCenter/amalthea/releases/tag/0.14.1>`_
+- `amalthea 0.14.2 <https://github.com/SwissDataScienceCenter/amalthea/releases/tag/0.14.2>`_
+- `amalthea 0.14.3 <https://github.com/SwissDataScienceCenter/amalthea/releases/tag/0.14.3>`_
+- `amalthea 0.14.4 <https://github.com/SwissDataScienceCenter/amalthea/releases/tag/0.14.4>`_
+
 0.60.0
 ------
 
@@ -74,11 +149,16 @@ Internal Changes
 **Bug Fixes**
 
 - **Data services**: Handle spaces in ``provider_id`` for connected services (`#482 <https://github.com/SwissDataScienceCenter/renku-data-services/pull/482>`__).
+- **csi-rclone**: Do not log potentially sensitive data in error messages.
+- **csi-rclone**: Properly handle encrypted secrets with the new annotation-based storage class.
+
 
 Individual Components
 ~~~~~~~~~~~~~~~~~~~~~
 
 - `renku-data-services 0.24.2 <https://github.com/SwissDataScienceCenter/renku-data-services/releases/tag/v0.24.2>`__
+- `csi-rclone 0.3.4 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.3.4>`__
+- `csi-rclone 0.3.5 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.3.5>`__
 
 0.59.1
 ------

diff --git a/cypress-tests/cypress/e2e/useSession.cy.ts b/cypress-tests/cypress/e2e/useSession.cy.ts
@@ -189,6 +189,8 @@ describe("Basic public project functionality", () => {
 
     // Stop the session -- mind that anonymous users cannot pause sessions
     cy.deleteSession({ fromSessionPage: true });
+
+    cy.robustLogin();
   });
 
   it("Start a new session on a project without permissions.", () => {

diff --git a/cypress-tests/cypress/support/utils/projectsV2.utils.ts b/cypress-tests/cypress/support/utils/projectsV2.utils.ts
@@ -23,7 +23,7 @@ export function getUserNamespaceAPIV2(): Cypress.Chainable<string | null> {
 
 /** Get a project by using only the API. */
 export function getProjectByNamespaceAPIV2(newProjectProps: ProjectIdentifierV2): Cypress.Chainable<any | null> {
-  return cy.request({ failOnStatusCode: false, method: "GET", url: `api/data/projects/${newProjectProps.namespace}/${newProjectProps.slug}` });
+  return cy.request({ failOnStatusCode: false, method: "GET", url: `api/data/namespaces/${newProjectProps.namespace}/projects/${newProjectProps.slug}` });
 }
 
 /** Create a project (if the project is missing) by using only the API. */

diff --git a/helm-chart/renku/requirements.yaml b/helm-chart/renku/requirements.yaml
@@ -23,7 +23,10 @@ dependencies:
     alias: jena
   - name: amalthea
     repository: "https://swissdatasciencecenter.github.io/helm-charts/"
-    version: "0.12.3"
+    version: "0.14.3"
+  - name: amalthea-sessions
+    repository: "https://swissdatasciencecenter.github.io/helm-charts/"
+    version: "0.14.4"
   - name: dlf-chart
     repository: "https://swissdatasciencecenter.github.io/datashim/"
     version: "0.3.9-renku-2"

diff --git a/helm-chart/renku/templates/_helpers.tpl b/helm-chart/renku/templates/_helpers.tpl
@@ -149,6 +149,10 @@ KC_DB_PASSWORD: {{ default (randAlphaNum 64) .Values.global.keycloak.postgresPas
 {{- end -}}
 {{- end -}}
 
+{{- define "renku.keycloakIssuerUrl" -}}
+{{- printf "%s/realms/%s" (include "renku.keycloakUrl" . | trimSuffix "/") (include "renku.keycloak.realm" .) -}}
+{{- end -}}
+
 {{/*
 Common labels
 */}}

diff --git a/helm-chart/renku/templates/core/cache-cleanup-job.yaml b/helm-chart/renku/templates/core/cache-cleanup-job.yaml
@@ -13,6 +13,9 @@ spec:
   jobTemplate:
     spec:
       template:
+        metadata:
+          labels:
+            app: core-cache-cleanup
         spec:
           containers:
           - name: {{ include "renku-core.fullname" $ }}-cache-cleanup-{{ $version.name }}

diff --git a/helm-chart/renku/templates/data-service/deployment.yaml b/helm-chart/renku/templates/data-service/deployment.yaml
@@ -43,6 +43,9 @@ spec:
             - name: http
               containerPort: 8000
               protocol: TCP
+          envFrom:
+            - secretRef:
+                name: {{ template "renku.notebooks.fullname" . }}
           env:
             - name: VERSION
               value: {{ .Values.dataService.image.tag | quote }}
@@ -70,10 +73,6 @@ spec:
               value: /secrets/encryptionKey/encryptionKey
             - name: SECRETS_SERVICE_PUBLIC_KEY_PATH
               value: /secrets/publicKey/publicKey
-            - name: SERVER_DEFAULTS
-              value: /etc/renku-data-service/server_options/server_defaults.json
-            - name: SERVER_OPTIONS
-              value: /etc/renku-data-service/server_options/server_options.json
             - name: K8S_NAMESPACE
               value: {{ .Release.Namespace | quote }}
             - name: GITLAB_URL
@@ -116,11 +115,21 @@ spec:
               value: {{ .Values.dataService.trustedProxies.proxiesCount | default "" | quote }}
             - name: REAL_IP_HEADER
               value: {{ .Values.dataService.trustedProxies.realIpHeader | default "" | quote }}
+            - name: KUBERNETES_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.namespace
+            - name: SESSIONS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.namespace
             {{- include "certificates.env.python" . | nindent 12 }}
             {{- include "certificates.env.grpc" . | nindent 12 }}
           volumeMounts:
             - name: server-options
-              mountPath: /etc/renku-data-service/server_options
+              mountPath: /etc/renku-notebooks/server_options
             - mountPath: "/secrets/encryptionKey"
               name: encryption-key
               readOnly: true

diff --git a/helm-chart/renku/templates/data-service/rbac.yaml b/helm-chart/renku/templates/data-service/rbac.yaml
@@ -19,6 +19,67 @@ rules:
       - patch
       - delete
       - create
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/log
+      - services
+      - endpoints
+      - secrets
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - secrets
+    verbs:
+      - delete
+  - apiGroups:
+      - apps
+    resources:
+      - statefulsets
+    verbs:
+      - get
+      - list
+      - watch
+      - patch
+  - apiGroups:
+      - ""
+    resources:
+      - secrets
+    verbs:
+      - create
+      - update
+      - delete
+      - patch
+  - apiGroups:
+      - {{ .Values.amalthea.crdApiGroup }}
+    resources:
+      - {{ .Values.amalthea.crdNames.plural }}
+    verbs:
+      - create
+      - update
+      - delete
+      - patch
+      - list
+      - get
+      - watch
+  - apiGroups:
+      - amalthea.dev
+    resources:
+      - amaltheasessions
+    verbs:
+      - create
+      - update
+      - delete
+      - patch
+      - list
+      - get
+      - watch
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole