Skip to content

Commit

Permalink
release 0.57.0 (#3732)
Browse files Browse the repository at this point in the history
* chore: create release 0.57.0

* feat!: gateway refactor (#3721)

The previous gateway components are replaced with a single API Gateway responsible for login and reverse proxying requests.

Details:
* Remove the `gateway-auth` Python component
* Update the `gateway-revproxy` Go component (now `gateway`)
* Update the `ui-server` to remove login responsibilities

---------

Co-authored-by: Tasko Olevski <[email protected]>

* feat: upgrade csi-rclone to 0.3.0 (#3753)

Release notes from csi-rclone:
- feat: get secret from pvc annotation by @olevski in #20
- feat: add support for decrypting storage credentials by @m-alisafaee
in #17
- fix: don't error out if unmounting fails, as it might block dependent
resources by @Panaetius in #16

* feat: Update search services to 0.5.0 (#3754)

* feat: update notebooks to 1.26.0 (#3757)

* fix: add environment variable for new redis stream to search (#3756)

* (docs) redirect to Community portal for Renku 2.0 docs (#3758)

* fix: bump csi rclone to 0.3.1 (#3759)

This fixes an error where the new version of the csi rclone was failing
when a secret annotation was not used to define the secret. This was
supposed to work but the storage class that is marked to use the PVC
annotation will fail if the annotation is not present. So we had to use
2 storage classes one that requires a PVC annotation for the secret and
another that is the same as the current one - which expects that the PVC
name and secret is the same.

* chore(ci): Update renku actions to v1.12.3 (#3766)

Co-authored-by: Rok Roškar <[email protected]>

* fix: implement browser logout from the gateway (#3764)

Add browser-based logout for Keycloak and Gitlab.

Previously, the logout flow would not end the Keycloak nor the GitLab browser session. This prevented users from switching accounts because their existing Keycloak and GitLab sessions would be re-used when logging in with the renku gateway.

* feat: update renku-data-services to v0.20.0 and renku-ui to 3.35.0 (#3765)

* data svc release v0.20.0

* bump UI -> 3.35.0

* update changelog

* Update CHANGELOG.rst

Co-authored-by: Laura <[email protected]>

* apply CHANGELOG suggestions

---------

Co-authored-by: Flora Thiebaut <[email protected]>
Co-authored-by: Rok Roškar <[email protected]>
Co-authored-by: Laura <[email protected]>
Co-authored-by: Andrea Cordoba <[email protected]>

* fix: bump gateway to 1.0.2 (#3767)

Fixes the `Authorization` header for the Renku 2.0 search service.

* chore: add general release wording

* chore: polish release 0.57.0 changelog

---------

Co-authored-by: Flora Thiebaut <[email protected]>
Co-authored-by: Tasko Olevski <[email protected]>
Co-authored-by: eikek <[email protected]>
Co-authored-by: M. Alisafaee <[email protected]>
Co-authored-by: Laura <[email protected]>
Co-authored-by: Rok Roškar <[email protected]>
Co-authored-by: Ralf Grubenmann <[email protected]>
Co-authored-by: Andrea Cordoba <[email protected]>
Co-authored-by: Rok Roškar <[email protected]>
  • Loading branch information
10 people authored Sep 16, 2024
1 parent 15a50f9 commit 2929f53
Show file tree
Hide file tree
Showing 29 changed files with 417 additions and 317 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/cron-jobs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: renku teardown
uses: SwissDataScienceCenter/renku-actions/cleanup-renku-ci-deployments@v1.11.3
uses: SwissDataScienceCenter/renku-actions/cleanup-renku-ci-deployments@v1.12.3
env:
GITLAB_TOKEN: ${{ secrets.DEV_GITLAB_TOKEN }}
RENKUBOT_KUBECONFIG: ${{ secrets.RENKUBOT_DEV_KUBECONFIG }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-helm-chart.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- name: Set version
id: vars
run: echo "tag=${GITHUB_REF#refs/*/}" >> $GITHUB_OUTPUT
- uses: SwissDataScienceCenter/renku-actions/publish-chart@v1.11.3
- uses: SwissDataScienceCenter/renku-actions/publish-chart@v1.12.3
env:
CHART_DIR: helm-chart/
CHART_NAME: renku
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-master-merges.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- id: set-version
run: |
echo "publish_version=${{ steps.bump-semver.outputs.new_version }}.$(echo ${{ github.sha }} | cut -c 1-7)" >> $GITHUB_ENV
- uses: SwissDataScienceCenter/renku-actions/publish-chart@v1.11.3
- uses: SwissDataScienceCenter/renku-actions/publish-chart@v1.12.3
env:
CHART_DIR: helm-chart/
CHART_TAG: "--tag ${{env.publish_version}}"
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/pull-request-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:
steps:
- uses: actions/[email protected]
- id: deploy-comment
uses: SwissDataScienceCenter/renku-actions/check-pr-description@v1.11.3
uses: SwissDataScienceCenter/renku-actions/check-pr-description@v1.12.3
with:
string: /deploy
pr_ref: ${{ github.event.number }}
Expand All @@ -79,7 +79,7 @@ jobs:
- uses: actions/[email protected]
- name: renku build and deploy
if: needs.check-deploy.outputs.pr-contains-string == 'true'
uses: SwissDataScienceCenter/renku-actions/deploy-renku@v1.11.3
uses: SwissDataScienceCenter/renku-actions/deploy-renku@v1.12.3
env:
DOCKER_PASSWORD: ${{ secrets.RENKU_DOCKER_PASSWORD }}
DOCKER_USERNAME: ${{ secrets.RENKU_DOCKER_USERNAME }}
Expand Down Expand Up @@ -122,7 +122,7 @@ jobs:
needs: [check-deploy, deploy-pr]
runs-on: ubuntu-22.04
steps:
- uses: SwissDataScienceCenter/renku-actions/test-renku@v1.11.3
- uses: SwissDataScienceCenter/renku-actions/test-renku@v1.12.3
with:
kubeconfig: ${{ secrets.RENKUBOT_DEV_KUBECONFIG }}
renku-release: ci-renku-${{ github.event.number }}
Expand Down Expand Up @@ -150,7 +150,7 @@ jobs:
]

steps:
- uses: SwissDataScienceCenter/renku-actions/test-renku-cypress@v1.11.3
- uses: SwissDataScienceCenter/renku-actions/test-renku-cypress@v1.12.3
if: github.event.action != 'closed' && needs.check-deploy.outputs.pr-contains-string == 'true' && needs.check-deploy.outputs.test-enabled == 'true'
with:
e2e-target: ${{ matrix.tests }}
Expand Down Expand Up @@ -183,7 +183,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: renku teardown
uses: SwissDataScienceCenter/renku-actions/cleanup-renku-ci-deployments@v1.11.3
uses: SwissDataScienceCenter/renku-actions/cleanup-renku-ci-deployments@v1.12.3
env:
HELM_RELEASE_REGEX: "^ci-renku-${{ github.event.number }}$"
GITLAB_TOKEN: ${{ secrets.DEV_GITLAB_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/renku-dev-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
github.event.client_payload.message == 'Helm test succeeded' }}
runs-on: ubuntu-20.04
steps:
- uses: SwissDataScienceCenter/renku-actions/test-renku@v1.11.3
- uses: SwissDataScienceCenter/renku-actions/test-renku@v1.12.3
with:
kubeconfig: ${{ secrets.RENKUBOT_DEV_KUBECONFIG }}
renku-release: renku
Expand Down
82 changes: 81 additions & 1 deletion CHANGELOG.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,85 @@
.. _changelog:

0.57.0
------

Renku `0.57.0` brings a suite of new features and improvements to the Renku 2.0 beta. As a main
highlight, you can now save and reuse the credentials for data sources. No more copy/paste on every
session launch! We have also made small improvements to sharing, search, and sessions in Renku 2.0.
For a full list of changes, see the list below.


NOTE to administrators: Upgrading the `csi-rclone` component will unmount all cloud storage for all
active or hibernated sessions. Therefore, we recommend notifying your users ahead of time when you
deploy this version of Renku and also if possible deploying the upgrade when there are fewer
sessions that use cloud storage or just fewer sessions in general. Once the upgrade is complete
users will be able to mount cloud storage as usual.

User-Facing Changes
~~~~~~~~~~~~~~~~~~~

**🌟 New Features**

- **UI**: Support saving and managing credentials for Renku 2.0 data sources (`#3266 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3266>`__).

**✨ Improvements**

- **Search Services**: Enable searching by prefix of indexed words
- **UI**: Add members to groups and projects in Renku 2.0 by username instead of email (`#3270 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3270>`__).
- **UI**: Enable sharing search URLs by reflecting the search query in the URL for Renku 2.0 (`#3245 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3245>`__).
- **UI**: Show the status of a session via a dynamic browser tab icon (`#3249 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3249>`__).
- **UI**: Display session details in session page in Renku 2.0 (`#3258 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3258>`__)
- **UI**: Set default namespace when creating a new Renku 2.0 project (`#3264 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3264>`__).

**🐞 Bug Fixes**

- **UI**: Fix issue in Renku 2.0 where launched sessions did not use the default storage size of the selected resource class (`#3295 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3295>`__).
- **UI**: Fix misnomers on the group creation page (`#3276 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3276>`__).
- **Data Services**: Fix connected services showing errors for anonymous users
- **Data Services**: Fix 500 error being raised when modifying a session launcher

Internal Changes
~~~~~~~~~~~~~~~~

**New Features**

- **csi-rclone**: Read credential secrets from PVC annotations
- **csi-rclone**: Update the CSI sidecar container versions
- **csi-rclone**: Add support for decrypting data storage secrets.
- **Gateway**: The API Gateway components have been refactored and simplified (`#709 <https://github.com/SwissDataScienceCenter/renku-gateway/pull/709>`__).
- **Notebooks**: Add a component for liveness detection
- **Notebooks**: Support for saving cloud storage secrets

**Improvements**

- **Search Services**: Reading all data service events from a single Redis stream. Processing from individual streams is kept.
- **Data Services**: Do not show user emails and use usernames instead for all interactions
- **UI**: The UI server has been refactored following the changes in the gateway (`#3271 <https://github.com/SwissDataScienceCenter/renku-ui/pull/3271>`__).

**Bug Fixes**

- **csi-rclone**: Do not crash on unmounting as it might block dependent resources
- **csi-rclone**: Use extra storage class when reading secrets from a PVC annotation
- **Data Services**: Fix group member changes not being sent to search
- **Data Services**: Fix Redis not being able to connect to the master node

Individual Components
~~~~~~~~~~~~~~~~~~~~~

- `csi-rclone 0.1.8 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.1.8>`__
- `csi-rclone 0.2.0 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.2.0>`__
- `csi-rclone 0.3.0 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.3.0>`__
- `csi-rclone 0.3.1 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.3.1>`__
- `renku-gateway 1.0.0 <https://github.com/SwissDataScienceCenter/renku-gateway/releases/tag/1.0.0>`_
- `renku-gateway 1.0.1 <https://github.com/SwissDataScienceCenter/renku-gateway/releases/tag/1.0.1>`_
- `renku-gateway 1.0.2 <https://github.com/SwissDataScienceCenter/renku-gateway/releases/tag/1.0.2>`_
- `renku-ui 3.34.0 <https://github.com/SwissDataScienceCenter/renku-ui/releases/tag/3.34.0>`_
- `renku-ui 3.35.0 <https://github.com/SwissDataScienceCenter/renku-ui/releases/tag/3.35.0>`_
- `renku-search 0.5.0 <https://github.com/SwissDataScienceCenter/renku-search/releases/tag/v0.5.0>`_
- `renku-notebooks 1.26.0 <https://github.com/SwissDataScienceCenter/renku-notebooks/releases/tag/1.26.0>`__
- `renku-data-services 0.20.0 <https://github.com/SwissDataScienceCenter/renku-data-services/releases/tag/v0.20.0>`__


0.56.3
------

Expand Down Expand Up @@ -48,7 +128,7 @@ Individual Components
0.56.1
------

Renku ``0.56.1`` fixes a bug where Amalthea would not start when the prometheus metrics or the
Renku ``0.56.1`` fixes a bug where Amalthea would not start when the prometheus metrics or the
audit log export functionality is enabled.

Internal Changes
Expand Down
2 changes: 2 additions & 0 deletions cypress-tests/cypress/e2e/useSession.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,8 @@ describe("Basic public project functionality", () => {

// Stop the session -- mind that anonymous users cannot pause sessions
cy.deleteSession({ fromSessionPage: true });

cy.robustLogin();
});

it("Start a new session on a project without permissions.", () => {
Expand Down
2 changes: 2 additions & 0 deletions cypress-tests/cypress/support/commands/login.ts
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,8 @@ function robustLogin(props?: RobustLoginProps) {
function logout() {
cy.get("#profile-dropdown").should("be.visible").click();
cy.get("#logout-link").should("be.visible").click();
// Make sure we fully log out
cy.wait(1_000);
}

export default function registerLoginCommands() {
Expand Down
7 changes: 7 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ Renku Documentation

-- Wikipedia

.. note::

**We're building the next version of Renku!** For documentation related to Renku 2.0, please see
our `Community Portal
<https://renku.notion.site/Renku-Community-Portal-2a154d7d30b24ab8a5968c60c2592d87>`_. To learn
more about the big changes coming in Renku, check out our `blog post <https://blog.renkulab.io/renku-2/>`_.

.. include:: ../README.rst
:start-after: renku:
:end-before: documentation:
Expand Down
2 changes: 2 additions & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ kwargs
ld
lefthand
lfs
liveness
LocalClient
localhost
Lucene
Expand Down Expand Up @@ -308,6 +309,7 @@ unmapped
unmerged
Unmount
unmount
unmounting
unpause
unpushed
unschedulable
Expand Down
9 changes: 9 additions & 0 deletions docs/tutorials.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@
Tutorials
---------

.. note::

**We're building the next version of Renku!** If you're looking for a tutorial for Renku 2.0,
please see the Renku 2.0 documentation on our `Community Portal
<https://renku.notion.site/Renku-Community-Portal-2a154d7d30b24ab8a5968c60c2592d87>`_ instead.
The tutorial linked below is outdated and refers to the legacy version of Renku that is no
longer under active development. To learn more about the big changes coming in Renku, check out
our `blog post <https://blog.renkulab.io/renku-2/>`_.

The following tutorials are available for getting acquainted with Renku. We recommend
you start with :ref:`first_steps`!

Expand Down
9 changes: 9 additions & 0 deletions docs/tutorials/01_firststeps.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@
Get Started on RenkuLab
=======================

.. note::

**We're building the next version of Renku!** If you're looking for a tutorial for Renku 2.0,
please see the Renku 2.0 documentation on our `Community Portal
<https://renku.notion.site/Renku-Community-Portal-2a154d7d30b24ab8a5968c60c2592d87>`_ instead.
This tutorial is outdated and refers to the legacy version of Renku that is no longer under
active development. To learn more about the big changes coming in Renku, check out our `blog
post <https://blog.renkulab.io/renku-2/>`_.

This tutorial will help you get started working on the Renkulab platform. We will
use Renku to realize a very small data science project: counting the
number of flights to Austin-Bergstrom International Airport in January, 2019. In this tutorial we will provide instructions for Python, Julia (in JupyterLab) and R (in RStudio).
Expand Down
2 changes: 1 addition & 1 deletion helm-chart/renku/requirements.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ dependencies:
condition: notebooks.cloudstorage.s3.installDatashim
- name: csi-rclone
repository: "https://swissdatasciencecenter.github.io/helm-charts/"
version: "0.1.7"
version: "0.3.1"
condition: global.csi-rclone.install
- name: solr
repository: "oci://registry-1.docker.io/bitnamicharts"
Expand Down
8 changes: 4 additions & 4 deletions helm-chart/renku/templates/gateway/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
{{/*
Template core service paths as a comma separated list
*/}}
{{- define "gateway.core.paths" -}}
{{- define "gateway.core.pathsYaml" -}}
{{- $paths := list -}}
{{- range $i, $k := (keys .Values.global.core.versions | sortAlpha) -}}
{{- $paths = mustAppend $paths (printf "/api/renku/%s" (get $.Values.global.core.versions $k).prefix) -}}
{{- if eq $k "latest" -}}
{{- $paths = mustAppend $paths "/api/renku" -}}
{{- end -}}
{{- end -}}
{{- join "," $paths | quote -}}
{{- $paths | toYaml -}}
{{- end -}}

{{/*
Template core service names as a comma separated list
*/}}
{{- define "gateway.core.serviceNames" -}}
{{- define "gateway.core.serviceNamesYaml" -}}
{{- $serviceNames := list -}}
{{- $coreBaseName := printf "%s-core" .Release.Name -}}
{{- range $i, $k := (keys .Values.global.core.versions | sortAlpha) -}}
Expand All @@ -25,7 +25,7 @@ Template core service names as a comma separated list
{{- $serviceNames = mustAppend $serviceNames $serviceName -}}
{{- end -}}
{{- end -}}
{{- join "," $serviceNames | quote -}}
{{- $serviceNames | toYaml -}}
{{- end -}}

{{/*
Expand Down
89 changes: 89 additions & 0 deletions helm-chart/renku/templates/gateway/configmap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "renku.fullname" . }}-gateway
labels:
app: {{ template "gateway.name" . }}
chart: {{ template "renku.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
config.yaml: |
server:
port: 8080
host: 0.0.0.0
rateLimits:
enabled: {{ .Values.gateway.rateLimits.general.enabled }}
rate: {{ .Values.gateway.rateLimits.general.average }}
burst: {{ .Values.gateway.rateLimits.general.burst }}
{{- with .Values.gateway.allowOrigin }}
allowOrigin:
{{- toYaml . | nindent 8 }}
{{- end }}
sessions:
authorizationVerifiers:
- issuer: {{ printf "%s/realms/%s" (include "renku.keycloakUrl" . | trimSuffix "/") (include "renku.keycloak.realm" .) }}
audience: renku
authorizedParty: renku
- issuer: {{ printf "%s/realms/%s" (include "renku.keycloakUrl" . | trimSuffix "/") (include "renku.keycloak.realm" .) }}
audience: renku
authorizedParty: renku-cli
revproxy:
renkuBaseUrl: {{ include "renku.baseUrl" . | quote }}
{{- if .Values.gitlab.enabled }}
externalGitlabUrl: ""
{{- else }}
externalGitlabUrl: {{ .Values.global.gitlab.url | default "" | quote }}
{{- end }}
k8sNamespace: {{ .Release.Namespace }}
renkuServices:
notebooks: {{ printf "http://%s-notebooks" .Release.Name | quote }}
kg: {{ printf "http://%s-knowledge-graph" .Release.Name | quote }}
webhook: {{ printf "http://%s-webhook-service" .Release.Name | quote }}
core:
serviceNames:
{{- include "gateway.core.serviceNamesYaml" . | nindent 12 }}
servicePaths:
{{- include "gateway.core.pathsYaml" . | nindent 12 }}
sticky: true
dataService: {{ printf "http://%s-data-service" .Release.Name | quote }}
keycloak: {{ include "renku.keycloakUrl" . | quote }}
uiserver: {{ printf "http://%s" (include "ui-server.fullname" .) | quote }}
search: {{ printf "http://%s-search-api" .Release.Name | quote }}
login:
renkuBaseUrl: {{ include "renku.baseUrl" . | quote }}
loginRoutesBasePath: "/api/auth"
defaultAppRedirectURL: {{ include "renku.baseUrl" . | quote }}
tokenEncryption:
enabled: true
providers:
renku:
issuer: {{ printf "%s/realms/%s" (include "renku.keycloakUrl" . | trimSuffix "/") (include "renku.keycloak.realm" .) }}
clientID: renku
scopes: ["profile", "email", "openid", "microprofile-jwt"]
callbackURI: {{ printf "%s/api/auth/callback" (include "renku.baseUrl" .) }}
usePKCE: false
gitlab:
issuer: {{ .Values.global.gitlab.url | quote }}
clientID: {{ .Values.gateway.gitlabClientId | default .Values.global.gateway.gitlabClientId | quote }}
scopes: ["openid", "api", "read_user", "read_repository"]
callbackURI: {{ printf "%s/api/auth/callback" (include "renku.baseUrl" .) }}
usePKCE: false
oldGitLabLogout: {{ .Values.gateway.oldGitLabLogout | default false }}
logoutGitLabUponRenkuLogout: {{ .Values.gateway.logoutGitLabUponRenkuLogout | default true }}
redis:
type: redis
addresses:
- {{ printf "%s:%d" .Values.global.redis.host (.Values.global.redis.port | int) | quote }}
isSentinel: {{ .Values.global.redis.sentinel.enabled }}
masterName: {{ .Values.global.redis.sentinel.masterSet | quote }}
dbIndex: {{ .Values.global.redis.dbIndex.gateway }}
monitoring:
sentry:
enabled: {{ .Values.gateway.sentry.enabled }}
environment: {{ .Values.gateway.sentry.environment }}
sampleRate: {{ .Values.gateway.sentry.sampleRate }}
prometheus:
enabled: {{ .Values.gateway.metrics.enabled }}
port: {{ .Values.gateway.metrics.port }}
---
Loading

0 comments on commit 2929f53

Please sign in to comment.