diff --git a/airflow/plural/docs/aws-secrets-backend.md b/airflow/plural/docs/aws-secrets-backend.md new file mode 100644 index 000000000..7e1650081 --- /dev/null +++ b/airflow/plural/docs/aws-secrets-backend.md @@ -0,0 +1,61 @@ +## Connecting to AWS Secrets Backend + +Airflow allows you the opportunity to connect to various services as a Secrets Backend as an alternative to using the +Airflow UI to manage connections. One of these services is [AWS Secrets Manager](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html). +Once you add below configurations, Airflow will be able to retrieve Secrets from AWS Secrets Manager (provided that they +have the same prefixes specified in the `KWARGS` config). + +In this scenario, the prefixes are `airflow/connections` & `airflow/variables`, so any values stored under the +`airflow/connections` prefix would be treated the same as an object stored in the `Admin >> Connections` menu of the +Airflow UI. Any values stored under the `airflow/variables` prefix would be treated the same as an object stored in the +`Admin >> Variables` menu of the Airflow UI. + +### edit values.yaml + +You'll then want to edit `airflow/helm/airflow/values.yaml` in your installation repo with something like: + +```yaml +airflow: + airflow: + airflow: + config: + AIRFLOW__SECRETS__BACKEND: airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend + AIRFLOW__SECRETS__BACKEND_KWARGS: '{"connections_prefix": "airflow/connections","variables_prefix": + "airflow/variables"}' +``` + +Alternatively, you should be able to do this in the configuration section for airflow in your plural console as well. + +### add policy to AWS role + +When installing the Airflow Application, Plural added a default role for Airflow. The role will be called +`-airflow`. You will need to add a policy to that role to allow it to access AWS Secrets Manager. You +can use this policy as a starting point: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "secretsmanager:GetRandomPassword", + "secretsmanager:ListSecrets" + ], + "Resource": "*" + }, + { + "Sid": "VisualEditor1", + "Effect": "Allow", + "Action": "secretsmanager:*", + "Resource": "arn:aws:secretsmanager:::secret:airflow/*" + } + ] +} +``` + +### redeploy + +From there, you should be able to run `plural build --only airflow && plural deploy --commit "use aws secrets manager +backend"` to use the secrets backend \ No newline at end of file diff --git a/airflow/plural/docs/running-dbt-via-cosmos.md b/airflow/plural/docs/running-dbt-via-cosmos.md new file mode 100644 index 000000000..0c4ff2387 --- /dev/null +++ b/airflow/plural/docs/running-dbt-via-cosmos.md @@ -0,0 +1,146 @@ +## Running dbt core in Airflow via Cosmos + +[Cosmos](https://github.com/astronomer/astronomer-cosmos) is an open source project that allows you to run dbt core +projects in Airflow natively. To date, it is probably one of the best ways to run dbt core in Airflow. + +### custom dockerfile + +In order to run dbt core effectively, we recommend you bake a new docker image against ours and then wiring it into your +installation. Please follow the [pip-packages](./pip-packages.md) guide for instructions on baking your own image. + +Airflow and dbt both share common dependencies (i.e. Jinja). This can cause dependency clashes between Airflow and your +dbt adapter when you upgrade them. To solve for this, we can put our dbt adapter in its own python virtual environment. +This is possible by adding the following step to your custom `Dockerfile`: + +```dockerfile +FROM docker.io/apache/airflow:2.6.3-python3.10 + +USER root +RUN apt-get -yq update \ + && apt-get -yq install --no-install-recommends \ + git \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +USER airflow + +COPY requirements.txt requirements.txt +RUN pip freeze | grep -i apache-airflow > protected-packages.txt \ + && pip install --constraint ./protected-packages.txt --no-cache-dir -r ./requirements.txt \ + && rm -rf ./protected-packages.txt ./requirements.txt + +## create virtual environments for dbt +RUN export PIP_USER=false && python -m venv dbt_venv && source dbt_venv/bin/activate && \ + pip install --no-cache-dir dbt-redshift==1.6.1 && deactivate && export PIP_USER=true + +``` + +In this example, we've installed the `dbt-redshift` adapter into the python virtual environment. However, you can swap +the adapter for the one that suites your needs (i.e. `dbt-bigquery`, `dbt-snowflake`, etc.) + +### add dbt project to your dags directory + +In your dags directory, add a folder called `dbt`. Within that folder, copy your dbt project. For example, if you were +going to add Fishtown Analytic's classic [Jaffle Shop project](https://github.com/dbt-labs/jaffle_shop), your project +directory would look something like this: + +```yaml +plural-airflow-repo +└── dags + └── dbt + └── jaffle_shop + ├── LICENSE + ├── README.md + ├── dbt_project.yml + ├── etc + │   ├── dbdiagram_definition.txt + │   └── jaffle_shop_erd.png + ├── models + │   ├── customers.sql + │   ├── docs.md + │   ├── orders.sql + │   ├── overview.md + │   ├── schema.yml + │   └── staging + │   ├── schema.yml + │   ├── stg_customers.sql + │   ├── stg_orders.sql + │   └── stg_payments.sql + └── seeds + ├── raw_customers.csv + ├── raw_orders.csv + └── raw_payments.csv +``` + +### point Cosmos class to the nested dbt project directory + +In your dags directory, add a `jaffle_shop.py` file to create a DAG, and add the following contents to it: + +```python +""" +## Jaffle Shop + +Example of using cosmos to run the jaffle shop dbt project +""" +import os +from datetime import datetime + +from airflow import DAG +from cosmos import DbtTaskGroup, ExecutionConfig, ProfileConfig, ProjectConfig +from cosmos.profiles.redshift.user_pass import RedshiftUserPasswordProfileMapping + +'''these next lines help to resolve the path to your dbt project in the plural airflow instance vs. local development''' + +# Dynamically retrieves the Airflow Home directory +airflow_home = os.getenv("AIRFLOW_HOME", "/usr/local/airflow") + +# I've set a local env variable ENVIRONMENT=DEV to determine if dag is running in plural airflow or local airflow +if os.getenv("ENVIRONMENT", "PROD") == "DEV": + # the project path when running Airflow locally + dbt_project_path = f"{airflow_home}/dags/dbt/jaffle_shop" +else: + # the project path in plural cluster + dbt_project_path = f"{airflow_home}/dags/repo/dags/dbt/jaffle_shop" + +# the path to the dbt executable that's within the venv created in Dockerfile +dbt_executable_path = f"{airflow_home}/dbt_venv/bin/dbt" + +# Profile mapping to connect dbt to a target +profile_mapping = RedshiftUserPasswordProfileMapping( + # airflow connection id to use for the dbt target + conn_id="redshift_default", + profile_args={ + # my redshift database name + "dbname": "dev", + # default schema to write to if one isn't specified in .yml or .sql dbt files + "schema": "a_default_schema_name" + } +) + +with DAG( + dag_id="jaffle_shop", + start_date=datetime(2023, 10, 6), + schedule=None, + doc_md=__doc__, + tags=["dbt", "redshift"], +): + DbtTaskGroup( + project_config=ProjectConfig( + dbt_project_path=dbt_project_path + ), + execution_config=ExecutionConfig( + dbt_executable_path=dbt_executable_path + ), + profile_config=ProfileConfig( + profile_name="jaffle_shop", # the default profile - recommended to be your dbt project name + target_name="cosmos_target", # the default target - recommended to just leave as cosmos_target + profile_mapping=profile_mapping, + ) + ) +``` + +This example uses a Redshift Data Warehouse as a target, but you can also configure profiles for other targets (i.e. +Snowflake, BigQuery, etc.). For more information, please review Cosmos Docs [here](https://astronomer.github.io/astronomer-cosmos/profiles/index.html). +After making these changes, you should see the DAG parse like so: + +![jaffle_shop_dag.png](https://github.com/astronomer/astronomer-cosmos/blob/main/docs/_static/jaffle_shop_task_group.png) diff --git a/bootstrap/helm/bootstrap/Chart.lock b/bootstrap/helm/bootstrap/Chart.lock index 50b6b0752..237323fb8 100644 --- a/bootstrap/helm/bootstrap/Chart.lock +++ b/bootstrap/helm/bootstrap/Chart.lock @@ -13,7 +13,7 @@ dependencies: version: 9.25.0 - name: aws-load-balancer-controller repository: https://aws.github.io/eks-charts - version: 1.4.8 + version: 1.6.1 - name: aws-ebs-csi-driver repository: https://kubernetes-sigs.github.io/aws-ebs-csi-driver version: 2.17.1 @@ -26,5 +26,5 @@ dependencies: - name: tigera-operator repository: https://docs.tigera.io/calico/charts version: v3.25.0 -digest: sha256:1d124ca9acb4e93009dfeb4273d149d075616babbad1fe3e5bb6c88540b5f96d -generated: "2023-03-07T15:21:37.729265+01:00" +digest: sha256:b7ee91be180afabfb812e9c8e7f7bfdfd2a1c4ebef9592ccd37e6eadd65409a2 +generated: "2023-10-05T13:27:49.248688-04:00" diff --git a/bootstrap/helm/bootstrap/Chart.yaml b/bootstrap/helm/bootstrap/Chart.yaml index 6cd12a2b7..789f04094 100644 --- a/bootstrap/helm/bootstrap/Chart.yaml +++ b/bootstrap/helm/bootstrap/Chart.yaml @@ -10,7 +10,7 @@ maintainers: email: mguarino46@gmail.com - name: David van der Spek email: david@plural.sh -version: 0.8.75 +version: 0.8.77 dependencies: - name: external-dns version: 6.14.1 @@ -30,7 +30,7 @@ dependencies: repository: https://kubernetes.github.io/autoscaler - name: aws-load-balancer-controller condition: aws-load-balancer-controller.enabled - version: 1.4.8 + version: 1.6.1 repository: https://aws.github.io/eks-charts - name: aws-ebs-csi-driver condition: aws-ebs-csi-driver.enabled diff --git a/bootstrap/helm/bootstrap/charts/aws-load-balancer-controller-1.4.8.tgz b/bootstrap/helm/bootstrap/charts/aws-load-balancer-controller-1.4.8.tgz deleted file mode 100644 index 321f4ef91..000000000 Binary files a/bootstrap/helm/bootstrap/charts/aws-load-balancer-controller-1.4.8.tgz and /dev/null differ diff --git a/bootstrap/helm/bootstrap/charts/aws-load-balancer-controller-1.6.1.tgz b/bootstrap/helm/bootstrap/charts/aws-load-balancer-controller-1.6.1.tgz new file mode 100644 index 000000000..f9497cb59 Binary files /dev/null and b/bootstrap/helm/bootstrap/charts/aws-load-balancer-controller-1.6.1.tgz differ diff --git a/bootstrap/helm/bootstrap/values.yaml b/bootstrap/helm/bootstrap/values.yaml index c44bdc14b..d0969e460 100644 --- a/bootstrap/helm/bootstrap/values.yaml +++ b/bootstrap/helm/bootstrap/values.yaml @@ -64,7 +64,7 @@ aws-load-balancer-controller: enabled: false image: repository: public.ecr.aws/eks/aws-load-balancer-controller # TODO: this should be migrated to our vendored images - tag: v2.4.7 + tag: v2.6.1 snapshot-validation-webhook: enabled: false diff --git a/bootstrap/terraform/aws-bootstrap/deps.yaml b/bootstrap/terraform/aws-bootstrap/deps.yaml index a7fc845cc..8b042feac 100644 --- a/bootstrap/terraform/aws-bootstrap/deps.yaml +++ b/bootstrap/terraform/aws-bootstrap/deps.yaml @@ -2,7 +2,7 @@ apiVersion: plural.sh/v1alpha1 kind: Dependencies metadata: description: Creates an EKS cluster and prepares it for bootstrapping - version: 0.1.54 + version: 0.1.55 spec: breaking: false dependencies: [] diff --git a/bootstrap/terraform/aws-bootstrap/main.tf b/bootstrap/terraform/aws-bootstrap/main.tf index 6bf1c9799..f1eee509d 100644 --- a/bootstrap/terraform/aws-bootstrap/main.tf +++ b/bootstrap/terraform/aws-bootstrap/main.tf @@ -37,19 +37,23 @@ module "vpc" { } module "cluster" { - source = "github.com/pluralsh/terraform-aws-eks?ref=output-service-cidr" - cluster_name = var.cluster_name - cluster_version = var.kubernetes_version - private_subnets = local.private_subnet_ids - public_subnets = local.public_subnet_ids - worker_private_subnets = local.worker_private_subnet_ids - vpc_id = local.vpc_id - enable_irsa = true - write_kubeconfig = false - create_eks = var.create_cluster - cluster_enabled_log_types = var.cluster_enabled_log_types - cluster_log_retention_in_days = var.cluster_log_retention_in_days - cluster_log_kms_key_id = var.cluster_log_kms_key_id + source = "github.com/pluralsh/terraform-aws-eks?ref=output-service-cidr" + cluster_name = var.cluster_name + cluster_version = var.kubernetes_version + private_subnets = local.private_subnet_ids + public_subnets = local.public_subnet_ids + worker_private_subnets = local.worker_private_subnet_ids + vpc_id = local.vpc_id + enable_irsa = true + write_kubeconfig = false + create_eks = var.create_cluster + cluster_enabled_log_types = var.cluster_enabled_log_types + cluster_log_retention_in_days = var.cluster_log_retention_in_days + cluster_log_kms_key_id = var.cluster_log_kms_key_id + cluster_endpoint_public_access = var.cluster_endpoint_public_access + cluster_endpoint_private_access = var.cluster_endpoint_private_access + cluster_encryption_config = var.cluster_encryption_config + cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs node_groups_defaults = {} diff --git a/bootstrap/terraform/aws-bootstrap/variables.tf b/bootstrap/terraform/aws-bootstrap/variables.tf index 416030b3e..0cc2c41a8 100644 --- a/bootstrap/terraform/aws-bootstrap/variables.tf +++ b/bootstrap/terraform/aws-bootstrap/variables.tf @@ -7,6 +7,34 @@ Name for the vpc for the cluster EOF } + +variable "cluster_endpoint_private_access" { + description = "Indicates whether or not the Amazon EKS private API server endpoint is enabled." + type = bool + default = false +} + +variable "cluster_endpoint_public_access" { + description = "Indicates whether or not the Amazon EKS public API server endpoint is enabled." + type = bool + default = true +} + +variable "cluster_endpoint_public_access_cidrs" { + description = "List of CIDR blocks which can access the Amazon EKS public API server endpoint." + type = list(string) + default = ["0.0.0.0/0"] +} + +variable "cluster_encryption_config" { + description = "Configuration block with encryption configuration for the cluster. See examples/secrets_encryption/main.tf for example format" + type = list(object({ + provider_key_arn = string + resources = list(string) + })) + default = [] +} + variable "cluster_enabled_log_types" { default = [] description = "A list of the desired control plane logging to enable. Supported options are: api, audit, authenticator, controllerManager, scheduler. For more information, see Amazon EKS Control Plane Logging documentation (https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)" diff --git a/mage/helm/mage/Chart.lock b/mage/helm/mage/Chart.lock index a2dedf52b..03e0f7fe7 100644 --- a/mage/helm/mage/Chart.lock +++ b/mage/helm/mage/Chart.lock @@ -4,6 +4,6 @@ dependencies: version: 0.1.5 - name: mageai repository: https://mage-ai.github.io/helm-charts - version: 0.1.2 -digest: sha256:dd45698821c408ea2e7d4092759b3349b81560831eecc5ae62585864c453f6c5 -generated: "2023-06-26T13:31:32.353565+02:00" + version: 0.1.4 +digest: sha256:85b3a36cafc18f811f99fceabc7ce5ee791d0994a10d03213979321a57a6e1e7 +generated: "2023-10-04T22:26:18.459861143-07:00" diff --git a/mage/helm/mage/Chart.yaml b/mage/helm/mage/Chart.yaml index c4b758d9e..9313eb5bf 100644 --- a/mage/helm/mage/Chart.yaml +++ b/mage/helm/mage/Chart.yaml @@ -3,10 +3,11 @@ name: mage description: helm chart for mageai type: application version: 0.1.10 -appVersion: 0.8.102 +appVersion: 0.9.31 dependencies: - name: postgres version: 0.1.5 repository: https://pluralsh.github.io/module-library - name: mageai - version: 0.1.2 + version: 0.1.4 + repository: https://mage-ai.github.io/helm-charts \ No newline at end of file diff --git a/mage/helm/mage/charts/mageai-0.1.4.tgz b/mage/helm/mage/charts/mageai-0.1.4.tgz new file mode 100644 index 000000000..7b042e3ee Binary files /dev/null and b/mage/helm/mage/charts/mageai-0.1.4.tgz differ diff --git a/mage/helm/mage/values.yaml b/mage/helm/mage/values.yaml index 454c44ea5..b912477ae 100644 --- a/mage/helm/mage/values.yaml +++ b/mage/helm/mage/values.yaml @@ -15,9 +15,9 @@ mageai: cert-manager.io/cluster-issuer: letsencrypt-prod image: repository: dkr.plural.sh/mage/mageai/mageai - tag: 0.8.102 + tag: 0.9.31 volumes: - name: mage-fs persistentVolumeClaim: - claimName: mageai-pvc + claimName: mageai-pvc \ No newline at end of file diff --git a/mage/repository.yaml b/mage/repository.yaml index 170023215..35a8646f4 100644 --- a/mage/repository.yaml +++ b/mage/repository.yaml @@ -9,4 +9,5 @@ homepage: https://www.mage.ai/ gitUrl: https://github.com/mage-ai/mage-ai contributors: - nico.guary@gmail.com - - walkoss@pm.me \ No newline at end of file + - walkoss@pm.me + - troyyyang@gmail.com \ No newline at end of file diff --git a/ray/helm/kuberay-operator/Chart.lock b/ray/helm/kuberay-operator/Chart.lock index 139a77337..4de922255 100644 --- a/ray/helm/kuberay-operator/Chart.lock +++ b/ray/helm/kuberay-operator/Chart.lock @@ -1,6 +1,6 @@ dependencies: - name: kuberay-operator - repository: https://kevin85421.github.io/kuberay - version: 0.3.0 -digest: sha256:abb43e05246ec58ef6137df26d2d1692f59066058695b84d17986e5622d82690 -generated: "2022-09-21T15:06:07.583379+02:00" + repository: https://ray-project.github.io/kuberay-helm/ + version: 0.6.0 +digest: sha256:68767f4de687430221785f64d5b752285141d2192cae4c91a55b13d40106d063 +generated: "2023-10-05T14:25:03.985572273-07:00" diff --git a/ray/helm/kuberay-operator/Chart.yaml b/ray/helm/kuberay-operator/Chart.yaml index e90255063..287d1a1c7 100644 --- a/ray/helm/kuberay-operator/Chart.yaml +++ b/ray/helm/kuberay-operator/Chart.yaml @@ -2,9 +2,9 @@ apiVersion: v2 name: kuberay-operator description: A Helm chart for Kubernetes type: application -version: 0.1.3 -appVersion: "v0.3.0" +version: 0.1.4 +appVersion: "v0.6.0" dependencies: - name: kuberay-operator - version: 0.3.0 - repository: https://kevin85421.github.io/kuberay + version: 0.6.0 + repository: https://ray-project.github.io/kuberay-helm/ diff --git a/ray/helm/kuberay-operator/charts/kuberay-operator-0.3.0.tgz b/ray/helm/kuberay-operator/charts/kuberay-operator-0.3.0.tgz deleted file mode 100644 index 293b2eda9..000000000 Binary files a/ray/helm/kuberay-operator/charts/kuberay-operator-0.3.0.tgz and /dev/null differ diff --git a/ray/helm/kuberay-operator/charts/kuberay-operator-0.6.0.tgz b/ray/helm/kuberay-operator/charts/kuberay-operator-0.6.0.tgz new file mode 100644 index 000000000..00851a027 Binary files /dev/null and b/ray/helm/kuberay-operator/charts/kuberay-operator-0.6.0.tgz differ diff --git a/ray/helm/kuberay-operator/crds/ray.io_rayclusters.yaml b/ray/helm/kuberay-operator/crds/ray.io_rayclusters.yaml index d2ce5baf3..062032188 100644 --- a/ray/helm/kuberay-operator/crds/ray.io_rayclusters.yaml +++ b/ray/helm/kuberay-operator/crds/ray.io_rayclusters.yaml @@ -16,7 +16,28 @@ spec: singular: raycluster scope: Namespaced versions: - - name: v1alpha1 + - additionalPrinterColumns: + - jsonPath: .status.desiredWorkerReplicas + name: desired workers + type: integer + - jsonPath: .status.availableWorkerReplicas + name: available workers + type: integer + - jsonPath: .status.state + name: status + type: string + - jsonPath: .metadata.creationTimestamp + name: age + type: date + - jsonPath: .status.head.podIP + name: head pod IP + priority: 1 + type: string + - jsonPath: .status.head.serviceIP + name: head service IP + priority: 1 + type: string + name: v1alpha1 schema: openAPIV3Schema: description: RayCluster is the Schema for the RayClusters API @@ -205,6 +226,113 @@ spec: resources required. type: object type: object + securityContext: + description: SecurityContext defines the security options the + container should be run with. + properties: + allowPrivilegeEscalation: + description: AllowPrivilegeEscalation controls whether a process + can gain more privileges than its parent process + type: boolean + capabilities: + description: The capabilities to add/drop when running containers. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. + type: boolean + procMount: + description: procMount denotes the type of proc mount to use + for the containers. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root filesystem. + Default is false. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a non-root + user. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the container. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. + type: string + type: + description: type indicates which kind of seccomp profile + will be applied. + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to all + containers. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. + type: string + type: object + type: object upscalingMode: description: UpscalingMode is "Conservative", "Default", or "Aggressive. enum: @@ -5599,12 +5727,15 @@ spec: type: object required: - rayStartParams - - serviceType - template type: object + headServiceAnnotations: + additionalProperties: + type: string + type: object rayVersion: - description: RayVersion is the version of ray being used. this affects - the command used to start ray + description: RayVersion is the version of ray being used. This determines + the autoscaler's image version. type: string workerGroupSpecs: description: WorkerGroupSpecs are the specs for the worker pods @@ -11122,6 +11253,14 @@ spec: type: string description: Service Endpoints type: object + head: + description: Head info + properties: + podIP: + type: string + serviceIP: + type: string + type: object lastUpdateTime: description: LastUpdateTime indicates last update timestamp for this cluster status. @@ -11138,6 +11277,14 @@ spec: each node group. format: int32 type: integer + observedGeneration: + description: observedGeneration is the most recent generation observed + for this RayCluster. + format: int64 + type: integer + reason: + description: Reason provides more information about current State + type: string state: description: 'INSERT ADDITIONAL STATUS FIELD - define observed state of cluster Important: Run "make" to regenerat' diff --git a/ray/helm/kuberay-operator/crds/ray.io_rayjobs.yaml b/ray/helm/kuberay-operator/crds/ray.io_rayjobs.yaml index 5b19903d1..46e3c1ca6 100644 --- a/ray/helm/kuberay-operator/crds/ray.io_rayjobs.yaml +++ b/ray/helm/kuberay-operator/crds/ray.io_rayjobs.yaml @@ -231,6 +231,115 @@ spec: compute resources required. type: object type: object + securityContext: + description: SecurityContext defines the security options + the container should be run with. + properties: + allowPrivilegeEscalation: + description: AllowPrivilegeEscalation controls whether + a process can gain more privileges than its parent process + type: boolean + capabilities: + description: The capabilities to add/drop when running + containers. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. + type: boolean + procMount: + description: procMount denotes the type of proc mount + to use for the containers. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root + filesystem. Default is false. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as + a non-root user. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the + container. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + properties: + localhostProfile: + description: localhostProfile indicates a profile + defined in a file on the node should be used. + type: string + type: + description: type indicates which kind of seccomp + profile will be applied. + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to + all containers. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA + admission webhook (https://github. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of + the GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container + should be run as a 'Host Process' container. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. + type: string + type: object + type: object upscalingMode: description: UpscalingMode is "Conservative", "Default", or "Aggressive. @@ -5861,12 +5970,15 @@ spec: type: object required: - rayStartParams - - serviceType - template type: object + headServiceAnnotations: + additionalProperties: + type: string + type: object rayVersion: - description: RayVersion is the version of ray being used. this - affects the command used to start ray + description: RayVersion is the version of ray being used. This + determines the autoscaler's image version. type: string workerGroupSpecs: description: WorkerGroupSpecs are the specs for the worker pods @@ -11656,10 +11768,15 @@ spec: of cluster Important: Run "make" to regenerat' type: string jobStatus: - description: JobStatus is the Ray Job Status. https://docs.ray.io/en/latest/cluster/jobs-package-ref. + description: JobStatus is the Ray Job Status. type: string message: type: string + observedGeneration: + description: observedGeneration is the most recent generation observed + for this RayJob. + format: int64 + type: integer rayClusterName: type: string rayClusterStatus: @@ -11680,6 +11797,14 @@ spec: type: string description: Service Endpoints type: object + head: + description: Head info + properties: + podIP: + type: string + serviceIP: + type: string + type: object lastUpdateTime: description: LastUpdateTime indicates last update timestamp for this cluster status. @@ -11696,6 +11821,14 @@ spec: of each node group. format: int32 type: integer + observedGeneration: + description: observedGeneration is the most recent generation + observed for this RayCluster. + format: int64 + type: integer + reason: + description: Reason provides more information about current State + type: string state: description: 'INSERT ADDITIONAL STATUS FIELD - define observed state of cluster Important: Run "make" to regenerat' diff --git a/ray/helm/kuberay-operator/crds/ray.io_rayservices.yaml b/ray/helm/kuberay-operator/crds/ray.io_rayservices.yaml index b51c8017b..70056ed2b 100644 --- a/ray/helm/kuberay-operator/crds/ray.io_rayservices.yaml +++ b/ray/helm/kuberay-operator/crds/ray.io_rayservices.yaml @@ -217,6 +217,115 @@ spec: compute resources required. type: object type: object + securityContext: + description: SecurityContext defines the security options + the container should be run with. + properties: + allowPrivilegeEscalation: + description: AllowPrivilegeEscalation controls whether + a process can gain more privileges than its parent process + type: boolean + capabilities: + description: The capabilities to add/drop when running + containers. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. + type: boolean + procMount: + description: procMount denotes the type of proc mount + to use for the containers. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root + filesystem. Default is false. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as + a non-root user. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the + container. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + properties: + localhostProfile: + description: localhostProfile indicates a profile + defined in a file on the node should be used. + type: string + type: + description: type indicates which kind of seccomp + profile will be applied. + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to + all containers. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA + admission webhook (https://github. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of + the GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container + should be run as a 'Host Process' container. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. + type: string + type: object + type: object upscalingMode: description: UpscalingMode is "Conservative", "Default", or "Aggressive. @@ -5847,12 +5956,15 @@ spec: type: object required: - rayStartParams - - serviceType - template type: object + headServiceAnnotations: + additionalProperties: + type: string + type: object rayVersion: - description: RayVersion is the version of ray being used. this - affects the command used to start ray + description: RayVersion is the version of ray being used. This + determines the autoscaler's image version. type: string workerGroupSpecs: description: WorkerGroupSpecs are the specs for the worker pods @@ -11736,6 +11848,14 @@ spec: type: string description: Service Endpoints type: object + head: + description: Head info + properties: + podIP: + type: string + serviceIP: + type: string + type: object lastUpdateTime: description: LastUpdateTime indicates last update timestamp for this cluster status. @@ -11752,6 +11872,15 @@ spec: of each node group. format: int32 type: integer + observedGeneration: + description: observedGeneration is the most recent generation + observed for this RayCluster. + format: int64 + type: integer + reason: + description: Reason provides more information about current + State + type: string state: description: 'INSERT ADDITIONAL STATUS FIELD - define observed state of cluster Important: Run "make" to regenerat' @@ -11760,7 +11889,7 @@ spec: serveDeploymentStatuses: items: description: ServeDeploymentStatus defines the current state - of Serve Deployment + of a Serve deployment properties: healthLastUpdateTime: description: Keep track of how long the service is healthy. @@ -11773,7 +11902,7 @@ spec: type: string name: description: Name, Status, Message are from Ray Dashboard - to represent the state of a serve deployment. + and represent a Serve deployment's state. type: string status: description: 'TODO: change status type to enum' @@ -11781,6 +11910,11 @@ spec: type: object type: array type: object + observedGeneration: + description: observedGeneration is the most recent generation observed + for this RayService. + format: int64 + type: integer pendingServiceStatus: description: Pending Service Status indicates a RayCluster will be created or is being created. @@ -11835,6 +11969,14 @@ spec: type: string description: Service Endpoints type: object + head: + description: Head info + properties: + podIP: + type: string + serviceIP: + type: string + type: object lastUpdateTime: description: LastUpdateTime indicates last update timestamp for this cluster status. @@ -11851,6 +11993,15 @@ spec: of each node group. format: int32 type: integer + observedGeneration: + description: observedGeneration is the most recent generation + observed for this RayCluster. + format: int64 + type: integer + reason: + description: Reason provides more information about current + State + type: string state: description: 'INSERT ADDITIONAL STATUS FIELD - define observed state of cluster Important: Run "make" to regenerat' @@ -11859,7 +12010,7 @@ spec: serveDeploymentStatuses: items: description: ServeDeploymentStatus defines the current state - of Serve Deployment + of a Serve deployment properties: healthLastUpdateTime: description: Keep track of how long the service is healthy. @@ -11872,7 +12023,7 @@ spec: type: string name: description: Name, Status, Message are from Ray Dashboard - to represent the state of a serve deployment. + and represent a Serve deployment's state. type: string status: description: 'TODO: change status type to enum' diff --git a/ray/helm/kuberay-operator/values.yaml b/ray/helm/kuberay-operator/values.yaml index 6d25d42b1..c2781d3cc 100644 --- a/ray/helm/kuberay-operator/values.yaml +++ b/ray/helm/kuberay-operator/values.yaml @@ -1,6 +1,6 @@ kuberay-operator: image: repository: dkr.plural.sh/ray/kuberay/operator - tag: 2a7f0f2 + tag: v0.6.0 metrics: enabled: true diff --git a/ray/helm/ray/Chart.lock b/ray/helm/ray/Chart.lock index f1ac79826..2ea75aefa 100644 --- a/ray/helm/ray/Chart.lock +++ b/ray/helm/ray/Chart.lock @@ -1,6 +1,6 @@ dependencies: - name: oauth2-proxy repository: https://oauth2-proxy.github.io/manifests - version: 6.2.7 -digest: sha256:48aff6bf5fe4978a7d837382579de9a468fec0d37c879ce7035fc2d1a23c1915 -generated: "2022-09-27T16:50:40.3297+02:00" + version: 6.12.1 +digest: sha256:93729f3b6ab0180ba08f03989328daeab2a80d7b71e530b5d681b4dc88c6178b +generated: "2023-06-02T16:38:57.184953+02:00" diff --git a/ray/helm/ray/Chart.yaml b/ray/helm/ray/Chart.yaml index 06b3c405a..a08b3ebb3 100644 --- a/ray/helm/ray/Chart.yaml +++ b/ray/helm/ray/Chart.yaml @@ -2,10 +2,10 @@ apiVersion: v2 name: ray description: helm chart for ray type: application -version: 0.1.8 -appVersion: "2.0.0" +version: 0.1.9 +appVersion: "2.7.0" dependencies: - name: oauth2-proxy - version: 6.2.7 + version: 6.12.1 repository: https://oauth2-proxy.github.io/manifests condition: oauth2-proxy.enabled diff --git a/ray/helm/ray/charts/oauth2-proxy-6.12.1.tgz b/ray/helm/ray/charts/oauth2-proxy-6.12.1.tgz new file mode 100644 index 000000000..a3c9cc249 Binary files /dev/null and b/ray/helm/ray/charts/oauth2-proxy-6.12.1.tgz differ diff --git a/ray/helm/ray/charts/oauth2-proxy-6.2.7.tgz b/ray/helm/ray/charts/oauth2-proxy-6.2.7.tgz deleted file mode 100644 index 0389a0fb1..000000000 Binary files a/ray/helm/ray/charts/oauth2-proxy-6.2.7.tgz and /dev/null differ diff --git a/ray/helm/ray/values.yaml b/ray/helm/ray/values.yaml index ce0134cae..9cdd25578 100644 --- a/ray/helm/ray/values.yaml +++ b/ray/helm/ray/values.yaml @@ -4,7 +4,7 @@ image: repository: dkr.plural.sh/ray/rayproject/ray - tag: 2.0.0 + tag: 2.7.0 pullPolicy: IfNotPresent nameOverride: "" @@ -101,7 +101,7 @@ workers: - groupName: small-group image: repository: dkr.plural.sh/ray/rayproject/ray - tag: 2.0.0 + tag: 2.7.0 replicas: 0 minReplicas: 0 maxReplicas: 10 @@ -164,7 +164,7 @@ oauth2-proxy: fullnameOverride: ray-oauth2-proxy image: repository: dkr.plural.sh/oauth2-proxy/oauth2-proxy/oauth2-proxy - tag: v7.3.0 + tag: v7.4.0 resources: limits: diff --git a/ray/repository.yaml b/ray/repository.yaml index 547a954d3..6b79312a1 100644 --- a/ray/repository.yaml +++ b/ray/repository.yaml @@ -12,3 +12,5 @@ oauthSettings: tags: - tag: AI - tag: automation +contributors: + - troyyyang@gmail.com diff --git a/ray/vendor_images.yaml b/ray/vendor_images.yaml index a25d5ad04..b1b01cbe7 100644 --- a/ray/vendor_images.yaml +++ b/ray/vendor_images.yaml @@ -1,7 +1,4 @@ docker.io: - images: - kuberay/operator: - - 2a7f0f2 images-by-tag-regex: # the below regex can be used to limit the regex range so that only versions >= v2.0.0 are matched rayproject/ray: ^v?(?:(?:[^0-1]{1})?(?:[2-9]{1}|[0-9]{2}))+\.[0-9]+?\.[0-9]+(?:-gpu)?$ diff --git a/renovate-on-prem/helm/renovate-on-prem/Chart.lock b/renovate-on-prem/helm/renovate-on-prem/Chart.lock index 3e8560743..5e995ff2f 100644 --- a/renovate-on-prem/helm/renovate-on-prem/Chart.lock +++ b/renovate-on-prem/helm/renovate-on-prem/Chart.lock @@ -1,6 +1,6 @@ dependencies: -- name: whitesource-renovate - repository: https://mend.github.io/renovate-on-prem - version: 3.1.4 -digest: sha256:ca1613ad4e7fb4d4716a67e43375c635dfc14687e42e617dc3086582f9e741e8 -generated: "2023-07-05T15:35:15.907830315Z" +- name: mend-renovate-ce + repository: https://mend.github.io/renovate-ce-ee + version: 6.1.1 +digest: sha256:20868fbc40e77d650d51cad5e64391551852d0cb3448aa111e4900e1525f3659 +generated: "2023-10-06T10:44:41.993067+02:00" diff --git a/renovate-on-prem/helm/renovate-on-prem/Chart.yaml b/renovate-on-prem/helm/renovate-on-prem/Chart.yaml index fa7091fa2..ab89abf65 100644 --- a/renovate-on-prem/helm/renovate-on-prem/Chart.yaml +++ b/renovate-on-prem/helm/renovate-on-prem/Chart.yaml @@ -2,9 +2,9 @@ apiVersion: v2 name: renovate-on-prem description: helm chart for renovate-on-prem type: application -version: 0.2.1 -appVersion: 4.3.0 +version: 0.3.0 +appVersion: 6.1.1 dependencies: -- name: whitesource-renovate - repository: https://mend.github.io/renovate-on-prem - version: 3.1.4 +- name: mend-renovate-ce + repository: https://mend.github.io/renovate-ce-ee + version: 6.1.1 diff --git a/renovate-on-prem/helm/renovate-on-prem/charts/mend-renovate-ce-6.1.1.tgz b/renovate-on-prem/helm/renovate-on-prem/charts/mend-renovate-ce-6.1.1.tgz new file mode 100644 index 000000000..6e534ba07 Binary files /dev/null and b/renovate-on-prem/helm/renovate-on-prem/charts/mend-renovate-ce-6.1.1.tgz differ diff --git a/renovate-on-prem/helm/renovate-on-prem/charts/whitesource-renovate-3.1.4.tgz b/renovate-on-prem/helm/renovate-on-prem/charts/whitesource-renovate-3.1.4.tgz deleted file mode 100644 index 383fbc641..000000000 Binary files a/renovate-on-prem/helm/renovate-on-prem/charts/whitesource-renovate-3.1.4.tgz and /dev/null differ diff --git a/renovate-on-prem/helm/renovate-on-prem/templates/runbooks.yaml b/renovate-on-prem/helm/renovate-on-prem/templates/runbooks.yaml index 90de34a6f..57ed2daef 100644 --- a/renovate-on-prem/helm/renovate-on-prem/templates/runbooks.yaml +++ b/renovate-on-prem/helm/renovate-on-prem/templates/runbooks.yaml @@ -16,18 +16,18 @@ spec: prometheus: format: cpu legend: $pod - query: sum(rate(container_cpu_usage_seconds_total{namespace="{{ .Release.Namespace }}",pod=~"{{ template "whitesource-renovate.fullname" (index .Subcharts "whitesource-renovate") }}.+"}[5m])) by (pod) + query: sum(rate(container_cpu_usage_seconds_total{namespace="{{ .Release.Namespace }}",pod=~"{{ template "mend-renovate.fullname" (index .Subcharts "mend-renovate-ce") }}.+"}[5m])) by (pod) - name: renovate-memory type: prometheus prometheus: format: memory legend: $pod - query: sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}",pod=~"{{ template "whitesource-renovate.fullname" (index .Subcharts "whitesource-renovate") }}.+"}) by (pod) + query: sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}",pod=~"{{ template "mend-renovate.fullname" (index .Subcharts "mend-renovate-ce") }}.+"}) by (pod) - name: renovate type: kubernetes kubernetes: resource: deployment - name: {{ template "whitesource-renovate.fullname" (index .Subcharts "whitesource-renovate") }} + name: {{ template "mend-renovate.fullname" (index .Subcharts "mend-renovate-ce") }} actions: - name: scale action: config @@ -36,28 +36,28 @@ spec: updates: - path: - renovate-on-prem - - whitesource-renovate + - mend-renovate-ce - resources - requests - cpu valueFrom: renovate-cpu - path: - renovate-on-prem - - whitesource-renovate + - mend-renovate-ce - resources - requests - memory valueFrom: renovate-memory - path: - renovate-on-prem - - whitesource-renovate + - mend-renovate-ce - resources - limits - cpu valueFrom: renovate-cpu-limit - path: - renovate-on-prem - - whitesource-renovate + - mend-renovate-ce - resources - limits - memory diff --git a/renovate-on-prem/helm/renovate-on-prem/values.yaml b/renovate-on-prem/helm/renovate-on-prem/values.yaml index 6b0b20fbb..ba96f8d28 100644 --- a/renovate-on-prem/helm/renovate-on-prem/values.yaml +++ b/renovate-on-prem/helm/renovate-on-prem/values.yaml @@ -1,9 +1,11 @@ secrets: {} -whitesource-renovate: +mend-renovate-ce: image: - repository: dkr.plural.sh/renovate-on-prem/whitesource/renovate - tag: 4.3.0 + repository: dkr.plural.sh/renovate-on-prem/mend/renovate-ce + tag: 6.1.1-full + renovate: + mendRnvAdminApiEnabled: true ingress: enabled: true ingressClassName: nginx diff --git a/renovate-on-prem/helm/renovate-on-prem/values.yaml.tpl b/renovate-on-prem/helm/renovate-on-prem/values.yaml.tpl index ff9af7f15..4f31e1408 100644 --- a/renovate-on-prem/helm/renovate-on-prem/values.yaml.tpl +++ b/renovate-on-prem/helm/renovate-on-prem/values.yaml.tpl @@ -4,7 +4,7 @@ global: - description: renovate public endpoint url: {{ .Values.hostname }} -whitesource-renovate: +mend-renovate-ce: ingress: hosts: - {{ .Values.hostname }} @@ -14,28 +14,29 @@ whitesource-renovate: - {{ .Values.hostname }} renovate: {{- if .Values.acceptTos }} - acceptWhiteSourceTos: "y" + mendRnvAcceptTos: "y" {{- end }} - licenseKey: {{ .Values.licenseKey }} - renovatePlatform: {{ .Values.platform }} + mendRnvLicenseKey: {{ .Values.licenseKey }} + mendRnvServerApiSecret: {{ dedupe . "renovate-on-prem.mend-renovate-ce.renovate.mendRnvServerApiSecret" (randAlphaNum 32) }} + mendRnvPlatform: {{ .Values.platform }} {{- if .Values.renovateEndpoint }} - renovateEndpoint: {{ .Values.renovateEndpoint }} + mendRnvEndpoint: {{ .Values.renovateEndpoint }} {{- else if eq .Values.platform "github" }} - renovateEndpoint: https://api.github.com/ + mendRnvEndpoint: https://api.github.com/ {{- else if eq .Values.platform "gitlab" }} - renovateEndpoint: https://gitlab.com/api/v4/ + mendRnvEndpoint: https://gitlab.com/api/v4/ {{- end }} {{- if .Values.githubAppId }} - githubAppId: {{ .Values.githubAppId | quote }} + mendRnvGithubAppId: {{ .Values.githubAppId | quote }} {{- end }} {{- if .Values.githubAppKey }} - githubAppKey: {{ .Values.githubAppKey | quote }} + mendRnvGithubAppKey: {{ .Values.githubAppKey | quote }} {{- end }} {{- if .Values.webhookSecret }} - webhookSecret: {{ .Values.webhookSecret }} + mendRnvWebhookSecret: {{ .Values.webhookSecret }} {{- end }} {{- if .Values.renovateToken }} - renovateToken: {{ .Values.renovateToken }} + mendRnvGitlabPat: {{ .Values.renovateToken }} {{- end }} {{- if .Values.githubComToken }} githubComToken: {{ .Values.githubComToken }} diff --git a/renovate-on-prem/vendor_images.yaml b/renovate-on-prem/vendor_images.yaml index 86e8b4efa..98e0c7532 100644 --- a/renovate-on-prem/vendor_images.yaml +++ b/renovate-on-prem/vendor_images.yaml @@ -1,4 +1,4 @@ -docker.io: +ghcr.io: images-by-tag-regex: # the below regex can be used to limit the regex range so that only versions >= 2.6.0 are matched - whitesource/renovate: (?:^v?2\.(?:(?:[^0-5]{1})?(?:[6-9]{1}|[0-9]{2}))+?\.[0-9]+$)|(?:^v?(?:(?:[^0-2]{1})?(?:[3-9]{1}|[0-9]{2}))+\.[0-9]+?\.[0-9]+$) + mend/renovate-ce: (?:^v?2\.(?:(?:[^0-5]{1})?(?:[6-9]{1}|[0-9]{2}))+?\.[0-9]+-full$)|(?:^v?(?:(?:[^0-2]{1})?(?:[3-9]{1}|[0-9]{2}))+\.[0-9]+?\.[0-9]+-full$)