From df492caafdfeb5a0c664717f93e863aaf5021f0b Mon Sep 17 00:00:00 2001 From: James Lott Date: Thu, 18 Aug 2022 16:35:57 -0400 Subject: [PATCH] Deploy Elavon SFTP ingest server into production (#1695) * docs(datasets and tables): added and revamped RT dataset section and docs with links out to dbt docs * docs(datasets and tables): added highlighting to important areas in dataset docs * (portfolio docs): add notebook tips * (portfolio docs): change order of sections * (portfolio docs): more description about decimals and rounding * (portfolio docs): fix formatting * (portfolio docs): fix formatting * start on gtfsrt-v2 * (portfolio_docs): fix typo * docs: testing broken action * switch rt node pool to c2 instances * get new calitp version * import gcs models from calitp-py! * airtable: add macro TODO * airtable: gtfs datasets mart & staging updates * airtable: reenable airtable warehouse resources * airtable: use correct incoming id names in staging tables * airtable: get tts actually working with prior updates * airtable: gtfs service data * airtable: add provider/gtfs table -- fixes #1487 * airtable: clean up some ctes * airtable: clean up references based on some schema changes * airtable: add relationship tests for foreign keys in mart * airtable: start renaming int to base * airtable: refactor staging tables to be historical; refactor get latest macro to enable daily extract selection * airtable: convert staging to views rather than tables * airtable: convert intermediate mapping tables to base * always compile, but only check dbt run success after docs/metabase * run tests even if run failed * airtable: define key as metabase PK * airtable: add equal row count tests for models with id mapping * airtable: rename map to bridge * update poetry.lock for dbt-metabase * airtable: latest-only-ify bridge tables * missed a couple * airtable: make mart latest-only * airtable: refactor dim service components * airtable: specify metabase FK columns * airtable: new fields & tables to address #1630 * airtable: make bridge tables date-aware and assorted small fixes * get us going! * airtable: address failing dbt tests -- minor tweaks * airtable: more failing dbt tests * airtable: refactor service components to handle duplicates * airtable: fix legacy airtable source definition to reference views * airtable: remove redundant metabase FK metadata * airtable: fix test syntax * airtable: use QUALIFY to simplify ranked queries * fix: make airtable gcs operator use timestamps rather than time string * fix(timestamp partitions): update calitp version to get schedule partition updates * warehouse (payments): migrated payments_views_staging cleaned dags to models as well as validation tables to tests * use new calitp version * fix(timestamp partitions): explicitly use isoformat string * style: rename CTEs to be more specific * farm surrogate key macro: coalesce nulls in macro itself * add notebook used to re-name a partition * chore: remove pyup config file no longer in use * chore: remove pyup ignore statement * airtable: use ts instead of time * add airtable mart to list of things synced to metabase * update metabase database names again * warehouse(payments_views_staging): split yml files into staging and source, added documentation for cleaned files, deleted old validation tables * warehouse(payments_views_staging): added generic tests, added composite unique tests from dbt_packages, added docs file with references, materialized staging tables as views * warehouse(payments_views_staging): added configuration to persist singular tests as tables in the warehouse * warehouse(payments_views): migrated airflow dags for payments views to its own model in dbt, added metadata and generic tests, added dbt references * print message if deploy is not set * round lat/lons, specify 4m accuracy, add new resources * print the documentation file being written * add coord system, disable shapes for now due to size limit * fix(fact daily trips timeout): wip incremental table * update to good stable version of sqlfluff * fix: make fact daily trips incremental -- WIP * pass and/or ignore new rules * linter * fact daily trips: remove dev incremental check * docs: update airtable prod maintenance instructions * docs: add new dags to dependency diagram * docs: add spacing to help w line wrapping * docs: more spaces for line wrapping... * dbt-metabase: update version in poetry; comment out failing relationship tests * warehouse(payments_views): got payments_rides working and migrated, added yml and metadata, added payments_views validation tests and persisted tables, added payments_views_refactored with intermedite tables and got that to work * get new calitp version * import gcs models from calitp-py! * missed a couple * get us going! * fix: make airtable gcs operator use timestamps rather than time string * fix(timestamp partitions): update calitp version to get schedule partition updates * fix(timestamp partitions): explicitly use isoformat string * use new calitp version * start experimenting with task queue options and metrics * get this working and test performance with greenlets * couple more metrics * wip testing with multiple consumers at high volume * start optimizing for lots of small tasks; have to make redis interaction fast * fix key str format * couple more libs * wip * wip on discussed changes * get the keys from environ for now * use new calitp py * print a bit more * we are just gonna get stuff in the env * commit this before I break anything * fmt * bump calitp-py * lint * rename v2 to v3 since 2.X tags already exist * kinda make this runnable * new node pool just dropped * get running in docker compose to kick the tires * start on RT v3 k8s * get the consumer working mostly? * label redis pod appropriately * tell consumer about temp rt secrets * that was dumb * ticker k8s! * set expire time on the huey instance * point consumer at svc account json * avoid pulling the stacktrace in * scrape on 9102 * bump to 16 workers per consumer * bump jupyterhub storage to 32gi * add these back! * add comment * bring in new calitp and fix tick rounding * improve metrics and labels * warehouse(payments): removed payemnts_rides_refactor from yml file * clean up labels * get secrets from secret manager sdk before the consumer starts... * missed this * fix secrets volume and adjust affinities * warehouse(payments): removed the airflow dags for the payments_views that were migrated, as well as the two test tables * warehouse(payments): removed the old intermediate tables from the dbt project yaml file * add content type header to bytes * ugh whitespace * warehouse: fixing linting error * warehouse: fixing linting error again * warehouse(dbt_project): added to-do comments in project config to remind where to move model schemas in the future * fix: update Mountain Transit URL * remove celery and gevent from pyproject deps Co-authored-by: Mjumbe Poe * we might as well specify huey app name by env as well just in case we end up on the same redis in the future * write to the prod bucket! * create a preprod version and deploy it * run fewer workers in preprod * move pull policies to patches, and only run 1 dev consumer * add redis considerations to readme * docs(datasets and tables): revised informationon dbt docs for views tables based on PR review * docs(datasets and tables): revised for readability * docs(datasets and tables): revised docs information for gtfs schedule based on PR review * docs(datasets and tables): fixed readability * docs(datasets and tables): added new formatting, added gtfs rt dbt docs instructions * docs(datasets and tables): revamped the overview page for datasets and tables * docs(datasets and tables): cleaned up readability * bump version and start adding more logging context * specifically log request errors that do not come from raise_for_status * set v3 image versions separately * bump to 8 workers and improve log formatting * formatting * fix string representation of exception type in logs * bump prod to 3.1 * oops * hotfix version * bump to 30m * warehouse(airflow): deleted the empty payments_views_staging dag directory * warehouse(airflow): deleted dummy_staging airflow task, removed gusty dependencies from other tables that relied on that task * docs(airflow): edited the production dags docs to reflect changes in payments staging views dags * docs(airflow): revised docs based on lauries comment re only listing enfoorced dependencies * Update new-team-member.md Fixed added missing meetings, deleted old meetings. deleted auto-assign * docs(datasets ans tables): reconfigured some pages for readability * docs(datasets and tables): re-reviewed and added clarity * fix (open data): align column publish metadata with open data dictionary -- suppress calitp hash, synthetic keys, and extraction date, add calitp_itp_id and url_number * docs(production maintenance): added n/a for dependencies for payments_views * docs(datasets and tables): created new page with content on how to use dbt docs, added to toc * docs(datasets and tables): removed information on how to navigate dbt docs in favor of the new page created, added info to warehouse schema sections, created dbt project cirectory sections * (analyst_docs): update gcloud commands * fix(open data): make test_metadata attribute optional to account for singular tests * docs(datasets and tables): reformatted for readability and conciseness * docs(datasets and tables): revisions based on Laurie's review * docs(datasets and tables): revised PR to put gtfs views tables used by ckan under the views doc * fix(open data): suppress publishing stop_times because of size limit issue * agencies.yml: update FCRTA and add Escalon Transit * agencies.yml: rename escalon transit to etrans * fix(airflow/gtfs_loader): replace non-utf-8 characters * feat(airtable): add new columns per request #1674 * fix(airtable data): address review comments PR #1677 * fix: add WeHo RT URLs * fix(ckan publishing): only add columns to data dictionary if they don't have publish.ignore set * update calitp py and change log * make docker compose work * specify buckets and bump version in dev * now do prod * change logging * add weho key * bump gtfs rt v3 version * bump calitp py * deploy new image to dev * get dev and prod working with bucket env vars * bump calitp py and expire cache every 5 minutes * deploy new cache clearing to prod/dev * make sure calitp is updated, load secrets in ticker too * fix docker compose, use new flags, deploy new image to dev * bump prod * add airtable age metric, bump version, scrape ticker * delete experimental fact_daily_trips_inc incremental table that was not functioning correctly (#1681) * docs: correct Transit Technology Stacks title (#1565) The Transit Technology Stacks header was not properly being linked to in the overview table. This fixes that. * fix: update GRaaS URLs (#1690) * New schedule pipeline validation job (#1648) * wip on validation in new schedule pipeline * bring in stuff from calitp storage, work on saving validations/outcomes * wip getting this working * use new calitp prerelease, fix filenames/content, remove break * oops * working! * update lockfile * unzip/validate schedule dag * remove this * bring in latest calitp-py * extra print * pass env vars into pod * fix lint * add readme * bring in latest calitp * fix print and formatting * bring the outcome-only classes over, and use env var for bucket * filter out nones for RT airtable records * bring in latest calitp py * get latest calitp * use new env var and rename validation job results * start updating airflow with new calitp py and using bucket env vars * test schedule downloader with new calitp * new calitp * handle new calitp, better logging * add env vars for new calitp * put prefix_bucket back for parse_and_validate_rt and document env var configuration * comments * use new version of caltip py with good gcsfs (#1693) * use new version of caltip py with good gcsfs * use the regular release * docs(agency): adding reference table for analysts to define agency, reference for pre-commit hooks (#1430) * docs(agency): adding reference table for analysts to define agency in their research * docs(agency): fixed table formatting error * docs(agency): fixed table formatting error plus pre-commit hooks * docs(pre-commit hooks): added information for using and troubleshooting pre-commit hooks * docs: formatting errors, added missing capitalization * docs: formatting table with list * docs: formatting table with no line break - attempt 1 * docs: clarified language and spacing in table * docs: clarified language in table * docs: removing extra information from agency table * docs: removing extra information from agency table pt 2 * docs: removing extra information from agency table pt 3 * docs: reworked table to include gtfs-provider-service relationships * docs: added space for the gtfs provider's services section * docs: added space for the gtfs provider's services section syntax corrections * docs: added space for the gtfs provider's services section syntax corrections again * docs: clarified information arounf gtfs provider relationships * docs: clarified information around gtfs provider relationships and intro content * docs: agency table revisions based on call with E * docs(agency reference): incorporated E's feedback in the copy, added warehouse table instead of airtable table * docs(agency reference): reformatted table * docs(warehouse): added new table information for analyst agency reference now that the airtable migration is complete and the table was created. added css styling to prevent table scrolling * docs: renamed python library file h1 to be more intuitive * docs(conf): added comments explaining the added css preventing horizontal scroll in markdown tables * docs(add to what_is_agency) * docs(warehouse): fixed some typos, errors, and formatting issues Co-authored-by: natam1 Co-authored-by: Charles Costanzo * we also have to pin a specific fsspec version directly in the requirements (#1694) * Create SFTP ingest component for Elavon data (#1692) * kubernetes: sftp-ingest-elavon: add server component * kubernetes: sftp-server: add sshd configuration This enables functionality like chroot'd logins and disabling of shell logins. * kubernetes: sftp-server: add readinessProbe Since the container is essentially built at startup, there is a sizeable time delta between container startup and ssh server startup. This addition helps the operator easily detect when installation is complete and the service is running. * kubernetes: sftp-server: add cluster service This enables cluster workloads to login using a DNS names. * kubernetes: sftp-server: refactor bootstrap script for better DRY * kubernetes: prod-sftp-ingest-elavon: create production localization * kubernetes: prod-sftp-ingest-elavon: add internet-service.yaml This exposes the SFTP port for inbound connections from the vendor. * ci: prod-sftp-ingest-elavon.env: enable prod deployment Co-authored-by: Charlie Costanzo Co-authored-by: tiffanychu90 Co-authored-by: Andrew Vaccaro Co-authored-by: Andrew Vaccaro Co-authored-by: Laurie Merrell Co-authored-by: Kegan Maher Co-authored-by: Laurie <55149902+lauriemerrell@users.noreply.github.com> Co-authored-by: evansiroky Co-authored-by: Mjumbe Poe Co-authored-by: tiffanychu90 <49657200+tiffanychu90@users.noreply.github.com> Co-authored-by: natam1 Co-authored-by: Charles Costanzo Co-authored-by: Github Action build-release-candidate --- ci/vars/releases/prod-sftp-ingest-elavon.env | 2 + kubernetes/apps/charts/jupyterhub/values.yaml | 2 +- .../archiver-app-vars.yaml | 8 ++ .../gtfs-rt-archiver-consumer.yaml | 72 +++++++++++++++++ .../gtfs-rt-archiver-ticker.yaml | 72 +++++++++++++++++ .../gtfs-rt-archiver-v3/kustomization.yaml | 5 ++ .../manifests/gtfs-rt-archiver-v3/redis.yaml | 57 +++++++++++++ .../sftp-server/intranet-service.yaml | 13 +++ .../manifests/sftp-server/kustomization.yaml | 8 ++ .../apps/manifests/sftp-server/sshd_config | 6 ++ .../apps/manifests/sftp-server/workload.yaml | 80 +++++++++++++++++++ .../kustomization.yaml | 6 ++ .../base-sftp-ingest-elavon/sftp-user.yaml | 6 ++ .../dev-sftp-ingest-elavon/kustomization.yaml | 8 ++ .../overlays/dev-sftp-ingest-elavon/ns.yaml | 4 + .../sftp-user-config.yaml | 6 ++ .../archiver-channel-vars.yaml | 9 +++ .../consumer.patch.yaml | 12 +++ .../kustomization.yaml | 17 ++++ .../overlays/gtfs-rt-archiver-v3-dev/ns.yaml | 5 ++ .../gtfs-rt-archiver-v3-dev/ticker.patch.yaml | 11 +++ .../archiver-channel-vars.yaml | 9 +++ .../kustomization.yaml | 13 +++ .../overlays/gtfs-rt-archiver-v3-prod/ns.yaml | 5 ++ .../kustomization.yaml | 5 ++ .../internet-service.yaml | 13 +++ .../kustomization.yaml | 15 ++++ .../overlays/prod-sftp-ingest-elavon/ns.yaml | 4 + .../patch-volume-size.json | 7 ++ .../sftp-user-config.yaml | 6 ++ 30 files changed, 485 insertions(+), 1 deletion(-) create mode 100644 ci/vars/releases/prod-sftp-ingest-elavon.env create mode 100644 kubernetes/apps/manifests/gtfs-rt-archiver-v3/archiver-app-vars.yaml create mode 100644 kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-consumer.yaml create mode 100644 kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-ticker.yaml create mode 100644 kubernetes/apps/manifests/gtfs-rt-archiver-v3/kustomization.yaml create mode 100644 kubernetes/apps/manifests/gtfs-rt-archiver-v3/redis.yaml create mode 100644 kubernetes/apps/manifests/sftp-server/intranet-service.yaml create mode 100644 kubernetes/apps/manifests/sftp-server/kustomization.yaml create mode 100644 kubernetes/apps/manifests/sftp-server/sshd_config create mode 100644 kubernetes/apps/manifests/sftp-server/workload.yaml create mode 100644 kubernetes/apps/overlays/base-sftp-ingest-elavon/kustomization.yaml create mode 100644 kubernetes/apps/overlays/base-sftp-ingest-elavon/sftp-user.yaml create mode 100644 kubernetes/apps/overlays/dev-sftp-ingest-elavon/kustomization.yaml create mode 100644 kubernetes/apps/overlays/dev-sftp-ingest-elavon/ns.yaml create mode 100644 kubernetes/apps/overlays/dev-sftp-ingest-elavon/sftp-user-config.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/archiver-channel-vars.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/consumer.patch.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/kustomization.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ns.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ticker.patch.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/archiver-channel-vars.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/kustomization.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/ns.yaml create mode 100644 kubernetes/apps/overlays/gtfs-rt-archiver-v3-release/kustomization.yaml create mode 100644 kubernetes/apps/overlays/prod-sftp-ingest-elavon/internet-service.yaml create mode 100644 kubernetes/apps/overlays/prod-sftp-ingest-elavon/kustomization.yaml create mode 100644 kubernetes/apps/overlays/prod-sftp-ingest-elavon/ns.yaml create mode 100644 kubernetes/apps/overlays/prod-sftp-ingest-elavon/patch-volume-size.json create mode 100644 kubernetes/apps/overlays/prod-sftp-ingest-elavon/sftp-user-config.yaml diff --git a/ci/vars/releases/prod-sftp-ingest-elavon.env b/ci/vars/releases/prod-sftp-ingest-elavon.env new file mode 100644 index 0000000000..d69b713dda --- /dev/null +++ b/ci/vars/releases/prod-sftp-ingest-elavon.env @@ -0,0 +1,2 @@ +RELEASE_DRIVER=kustomize +RELEASE_KUSTOMIZE_DIR=kubernetes/apps/overlays/prod-sftp-ingest-elavon diff --git a/kubernetes/apps/charts/jupyterhub/values.yaml b/kubernetes/apps/charts/jupyterhub/values.yaml index fd6ab395f5..69eb8e1200 100644 --- a/kubernetes/apps/charts/jupyterhub/values.yaml +++ b/kubernetes/apps/charts/jupyterhub/values.yaml @@ -32,7 +32,7 @@ jupyterhub: hub: db: pvc: - storage: 8Gi + storage: 32Gi config: GitHubOAuthenticator: # client_id: in existingSecret diff --git a/kubernetes/apps/manifests/gtfs-rt-archiver-v3/archiver-app-vars.yaml b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/archiver-app-vars.yaml new file mode 100644 index 0000000000..f8a89eb7d6 --- /dev/null +++ b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/archiver-app-vars.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: archiver-app-vars +data: + CALITP_HUEY_REDIS_HOST: redis + CALITP_USER: pipeline + GOOGLE_APPLICATION_CREDENTIALS: /secrets/gtfs-feed-secrets/google_application_credentials.json diff --git a/kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-consumer.yaml b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-consumer.yaml new file mode 100644 index 0000000000..1834a6fcb1 --- /dev/null +++ b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-consumer.yaml @@ -0,0 +1,72 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gtfs-rt-archiver-consumer + labels: + name: gtfs-rt-archiver-consumer +spec: + replicas: 3 + strategy: + type: Recreate + selector: + matchLabels: + name: gtfs-rt-archiver-consumer + template: + metadata: + labels: + name: gtfs-rt-archiver-consumer + annotations: + prometheus.io/port: "9102" + prometheus.io/scrape: "true" + spec: + containers: + - name: app + image: gtfs-rt-archiver + command: ["python"] + args: ["-m", "gtfs_rt_archiver_v3.consumer", "--load-env-secrets"] + envFrom: + - configMapRef: + name: archiver-app-vars + - configMapRef: + name: archiver-channel-vars + - secretRef: + name: gtfs-feed-secrets + volumeMounts: + - name: gtfs-feed-secrets + mountPath: /secrets/gtfs-feed-secrets + resources: + requests: + memory: 512Mi + cpu: 1 + limits: + memory: 1Gi + volumes: + - name: gtfs-feed-secrets + secret: + secretName: gtfs-feed-secrets + tolerations: + - key: resource-domain + operator: Equal + value: gtfsrtv3 + effect: NoSchedule + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: resource-domain + operator: In + values: + - gtfsrtv3 + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchExpressions: + - key: name + operator: In + values: + - gtfs-rt-archiver-consumer diff --git a/kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-ticker.yaml b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-ticker.yaml new file mode 100644 index 0000000000..d28e7ab992 --- /dev/null +++ b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/gtfs-rt-archiver-ticker.yaml @@ -0,0 +1,72 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gtfs-rt-archiver-ticker + labels: + name: gtfs-rt-archiver-ticker +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + name: gtfs-rt-archiver-ticker + template: + metadata: + labels: + name: gtfs-rt-archiver-ticker + annotations: + prometheus.io/port: "9102" + prometheus.io/scrape: "true" + spec: + containers: + - name: app + image: gtfs-rt-archiver + command: ["python"] + args: ["-m", "gtfs_rt_archiver_v3.ticker", "--load-env-secrets"] + envFrom: + - configMapRef: + name: archiver-app-vars + - configMapRef: + name: archiver-channel-vars + - secretRef: + name: gtfs-feed-secrets + volumeMounts: + - name: gtfs-feed-secrets + mountPath: /secrets/gtfs-feed-secrets + resources: + requests: + memory: 512Mi + cpu: 1 + limits: + memory: 1Gi + volumes: + - name: gtfs-feed-secrets + secret: + secretName: gtfs-feed-secrets + tolerations: + - key: resource-domain + operator: Equal + value: gtfsrtv3 + effect: NoSchedule + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: resource-domain + operator: In + values: + - gtfsrtv3 + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchExpressions: + - key: name + operator: In + values: + - redis diff --git a/kubernetes/apps/manifests/gtfs-rt-archiver-v3/kustomization.yaml b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/kustomization.yaml new file mode 100644 index 0000000000..f90a45e057 --- /dev/null +++ b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/kustomization.yaml @@ -0,0 +1,5 @@ +resources: +- archiver-app-vars.yaml +- gtfs-rt-archiver-consumer.yaml +- gtfs-rt-archiver-ticker.yaml +- redis.yaml diff --git a/kubernetes/apps/manifests/gtfs-rt-archiver-v3/redis.yaml b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/redis.yaml new file mode 100644 index 0000000000..61abe4c945 --- /dev/null +++ b/kubernetes/apps/manifests/gtfs-rt-archiver-v3/redis.yaml @@ -0,0 +1,57 @@ +# Cobbled together from the following: +# https://www.containiq.com/post/deploy-redis-cluster-on-kubernetes +# maybe we should just use the bitnami chart... +--- +apiVersion: v1 +kind: Service +metadata: + name: redis +spec: + ports: + - port: 6379 + name: redis + clusterIP: None + selector: + app: redis +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: redis + labels: + app: redis +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: redis + template: + metadata: + labels: + app: redis + spec: + containers: + - name: app + image: redis:5.0.4 + resources: + requests: + memory: 512Mi + cpu: 1 + limits: + memory: 1Gi + tolerations: + - key: resource-domain + operator: Equal + value: gtfsrtv3 + effect: NoSchedule + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: resource-domain + operator: In + values: + - gtfsrtv3 diff --git a/kubernetes/apps/manifests/sftp-server/intranet-service.yaml b/kubernetes/apps/manifests/sftp-server/intranet-service.yaml new file mode 100644 index 0000000000..52817b5ea8 --- /dev/null +++ b/kubernetes/apps/manifests/sftp-server/intranet-service.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: sftp-intranet + labels: + component: sftp-server +spec: + ports: + - port: 22 + name: sftp + clusterIP: None + selector: + component: sftp-server diff --git a/kubernetes/apps/manifests/sftp-server/kustomization.yaml b/kubernetes/apps/manifests/sftp-server/kustomization.yaml new file mode 100644 index 0000000000..d3cba630b9 --- /dev/null +++ b/kubernetes/apps/manifests/sftp-server/kustomization.yaml @@ -0,0 +1,8 @@ +resources: +- workload.yaml +- intranet-service.yaml + +configMapGenerator: + - name: sftp-sshd + files: + - sshd_config diff --git a/kubernetes/apps/manifests/sftp-server/sshd_config b/kubernetes/apps/manifests/sftp-server/sshd_config new file mode 100644 index 0000000000..8f65bbda01 --- /dev/null +++ b/kubernetes/apps/manifests/sftp-server/sshd_config @@ -0,0 +1,6 @@ +Port 22 +Include /etc/ssh/sshd_config.d/*.conf +AuthorizedKeysFile .ssh/authorized_keys +Subsystem sftp internal-sftp +ForceCommand internal-sftp +ChrootDirectory /sftp diff --git a/kubernetes/apps/manifests/sftp-server/workload.yaml b/kubernetes/apps/manifests/sftp-server/workload.yaml new file mode 100644 index 0000000000..0d45e399f3 --- /dev/null +++ b/kubernetes/apps/manifests/sftp-server/workload.yaml @@ -0,0 +1,80 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: sftp-server +spec: + selector: + matchLabels: + component: sftp-server + serviceName: sftp-intranet + replicas: 1 + template: + metadata: + labels: + component: sftp-server + spec: + containers: + - name: server + image: fedora:latest + env: + - name: SFTP_UID + value: '1000' + - name: SFTP_AUTHORIZED_KEYS_SRC + value: /config/user/authorized_keys + - name: SFTP_SSHD_CONFIG_SRC + value: /config/server/sshd_config + envFrom: + - configMapRef: + name: sftp-user + ports: + - containerPort: 22 + name: sftp + volumeMounts: + - name: data + mountPath: /sftp/data + - name: user-config + mountPath: /config/user + - name: server-config + mountPath: /config/server + command: [ /bin/bash ] + args: + - -c + - | + test "$SFTP_USER" || SFTP_USER=sftp + test "$SFTP_USER_HOME" || SFTP_USER_HOME=/home/$SFTP_USER + test "$SFTP_USER_SSH_HOME" || SFTP_USER_SSH_HOME=$SFTP_USER_HOME/.ssh + dnf install -y openssh-server + groupadd -g $SFTP_UID $SFTP_USER + useradd -g $SFTP_UID -md $SFTP_USER_HOME -s /bin/bash $SFTP_USER + mkdir -p -m 0700 "$SFTP_USER_SSH_HOME" + if [[ -e $SFTP_AUTHORIZED_KEYS_SRC ]]; then + (umask 077; cp $SFTP_AUTHORIZED_KEYS_SRC "$SFTP_USER_SSH_HOME"/authorized_keys) + else + echo "warn: no $SFTP_AUTHORIZED_KEYS_SRC file; user $SFTP_USER may not be able to login" + fi + chown -R $SFTP_USER:$SFTP_USER "$SFTP_USER_SSH_HOME" + cp "$SFTP_SSHD_CONFIG_SRC" /etc/ssh/sshd_config + ssh-keygen -A + exec /usr/sbin/sshd -e -D + readinessProbe: + tcpSocket: + port: 22 + initialDelaySeconds: 5 + periodSeconds: 5 + securityContext: + fsGroup: 1000 + volumes: + - name: user-config + configMap: + name: sftp-user-config + - name: server-config + configMap: + name: sftp-sshd + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ 'ReadWriteOnce' ] + resources: + requests: + storage: 1Gi diff --git a/kubernetes/apps/overlays/base-sftp-ingest-elavon/kustomization.yaml b/kubernetes/apps/overlays/base-sftp-ingest-elavon/kustomization.yaml new file mode 100644 index 0000000000..44d37c5a72 --- /dev/null +++ b/kubernetes/apps/overlays/base-sftp-ingest-elavon/kustomization.yaml @@ -0,0 +1,6 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- ../../manifests/sftp-server +- sftp-user.yaml diff --git a/kubernetes/apps/overlays/base-sftp-ingest-elavon/sftp-user.yaml b/kubernetes/apps/overlays/base-sftp-ingest-elavon/sftp-user.yaml new file mode 100644 index 0000000000..684474a83b --- /dev/null +++ b/kubernetes/apps/overlays/base-sftp-ingest-elavon/sftp-user.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: sftp-user +data: + SFTP_USER: elavon diff --git a/kubernetes/apps/overlays/dev-sftp-ingest-elavon/kustomization.yaml b/kubernetes/apps/overlays/dev-sftp-ingest-elavon/kustomization.yaml new file mode 100644 index 0000000000..4eb9118767 --- /dev/null +++ b/kubernetes/apps/overlays/dev-sftp-ingest-elavon/kustomization.yaml @@ -0,0 +1,8 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: dev-sftp-ingest-elavon + +resources: +- ../base-sftp-ingest-elavon +- ns.yaml +- sftp-user-config.yaml diff --git a/kubernetes/apps/overlays/dev-sftp-ingest-elavon/ns.yaml b/kubernetes/apps/overlays/dev-sftp-ingest-elavon/ns.yaml new file mode 100644 index 0000000000..d5a5446865 --- /dev/null +++ b/kubernetes/apps/overlays/dev-sftp-ingest-elavon/ns.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: dev-sftp-ingest-elavon diff --git a/kubernetes/apps/overlays/dev-sftp-ingest-elavon/sftp-user-config.yaml b/kubernetes/apps/overlays/dev-sftp-ingest-elavon/sftp-user-config.yaml new file mode 100644 index 0000000000..e37d9d4747 --- /dev/null +++ b/kubernetes/apps/overlays/dev-sftp-ingest-elavon/sftp-user-config.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: sftp-user-config +data: + authorized_keys: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvdNntkalKcXm7lkZzSZlsEtXVyPfQIu/D5HXZbpXIkF1qMr5vGehACgoqe1IeZY/Y/u5M4uJVcrc5S7OZc5rX6E+A30Zjhck4HPfoHjPf8jRfwvZ9EmV5BncJuqCytkWx6oDi1boS8IchKh5H+YslRKe9DYD38igB5aMfrTLjsTHfAm76qD4VtuqhPuNMwK2C66cMVFhIqm8g6BcurncdiVWyP8OcekZw+A527N5FeGw989ZPRhS/AMeZpCKed2z8n04wFgXezIeA6aWMKbh5WEKEe8/O8xFnUTVR1JPuGGZ8OFDXNVUBegiZo3XvaH/RGiBQPMdXUU14Sf6+vrqx' # jlott diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/archiver-channel-vars.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/archiver-channel-vars.yaml new file mode 100644 index 0000000000..d0bb9fae59 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/archiver-channel-vars.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: archiver-channel-vars +data: + AIRFLOW_ENV: development + CALITP_BUCKET__AIRTABLE: "gs://test-calitp-airtable" + CALITP_BUCKET__GTFS_RT_RAW: "gs://test-calitp-gtfs-rt-raw" + HUEY_CONSUMER_WORKERS: "16" diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/consumer.patch.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/consumer.patch.yaml new file mode 100644 index 0000000000..a11ad89357 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/consumer.patch.yaml @@ -0,0 +1,12 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gtfs-rt-archiver-consumer +spec: + replicas: 1 + template: + spec: + containers: + - name: app + imagePullPolicy: Always diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/kustomization.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/kustomization.yaml new file mode 100644 index 0000000000..af7c19db53 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/kustomization.yaml @@ -0,0 +1,17 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: gtfs-rt-v3-dev + +resources: +- ns.yaml +- archiver-channel-vars.yaml +- ../gtfs-rt-archiver-v3-release + +patches: +- consumer.patch.yaml +- ticker.patch.yaml + +images: +- name: 'gtfs-rt-archiver' + newName: 'ghcr.io/cal-itp/data-infra/gtfs-rt-archiver' + newTag: '3.1.7' diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ns.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ns.yaml new file mode 100644 index 0000000000..820a8e7586 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ns.yaml @@ -0,0 +1,5 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: gtfs-rt-v3-dev diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ticker.patch.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ticker.patch.yaml new file mode 100644 index 0000000000..76d96f8cbd --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-dev/ticker.patch.yaml @@ -0,0 +1,11 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gtfs-rt-archiver-ticker +spec: + template: + spec: + containers: + - name: app + imagePullPolicy: Always diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/archiver-channel-vars.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/archiver-channel-vars.yaml new file mode 100644 index 0000000000..891f464dc9 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/archiver-channel-vars.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: archiver-channel-vars +data: + AIRFLOW_ENV: cal-itp-data-infra + CALITP_BUCKET__AIRTABLE: "gs://calitp-airtable" + CALITP_BUCKET__GTFS_RT_RAW: "gs://calitp-gtfs-rt-raw" + HUEY_CONSUMER_WORKERS: "16" diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/kustomization.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/kustomization.yaml new file mode 100644 index 0000000000..e42a68d453 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/kustomization.yaml @@ -0,0 +1,13 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: gtfs-rt-v3 + +resources: +- ns.yaml +- archiver-channel-vars.yaml +- ../gtfs-rt-archiver-v3-release + +images: +- name: 'gtfs-rt-archiver' + newName: 'ghcr.io/cal-itp/data-infra/gtfs-rt-archiver' + newTag: '3.1.7' diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/ns.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/ns.yaml new file mode 100644 index 0000000000..378f7a9bf0 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod/ns.yaml @@ -0,0 +1,5 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: gtfs-rt-v3 diff --git a/kubernetes/apps/overlays/gtfs-rt-archiver-v3-release/kustomization.yaml b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-release/kustomization.yaml new file mode 100644 index 0000000000..9ace0ed667 --- /dev/null +++ b/kubernetes/apps/overlays/gtfs-rt-archiver-v3-release/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- '../../manifests/gtfs-rt-archiver-v3' diff --git a/kubernetes/apps/overlays/prod-sftp-ingest-elavon/internet-service.yaml b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/internet-service.yaml new file mode 100644 index 0000000000..83d5026c5b --- /dev/null +++ b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/internet-service.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: sftp-internet + labels: + component: sftp-server +spec: + type: LoadBalancer + ports: + - port: 22 + name: sftp + selector: + statefulset.kubernetes.io/pod-name: sftp-server-0 diff --git a/kubernetes/apps/overlays/prod-sftp-ingest-elavon/kustomization.yaml b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/kustomization.yaml new file mode 100644 index 0000000000..001ae62ac0 --- /dev/null +++ b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/kustomization.yaml @@ -0,0 +1,15 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: prod-sftp-ingest-elavon + +resources: +- ../base-sftp-ingest-elavon +- ns.yaml +- sftp-user-config.yaml +- internet-service.yaml + +patches: +- path: patch-volume-size.json + target: + kind: StatefulSet + name: sftp-server diff --git a/kubernetes/apps/overlays/prod-sftp-ingest-elavon/ns.yaml b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/ns.yaml new file mode 100644 index 0000000000..b903e246c1 --- /dev/null +++ b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/ns.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: prod-sftp-ingest-elavon diff --git a/kubernetes/apps/overlays/prod-sftp-ingest-elavon/patch-volume-size.json b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/patch-volume-size.json new file mode 100644 index 0000000000..2e52a18738 --- /dev/null +++ b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/patch-volume-size.json @@ -0,0 +1,7 @@ +[ + { + "op": "replace", + "path": "/spec/volumeClaimTemplates/0/spec/resources/requests/storage", + "value": "50Gi" + } +] diff --git a/kubernetes/apps/overlays/prod-sftp-ingest-elavon/sftp-user-config.yaml b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/sftp-user-config.yaml new file mode 100644 index 0000000000..8eefa36acf --- /dev/null +++ b/kubernetes/apps/overlays/prod-sftp-ingest-elavon/sftp-user-config.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: sftp-user-config +data: + authorized_keys: ''