Merge branch 'main' into agent-runner-update-refactor

.github/ISSUE_TEMPLATE/bug-template.md

@@ -0,0 +1,46 @@
+---
+name: Bug template
+about: Template that will help you to submit a visible and actionable bug report.
+title: 'Bug: '
+labels: t/bug
+assignees: ''
+
+---
+
+## Environment
+
+<!--
+e.g. local development, staging, production
+-->
+
+## Steps to reproduce
+
+<!--
+If you can, try to reproduce the bug. If e.g. this occurred in production under
+rare circumstances, indicate as such. Any related events can be included in the
+later "Actual result" section.
+
+If this occurred while you were testing, but you don't have logs for it, say
+what kind of testing you were doing! Provide any scripts / commands that were
+relevant.
+-->
+
+## Expected result
+
+<!--
+What did you expect to happen?
+
+Try to avoid vague statements like "operation should succeed". Instead, say more
+specifically what the success should look like (e.g. "VM should scale up").
+-->
+
+## Actual result
+
+<!--
+Remember to include logs, graphs, and other supporting information to show
+how/why things did not function as expected.
+-->
+
+## Other logs, links
+
+- ...

.github/ISSUE_TEMPLATE/epic-template.md

@@ -0,0 +1,51 @@
+---
+name: Epic Template
+about: A set of related tasks contributing towards specific outcome, comprising of
+  more than 1 week of work.
+title: 'Epic: '
+labels: t/Epic
+assignees: ''
+
+---
+
+## Motivation
+
+<!--
+Why is this worthwhile?
+
+Provide any relevant context, either written here or with links to slack, other issues, etc.
+-->
+
+## DoD
+
+<!--
+DoD = "definition of done".
+
+Put any ideas you have for how the changes should look *externally* - i.e. to a
+user of the system or component that needs changing. For each idea, give the
+DoD: "what external changes need to be made for the feature to be complete?"
+-->
+
+
+## Implementation ideas
+
+<!--
+If you have ideas about how the feature could be implemented, please share!
+
+If not, leave this section here so that it can be filled in by others.
+-->
+
+TODO
+
+
+## Tasks
+
+```[tasklist]
+- [ ] ...
+- [ ] List tasks as they're created for this Epic
+```
+
+
+## Other related tasks, Epics, and links
+
+-

.github/ISSUE_TEMPLATE/feature-request-template.md

@@ -0,0 +1,37 @@
+---
+name: Feature request template
+about: Template that will help you to submit a visible and actionable feature request.
+title: 'Feature: '
+labels: t/feature
+assignees: ''
+
+---
+
+## Problem description / Motivation
+
+<!--
+Feature requests are for when something isn't a bug (because there's nothing
+*unexpected* per se), but there's still room for improvement.
+
+This area is the most important part! Sometimes we think one solution is best,
+but gradually realize that another is simpler. This is only possible if the
+problem description is clear about what is *actually* required.
+-->
+
+## Feature idea(s) / DoD
+
+<!--
+DoD = "definition of done".
+
+Put any ideas you have for how the changes should look *externally* - i.e. to a
+user of the system or component that needs changing. For each idea, give the
+DoD: "what external changes need to be made for the feature to be complete?"
+-->
+
+## Implementation ideas
+
+<!--
+If you have ideas about how the feature could be implemented, please share!
+
+If not, feel free to remove this section.
+-->

.github/workflows/e2e-test.yaml

@@ -8,6 +8,7 @@ on:
 jobs:
   e2e-tests:
     strategy:
+      fail-fast: false
       matrix:
         cluster:
           - k3d
@@ -64,8 +65,11 @@ jobs:
       - run: make ${{ matrix.cluster }}-setup
 
       - run: make deploy
+        timeout-minutes: 10
       - run: make example-vms
+        timeout-minutes: 10
       - run: make e2e
+        timeout-minutes: 15
 
       - name: Get k8s logs and events
         if: always()

.github/workflows/release.yaml

@@ -7,7 +7,8 @@ on:
 env:
   AGENT_IMAGE: "neondatabase/autoscaler-agent"
   SCHED_IMAGE: "neondatabase/autoscale-scheduler"
-  INFORMANT_IMAGE: "neondatabase/vm-informant"
+  MONITOR_IMAGE: "neondatabase/vm-monitor"
+  VM_MONITOR_BRANCH: "main"
 
   KUSTOMIZE_VERSION:        "4.5.7"
   CONTROLLER_TOOLS_VERSION: "0.10.0"
@@ -50,7 +51,10 @@ jobs:
         run: go build ./...
 
       - name: build binaries
-        run:  make VM_INFORMANT_IMG=${{ env.INFORMANT_IMAGE }}:${{ steps.get_vcs_info.outputs.version }} build
+        run: |
+          make \
+            VM_MONITOR_IMG=${{ env.MONITOR_IMAGE }}:${{ steps.get_vcs_info.outputs.version }} \
+            build
 
       - name: docker - install qemu
         uses: docker/setup-qemu-action@v2
@@ -119,16 +123,17 @@ jobs:
           build-args: |
             GIT_INFO=${{ steps.get_vcs_info.outputs.git_info }}
 
-      - name: build and push vm-informant image
+      - name: build and push vm-monitor image
         uses: docker/build-push-action@v3
         with:
           context: .
           platforms: linux/amd64
           push: true
-          file: build/vm-informant/Dockerfile
-          tags: ${{ env.INFORMANT_IMAGE }}:${{ steps.get_vcs_info.outputs.version }}
+          file: build/vm-monitor/Dockerfile
+          tags: ${{ env.MONITOR_IMAGE }}:${{ steps.get_vcs_info.outputs.version }}
           build-args: |
             GIT_INFO=${{ steps.get_vcs_info.outputs.git_info }}
+            BRANCH=${{ env.VM_MONITOR_BRANCH }}
 
       - name: render kubernetes resources
         uses: stefanprodan/kube-tools@v1
@@ -149,12 +154,12 @@ jobs:
             kustomize build deploy/scheduler > rendered_manifests/autoscale-scheduler.yaml
             kustomize build deploy/agent > rendered_manifests/autoscaler-agent.yaml
 
-      # Because we want a docker image for the VM informant, the easiest way for us to also provide
+      # Because we want a docker image for the VM monitor, the easiest way for us to also provide
       # a binary is by just extracting it from the container image itself.
-      - name: extract vm-informant binary
+      - name: extract vm-monitor binary
         run: |
-          ID=$(docker create ${{ env.INFORMANT_IMAGE }}:${{ steps.get_vcs_info.outputs.version }})
-          docker cp $ID:/usr/bin/vm-informant bin/vm-informant
+          ID=$(docker create ${{ env.MONITOR_IMAGE }}:${{ steps.get_vcs_info.outputs.version }})
+          docker cp $ID:/usr/bin/vm-monitor bin/vm-monitor
           docker rm -f $ID
 
       - name: build and push cluster-autoscaler image
@@ -173,10 +178,11 @@ jobs:
           files: |
             bin/vm-builder
             bin/vm-builder-generic
-            bin/vm-informant
+            bin/vm-monitor
             rendered_manifests/autoscale-scheduler.yaml
             rendered_manifests/autoscaler-agent.yaml
             rendered_manifests/neonvm.yaml
             rendered_manifests/multus.yaml
             rendered_manifests/multus-eks.yaml
             rendered_manifests/whereabouts.yaml
+            deploy/vmscrape.yaml

.golangci.yml

@@ -9,7 +9,7 @@ run:
 issues:
   exclude:
     # ChanMutex contains only a channel, which *is* safe to copy
-    - 'copylocks: return copies lock value: github\.com/neondatabase/autoscaling/pkg/util\.ChanMutex'
+    - 'copylocks: .* copies lock value.*: github\.com/neondatabase/autoscaling/pkg/util\.ChanMutex'
 
 output:
   format: colored-line-number
@@ -54,8 +54,8 @@ linters-settings:
     exclude:
       - '^net/http\.(Client|Server)'
       - '^net\.TCPAddr$'
-      # metav1.{CreateOptions,GetOptions,ListOptions,WatchOptions,PatchOptions}
-      - '^k8s\.io/apimachinery/pkg/apis/meta/v1\.(Create|Get|List|Watch|Patch)Options$'
+      # metav1.{CreateOptions,GetOptions,ListOptions,WatchOptions,PatchOptions,DeleteOptions}
+      - '^k8s\.io/apimachinery/pkg/apis/meta/v1\.(Create|Get|List|Watch|Patch|Delete)Options$'
       - '^k8s\.io/apimachinery/pkg/apis/meta/v1\.ObjectMeta$'
       - '^k8s\.io/apimachinery/pkg/api/resource\.Quantity$'
       - '^github.com/prometheus/client_golang/prometheus(/.*)?\.\w+Opts$'

ARCHITECTURE-network-diagram.org

@@ -32,29 +32,29 @@ awk '/#\+BEGIN_SRC/{flag=1;next}/#\+END_SRC/{flag=0}flag' ARCHITECTURE-network-d
                   |                      +---------|  autoscaler agent  |
                   |                                |                    |
                   |                                | (one per K8s node) |
-                  |                                +-----------------*--+
-                  |                                  |           |   ^ random port
-                  |                                  |           |   |  (per VM)
-                  |                                  |           |   |
-+=================|==================================|===========|===|======+
-: K8s pod         |                                  |           |   |      :
-:             QMP |                                  |           |   |      :
-:           20183 V                                  |           |   |      :
-: +---------------*----------------------------------|-----------|---|---+  :
-: |                                                  |           |   |   |  :
-: | QEMU process                                     |           |   |   |  :
-: |                                                  |           |   |   |  :
-: |                                                  |           |   |   |  :
-: |            compute_ctl    postgres       metrics | informant |   |   |  :
-: |               mgmt API    postgres    prometheus | informant |   |   |  :
-: |                   3080        5432          9100 V     10301 V   |   |  :
+                  |                                +--------------------+
+                  |                                  |           |
+                  |                                  |           |    
+                  |                                  |           |    
++=================|==================================|===========|==========+
+: K8s pod         |                                  |           |          :
+:             QMP |                                  |           |          :
+:           20183 V                                  |           |          :
+: +---------------*----------------------------------|-----------|-------+  :
+: |                                                  |           |       |  :
+: | QEMU process                                     |           |       |  :
+: |                                                  |           |       |  :
+: |                                                  |           |       |  :
+: |            compute_ctl    postgres       metrics |   monitor |       |  :
+: |               mgmt API    postgres    prometheus | websocket |       |  :
+: |                   3080        5432          9100 V     10301 V       |  :
 : +------------------------*-----------*-------------*-----------*-------+  :
 : | VM                                                                   |  :
 : |                                                                      |  :
 : | Inside the VM runs:                                                  |  :
 : |    - compute_ctl (listens on port 3080)                              |  :
+: |    - VM monitor (port 10301 via websocket)                           |  :
 : |    - Postgres (port 5432)                                            |  :
-: |    - VM informant (port 10301)                                       |  :
 : |    - vector (metrics on port 9100)                                   |  :
 : |                                                                      |  :
 : +----------------------------------------------------------------------+  :