Sidecar usage.

[PaulRudin]

I was browsing the sidecar documentation, and I'm not quite clear in the intended usage pattern. If I want to share memory between containers from two pods on the same node, presumably I only need and want one vineyard process on that node to keep hold of the shared memory. So the expectation would be that one of those pods uses the sidecar, and the other just connect via a local socket? There would be no point in both pods using a sidecar?

[sighingnow]

There would be no point in both pods using a sidecar?

The answer is yes. The sidecar is designed to simply the deployment of vineyardd cluster along with users' applications on Kubernetes. One of the possible use cases is the application consists of more than one pod (e.g., different kinds of containerized analytical engines that want to share the same piece of large immutable data), where the sidecar could be helpful to simply the deployment process.

[dashanji]

Hi @PaulRudin, would you please tell us what your application or scenario is and what you want to do with the sidecar? In this way, we can continue to improve this feature and serve your application or scenario better.

Looking forward to your reply.

[PaulRudin]

At the moment I have two daemonsets. One for vineyard and one for a "producer" application, that continually recomputes some data. The producer puts objects in vineyard. On the same node I have a few replicas of a consumer application, which read data from vineyard. ( I use redis to propagate information about updates). The data consists of a few numpy arrays of the order of a few gigbytes.

This setup works fine - I'm just use affinities, taints and tolerations to ensure things end up on nodes together.

It occurred to me that it might be more straightforward just to use the sidecar along with the "producer" daemonset.

[PaulRudin]

If nothing else, it would save the chore of ensuring that the two daemonsets are always co-located on the same nodes.

[dashanji]

Yeah, deploying the vineyard through the sidecar is very suitable for your scenario. Also, we have a command line tool to simplify the installation of the sidecar, please refer to https://github.com/v6d-io/v6d/blob/main/k8s/cmd/README.md#vineyardctl-inject. If you have any questions about it, feel free to report them here. We'll reply as soon as possible.

[PaulRudin]

I had a go at changing my existing config to using the sidecar today. I'm not sure what I'm missing, but the sidecar wasn't injected into the producer daemonset's pods.

In any case I have a separate concern. My current setup relies on use host volume mounts for the socket to be accessible to both the producer daemonset's pod and the pods for the consumer application. Is this going to work with the sidecar?

[dashanji]

@PaulRudin

I had a go at changing my existing config to using the sidecar today. I'm not sure what I'm missing, but the sidecar wasn't injected into the producer daemonset's pods.

Would you mind listing your workload yaml here for debugging?

In any case I have a separate concern. My current setup relies on use host volume mounts for the socket to be accessible to both the producer daemonset's pod and the pods for the consumer application. Is this going to work with the sidecar?

If you want to access the socket with several applications, I recommend deploying the vineyard cluster independently, so that you could deploy as many applications as you want on the vineyard cluster with the host volume.

There are some tools to simplify the above process.

• Deploy the vineyard deployment via vineyardctl deploy vineyard-deployment. If you want to deploy vineyard cluster as a daemonset, you could set the replicas as the number of kubernetes nodes.
• Add the host volume(the same as the vineyard deployment) to your applications yaml.
• Schedule your applications on the vineyard node so that they can access the vineyard socket on the same host via vineyardctl schedule.

[PaulRudin]

I'm now looking at this again. I thought I'd start again, this time I'm trying with the default sidecar, rather than the custom one I had previously. So I added the label and annotation to the deployment:

...
  template:
    metadata:
      annotations:
        sidecar.v6d.io/name: default
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: plfpl-testing-load
        build: 52cfaaa31
        ismfg: rtweb
        sidecar.v6d.io/enabled: "true"
...

And now when I describe the replicaset I see:

Warning  FailedCreate  67s (x22 over 20m)  replicaset-controller  Error creating: admission webhook "mpod.sidecar.kb.io" denied the request: Sidecar.k8s.v6d.io "app.kubernetes.io/instance-plfpl │
│ -testing-load-default-sidecar" is invalid: metadata.name: Invalid value: "app.kubernetes.io/instance-plfpl-testing-load-default-sidecar": a lowercase RFC 1123 subdomain must consist of lower case │
│  alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[ │
│ -z0-9[])?)*') 

So it seems that the operator is deriving an invalid name?

[PaulRudin]

I edited the deployment to get rid of the " app.kubernetes.io/instance:" label. Now I see some pods being created, but they don't have the sidecar injected. Here's the deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "88"
    meta.helm.sh/release-name: plfpl-testing-load
    meta.helm.sh/release-namespace: plfpl-testing-load
  creationTimestamp: "2023-05-13T06:56:45Z"
  generation: 116
  labels:
    app.kubernetes.io/managed-by: Helm
    build: 52cfaaa31
    helm.sh/chart: ism_game-0.1.0
  name: ismfg-rtweb
  namespace: plfpl-testing-load
  resourceVersion: "1090399197"
  uid: 262d53cf-f94c-4a13-967f-a9000efcfd96
spec:
  progressDeadlineSeconds: 600
  replicas: 8
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      ismfg: rtweb
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 75%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        sidecar.v6d.io/name: default
      creationTimestamp: null
      labels:
        build: 52cfaaa31
        ismfg: rtweb
        sidecar.v6d.io/enabled: "true"
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                - ismfg-staging-realtime-web-nodepool
      automountServiceAccountToken: false
      containers:
      - env:
        - name: URL_ROOT
          value: /rt/api
        - name: REDIS_URL
          value: redis://10.89.120.196:6379
        - name: LOGLEVEL
          value: DEBUG
        - name: ENABLE_SENTRY
          value: "0"
        - name: SENTRY_TRACES_SAMPLE_RATE
          value: "0.1"
        - name: SENTRY_PROFILES_SAMPLE_RATE
          value: "0.1"
        - name: SENTRY_ENVIRONMENT
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: eu.gcr.io/ismfg-staging/rtweb_plfpl-testing-load:52cfaaa31
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 1
          httpGet:
            httpHeaders:
            - name: Host
              value: KubernetesProbe
            path: /healthz
            port: 8000
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: ismfg-web
        ports:
        - containerPort: 8000
          protocol: TCP
        readinessProbe:
          failureThreshold: 5
          httpGet:
            httpHeaders:
            - name: Host
              value: KubernetesProbe
            path: /ready
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 10
        resources:
          limits:
            cpu: "1"
            memory: 500Mi
          requests:
            cpu: "1"
            memory: 500Mi
        startupProbe:
          failureThreshold: 20
          httpGet:
            httpHeaders:
            - name: Host
              value: KubernetesProbe
            path: /ready
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: rt_web
        operator: Exists

[dashanji]

Thanks for the feedback, It looks like a bug and I will check it ASAP.

[PaulRudin]

If you want to access the socket with several applications, I recommend deploying the vineyard cluster independently, so that you could deploy as many applications as you want on the vineyard cluster with the host volume.

When you say "independently" do you mean without using the operator?

[dashanji]

I think it depends on whether you need to manage the lifecycle of vineyard cluster. If you want to regard vineyard as a service that will last on the kubernetes cluster for a long time, it's better to deploy an operator for easy upgrading, etc. In a word, it mainly depends on whether you like to operate CRDs or YAMLs😊😊😊.

[PaulRudin]

yeah, I'd prefer to regard vineyard as a long lived service. But I just want to have a single vineyard instance on a given number of nodes, each with several clients running on the same node, and to this end I've used a daemonset with appropriate taints and tolerations previously. If I'm going to use the operator, how do I make the cluster specification so that I get vineyard pods on just the nodes that I need?

[dashanji]

it's a good feature, we could add the taints and tolerations as part fields of the Vineyardd CRD.

[PaulRudin]

yeah - maybe in general it's possible to have a specification of fields to merge into the created pods, that way you wouldn't need to anticipate exactly the fields that people might want?

[dashanji]

Good idea, we could add a field that allows users to specify any deployment fields including taints and tolerations, etc.

[dashanji]

@PaulRudin Hi, I have fixed the bug via #1438 and pushed the image to the Docker Hub, you could try the newest vineyard operator image.

[PaulRudin]

I'll take a look presently - although based on this discussion I've moved away from using the operator and sidecar anyway. I've made a daemonset for vineyardd and used a nodeSelector to ensure that I only get pods on particular nodes.

[dashanji]

Ok, no problem, then you need to deploy your applications on the same nodes with Vineyard manually.

I just opened issue #1439 which addresses such demands. FYI.

[dashanji]

Hi @PaulRudin, we have posted a blog about how to use data sharing with Vineyard on Kubernetes, which may simplify your usage with Vineyard. Please don't hesitate to reach out if you have any further questions or concerns.

https://v6d.io/tutorials/kubernetes/data-sharing-with-vineyard-on-kubernetes.html