Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

launching Kafka components using StatefulSet insted of Deployments #744

Open
ElyRonen opened this issue Jan 20, 2022 · 2 comments · May be fixed by #751
Open

launching Kafka components using StatefulSet insted of Deployments #744

ElyRonen opened this issue Jan 20, 2022 · 2 comments · May be fixed by #751
Labels
community question Further information is requested

Comments

@ElyRonen
Copy link

Hello,
My name is Ely, I am a customer success engineer at Statehub, we provide a global PersistentVolumes allowing stateful applications on Kubernetes to access their PVs across clusters (regions and clouds) to access

I’m currently creating a blog post about launching Kafka cluster using Koperator while leveraging our technology for a stateful Kafka cluster that will allow fast failover between one Kubernetes cluster to the other.

While testing your guide and launching a fully functional example Kafka cluster I noticed that all the components launched are using deployments and not StatefulSet, thus providing the PVC a name with a suffix that is auto-generated.

Is it possible to deploy the Kafka and Zookeeper pods using a statefulSet with PersistentVolumeClaims instead of a deployment?
Providing a global name that will be recognized across clusters will allow us to launch a new Kafka cluster using the data and volumes from the previous cluster. in order for that to happen, we need zookeeper and Kafka volumes to have a unique and global ID and not be generated.
I browsed through the repository but couldn’t find an example of a config that will use STS instead of deployment.

I’d appreciate your opinion,
Thanks in advance,
Ely
@baluchicken baluchicken added community question Further information is requested labels Jan 21, 2022
@adamantal
Copy link
Contributor

adamantal commented Jan 21, 2022 •
edited
Loading

Hi Ely,

Thanks for opening this issue! We are happy that you're using the operator.

Firstly, As opposed to other Kafka operators, Koperator runs the Kafka brokers as bare Kubernetes Pods instead of using Deployments or StatefulSets.
Secondly, the Koperator takes into account any volume related constraint that you provide through the KafkaCluster CR configuration. More concretely you can set the properties of the PVC generated for each broker in brokerConfig.storageConfigs.pvcSpec.
As an example please check simplekafkacluster.yaml with the default settings:

      storageConfigs:
        - mountPath: "/kafka-logs"
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi

If you have any further questions, please let us know. We'd also be happy if you can send us the blog post, when you're finished with it.

Thanks,
Adam

zaidenwerg pushed a commit to zaidenwerg/koperator that referenced this issue Jan 23, 2022
ISSUE-744 PVCs are created with generated names
0f41e14 
Fixes banzaicloud#744

Create PersistentVolumeClaim with Name instead of Generated Name
Will allow kafka k8s cluster switch over using global storage classes that span multiple kubernetes clusters.
@zaidenwerg zaidenwerg linked a pull request Jan 23, 2022 that will close this issue
Open
5 tasks
@baluchicken
Copy link
Member

baluchicken commented Jan 25, 2022 •
edited
Loading

Hi @ElyRonen please look my answer here how can you create manually your own PV and PVC for the operator.
#572 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community question Further information is requested
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Pvcglobaluniqeid zaidenwerg/koperator
3 participants
@baluchicken @adamantal @ElyRonen