Skip to content

Latest commit

 

History

History
199 lines (128 loc) · 6.81 KB

INSTALL.md

File metadata and controls

199 lines (128 loc) · 6.81 KB
Raw

FADI - Installation

This page describes how to install the FADI platform

  1. on a laptop/workstation, using Minikube
  2. on a generic Kubernetes cluster
  3. on Google Kubernetes Engine (GKE)

The last sections describes how to automate the deployment using Gitlab-CI, and configure various parts of FADI (user management, ...).

Once FADI is installed, head to the user guide

The deployment of the FADI stack is achieved with:

1. Local installation

This type of installation provides a quick way to test the platform, and also to adapt it to your needs.

1.1. Prerequisites

The following tools need to be installed on the host system:

Make sure that the computer you are installing FADI to is powerful enough (currently, 6vcpu's and 12GB RAM are needed for comfortable use).

1.2. Local deployment

Delete any previously created Minikube installation:

minikube delete

Start Minikube:

minikube start --cpus 6 --memory 12288 --disk-size=40GB

Notes:

  • the vm driver can be specified as such: --vm-driver kvm2 or --vm-driver virtualbox
  • a partial FADI stack would need less ressources, edit the config file accordingly to disable services that are not needed.

To get the Kubernetes dashboard, type:

minikube dashboard

This will open a browser window with the Kubernetes Dashboard, it should look like this:

Minikube initial dashboard

Clone this repository:

git clone https://github.com/cetic/fadi.git fadi
cd fadi

Note for Mac users : you need to change the network interface in the Minikube vm: in the VirtualBox GUI, go to minikube->Configuration->Network->Interface 1->advanced and change Interface Type to PCnet-FAST III (the minikube vm should be shut down in order to be able to change the network interface: minikube stop

Launch the Helm script, this will deploy all the FADI services on the Minikube cluster (and may take some time).

kubectl config set-context minikube
minikube addons enable ingress
cd helm
# you can edit values.yaml file to customise the stack
./deploy.sh
# see deploy.log for connection information to the various services

You can check everything is deploying/-ed in the Kubernetes dashboard:

Kubernetes FADI dashboard

To access a service in your browser, type for instance:

minikube service list
minikube service -n fadi fadi-nifi

To access a service in your browser, you can also setup entries in your /etc/hosts file to route http://fadi.minikube to our cluster.

In production you would want to set up real DNS entries. You can get the IP address of your minikube instance by running minikube ip:

echo "$(minikube ip) fadi.minikube" | sudo tee -a /etc/hosts

Reply this operation for all the different services. For instance, for superset:

echo "$(minikube ip) superset.fadi.minikube" | sudo tee -a /etc/hosts

You can list all the addresses by typing:

kubectl get ingress -n fadi

We should now be able to visit the different services behind http://fadi.minikube. For instance, you can access Grafana in the browser at http://grafana.fadi.minikube.

To update the FADI stack, re-type:

cd helm
./deploy.sh

To delete the FADI stack, type:

cd helm
./teardown.sh

2. Deployment on a generic Kubernetes cluster

First, clone this repository.

git clone https://github.com/cetic/fadi.git fadi
cd fadi

Set your Kubernetes context:

kubectl config set-context <your-k8s-context>

Finally, you can deploy the full FADI stack by typing:

cd helm
./deploy.sh

Note that depending on your workstation size and network connection, this could take some time (at least 5 minutes)

Now that you have a sandbox with FADI on your workstation, you can try it with a simple use case

3. Deployment on GKE

GKE is a managed Kubernetes offer by the Google Cloud Platform (GCP).

"Kubernetes Engine is a managed, production-ready environment for deploying containerized applications. It brings our latest innovations in developer productivity, resource efficiency, automated operations, and open source flexibility to accelerate your time to market."

The creation of a GKE environment can be done with Terraform or manually.

See the Terraform scripts for the creation of the Kubernetes cluster here and its documentation here.

To manually create a Kubernetes cluster (GKE):

  1. Visit the Google Kubernetes Engine menu in GCP Console.
  2. Click Create cluster.
  3. Choose the Standard cluster template or choose an appropriate template for your workload.
  4. From the Cluster Version drop-down menu, select the desired GKE version to run in the cluster.

It is also possible to create the Kubernetes cluster in command line, see: https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster

4. Troubleshooting

  • Installation logs are located in the helm/deploy.log file.
  • Enable local monitoring in minikube: minikube addons enable metrics-server

5. Continuous integration (CI) and deployment (CD)

See .gitlab-ci.sample.yml for an example CI setup with Gitlab-CI.

6. Additional configuration

See the user management documentation for additional information on how to configure FADI user identification and authorization (LDAP, RBAC, ...).