diff --git a/Vagrant-Minio-Infra/README.md b/Vagrant-Minio-Infra/README.md new file mode 100644 index 0000000..57d2cd9 --- /dev/null +++ b/Vagrant-Minio-Infra/README.md @@ -0,0 +1,313 @@ +# Minio Cluster With Vagrant + +The `qemu/kmv` virtualization and `vagrant` need to be installed in Host. + +## Download `Minio` Package + +Download `mino` package in Host, It will be copied to Vagrant VMs. + +```bash + curl -L https://dl.min.io/server/minio/release/linux-amd64/minio_20240501011110.0.0_amd64.deb -o /tmp/minio_20240501011110.0.0_amd64.deb +``` + + > For the latest version check the Minio Download page for Linux [Minio-Download-Page][1] + > + > If You can't download , set proxy for `curl` `--proxy "socks5h://:"` + > +example : + + ```bash + curl --proxy "socks5h://127.0.0.1:10808" -L https://example.com/package.deb -o /path/to/package + ``` + +## Vagrant + +Vagrant VMs IPs : + +> It Depends On Host Virtualization `qemu/kvm` IP Range. it could be changed. +> +> To find the ip range (by default the link name is `virbr0`) , invoke `ip addr show virbr0`. + +- 192.168.122.31 +- 192.168.122.32 +- 192.168.122.33 +- 192.168.122.34 + +### Vagrantfile + +This Vagrantfile is for `libvirt` via `qemu/kvm` Virtualization. + +Please Read the Comments in Vagrantfile before applying it. +Invoke `vagrant up` and wait all VMs be provisioned. + +```vagrantfile +# Latest Version Of Minio Is Present in Host +MINIO_DEB_LATEST_VERSION="minio_20240501011110.0.0_amd64.deb" + +Vagrant.configure("2") do |config| + config.vm.box = "generic/ubuntu2204" + config.ssh.insert_key = false + config.vm.synced_folder ".", "/vagrant", disabled: true + # Copy Minio-Package From Host To VMs + minio_deb_package_path = "/tmp/#{MINIO_DEB_LATEST_VERSION}" + config.vm.provision "file", source: minio_deb_package_path , destination: minio_deb_package_path + # Step by Step Shell commands to prepare the VMs + # The Vagrant User is in Sudoer Group by default. + config.vm.provision "shell", inline: <<-SHELL + # Add Your Host SSH Key To VMs to connect them via ssh + echo '' >> /home/vagrant/.ssh/authorized_keys + # Changing Mod and OwnerShip of authorized key to be readable just for vagrant user + chmod 600 /home/vagrant/.ssh/authorized_keys + chown vagrant:vagrant /home/vagrant/.ssh/authorized_keys + # Adding Proxy Server Address for Apt Because of Filterring If You Need because Of Not Free Internet + # If You Have Free Internet , Comment below line + echo 'Acquire::http::Proxy "socks5h://:";' >> /etc/apt/apt.conf.d/proxy + # Update And Upgrade VMs + sudo apt update -y && sudo apt upgrade -y + # Adding Minio servers IP to /ets/host to implement DNS Server. + # The Static IPs which going to be assigned to VMs. + sudo echo -en "192.168.122.31 minio-01\n192.168.122.32 minio-02\n192.168.122.33 minio-03\n192.168.122.34 minio-04\n" >> /etc/hosts + # Adding User and group for minio + sudo useradd -M -r -s /usr/sbin/nologin minio-user + # changing TimeZone To Asia/Tehran + sudo timedatectl set-timezone Asia/Tehran + # Installing Minio Which it's package is downloaded in host and copied to VMs. + sudo dpkg -i #{minio_deb_package_path } + # Creating Directory For Mounting disks in vms from their /etc/fstab after provisioning the VMs. + sudo mkdir /mnt/disk{1..2} + SHELL + + config.vm.provider :libvirt do |libvirt| + libvirt.memory = 2048 + libvirt.cpus = 2 # Set the number of CPUs + libvirt.storage :file, :size => '2G', :type => 'qcow2' # Additional Disk + libvirt.storage :file, :size => '2G', :type => 'qcow2' + # v.cpusize = 1024 # Set the CPU execution cap (in percentage) + end + + # Minio Server 1 + config.vm.define "Minio-Server-1" do |app1| + app1.vm.hostname = "minio-01" + app1.vm.network :private_network, type: "static", ip: "192.168.122.31", bridge: "virbr0" + end + + # Minio Server 2 + config.vm.define "Minio-Server-2" do |app2| + app2.vm.hostname = "minio-02" + app2.vm.network :private_network, type: "static", ip: "192.168.122.32", bridge: "virbr0" + end + + # Minio Server 3 + config.vm.define "Minio-Server-3" do |app3| + app3.vm.hostname = "minio-03" + app3.vm.network :private_network, type: "static", ip: "192.168.122.33", bridge: "virbr0" + end + + # Minio Server 4 + config.vm.define "Minio-Server-4" do |app4| + app4.vm.hostname = "minio-04" + app4.vm.network :private_network, type: "static", ip: "192.168.122.34", bridge: "virbr0" + end +end +``` + +## Connect To VMs + +There is two way to connect to Provisioned VMs: + +1 - `ssh vagrant@VM-IP` + > The Host public ssh key has to be added to VMs via Vagrantfile. + + example: + + ```bash + ssh vagrant@192.168.121.34 + ``` + +2 - `vagrant ssh `

+ example: + + ```bash + vagrant ssh Minio-Server-4 + ``` + +## Mount Disks + +- At first invoke `lsblk` to list the disks which is added to VMs via Vagrant + + The `vdb` and `vdc` disks were added. + + ```txt + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS + loop0 7:0 0 63.4M 1 loop /snap/core20/1974 + loop1 7:1 0 53.3M 1 loop /snap/snapd/19457 + loop2 7:2 0 111.9M 1 loop /snap/lxd/24322 + loop3 7:3 0 38.7M 1 loop /snap/snapd/21465 + loop4 7:4 0 87M 1 loop /snap/lxd/28373 + vda 252:0 0 128G 0 disk + ├─vda1 252:1 0 1M 0 part + ├─vda2 252:2 0 2G 0 part /boot + └─vda3 252:3 0 126G 0 part + └─ubuntu--vg-ubuntu--lv 253:0 0 63G 0 lvm / + vdb 252:16 0 2G 0 disk + vdc 252:32 0 2G 0 disk + ``` + +- become root/sudoer via `sudo su` and make `XFS` File system and add Labels `DISK1` and `DISK2` on the two disks (Minio Highly Recommends to use `XFS`) + + ```bash + mkfs.xfs /dev/vdb -L DISK1 + mkfs.xfs /dev/vdc -L DISK2 + ``` + +- Add Two `XFS` Disk at the end of `/etc/fstab` file to be permanently mounted in reboot , `vi /etc/fstab` + + ```conf + LABEL=DISK1 /mnt/disk1 xfs defaults,noatime 0 2 + LABEL=DISK2 /mnt/disk2 xfs defaults,noatime 0 2 + ``` + +- Mount Disks from `/etc/fstab` , the `/mnt/disk1` and `/mnt/disk2` have been created via Vagrant in VMs. + + ```bash + mount -av + ``` + +- Create `minio` Directory in `/mnt/disk1` and `/mnt/disk2`. + + ```bash + mkdir /mnt/disk1/minio + mkdir /mnt/disk2/minio + ``` + +- Change Ownership of those Directories to `minio-user` + + ```bash + chown minio-user:minio-user /mnt/disk1/minio /mnt/disk2/minio + ``` + +## [ Minio Systemd Service][2] + +After connecting to each VM, add a minio systemd service in `/etc/systemd/system` + +```bash +vi /etc/systemd/system/minio.service +``` + +The User and Group in service file has to be `minio-user` + +```conf +[Unit] +Description=MinIO +Documentation=https://docs.min.io +Wants=network-online.target +After=network-online.target +AssertFileIsExecutable=/usr/local/bin/minio +AssertFileNotEmpty=/etc/default/minio + +[Service] +Type=notify + +WorkingDirectory=/usr/local/ + +User=minio-user +Group=minio-user +ProtectProc=invisible + +EnvironmentFile=-/etc/default/minio +ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi" +ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES + +# Let systemd restart this service always +Restart=always + +# Specifies the maximum file descriptor number that can be opened by this process +LimitNOFILE=1048576 + +# Specifies the maximum number of threads this process can create +TasksMax=infinity + +# Disable timeout logic and wait until process is stopped +TimeoutSec=infinity + +SendSIGKILL=no + +[Install] +WantedBy=multi-user.target +``` + +## Environments For Minio + +in the above systemd service, Minio loads the environment variables file from `/etc/default/minio` , create that file and add below contents to it: + +```conf +MINIO_VOLUMES="http://minio-0{1...4}:9000/mnt/disk{1...2}/minio" +MINIO_OPTS="--console-address :9001" +MINIO_ROOT_USER= +MINIO_ROOT_PASSWORD= +MINIO_SERVER_URL="http://minio-01:9000" +``` + +## Start And Enable Minio Service + +```bash +systemctl start minio.service +``` + +Enable The service in Reboot/Restart. + +```bash +systemctl enable minio.service +``` + +status of the minio service , it has to be in active and running status. + +```bash +systemctl status minio.service +``` + +```txt + Loaded: loaded (/etc/systemd/system/minio.service; disabled; vendor preset: enabled) + Active: active (running) since Sun 2024-05-05 19:32:26 +0330; 2h 9min ago + Docs: https://docs.min.io + Process: 4198 ExecStartPre=/bin/bash -c if [ -z "${MINIO_VOLUMES}" ]; then echo "Variable MINIO_VOLUMES not set in /etc/default/minio"; exit 1; fi (code=exited, status=0/SUCCESS) + Main PID: 4199 (minio) + Tasks: 11 + Memory: 245.8M + CPU: 16.618s + CGroup: /system.slice/minio.service + └─4199 /usr/local/bin/minio server --console-address :9001 http://minio-0{1...4}:9000/mnt/disk{1...2}/minio + +May 05 19:32:26 minio-04 minio[4199]: All MinIO sub-systems initialized successfully in 2.583099ms +May 05 19:32:26 minio-04 systemd[1]: Started MinIO. +May 05 19:32:26 minio-04 minio[4199]: MinIO Object Storage Server +May 05 19:32:26 minio-04 minio[4199]: Copyright: 2015-2024 MinIO, Inc. +May 05 19:32:26 minio-04 minio[4199]: License: GNU AGPLv3 +May 05 19:32:26 minio-04 minio[4199]: Version: RELEASE.2024-05-01T01-11-10Z (go1.21.9 linux/amd64) +May 05 19:32:26 minio-04 minio[4199]: API: http://minio-01:9000 +May 05 19:32:26 minio-04 minio[4199]: WebUI: http://192.168.122.34:9001 http://192.168.122.9:9001 http://127.0.0.1:9001 +May 05 19:32:26 minio-04 minio[4199]: Docs: https://min.io/docs/minio/linux/index.html +May 05 19:32:26 minio-04 minio[4199]: Status: 8 Online, 0 Offline. + +``` + +## Login To Minio (Web UI) + +in Browser type the minio UI URL , can be reached with VMs IP and its Port and Login via the Username And Password Provided in `/etc/default/mini` file. + +```txt +192.168.121.31:9000 +``` + +## Resources + +- [`qemu/kvm` installation][3] +- [Vagrant Installation][4] +- [VagrantBoxes][5] + + +[1]: https://min.io/download?license=agpl&platform=linux#/linux +[2]: https://github.com/minio/minio-service/blob/master/linux-systemd/minio.service +[3]: https://docs.fedoraproject.org/en-US/quick-docs/virtualization-getting-started +[4]: https://developer.hashicorp.com/vagrant/install +[5]: https://app.vagrantup.com/boxes/search \ No newline at end of file diff --git a/Vagrant-Minio-Infra/Vagrantfile b/Vagrant-Minio-Infra/Vagrantfile new file mode 100644 index 0000000..abf4631 --- /dev/null +++ b/Vagrant-Minio-Infra/Vagrantfile @@ -0,0 +1,68 @@ +# Latest Version Of Minio Is Present in Host +MINIO_DEB_LATEST_VERSION="minio_20240501011110.0.0_amd64.deb" + +Vagrant.configure("2") do |config| + config.vm.box = "generic/ubuntu2204" + config.ssh.insert_key = false + config.vm.synced_folder ".", "/vagrant", disabled: true + # Copy Minio-Package From Host To VMs + minio_deb_package_path = "/tmp/#{MINIO_DEB_LATEST_VERSION}" + config.vm.provision "file", source: minio_deb_package_path , destination: minio_deb_package_path + # Step by Step Shell commands to prepare the VMs + # The Vagrant User is in Sudoer Group by default. + config.vm.provision "shell", inline: <<-SHELL + # Add Your Host SSH Key To VMs to connect them via ssh + echo '' >> /home/vagrant/.ssh/authorized_keys + # Changing Mod and OwnerShip of authorized key to be readable just for vagrant user + chmod 600 /home/vagrant/.ssh/authorized_keys + chown vagrant:vagrant /home/vagrant/.ssh/authorized_keys + # Adding Proxy Server Address for Apt Because of Filterring If You Need because Of Not Free Internet + # If You Have Free Internet , Comment below line + echo 'Acquire::http::Proxy "socks5h://:";' >> /etc/apt/apt.conf.d/proxy + # Update And Upgrade VMs + sudo apt update -y && sudo apt upgrade -y + # Adding Minio servers IP to /ets/host to implement DNS Server. + # The Static IPs which going to be assigned to VMs. + sudo echo -en "192.168.122.31 minio-01\n192.168.122.32 minio-02\n192.168.122.33 minio-03\n192.168.122.34 minio-04\n" >> /etc/hosts + # Adding User and group for minio + sudo useradd -M -r -s /usr/sbin/nologin minio-user + # changing TimeZone To Asia/Tehran + sudo timedatectl set-timezone Asia/Tehran + # Installing Minio Which it's package is downloaded in host and copied to VMs. + sudo dpkg -i #{minio_deb_package_path } + # Creating Directory For Mounting disks in vms from their /etc/fstab after provisioning the VMs. + sudo mkdir /mnt/disk{1..2} + SHELL + + config.vm.provider :libvirt do |libvirt| + libvirt.memory = 2048 + libvirt.cpus = 2 # Set the number of CPUs + libvirt.storage :file, :size => '2G', :type => 'qcow2' # Additional Disk + libvirt.storage :file, :size => '2G', :type => 'qcow2' + # v.cpusize = 1024 # Set the CPU execution cap (in percentage) + end + + # Minio Server 1 + config.vm.define "Minio-Server-1" do |app1| + app1.vm.hostname = "minio-01" + app1.vm.network :private_network, type: "static", ip: "192.168.122.31", bridge: "virbr0" + end + + # Minio Server 2 + config.vm.define "Minio-Server-2" do |app2| + app2.vm.hostname = "minio-02" + app2.vm.network :private_network, type: "static", ip: "192.168.122.32", bridge: "virbr0" + end + + # Minio Server 3 + config.vm.define "Minio-Server-3" do |app3| + app3.vm.hostname = "minio-03" + app3.vm.network :private_network, type: "static", ip: "192.168.122.33", bridge: "virbr0" + end + + # Minio Server 4 + config.vm.define "Minio-Server-4" do |app4| + app4.vm.hostname = "minio-04" + app4.vm.network :private_network, type: "static", ip: "192.168.122.34", bridge: "virbr0" + end +end \ No newline at end of file diff --git a/Vagrant-Minio-Infra/etc/default/minio b/Vagrant-Minio-Infra/etc/default/minio new file mode 100644 index 0000000..22269e5 --- /dev/null +++ b/Vagrant-Minio-Infra/etc/default/minio @@ -0,0 +1,5 @@ +MINIO_VOLUMES="http://minio-0{1...4}:9000/mnt/disk{1...2}/minio" +MINIO_OPTS="--console-address :9001" +MINIO_ROOT_USER=minioadmin +MINIO_ROOT_PASSWORD=minioadminpassword +MINIO_SERVER_URL="http://minio-01:9000" diff --git a/Vagrant-Minio-Infra/etc/systemd/system/minio.service b/Vagrant-Minio-Infra/etc/systemd/system/minio.service new file mode 100644 index 0000000..a4634d9 --- /dev/null +++ b/Vagrant-Minio-Infra/etc/systemd/system/minio.service @@ -0,0 +1,37 @@ +[Unit] +Description=MinIO +Documentation=https://docs.min.io +Wants=network-online.target +After=network-online.target +AssertFileIsExecutable=/usr/local/bin/minio +AssertFileNotEmpty=/etc/default/minio + +[Service] +Type=notify + +WorkingDirectory=/usr/local/ + +User=minio-user +Group=minio-user +ProtectProc=invisible + +EnvironmentFile=-/etc/default/minio +ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi" +ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES + +# Let systemd restart this service always +Restart=always + +# Specifies the maximum file descriptor number that can be opened by this process +LimitNOFILE=1048576 + +# Specifies the maximum number of threads this process can create +TasksMax=infinity + +# Disable timeout logic and wait until process is stopped +TimeoutSec=infinity + +SendSIGKILL=no + +[Install] +WantedBy=multi-user.target