Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MISC] Adding documentation for AWS and Azure deployment #256

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 207 additions & 0 deletions docs/how-to/h-deploy-aws.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
# How to deploy on AWS

[Amazon Web Services](https://aws.amazon.com/) is a popular subsidiary of Amazon that provides on-demand cloud computing platforms on a metered pay-as-you-go basis. Access the AWS web console at [console.aws.amazon.com](https://console.aws.amazon.com/).

## Summary
deusebio marked this conversation as resolved.
Show resolved Hide resolved

* [Install AWS and Juju tooling](#install-aws-and-juju-tooling)
* [Authenticate](#authenticate)
* [Bootstrap Juju controller on AWS EC2](#bootstrap-juju-controller-on-aws-ec2)
* [Deploy charms](#deploy-charms)
* [Clean up](#clean-up)

---
deusebio marked this conversation as resolved.
Show resolved Hide resolved

## Install AWS and Juju tooling

Install Juju via snap:
deusebio marked this conversation as resolved.
Show resolved Hide resolved

```shell
sudo snap install juju
deusebio marked this conversation as resolved.
Show resolved Hide resolved
```

Follow the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) for guidance on how to install the Amazon Web Services CLI.

To check whether both Juju and AWS CLI are correctly installed, run commands to display their versions:

```console
~$ juju version
3.5.4-genericlinux-amd64

~$ aws --version
aws-cli/2.13.25 Python/3.11.5 Linux/6.2.0-33-generic exe/x86_64.ubuntu.23 prompt/off
```

### Authenticate
deusebio marked this conversation as resolved.
Show resolved Hide resolved

[Create an IAM account](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html) or use legacy user access keys and secret key to operate AWS EC2:

```shell
mkdir -p ~/.aws && cat <<- EOF > ~/.aws/credentials.yaml
credentials:
aws:
NAME_OF_YOUR_CREDENTIAL:
auth-type: access-key
access-key: SECRET_ACCESS_KEY_ID
secret-key: SECRET_ACCESS_KEY_VALUE
EOF
```

<!--- TODO, teach Juju to use `aws configure` format:
deusebio marked this conversation as resolved.
Show resolved Hide resolved
```shell
~$ aws configure
AWS Access Key ID [None]: SECRET_ACCESS_KEY_ID
AWS Secret Access Key [None]: SECRET_ACCESS_KEY_VALUE
Default region name [None]: eu-west-3
Default output format [None]:
```
Check AWS credentials:
```shell
~$ aws sts get-caller-identity
{
"UserId": "1234567890",
"Account": "1234567890",
"Arn": "arn:aws:iam::1234567890:root"
}
```
--->

## Bootstrap Juju controller on AWS EC2

Add AWS credentials to Juju:
deusebio marked this conversation as resolved.
Show resolved Hide resolved

```shell
juju add-credential aws -f ~/.aws/credentials.yaml
```

Bootstrap Juju controller ([check all supported configuration options](https://juju.is/docs/juju/amazon-ec2)):
deusebio marked this conversation as resolved.
Show resolved Hide resolved

```shell
juju bootstrap aws <CONTROLLER_NAME>
```

[details="Output example"]
deusebio marked this conversation as resolved.
Show resolved Hide resolved
```shell
> juju bootstrap aws
Creating Juju controller "aws-us-east-1" on aws/us-east-1
Looking for packaged Juju agent version 3.5.4 for amd64
Located Juju agent version 3.5.4-ubuntu-amd64 at https://juju-dist-aws.s3.amazonaws.com/agents/agent/3.5.4/juju-3.5.4-linux-amd64.tgz
Launching controller instance(s) on aws/us-east-1...
- i-0f4615983d113166d (arch=amd64 mem=8G cores=2)
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 54.226.221.6:22
Attempting to connect to 172.31.20.34:22
Connected to 54.226.221.6
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 54.226.221.6 to verify accessibility...

Bootstrap complete, controller "aws-us-east-1" is now available
Controller machines are in the "controller" model

Now you can run
juju add-model <model-name>
to create a new model to deploy workloads.
```
[/details]
deusebio marked this conversation as resolved.
Show resolved Hide resolved

## Deploy charms

Create a new Juju model, if needed:
deusebio marked this conversation as resolved.
Show resolved Hide resolved

```shell
juju add-model <MODEL_NAME>
```
> (Optional) Increase the debug level if you are troubleshooting charms:
deusebio marked this conversation as resolved.
Show resolved Hide resolved
> ```shell
> juju model-config logging-config='<root>=INFO;unit=DEBUG'
> ```
deusebio marked this conversation as resolved.
Show resolved Hide resolved

Then, Charmed Kafka can be deployed as usual. However, note that the smallest instance types on AWS may not have enough resources for hosting
a Kafka broker. We recommend selecting an instance type that provides at the very least 8 GB of RAM and 4 cores.
For more guidance on production environment sizing, see the [Requirements page](/t/charmed-kafka-reference-requirements/10563).
You can find more information about the available instance types in the [AWS documentation](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Instances:instanceState=running),
ensuring that you select the correct AWS.

Deploy and integrate Kafka and ZooKeeper, for example:

deusebio marked this conversation as resolved.
Show resolved Hide resolved
```shell
juju deploy zookeeper -n3 --channel 3/stable
deusebio marked this conversation as resolved.
Show resolved Hide resolved
juju deploy kafka -n3 --constraints "instance-type=m7i.xlarge" --channel 3/stable
juju integrate kafka zookeeper
```

We also recommend to deploy a [Data Integrator](https://charmhub.io/data-integrator) for creating an admin user to manage the content of the Kafka cluster:

```shell
juju deploy data-integrator admin \
--config extra-user-roles=admin \
--config topic-name=admin-topic
```

And integrate it with the Kafka application:

```shell
juju integrate kafka admin
```

For more information on Data Integrator and how to use it, please refer to the [how-to manage applications](/t/charmed-kafka-how-to-manage-app/10285) guide.

## Clean up

[note type="caution"]
Always clean AWS resources that are no longer necessary! Abandoned resources are tricky to detect and they can become expensive over time.
deusebio marked this conversation as resolved.
Show resolved Hide resolved
[/note]

deusebio marked this conversation as resolved.
Show resolved Hide resolved
To list all controllers use the `juju controllers` command.

To destroy the Juju controller and remove AWS instance (**Warning**: all your data will be permanently deleted):

```shell
juju destroy-controller <CONTROLLER_NAME> --destroy-all-models --destroy-storage --force
deusebio marked this conversation as resolved.
Show resolved Hide resolved
```

Use `juju list-controllers` to list the available controllers the client has registered to.

After destroying the controller, check and manually delete all unnecessary AWS EC2 instances, to show the list of all your EC2 instances run the following command (make sure to use the correct region):

```shell
aws ec2 describe-instances --region us-east-1 --query "Reservations[].Instances[*].{InstanceType: InstanceType, InstanceId: InstanceId, State: State.Name}" --output table
```

[details="Output example"]
deusebio marked this conversation as resolved.
Show resolved Hide resolved
```shell
-------------------------------------------------------
| DescribeInstances |
+---------------------+----------------+--------------+
| InstanceId | InstanceType | State |
+---------------------+----------------+--------------+
| i-0f374435695ffc54c| m7i.xlarge | terminated |
| i-0e1e8279f6b2a08e0| m7i.xlarge | terminated |
| i-061e0d10d36c8cffe| m7i.xlarge | terminated |
| i-0f4615983d113166d| m7i.xlarge | terminated |
+---------------------+----------------+--------------+
```
[/details]

List your Juju credentials with the `juju credentials` command:

```shell
...
Client Credentials:
Cloud Credentials
aws NAME_OF_YOUR_CREDENTIAL
...
```
Remove AWS EC2 CLI credentials from Juju:
deusebio marked this conversation as resolved.
Show resolved Hide resolved

```shell
juju remove-credential aws NAME_OF_YOUR_CREDENTIAL
```

Finally, remove AWS CLI user credentials (to avoid forgetting and leaking):
deusebio marked this conversation as resolved.
Show resolved Hide resolved
deusebio marked this conversation as resolved.
Show resolved Hide resolved

```shell
rm -f ~/.aws/credentials.yaml
```
Loading
Loading