Skip to content

Commit

Permalink
Merge pull request #60 from Releem/dev
Browse files Browse the repository at this point in the history
Multiple improvements
  • Loading branch information
drupaladmin authored Jan 30, 2021
2 parents 63e6d20 + 995d4d4 commit 6d0bd5a
Show file tree
Hide file tree
Showing 4 changed files with 172 additions and 59 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.devcontainer/*
test-project/test.sh
.DS_Store
releem-install.log
15 changes: 14 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
MySQL Configurer releases
---
Information about releases of the MySQLConfigurer script and online service.
Information about releases of the Releem MySQLConfigurer.

Releem MySQLConfigurer 0.5.0, 2021-01-30
- Added simple one step installation process. Closes #23.
- Improved documentation.
- Improved and published tests description at [releem.com](https://releem.com/blog/how-to-improve-performance-mysql57-default-configuration). Closes #31.
- Fixed problem with timeout variables. Closes #29.
- Added calculation of the 'max_allowed_packet' variable.
- Added calculation of the 'read_rnd_buffer_size' variable.
- Improved calcualtion of the 'sort_buffer_size' variable.
- Improved calculation of the 'innodb_buffer_pool_size' variable.
- Improved calculation of the 'key_buffer_size' variable.
- Improved calculation of the 'innodb_buffer_pool_instances' variable. Closes #37.


Releem MySQLConfigurer 0.4.0, 2020-11-21
- Improved documentation
Expand Down
105 changes: 47 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ To get your Releem API Key please [sign up](https://releem.com/?utm_source=githu
Join the Releem Community on [Slack](https://mysqlcommunity.slack.com/archives/C01FFDYTWTW) and [Telegram](https://t.me/releemhq).

## Features
- Fully automated MySQL performance optimized configuration creation.
- Fully automated MySQL performance configuration tuning.
- **MySQLConfigurer** recommended configuration deliver a [30% boost](#Tests) to MySQL performance compare to the default configuration.
- **MySQLConfigurer** supports 19 parameters of MySQL/Percona/MariaDB server.
- With **MySQLConfigurer** you could prepare configuration file for your MySQL server just in [60 seconds](https://youtu.be/QluJpSl6dGk).
- You could use **MySQLConfigurer** for getting the recommended values for your server and insert in your configuration.
- **MySQLConfigurer** supports 25 parameters of MySQL/Percona/MariaDB server.
- Using **MySQLConfigurer** you can prepare configuration file for your MySQL server just in [60 seconds](https://youtu.be/QluJpSl6dGk).
- You could use **MySQLConfigurer** to getting the recommended values for your server and insert in your configuration.

## Warning
**Always** test recommended configuration on staging environments, and **always** keep in mind that improvements in one area can **negatively** affect MySQL in other areas.
Expand Down Expand Up @@ -67,48 +67,31 @@ Two configurations were tested, the MySQL default configuration and the configur
Recommended configuration delivered a 30% boost to MySQL performance compared to the default configuration. Follow this [link](https://releem.com/blog/how-to-improve-performance-mysql57-default-configuration) to see test results.

## Options
**-k [Releem API KEY]** - used for authorization to Releem platform. To get your Releem API Key please [sign up](https://releem.com/?utm_source=github&utm_medium=link&utm_campaign=signup#rec221377760).
**-k [Releem API KEY]** - API Key to Releem platform. To get your Releem API Key please [sign up](https://releem.com/?utm_source=github&utm_medium=link&utm_campaign=signup#rec221377760).

**-m [MYSQL_MEMORY_LIMIT]** - set maximum memory limit for MySQL. Used when there are installed different applications on the server.

## Installation

One step installation to /opt/releem
```
RELEEM_API_KEY=[YOUR_RELEEM_API_KEY] bash -c "$(curl -L https://releem.s3.amazonaws.com/install.sh)"
```

## Usage
1. Install dependencies

a. Debian/Ubuntu
* `apt-get install -y wget curl net-tools libjson-perl`

b. Centos/RedHat
* `yum install -y wget curl net-tools perl-JSON perl-Data-Dumper`

c. Other OS
* `Requires installation of similar packages on your OS`
2. Download mysqlconfigurer.sh
```bash
wget https://releem.s3.amazonaws.com/mysqlconfigurer.sh
```
or
```bash
curl -o mysqlconfigurer.sh https://releem.s3.amazonaws.com/mysqlconfigurer.sh
```
3. Run mysqlconfigurer.sh

1. To run mysqlconfigurer.sh execute folowing command
```bash
/bin/bash mysqlconfigurer.sh -k [RELEEM_API_KEY]
/bin/bash /opt/releem/mysqlconfigurer.sh -k [RELEEM_API_KEY]
```
- Use -m [MYSQL_MEMORY_LIMIT] - to set maximum memory limit for MySQL. Used when there are different applications installed on the server.
- **RELEEM_API_KEY** - To get your Releem API Key please [sign up](https://releem.com/?utm_source=github&utm_medium=link&utm_campaign=signup#rec221377760).
4. In the /tmp/.mysqlconfigurer folder you could see
```bash
root@mysqlconfigurer# ls -l /tmp/.mysqlconfigurer/
total 264
-rw-r--r-- 1 root root 479 Dec 19 06:03 z_aiops_mysql.cnf
-rw-r--r-- 1 root root 226002 Dec 18 16:44 mysqltuner.pl
-rw-r--r-- 1 root root 33410 Dec 18 16:44 mysqltunerreport.json
```
- **mysqltunerreport.json** - the MySQLTuner report file in the JSON format
- **z_aiops_mysql.cnf** - recommended MySQL config file downloaded from api.server-support.com

5. **Only if you need to increase `open_files_limit` variable.** Perform the folowing steps to safely setup `open_files_limit` in MySQL
2. Recommended MySQL configuration file is /tmp/.mysqlconfigurer/z_aiops_mysql.cnf

3. **Only if you need to increase `open_files_limit` variable.** Perform the folowing steps to safely setup `open_files_limit` in MySQL

5.1. Find out if any other .conf files are being used with MySQL that overrides the values for open limits. Run `systemctl status mysqld/mysql/mariadb` command and it will show something like this
3.1. Find out if any other .conf files are being used with MySQL that overrides the values for open limits. Run `systemctl status mysqld/mysql/mariadb` command and it will show something like this
```
Drop-In:
/etc/systemd/system/(mysqld/mysql/mariadb).service.d
Expand All @@ -119,19 +102,19 @@ Recommended configuration delivered a 30% boost to MySQL performance compared to

`mysqld/mysql/mariadb` is selected depending on the name of the running service name on the server, which is also defined in the output of the command `systemctl status mysqld/mysql/mariadb`

5.2. Edit the file and add the following and change `[table_open_cache]` to your value
3.2. Edit the file and add the following and change `[table_open_cache]` to your value
```
[Service]
LimitNOFILE=([table_open_cache] * 2)
```
- **`open_files_limit` should be no less than `[table_open_cache] * 2`.**

5.3. Run the following command to apply the changes.
3.3. Run the following command to apply the changes.
`systemctl daemon-reload`

5.4. Reboot your mysql server.
3.4. Reboot your mysql server.

5.5. After the successful reboot of the server, we will again run below SQL Queries.
3.5. After the successful reboot of the server, we will again run below SQL Queries.

```
SHOW VARIABLES LIKE 'open_files_limit';
Expand All @@ -148,7 +131,7 @@ Recommended configuration delivered a 30% boost to MySQL performance compared to
1 row in set (0.00 sec)
```

6. Perform the following steps to safely apply recommended configuration:
4. Perform the following steps to safely apply recommended configuration:

**WARNING!** **In case of change 'innodb_log_file_size' only in MySQL 5.6.7 or earlier** set parameter 'innodb_fast_shutdown' to 1 ([Official documentation](https://dev.mysql.com/doc/refman/5.6/en/innodb-redo-log.html)), stop MySQL server, copy old log files into a safe place and delete it from log directory, copy recommended configuration and start MySQL server:
```bash
Expand All @@ -169,25 +152,31 @@ Recommended configuration delivered a 30% boost to MySQL performance compared to
Example of the recommended configuration file /tmp/.mysqlconfigurer/z_aiops_mysql.cnf:
```
[mysqld]
query_cache_type = 1 ### Previous value : OFF
query_cache_size = 128M ### Previous value : 16777216
query_cache_limit = 16M ### Previous value : 1048576
query_cache_type = 1 ### Previous value : ON
query_cache_size = 128M ### Previous value : 134217728
query_cache_limit = 16M ### Previous value : 16777216
thread_cache_size = 8 ### Previous value : 8
key_buffer_size = 196M ### Previous value : 268435456
sort_buffer_size = 24M ### Previous value : 1048576
bulk_insert_buffer_size = 2M ### Previous value : 8388608
myisam_sort_buffer_size = 24M ### Previous value : 67108864
innodb_buffer_pool_instances = 3 ### Previous value : 1
innodb_buffer_pool_size = 3019898880 ### Previous value : 1073741824
max_heap_table_size = 256M ### Previous value : 16777216
tmp_table_size = 256M ### Previous value : 16777216
join_buffer_size = 8M ### Previous value : 262144
key_buffer_size = 205520896 ### Previous value : 205520896
max_allowed_packet = 1073741824 ### Previous value : 67108864
sort_buffer_size = 16777216 ### Previous value : 25165824
read_rnd_buffer_size = 4194304 ### Previous value : 4194304
bulk_insert_buffer_size = 8M ### Previous value : 2097152
myisam_sort_buffer_size = 8388608 ### Previous value : 25165824
innodb_buffer_pool_instances = 2 ### Previous value : 3
innodb_buffer_pool_size = 3019898880 ### Previous value : 3019898880
max_heap_table_size = 256M ### Previous value : 268435456
tmp_table_size = 256M ### Previous value : 268435456
join_buffer_size = 8M ### Previous value : 8388608
max_connections = 151 ### Previous value : 151
interactive_timeout = 1200 ### Previous value : 28800
wait_timeout = 1200 ### Previous value : 28800
table_open_cache = 65536 ### Previous value : 4096
table_open_cache = 3072 ### Previous value : 3072
table_definition_cache = 1920 ### Previous value : 1920
innodb_flush_log_at_trx_commit = 2 ### Previous value : 2
innodb_log_file_size = 805306368 ### Previous value : 67108864
innodb_log_file_size = 377487360 ### Previous value : 805306368
innodb_write_io_threads = 4 ### Previous value : 4
innodb_read_io_threads = 4 ### Previous value : 4
innodb_file_per_table = 1 ### Previous value : ON
innodb_flush_method = O_DIRECT ### Previous value :
innodb_thread_concurrency = 0 ### Previous value : 0
```

## Contribute
Expand Down
110 changes: 110 additions & 0 deletions install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
#!/bin/bash
# (C) Releem, Inc 2020
# All rights reserved

# Releem installation script: install and set up the mysqlconfigurer on supported Linux distributions
# using the package manager.

set -e
install_script_version=1.0.0
logfile="releem-install.log"

WORKDIR="/opt/releem"
CONF="$WORKDIR/releem.conf"

# Read configuration


# Set up a named pipe for logging
npipe=/tmp/$$.tmp
mknod $npipe p

# Log all output to a log for error checking
tee <$npipe $logfile &
exec 1>&-
exec 1>$npipe 2>&1
trap 'rm -f $npipe' EXIT

function on_error() {
printf "\033[31m$ERROR_MESSAGE
It looks like you hit an issue when trying to install the Releem.
If you're still having problems, please send an email to [email protected]
with the contents of releem-install.log and we'll do our very best to help you
solve your problem.\n\033[0m\n"
}
trap on_error ERR

apikey=
if [ -n "$RELEEM_API_KEY" ]; then
apikey=$RELEEM_API_KEY
fi

if [ ! "$apikey" ]; then
printf "\033[31mReleem API key is not available in RELEEM_API_KEY environment variable. Please sigh up at https://releem.com\033[0m\n"
exit 1;
fi

# OS/Distro Detection
# Try lsb_release, fallback with /etc/issue then uname command
KNOWN_DISTRIBUTION="(Debian|Ubuntu|RedHat|CentOS|Amazon)"
DISTRIBUTION=$(lsb_release -d 2>/dev/null | grep -Eo $KNOWN_DISTRIBUTION || grep -Eo $KNOWN_DISTRIBUTION /etc/issue 2>/dev/null || grep -Eo $KNOWN_DISTRIBUTION /etc/Eos-release 2>/dev/null || grep -m1 -Eo $KNOWN_DISTRIBUTION /etc/os-release 2>/dev/null || uname -s)

if [ -f /etc/debian_version ] || [ "$DISTRIBUTION" == "Debian" ] || [ "$DISTRIBUTION" == "Ubuntu" ]; then
OS="Debian"
elif [ -f /etc/redhat-release ] || [ "$DISTRIBUTION" == "RedHat" ] || [ "$DISTRIBUTION" == "CentOS" ] || [ "$DISTRIBUTION" == "Amazon" ]; then
OS="RedHat"
# Some newer distros like Amazon may not have a redhat-release file
elif [ -f /etc/system-release ] || [ "$DISTRIBUTION" == "Amazon" ]; then
OS="RedHat"
# Arista is based off of Fedora14/18 but do not have /etc/redhat-release
elif [ -f /etc/Eos-release ] || [ "$DISTRIBUTION" == "Arista" ]; then
OS="RedHat"
fi

# Root user detection
if [ "$(echo "$UID")" = "0" ]; then
sudo_cmd=''
else
sudo_cmd='sudo'
fi

# Install the necessary package sources
if [ "$OS" = "RedHat" ]; then
echo -e "\033[34m\n* Installing dependencies...\n\033[0m"

if [ -x "/usr/bin/dnf" ]; then
package_manager='dnf'
else
package_manager='yum'
fi

$sudo_cmd $package_manager -y install net-tools perl-JSON perl-Data-Dumper perl-Getopt-Long

elif [ "$OS" = "Debian" ]; then

printf "\033[34m\n* Installing dependences...\n\033[0m\n"

$sudo_cmd apt-get update
$sudo_cmd apt-get install -y --force-yes curl net-tools libjson-perl

else
printf "\033[31mYour OS or distribution are not supported by this install script.\033[0m\n"
exit;
fi

# Set the configuration
if [ ! -e $CONF ]; then
$sudo_cmd mkdir $WORKDIR
fi
if [ "$apikey" ]; then
printf "\033[34m\n* Adding your API key to the Agent configuration: $CONF\n\033[0m\n"
$sudo_cmd echo "export apikey=$apikey" > $CONF
fi

# Secure the configuration file
$sudo_cmd chmod 640 $CONF

printf "\033[31mDownloading Releem MySQLConfigurer...\033[0m\n"
curl -o $WORKDIR/mysqlconfigurer.sh https://releem.s3.amazonaws.com/mysqlconfigurer.sh

0 comments on commit 6d0bd5a

Please sign in to comment.