Old Doc Pages

BetterBandwidth Concepts

BetterBandwidth uses 'tc' (a built in Linux tool) to provide traffic shaping for users that have their own network interface. The tc tool on its own does not provide a mechanism for limiting bandwidth usage by uid/user on a shared network interface. BetterLinux has added this functionality into its BetterBandwidth feature. Now admins can limit bandwidth for a single user on a shared server that has a corporate internet connection.

A common problem for web hosts is that one single user with a 1Gbit connection could kill the entire TCP stack. Often the host is unaware that it happens because they use 5 min bandwidth averages to monitor bandwidth on their 1 gig or 10 gig port. But all it takes is a 30 second spike in bandwidth out of 5 mins to make customers angry. And because of the 5 minute average it looks like the peak was around 50Mbits when it easily could have been 500Mbits. Customers will blame it on the server or the hosting company.

Many hosting companies give out IPs just to be able to limit bandwidth. This is an expensive approach. The BetterLinux BetterBandwidth feature can do it without giving out dedicated IPs.

BetterLinux has an apache mod called mod_betterlinux that allows BetterBandwidth to limit bandwidth for all ports for an individual user or group of users like email, FTP, etc is all included. This is a tremendous advantage for a host. BetterBandwidth monitors and controls individual, group, and system-wide bandwidth usage in multi-tenant Linux environments. Administrators can now limit outbound bandwidth as if each user had a dedicated IP address. This lets admins consolidate multiple users onto a single IP, which innovatively addresses the fast-increasing IPV4 IP address shortage.

Configure BetterBandwidth

Follow the BetterBandwidth HOWTO guide for configuring bandwidth limiting

Bandwidth limiting

BetterBandwidth limiting uses the mod_betterlinux Apache module located in in the betterlinux-core-*.rpm , as well as the linux tc method of using the uid classifier to categorize specific user processes. The betterlinux-core-*.rpm is installed by the bl-install script by default and auto-configured for cPanel systems during the betterlinux-cpanel-install script.

cPanel Bandwidth Limiting Setup

During the betterlinux-cpanel-install script installation, the user will answer Y/N to this question:

Install BetterLinux Apache throttling? [y/N]

Selecting yes, will auto-configure and load the apache module mod_betterlinux on a cPanel system.

After the installation is complete, the following parameters and arguments need to be added to the BetterLinux configuration files:

• In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example:

interface interface-types eth0

• In /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format:

bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to:

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

non-cPanel Bandwidth Limiting Setup

• Depending on the distribution, edit the apache conf or create a new one under conf.d and add the following:

LoadModule betterlinux_module /usr/lib64/httpd/modules/mod_betterlinux.so
<IfModule betterlinux_module>
 Betterlinux On
</IfModule>

Then restart apache.

Using this command will check the mod_betterlinux syntax in the apache .conf file:

apachectl -t -D DUMP_MODULES

• Configure BetterCPU (cpud) by adding or uncommenting the following parameter located in /etc/betterlinux/cpud.conf.d/main.conf

 approved_accounting_program /usr/sbin/httpd apache

Please note, the full path to the httpd program is required, and it must be the actual program and not a symlink. Also, be sure to list the user it runs as, whether that is apache, httpd, or different username.

• In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example:

interface interface-types eth0

• In /etc/betterlinux/cpud.conf.d/main.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format:

bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to:

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

Assigning IP addresses with the dedicated IP address mapper

With multi-homed systems, incoming traffic is directed to the IP address where DNS resolves the domain name. A system with multiple virtual hosts can accept traffic to secondary addresses which, in turn, are assigned to secondary addresses, which are assigned to the system's network adapters.

Without some modification, responses sent from a server in a multi-homed system originate from the host's primary IP address. This makes it difficult to apply bandwidth or connection limits using a server's IP address. Most often, hosting providers work around this problem by implementing some sort of full virtualization solution that provides a unique virtual instance for each client. <KGO> What is the drawback of this solution?

BetterBandwidth uses the dedicated kernel module to provide a more cost-effective, efficient solution for managing bandwidth in multi-homed environments. When used in conjunction with the BetterBandwidth dedicated daemon, the dedicated kernel module allows administrators to assign outgoing traffic to a specific secondary address which, in turn, is assigned to a specific user--all without the overhead of a fully virtualized system. (Both IPv4 and IPv6 are supported.)

To enable this functionality:

1. Ensure the dedicated kernel module is loaded:
   

   modprobe dedicated

2. Edit /etc/betterlinux/ipv4 and/or /etc/betterlinux/ipv6 to include the specific mappings.
3. Restart dedicatedd:
   

   service dedicatedd restart

The IPv4 and IPv6 mapping files are in the following format. Each line contains a single entry.

username|uid: address

The address field can contain either a raw IPv4 or IPv6 address or a DNS name. If the DNS name resolves to more than one address, the kernel randomly selects one of the addresses.

Once configured, the active mappings for IPv4 addresses can be viewed using the following command:

cat /proc/sys/net/ipv4/dedicated

To view mappings for IPv6 addresses, use the following command:

cat /proc/sys/net/ipv6/dedicated

Note: The command output displays the uid rather than the username.

BetterCPU Configuration

Refer to the BetterCPU HOWTOs section for details on configuration and setup.

BetterCPU Concepts

BetterCPU is designed to maximize cpu utilization for shared hosting environments and at the same time prevent one or many users from abusing the cpu utilization. BetterCPU makes it very easy to quickly identify the CPU hogs on the system and keep them under control and then decide if these users need to moved to a higher paying account that provides them with the resources they need to run their business. This is accomplished by defining BetterLinux groups of users or mapping BetterLinux groups to cPanel packages. Once the user groups are configured, then throttling limits are defined for CPU.

Defining jail cores

Before limits are configured, it's important to understand BetterCPU jails. The jails are elastic by nature and only exist if they are needed by the BetterCPU controller to temporarily limit an abusive user process. Think of process jailing as analogous to law enforcement:

• A user is a “citizen”
• The performance thresholds are the “law”
• BetterCPU enforces the laws

A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.

If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.

When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.

Configuration of the jail is done using the parameters:

jail_cnt
max_jail_cnt

Commenting both of the parameters out sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1. BetterCPU then auto-scales the jails the system needs based on demand for jail cores. However, the jail cores will never take a priority over non-jail cores.

The jail_cnt can be defined using two different methods:

• If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jai_cnt. For example:

jail_cnt 2

The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems. If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.

• is expressed as one or more pairs of numbers. For example:

jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail; an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system. The 32 core system will be assigned 6 cores as the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails, and is written in the same manner as the jail_cnt parameter. Setting the max_jail_cnt parameter will fix the ultimate size of the jail. Fixing the jail size will determines the amount of the cpu utilization abusive users will share in the jail core(s).

By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.

Defining BetterCPU Throttling Limits

The BetterCPU throttling limits are defined by a percentage of one core for each user.

Once the jail cores are defined, BetterCPU provide two throttling parameters that define when a user is moved into and out of the jail core(s). The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set jail throttling limits for groups of uids defined in groups.conf or betterlinux-cpud.conf on cPanel installs. The cpu_percent sets the percentage a user (uid) can use of one core on the system. The values for cpu_percent in start_throttling and stop_throttling are based on one core and are not adjusted when cores are added or removed from the jail.

When a throttled user’s total utilization drops below the stop_throttling for cpu_percent and "seconds" settings, then the process is removed from the jail.

For example, consider an 8-core system with the following values configured:

start_throttling group=example-group cpu_percent=50,time=3
stop_throttling group=example-group cpu_percent=25,time=10

A user whose processes exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.

Throttling Users

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. By default, the non-cPanel installation (betterlinux-install) does not configure BetterCPU to throttle users. However, the cPanel installation (betterlinux-cpanel-install) script does configure and start BetterCPU throttling for all cPanel users on the server. The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).

Configuring the Jail Cores

Throttling is configured using parameters located in:

non-cPanel install: /etc/betterlinux/cpud.conf.d/rules.conf:
cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

jail_cnt
max_jail_cnt

These settings are designed so that the same configuration file may be used on different systems with different numbers of cores. However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file. When these parameters are removed, the cpud daemon uses the default settings wich is jail_cnt=1 and max_jail_cnt=total cores minus 1 and dynamically auto-scales up or down the number of jail cores needed for the system.

Syntax:

jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]

The jail_cnt parameter is defined using one of two methods:

• If a fixed number of cores for jail is desired, make the assignment using jail_cnt as in this example:

jail_cnt 2

In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core. If the system is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.

• If the number of cores for jail varies based on the total number of cores in the system, use the following format:

jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. For example, using the values specified in the previous example, a 32-core system will use the jail_cnt value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails and is formatted identically to the jail_cnt parameter. By default, only the value of jail_cnt is used, but when a max_jail_cnt is specified, the jail is expanded when less than the value in jail_min_idle is available per jail core.

The jail_min_idle value is the minimum amount of idle utilization to maintain on the jail cores if jail_max_cnt has not been reached. If the average free utilization on each jail core is less than jail_min_idle, the jail is expanded until either the jail_min_idle value is maintained or jail_max_cnt is reached.

Syntax:

max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}

Example

max_jail_cnt 2:1 4:2 8:2 16:2

The result on a dual core system is no different from jail_cnt. But for 4-core machines and above, if the max_jail_cnt value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded.

max_jail_cnt 2

The result will always set the number of maximum jails to two. The maximum number of jails are set at two no matter how many cores may be on the system. The exception being, when they system has 2 cores it will set the max jail core to one.

In order for the jail to expand, add a jail_min_idle value to denote how much free utilization should be maintained in the jail:

jail_min_idle 5

BetterCPU will attempt to maintain at least 5% free CPU per jail core, and will expand the jail if max_jail_cnt has not been reached.

Defining User Groups

For more details on groups please see the Defining User Groups section of the documentation.

The BetterLinux cPanel install script will discover all the cPanel users on the system and create a user group called "cpanel-users" that contains a list of all the users assigned to a cPanel package.

The bl-install script will not create user groups, they need to be created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, or files containing UIDs or usernames, CPU usage levels can be precisely defined for different groups, and durations and conditions for starting/stopping throttling can be defined. The group names defined in /etc/betterlinux/cpud.conf.d/groups.conf are used in the settings for the start_throttling and stop_throttling parameters.

Defining User Groups

When planning a strategy for defining groups, use the following steps:

1. Decide how users/customers should be organized into groups or cPanel packages. This organization could be based on needs, paid service levels, or any other criteria.
2. Ideally, user UIDs ranges or a file that contains a list of UIDs should be used to define users with similar requirements. The BetterLinux cPanel hooks feature will automatically create a uid text file with a name that matches the cPanel package and will add newly created cPanel users to the file.
3. Edit /etc/betterlinux/cpud.conf.d/groups.conf or the BetterLinux cPanel hooks feature will do this automatically.
4. For example:
   uid package-normal-users /etc/betterlinux/cpanel/package-normal-users.txt
   uid package-premium-users /etc/betterlinux/cpanel/package-premium-users.txt
5. Save the groups.conf file
6. Restart the BetterCPU daemon:
   service cpud restart

Once this parameter change is in place, throttling limits that meet each group's SLA can be defined for each group.

For non-cPanel servers, the users will likely already be in a group defined in /etc/group. If this is the case, it is fairly easy to use that group definition within the groups.conf file. Use the following command to extract the group definition and reformat it for inclusion in the /etc/betterlinux/cpud.conf.d/groups.conf file. When defining the syntax, simply replace {group_name} with the name of the group defined in /etc/group.

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting CPU Core Usage Limits

Once jail_cnt and jail_cnt_max are defined and groups are set up, it is necessary to configure the throttling parameters that define when user groups are moved into and out of the jail cores. The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set throttling limits for groups defined in /etc/betterlinux/cpud.conf.d/groups.conf. The cpu_percent value sets the percentage an individual user (uid) can use of the total non-jail cores on the system.

To understand how this works, it is important to understand how CPU utilization is measured in a multi-core system. In Linux, multithreaded processes can take more than 100% utilization because overall CPU utilization is measured as 100% * number_of_cores. So in an 8 core system, if all CPUs are running at maximum utilization, the system's overall utilization is 800% (100% * 8 cores).

The values for cpu_percent in start_throttling and stop_throttling are based on a normalized utilization assigned to a single core (rather than being spread across all cores). This means that while an 8 core system can have a maximum of 800% utilization, the value used for the cpu_percent value represents the fraction of a single core that could be used. Specifying a value of 80 for cpu_percent means that on a system with no cores presently allocated to jail, when the user exceeds a value of 80% * 8 = 640% on an 8 core system, a core is allocated to the jail and the user's processes are throttled. When a core is allocated to jail, the percentage used to determine if the threshold has been exceeded is 80% * 7 = 560%. If a second core is allocated to jail, this threshold drops to 80% * 6 = 480%.

What this means is that - using the example of cpu_percent set to 80 - any distribution of utilization across all 8 cores of an 8 core system that adds up to more than 640% utilization will trigger the allocation of a core for throttling and that user's processes will be moved to that single core. The utilization could be 100% of 6 cores and 40% of a seventh core, it could be 80% utilization distributed across all 8 cores, or it could be any variation where the user's total utilization exceeds the threshold.

When a non-throttled user's total process utilization exceeds cpu_percent for seconds seconds as defined in the start_throttling parameter, the user's processes are moved into the jail cores.

When a throttled user’s total utilization drops below the cpu_percent value for seconds seconds as defined in the stop_throttling parameter, the user's processes are moved out of the jail back onto the non-jailed cores.

To define these limits:

• Edit the users.conf located in /etc/betterlinux/cpud.conf.d.
• Add the following parameters users.conf, with appropriate limits to the system's needs:

start_throttling group=example-group cpu_percent=50,seconds=3
stop_throttling group=example-group cpu_percent=25,seconds=10

• Save users.conf.
• Restart cpud:

service cpud restart

With these settings, a user in the group defined with processes that exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system. When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.

Start the BetterCPU Daemon (cpud)

BetterCPU is now configured with basic jail core and throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service cpud {start|stop|status|reload|restart|condrestart}
 start = starts cpud daemon
 stops = stops cpud daemon
 status = shows the status of cpud daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Start the cpud daemon with the following command:

service cpud start

Start the BetterLinux Stats Program (blstat)

In order to see user processes that are in the jail cores, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterI/O. Press 'c' to see the stats for BetterCPU. Press '?' to display general help for the utility.

Limiting Execution Time Using time_monitor

BetterCPU can be used to limit the amount of time programs can execute. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting execution time are:

1. Define a program or user group
2. Add time_monitor to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is as follows:

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example:

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. The user groups are structured as follows:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for time_monitor execution limiting are identical to user groups used for other functions in the BetterLinux suite.

Add time_monitor to main.conf

Limiting the execution time for a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again. The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.

To define the limit, add the following line to the main.conf file:

time_monitor groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer 10

In this example, we use the sample "bad_progs" program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash. If the program has run for 5 seconds or longer, /root/killer is called with the PID of the program, and the script kills the program after logging the date and time.

The final parameter (10) tells cpud to not run the script more than once every 10 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring the execution time, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:

time_monitor_prog_exclude groups=exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

Limiting Open Files Using file_limit

BetterCPU can be used to limit the number of concurrent open files a user can have at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent open files are:

1. Define a user group
2. Add file_limit to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When open files are limited, the file_limit_override and file_limit_exclude options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the following format used is:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting open files with file_limit are identical to user groups used for other functions in the BetterLinux suite.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Processes Using process_limit

BetterCPU can be used to limit the number of concurrent processes a user can execute at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent processes are:

1. Define a user group
2. Add process_limit to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the format used is:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting concurrent processes with process_limit are identical to user groups used for other functions in the BetterLinux suite.

Add process_limit to main.conf

Limiting concurrent processes per user is very straightforward.

To apply a limit, once a user group is defined, add a line to apply the desired restiction to the main.conf file:

process_limit group=bad_users limit=100

This limits the users in the bad_users group to 100 concurrent processes each at any given time.

To completely exclude specific programs from having limits applied, create a program group in groups.conf with the programs in it:

prog exclude_process_limits /bin/cron,/bin/bash

Then add a process_limit_exclude line to main.conf:

process_limit_exclude group=exclude_process_limits

If a user limit should be overridden for particular programs, specify the user group as part of a process_limit_override command and specify the specific programs that should have the different limit in main.conf:

process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd

This overrides the existing limits for bad_users for instances of /usr/sbin/sshd and sets those limits to 120.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Memory Utilization Using memory_limit

BetterCPU can be used to limit the amount of RSS memory programs can use. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting RSS memory usage are:

1. Define a program or user group
2. Add memory_limit to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is:

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example:

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. These user groups are structured as:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for RSS memory usage limiting are identical to user groups used for other functions in the BetterLinux suite.

Add memory_limit to main.conf

Limiting the RSS memory used by a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again. The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.

To define the limit, add the following line to the main.conf file:

memory_limit group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run=30

In this example, we use the sample bad_progs program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB). If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.

The final parameter (30) tells cpud to not run the script more than once every 30 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:

memory_limit_prog_exclude exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

BetterIO Configuration

Refer to the BetterIO HOWTOs section for details on configuring BetterIO.

BetterIO Concepts

The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.

In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.

From a CPU perspective, this is referred to as “Waiting on I/O”, or iowait%. If the I/O channel is not also managed, then the system will not be able to run optimally.

When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.

There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To monitor these factors, one can use the iostat command or blstat (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.

What can BetterIO tune to affect block device utilization?

At a high level, tuning is applied to I/O operations per second (iops) and bandwidth (bw). These two parameters can be tuned by applying minimum and maximum values in /etc/iothrottled/iothrottled.conf.d/rules.conf.

Those system-wide minimum and maximum values can be refined by applying user limits and program limits, which are defined in /etc/iothrottled/user_limits.conf and /etc/iothrottled/prog_limits.conf, respectively.

The limits defined can be scaled to allow the block device channel to be utilized more effectively when it is under-utilized. The auto_scale parameter defines how scaling is applied to keep both the bandwidth utilization and iops within a desired performance range.

Setting I/O Bandwidth Limits Per-User or by User-Group

Individual user or group limits can be set using the configuration file /etc/iothrottled/user_limits.conf.

This file uses a single line per entry:

[user|group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or mount points to include or exclude.

For example, the file might contain a line that reads:

testuser bw max sda,sdb,sdc 10000000

This would mean that the user testuser has their maximum bandwidth (bw) limited on devices sda, sdb, and sdc to 10000000 B/s (Bytes per Second).

The first field can be a user or group name. Groups are defined in the file /etc/betterlinux/groups.conf. If a group and a user have the same name, the limit applies to the user.

Setting I/O Bandwidth Limits by Program-Group

More granular control over individual programs’ usage of block device I/O rates and data rates can be achieved using the /etc/betterlinux/iothrottled.conf.d/prog_limits.conf file.

This file is similar to the user_limits.conf file, except that it specifies a program group name. The line format for this file is:

[program_group_name] [S|G] [bw|ops] [device/mount point list] [limit]

The fields that match up with fields in prog_limits.conf are defined the same way as in user_limits.conf.

The prognam_group_name field contains the names of the program or programs which is defined in groups.conf.

If the third field contains an “S”, the limit applies to each instance of the program started by the use or group. A “G” means the limit applies to all instances together.

For example: NOTE: In all the examples below we assume to have the group "tar_progs" defined in /etc/betterlinux/iothrottled.conf.d/groups.conf:

prog tar_progs /bin/tar,/usr/bin/tar

If our goal is to limit each individual instance of "tar" to a 5 MB/s I/O rate we would use the following line.

tar_progs S bw max inc="^/" 5000000

NOTE: all the "/bin/tar" and "/usr/bin/tar" instances will be affected by this rule if the UID that runs them is not included in any UID group. Remember that UID groups always win against PROG groups.

If our goal was to group ALL instances of tar to an aggregate bandwidth of 5 MB/s then you would change the "S" (Single process) in the example above to "G" (For grouping all tar processes together). This option is helpful when you want to limit one or more instances together. So, if you had one tar running or 50 tars the total disk I/O bandwidth would never exceed 5 MB/s.

The syntax for this scenario would be as follows:

tar_progs G bw max inc="^/" 5000000

If our goal was to limit each instance of tar to a disk I/O rate of 5 MB/s, but would only like to enforce this limit on any filesystem mounted under /home (and any sub-directories) but have no limits enforced on other drives then the syntax would be as follows.

tar_progs G bw max inc="^/home" 5000000

You can include multiple block devices (hard drives) or mount points to be included in the limits that you have defined as shown below:

tar_progs G bw max inc="/home",sdb,dm-1,inc="/var/matt",inc="/etc" 5000000

In addition, you can specific drives to be excluded from the limits that you have defined. This syntax is shown below.

tar_progs G bw max inc="/",exc="^/media|^/mnt" 5000000

For example:

tar_progs S bw sda 1000000

This applies a bandwidth limit of 1000000 bytes/s to each instance of the /bin/tar or /usr/bin/tar commands run by any user not included in a UID group. If one of these users starts two concurrent instances of the tar command, each one is limited to 1000000 b/s of block device I/O bandwidth. If this line reads:

tar_progs G bw sda 1000000

The bandwidth limit is applied to all instances of tar that the users launch. In this case, if the users launch two concurrent instances of the tar command, the total bandwidth both instances can use together is 1000000 B/s.

Below are some configuration rule examples of how to limit I/O bandwidth for certain programs.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied. The inc= and exc= directives take an extended regular expression for mount points. For example, the line:

tar_progs G bw max sda,inc="/home|/backup",exc="/mnt1|/mnt2" 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that have "/home" or "/backup" in their mount point (that would also include, for example, "/home2", "/media/home3", excluding all the mount points that have "/mnt1" or "/mnt2" (like "/mnt21", "/mnt/mnt1", ...). The values are regular expressions and they can appear anywhere within the mount point). Including ^ and $ in the regex will force a match at the begining or end.

The evaluation of these expressions always follows the order of:

• Explicit device names
• Devices mounted in the "inc" list
• Devices mounted in the "exc" list

Exclusions will ultimately override the other values in the list. Consider the following list of mounted devices:

/dev/sda mounted at /
/dev/sdb mounted at /home
/dev/sdc mounted at /media/backups
/dev/sdd mounted at /usr
/dev/sde mounted at /mnt/home
/dev/sdf mounted at /srv/www
/dev/sdg mounted at /srv/ftp
/dev/sdh mounted at /srv/home

With the following restrictions in user_limits.conf:

tar_progs G bw max sda,sdd,inc="/home",exc="/srv" 1000000

Programs in group "tar_progs" will be limited to 1 MB/s on /home and /mnt/home, but not on /srv/home, due to the exc="/srv" rule.

How do the block device inclusion/exclusion lists work?

Simple configuration of BetterIO uses device names in a list in the user_limits.conf and prog_limits.conf files. For example:

testuser bw max sda,sdb,sdc 1000000

This applies the limit of 1000000 B/s for the user testuser on devices /dev/sda, /dev/sdb, and /dev/sdc.

Modern Linux systems, however, do not always guarantee the assignment of devices, which can lead to issues with limits not being applied to the correct device.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied.

The inc= and exc=' directives take an extended regular expression for mount points. For example, the line:

testuser bw max sda,inc=”/home|/backup”,exc=”/mnt1|/mnt2” 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that are mounted at “/home” and “/backup” (and would also include, for example, “/home2”, “/media/home3” - the values are regular expressions and can appear anywhere within the mount point), but would exclude any mount point that includes “/mnt1” or “/mnt2” (including “/mnt21”, “/mnt/mnt1” - as with the “inc=” directive). Including ^ and $ in the regex will force a match at the begining or end.

Another example for this using regular expression that will apply the limit of 1MiB/s to any mounted block device except those mounted under /var or /tmp

testuser bw max inc=”^/”,exc=”^/var|^/tmp” 1000000

In this example we want to limit all the mounted block devices, except the temporary mounted devices, usually mounted under /mnt or /media.

• add the line below to /etc/betterlinux/groups.conf:

testgroup uid 500-1000

• Then add the line below to /etc/betterlinux/iothrottled/user_limits.conf:

testgroup bw max inc=”^/”,exc=”^/mnt|^/media” 1000000

The evaluation of these expressions always follows the order of:

1. Explicit device names
2. Devices mounted in the “inc” list
3. Devices mounted in the “exc” list

Exclusions will ultimately override the other values in the list.

Consider the following list of mounted devices:

/dev/sda1 mounted at /

/dev/sda2 mounted at /home

/dev/sdb1 mounted at /media/backups

/dev/sdb2 mounted at /usr

/dev/sdc1 mounted at /mnt/home

/dev/sdc2 mounted at /srv/www

/dev/sdd1 mounted at /srv/ftp

With the following restrictions in user_limits.conf:

testuser bw max sda,sdd,inc=”/home”,exc=”/srv” 1000000

The result would be:

sda would be included

sdb would be excluded

sdc would be excluded

sdd would be excluded

So the only device controlled by BetterIO in this scenario would be /dev/sda. The controls are applied per block device, not on a per-partition basis - so if a partition is excluded from control, all partitions on that block device will be excluded.

When configuring your system, it is important to take this into consideration - you do not want your swap partition to be throttled by BetterIO, but if you put swap on /dev/sda and control /dev/sda with BetterIO, you may end up impacting performance on the system in ways you may not intend to.

Examples include/exclude:

/dev/sda1 on /

/dev/sda2 on /home

/dev/sdb1 on /home2

/dev/sdc1 on /home3

/dev/sdd1 on /home4

testuser bw max inc="/home",exc="/home2"

The rexeg's are matched against the left of the mount point. So the include matches /home1, /home2, /home3, /home4 and the exclude matches /home2 only. So the result is:

/dev/sda included

/dev/sdb excluded

/dev/sdc included

/dev/sdd included

Example Exclude:

/dev/sda1 on /

/dev/sda2 on /home

/dev/sdb1 on /src

/dev/sdb2 on /usr

/dev/sdc1 on /extra

testuser bw max exc="/src"

/dev/sda included

/dev/sdb excluded

/dev/sdc included

How does auto scaling work?

The auto_scale parameter is an on/off toggle switch for enabling or disabling the auto scale feature. The auto scaling algorithm is applied to overall bandwidth utilization or iops giving the system administrator the ability provide more bandwidth to a defined set of users when it's available.

When setting the value for this parameter, an integer is entered that represents the on/off values:

0 = auto scaling off

1 = auto scaling on

Auto scaling is controlled by these parameters:

scale_pct_hi

scale_pct_low

scale_pct_periods

Additionally, the following parameters apply to auto scaling:

scale_min

scale_max

scale_up_inc

scale_down_inc

To understand how these parameters are related to each other, consider the following scenario:

With the following line in /etc/iothrottled/user_limits.conf:

testuser bw max sda 1000000

And in /etc/iothrottled/iothrottled.conf.d/rules.conf:

auto_scale 1

scale_pct_hi 80

scale_pct_low 20

scale_pct_periods 2

scale_min 25

scale_max 500

scale_up_inc 20

scale_down_inc 5

If the overall block device utilization is below 20% (scale_pct_low), then the limit applied to testuser is increased by 20% (scale_up_inc) of 1000000 (taken from user_limits.conf for this user), or 200000 B/s (Bytes per Second).

If overall block device utilization goes over 80% (scale_pct_hi), then the limit applied to testuser is decreased by 5% (scale_down_inc) of 1000000 (from the user’s limit in user_limits.conf), or 50000 B/s.

The increase/decrease in the scaling factor is tracked as well, so on the first cycle, the scaling factor is 120%. The second iteration scales up to 140%, and so on, until the limit set by scale_max is reached.

Similarly, if scaling down were to be applied instead, that scaling would drop by 5% each iteration until the scale factor reached the scale_min limit of 25%.

The result is that BetterIO attempts to keep the block device channel utilization between 20% and 80% at all times, and adjusts the utilization limits for all users in order to achieve that goal. If one were to graph the utilization over time, the goal is to have that graph look something like this:

Whenever the utilization drops below the threshold at 20% or goes above the threshold at 80%, the adjustments kick in so the utilization falls in the area between the two thresholds. The majority of the time, that device’s utilization should be between 20% and 80% with this configuration.

Those adjustments are subject to the scale_min and scale_max parameters. Having limits on the amount of scaling will prevent BetterIO from attempting to scale up without a limit when there is no I/O taking place. If it were allowed to do that, then a period with no I/O could end up with a scaling factor in excess of 1000%, and the process of scaling down would result in it taking a significant amount of time with no effective limit in place, which would make the available bandwidth the bottleneck for system performance until the scaling factor returned to a reasonable range.

How can one test to see if BetterIO is controlling devices?

A simple and quite dramatic way to see BetterIO limiting I/O is to set a lower threshold for a user that is very small. For example, create a user called testuser and log in as that user. Run the following command:

sync; dd if=/dev/zero of=temp.fil bs=1M count=10 oflag=direct; rm temp.fil

The results from that command for an unmanaged user would be similar to this screenshot:

If we then modify the /etc/iothrottled/user_limits.conf file to include the line:

testuser bw max inc=”/” 500000

and rerun the dd command above, the following output results:

The drop in throughput from 350-400 MB/s to 500-600 kB/s shows the effect of the change in bandwidth limits for this user.

Are there any other considerations to keep in mind?

With BetterIO, the limits are set on a per-block device basis. It is important to keep in mind that I/O channels are not always assigned in a 1:1 ratio with block devices. If a series of storage devices are connected to the system by a Fibre Channel connection, the channel bandwidth available for all of those devices needs to be taken into consideration across all of the devices.

If devices /dev/sda, /dev/sdb, and /dev/sdc are all on the same disk controller, and the maximum throughput of that controller is 1 Gbps, then all three disks cannot run at 1 Gbps even if the drive electronics would permit that.

If you then have 10 users limited to 100 Mbps each, then performance will effectively be running without any throttling because the disk channel will be oversubscribed if all of the users are attempting to fully utilize their allocated bandwidth to each of these three devices. At the same time, these users are still limited to 100 Mbps each and are unable to monopolize the resources.

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. Once the the system is rebooted, configure BetterIO and start the iothrottled daemon. By default, BetterIO does not throttle any users. The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).

Defining user groups

User-created groups do not exist on the system until they have been created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined. The group names defined in /etc/betterlinux/iothrottled.conf.d/groups.conf are used in the settings for setting per-group I/O limits.

To define groups:

1. Choose a group name.
2. Populate the group with UIDs, UID ranges, user names, or files containing UIDs or usernames
   Note: A group may contain one or more usernames, UIDs, or ranges.

The following examples illustrate the command syntax:

Single range:
 Syntax:  uid {group name} {uid range}

 Example: uid example-group 0-499
This example includes all UIDs from 0-499 as members of the group example-group

 Example: uid example-group 800-
This example includes all UIDs from 800 and above as members of the group example-group
 
Multiple UID ranges:
 Syntax:  uid {group name} {uid range}[,{uid range2}...]

 Example: uid example-group 0-499,600-699
This example includes all UIDs from 0-499 and from 600-699 as members of the group example-group

When planning a strategy for defining groups, use the following steps:

1. Decide how users/customers should be organized into groups.
   • This organization could be based on needs, paid service levels, or any other criteria.
   • Ideally, user UIDs ranges should be used to define users with similar requirements.
2. Edit /etc/betterlinux/iothrottled.conf.d/groups.conf.
   • For example:
     uid normal-users 500-999
     uid premium-users 1000-1999
3. Save the groups.conf file.
4. Restart the BetterIO daemon:
   service iothrottled restart

Once this parameter change is in place, you can define throttling limits that meet each group's SLA.

If the users are already in a group defined in /etc/group, it is fairly easy to use that group definition within the groups.conf file. Using the following command will extract the group definition and reformat it for inclusion in the /etc/betterlinux/iothrottled.conf.d/groups.conf file. Just replace {group_name} with the group name defined in /etc/group.

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting I/O Bandwidth Limits

Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis. To do this, edit the user_limits.conf file located in /etc/betterlinux/iothrottled.conf.d. This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/groups.conf.

The user_limits.conf file uses a single line per entry:

[group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.

For example:

example-group bw max sda,sdb,sdc 10000000

This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.

Another example:

io_standard_users bw max inc="^/",exc="^/home" 5000000

This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.

After updating the user_limits.conf file with limits applicable to the system, save the file and restart iothrottled.

Note:

• If a group and a user have the same name, the limit applies to the user.
• Only one limit per device can be set. It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.

Please see the BetterIO documentation for more configuration options for I/O bandwidth limiting features by program and inclusion/exclusion options.

Tuning I/O Scaling and Balancing

The auto_scale and seek_balance features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in user_limits.conf.

These settings are contained in /etc/betterlinux/iothrottled.conf.d/scale.conf. The configuration parameters define how these settings affect the I/O bandwidth. Details on how these settings work can be found in the user documentation for scaling and balancing.

Start the BetterIO daemon (iothrottled)

BetterIO is now configured with basic throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service iothrottled {start|stop|status|reload|restart|condrestart}
 start = starts iothrottled daemon
 stops = stops iothrottled daemon
 status = shows the status of iothrottled daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Use the following command to start the iothrottled daemon:

service iothrottled start

Start the BetterLinux stats program (blstat)

To see user processes that are being throttled by BetterIO, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterIO. Press 'i' to see the stats for BetterIO. Press '?' to display general help for the utility.

BetterMySQL Concepts

BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.

It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated. Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.

Install and Configure BetterMySQL

BetterMySQL cannot be installed unless MySQL is already on the system. The BetterMySQL HOWTO provides a step by step guid for installing and configuring BetterMySQL on Linux and Linux with cPanel.

Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the BetterCPU functional documentation. BetterMySQL’s configuration file serves only to enable BetterMySQL.

Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set. For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit:

start_throttling group=cpanel_users cpu_percent=50 seconds=3
stop_throttling group=cpanel_users cpu_percent=30 seconds=5

Same logic apply's to any BetterIO limits that have been set. In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled.

cpanel_users bw max inc="^/" 5000000

Configuring BetterMySQL for cPanel Support

BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.

Installing the BetterLinux version of MySQL for cPanel

BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL. If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script:

/etc/betterlinux/cpanel/bl-cpmysql-install

Configuring BetterMySQL for cPanel support

To provide and configure per-user query throttling of MySQL users a mapping of MySQL users to Linux users must be configured. This mapping and other configuration options are auto-configured by either the bl-cpanel-install script on first time installation or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the bl-hooks script can be used after a manual installation of BetterMySQL. Example syntax for running the bl-hooks script:

/etc/betterlinux/cpanel/bl-hooks --inint

Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.

• usersmap.conf contains the mapping of cPanel users to UIDs
• mysql.conf contains the parameter location of the usermap.conf file
• turns on BetterMySQL throttling by adding the approved_accounting parameter to /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

approved_accounting_program program=/usr/libexec/mysqld user=mysql

• Restarts the cpud daemon to enable BetterMySQL throttling.

service cpud restart

Configuring BetterMySQL for Linux Servers

BetterMySQL configuration is a manual setup for Linux servers

Installing the BetterLinux version of MySQL non-cPanel

After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command:

yum install mysql-server

Configuring BetterMySQL for non-Cpanel Linux servers

To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.

The command format is as follows:

bl_usermap_filename [usermap_file_path]

Example:

bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

Now create the MySQL mapping to Linux user mappings located in the following file:

/etc/betterlinux/mysql.conf.d/usermap.conf

The usermap file is a tab-delimited file that contains the following information:

[mysqluser]	[linux_user]	[uid]

The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.

For example, the following file will create mappings for testuser1 and testuser2. For testuser1, the uid will be determined when that user makes a query against the database.

testuser1	testuser1
testuser2	testuser2	501

The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;":

mysql> show user_map;
+-----------+-----------+------+
| From_user | Unix_user | Uid  |
+-----------+-----------+------+
| testuser2 | testuser2 |  501 |
| testuser1 | testuser1 |  500 |
+-----------+-----------+------+
2 rows in set (0.00 sec)

The approved_accounting parameter turns on the BetterMySQL throttling and is located in:

/etc/betterlinux/cpud.conf.d/main.conf

The syntax is as follows for non-cPanel servers:

approved_accounting_program program=/usr/sbin/mysqld user=mysql

Restart the cpud daemon to enable the configuration settings:

service cpud restart

Monitoring MySQL Utilization with myusertop

BetterMySQL includes the myusertop utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.

Once myusertop is running, press '?' to view the built-in help.

The output is similar to the output of the standard 'top' utility:

mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
Time: CPU 448          Busy 529         CPU/Busy 0.8477   RowsExamined:   1,476m
Qrys: Select 28,456    Update 0         Other 5,250         Adj-RowsEx:   1,251m
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
      h=Help m=MySQLUsersOn/Off                                           q=quit
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx   %CPU
root                   0     2 0.0075   0.0074   1.0060     150      151  100.00
  root                 0     2 0.0075   0.0074   1.0060     150      151  100.00
testuser2            501     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser2          501     0 0.0000   0.0000   0.0000       0        0    0.00
testuser1            500     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser1          500     0 0.0000   0.0000   0.0000       0        0    0.00

In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization. If it were divided equally between the three users, myusertop would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.

If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.