Old Doc Pages

BetterBandwidth Concepts

BetterBandwidth uses 'tc' (a built in Linux tool) to provide traffic shaping for users that have their own network interface. The tc tool on its own does not provide a mechanism for limiting bandwidth usage by uid/user on a shared network interface. BetterLinux has added this functionality into its BetterBandwidth feature. Now admins can limit bandwidth for a single user on a shared server that has a corporate internet connection.

A common problem for web hosts is that one single user with a 1Gbit connection could kill the entire TCP stack. Often the host is unaware that it happens because they use 5 min bandwidth averages to monitor bandwidth on their 1 gig or 10 gig port. But all it takes is a 30 second spike in bandwidth out of 5 mins to make customers angry. And because of the 5 minute average it looks like the peak was around 50Mbits when it easily could have been 500Mbits. Customers will blame it on the server or the hosting company.

Many hosting companies give out IPs just to be able to limit bandwidth. This is an expensive approach. The BetterLinux BetterBandwidth feature can do it without giving out dedicated IPs.

BetterLinux has an apache mod called mod_betterlinux that allows BetterBandwidth to limit bandwidth for all ports for an individual user or group of users like email, FTP, etc is all included. This is a tremendous advantage for a host. BetterBandwidth monitors and controls individual, group, and system-wide bandwidth usage in multi-tenant Linux environments. Administrators can now limit outbound bandwidth as if each user had a dedicated IP address. This lets admins consolidate multiple users onto a single IP, which innovatively addresses the fast-increasing IPV4 IP address shortage.

Configure BetterBandwidth

Follow the BetterBandwidth HOWTO guide for configuring bandwidth limiting

Bandwidth limiting

BetterBandwidth limiting uses the mod_betterlinux Apache module located in in the betterlinux-core-*.rpm , as well as the linux tc method of using the uid classifier to categorize specific user processes. The betterlinux-core-*.rpm is installed by the bl-install script by default and auto-configured for cPanel systems during the betterlinux-cpanel-install script.

cPanel Bandwidth Limiting Setup

During the betterlinux-cpanel-install script installation, the user will answer Y/N to this question:

Install BetterLinux Apache throttling? [y/N]

Selecting yes, will auto-configure and load the apache module mod_betterlinux on a cPanel system.

After the installation is complete, the following parameters and arguments need to be added to the BetterLinux configuration files:

• In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example:

interface interface-types eth0

• In /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format:

bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to:

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

non-cPanel Bandwidth Limiting Setup

• Depending on the distribution, edit the apache conf or create a new one under conf.d and add the following:

LoadModule betterlinux_module /usr/lib64/httpd/modules/mod_betterlinux.so
<IfModule betterlinux_module>
 Betterlinux On
</IfModule>

Then restart apache.

Using this command will check the mod_betterlinux syntax in the apache .conf file:

apachectl -t -D DUMP_MODULES

• Configure BetterCPU (cpud) by adding or uncommenting the following parameter located in /etc/betterlinux/cpud.conf.d/main.conf

 approved_accounting_program /usr/sbin/httpd apache

Please note, the full path to the httpd program is required, and it must be the actual program and not a symlink. Also, be sure to list the user it runs as, whether that is apache, httpd, or different username.

• In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example:

interface interface-types eth0

• In /etc/betterlinux/cpud.conf.d/main.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format:

bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to:

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

Assigning IP addresses with the dedicated IP address mapper

With multi-homed systems, incoming traffic is directed to the IP address where DNS resolves the domain name. A system with multiple virtual hosts can accept traffic to secondary addresses which, in turn, are assigned to secondary addresses, which are assigned to the system's network adapters.

Without some modification, responses sent from a server in a multi-homed system originate from the host's primary IP address. This makes it difficult to apply bandwidth or connection limits using a server's IP address. Most often, hosting providers work around this problem by implementing some sort of full virtualization solution that provides a unique virtual instance for each client. <KGO> What is the drawback of this solution?

BetterBandwidth uses the dedicated kernel module to provide a more cost-effective, efficient solution for managing bandwidth in multi-homed environments. When used in conjunction with the BetterBandwidth dedicated daemon, the dedicated kernel module allows administrators to assign outgoing traffic to a specific secondary address which, in turn, is assigned to a specific user--all without the overhead of a fully virtualized system. (Both IPv4 and IPv6 are supported.)

To enable this functionality:

1. Ensure the dedicated kernel module is loaded:
   

   modprobe dedicated

2. Edit /etc/betterlinux/ipv4 and/or /etc/betterlinux/ipv6 to include the specific mappings.
3. Restart dedicatedd:
   

   service dedicatedd restart

The IPv4 and IPv6 mapping files are in the following format. Each line contains a single entry.

username|uid: address

The address field can contain either a raw IPv4 or IPv6 address or a DNS name. If the DNS name resolves to more than one address, the kernel randomly selects one of the addresses.

Once configured, the active mappings for IPv4 addresses can be viewed using the following command:

cat /proc/sys/net/ipv4/dedicated

To view mappings for IPv6 addresses, use the following command:

cat /proc/sys/net/ipv6/dedicated

Note: The command output displays the uid rather than the username.

BetterCPU Configuration

Refer to the BetterCPU HOWTOs section for details on configuration and setup.

BetterCPU Concepts

BetterCPU is designed to maximize cpu utilization for shared hosting environments and at the same time prevent one or many users from abusing the cpu utilization. BetterCPU makes it very easy to quickly identify the CPU hogs on the system and keep them under control and then decide if these users need to moved to a higher paying account that provides them with the resources they need to run their business. This is accomplished by defining BetterLinux groups of users or mapping BetterLinux groups to cPanel packages. Once the user groups are configured, then throttling limits are defined for CPU.

Defining jail cores

Before limits are configured, it's important to understand BetterCPU jails. The jails are elastic by nature and only exist if they are needed by the BetterCPU controller to temporarily limit an abusive user process. Think of process jailing as analogous to law enforcement:

• A user is a “citizen”
• The performance thresholds are the “law”
• BetterCPU enforces the laws

A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.

If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.

When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.

Configuration of the jail is done using the parameters:

jail_cnt
max_jail_cnt

Commenting both of the parameters out sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1. BetterCPU then auto-scales the jails the system needs based on demand for jail cores. However, the jail cores will never take a priority over non-jail cores.

The jail_cnt can be defined using two different methods:

• If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jai_cnt. For example:

jail_cnt 2

The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems. If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.

• is expressed as one or more pairs of numbers. For example:

jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail; an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system. The 32 core system will be assigned 6 cores as the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails, and is written in the same manner as the jail_cnt parameter. Setting the max_jail_cnt parameter will fix the ultimate size of the jail. Fixing the jail size will determines the amount of the cpu utilization abusive users will share in the jail core(s).

By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.

Defining BetterCPU Throttling Limits

The BetterCPU throttling limits are defined by a percentage of one core for each user.

Once the jail cores are defined, BetterCPU provide two throttling parameters that define when a user is moved into and out of the jail core(s). The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set jail throttling limits for groups of uids defined in groups.conf or betterlinux-cpud.conf on cPanel installs. The cpu_percent sets the percentage a user (uid) can use of one core on the system. The values for cpu_percent in start_throttling and stop_throttling are based on one core and are not adjusted when cores are added or removed from the jail.

When a throttled user’s total utilization drops below the stop_throttling for cpu_percent and "seconds" settings, then the process is removed from the jail.

For example, consider an 8-core system with the following values configured:

start_throttling group=example-group cpu_percent=50,time=3
stop_throttling group=example-group cpu_percent=25,time=10

A user whose processes exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.

Throttling Users

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. By default, the non-cPanel installation (betterlinux-install) does not configure BetterCPU to throttle users. However, the cPanel installation (betterlinux-cpanel-install) script does configure and start BetterCPU throttling for all cPanel users on the server. The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).

Configuring the Jail Cores

Throttling is configured using parameters located in:

non-cPanel install: /etc/betterlinux/cpud.conf.d/rules.conf:
cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

jail_cnt
max_jail_cnt

These settings are designed so that the same configuration file may be used on different systems with different numbers of cores. However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file. When these parameters are removed, the cpud daemon uses the default settings wich is jail_cnt=1 and max_jail_cnt=total cores minus 1 and dynamically auto-scales up or down the number of jail cores needed for the system.

Syntax:

jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]

The jail_cnt parameter is defined using one of two methods:

• If a fixed number of cores for jail is desired, make the assignment using jail_cnt as in this example:

jail_cnt 2

In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core. If the system is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.

• If the number of cores for jail varies based on the total number of cores in the system, use the following format:

jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. For example, using the values specified in the previous example, a 32-core system will use the jail_cnt value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails and is formatted identically to the jail_cnt parameter. By default, only the value of jail_cnt is used, but when a max_jail_cnt is specified, the jail is expanded when less than the value in jail_min_idle is available per jail core.

The jail_min_idle value is the minimum amount of idle utilization to maintain on the jail cores if jail_max_cnt has not been reached. If the average free utilization on each jail core is less than jail_min_idle, the jail is expanded until either the jail_min_idle value is maintained or jail_max_cnt is reached.

Syntax:

max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}

Example

max_jail_cnt 2:1 4:2 8:2 16:2

The result on a dual core system is no different from jail_cnt. But for 4-core machines and above, if the max_jail_cnt value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded.

max_jail_cnt 2

The result will always set the number of maximum jails to two. The maximum number of jails are set at two no matter how many cores may be on the system. The exception being, when they system has 2 cores it will set the max jail core to one.

In order for the jail to expand, add a jail_min_idle value to denote how much free utilization should be maintained in the jail:

jail_min_idle 5

BetterCPU will attempt to maintain at least 5% free CPU per jail core, and will expand the jail if max_jail_cnt has not been reached.

Defining User Groups

For more details on groups please see the Defining User Groups section of the documentation.

The BetterLinux cPanel install script will discover all the cPanel users on the system and create a user group called "cpanel-users" that contains a list of all the users assigned to a cPanel package.

The bl-install script will not create user groups, they need to be created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, or files containing UIDs or usernames, CPU usage levels can be precisely defined for different groups, and durations and conditions for starting/stopping throttling can be defined. The group names defined in /etc/betterlinux/cpud.conf.d/groups.conf are used in the settings for the start_throttling and stop_throttling parameters.

Defining User Groups

When planning a strategy for defining groups, use the following steps:

1. Decide how users/customers should be organized into groups or cPanel packages. This organization could be based on needs, paid service levels, or any other criteria.
2. Ideally, user UIDs ranges or a file that contains a list of UIDs should be used to define users with similar requirements. The BetterLinux cPanel hooks feature will automatically create a uid text file with a name that matches the cPanel package and will add newly created cPanel users to the file.
3. Edit /etc/betterlinux/cpud.conf.d/groups.conf or the BetterLinux cPanel hooks feature will do this automatically.
4. For example:
   uid package-normal-users /etc/betterlinux/cpanel/package-normal-users.txt
   uid package-premium-users /etc/betterlinux/cpanel/package-premium-users.txt
5. Save the groups.conf file
6. Restart the BetterCPU daemon:
   service cpud restart

Once this parameter change is in place, throttling limits that meet each group's SLA can be defined for each group.

For non-cPanel servers, the users will likely already be in a group defined in /etc/group. If this is the case, it is fairly easy to use that group definition within the groups.conf file. Use the following command to extract the group definition and reformat it for inclusion in the /etc/betterlinux/cpud.conf.d/groups.conf file. When defining the syntax, simply replace {group_name} with the name of the group defined in /etc/group.

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting CPU Core Usage Limits

Once jail_cnt and jail_cnt_max are defined and groups are set up, it is necessary to configure the throttling parameters that define when user groups are moved into and out of the jail cores. The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set throttling limits for groups defined in /etc/betterlinux/cpud.conf.d/groups.conf. The cpu_percent value sets the percentage an individual user (uid) can use of the total non-jail cores on the system.

To understand how this works, it is important to understand how CPU utilization is measured in a multi-core system. In Linux, multithreaded processes can take more than 100% utilization because overall CPU utilization is measured as 100% * number_of_cores. So in an 8 core system, if all CPUs are running at maximum utilization, the system's overall utilization is 800% (100% * 8 cores).

The values for cpu_percent in start_throttling and stop_throttling are based on a normalized utilization assigned to a single core (rather than being spread across all cores). This means that while an 8 core system can have a maximum of 800% utilization, the value used for the cpu_percent value represents the fraction of a single core that could be used. Specifying a value of 80 for cpu_percent means that on a system with no cores presently allocated to jail, when the user exceeds a value of 80% * 8 = 640% on an 8 core system, a core is allocated to the jail and the user's processes are throttled. When a core is allocated to jail, the percentage used to determine if the threshold has been exceeded is 80% * 7 = 560%. If a second core is allocated to jail, this threshold drops to 80% * 6 = 480%.

What this means is that - using the example of cpu_percent set to 80 - any distribution of utilization across all 8 cores of an 8 core system that adds up to more than 640% utilization will trigger the allocation of a core for throttling and that user's processes will be moved to that single core. The utilization could be 100% of 6 cores and 40% of a seventh core, it could be 80% utilization distributed across all 8 cores, or it could be any variation where the user's total utilization exceeds the threshold.

When a non-throttled user's total process utilization exceeds cpu_percent for seconds seconds as defined in the start_throttling parameter, the user's processes are moved into the jail cores.

When a throttled user’s total utilization drops below the cpu_percent value for seconds seconds as defined in the stop_throttling parameter, the user's processes are moved out of the jail back onto the non-jailed cores.

To define these limits:

• Edit the users.conf located in /etc/betterlinux/cpud.conf.d.
• Add the following parameters users.conf, with appropriate limits to the system's needs:

start_throttling group=example-group cpu_percent=50,seconds=3
stop_throttling group=example-group cpu_percent=25,seconds=10

• Save users.conf.
• Restart cpud:

service cpud restart

With these settings, a user in the group defined with processes that exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system. When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.

Start the BetterCPU Daemon (cpud)

BetterCPU is now configured with basic jail core and throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service cpud {start|stop|status|reload|restart|condrestart}
 start = starts cpud daemon
 stops = stops cpud daemon
 status = shows the status of cpud daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Start the cpud daemon with the following command:

service cpud start

Start the BetterLinux Stats Program (blstat)

In order to see user processes that are in the jail cores, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterI/O. Press 'c' to see the stats for BetterCPU. Press '?' to display general help for the utility.

Limiting Execution Time Using time_monitor

BetterCPU can be used to limit the amount of time programs can execute. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting execution time are:

1. Define a program or user group
2. Add time_monitor to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is as follows:

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example:

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. The user groups are structured as follows:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for time_monitor execution limiting are identical to user groups used for other functions in the BetterLinux suite.

Add time_monitor to main.conf

Limiting the execution time for a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again. The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.

To define the limit, add the following line to the main.conf file:

time_monitor groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer 10

In this example, we use the sample "bad_progs" program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash. If the program has run for 5 seconds or longer, /root/killer is called with the PID of the program, and the script kills the program after logging the date and time.

The final parameter (10) tells cpud to not run the script more than once every 10 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring the execution time, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:

time_monitor_prog_exclude groups=exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

Limiting Open Files Using file_limit

BetterCPU can be used to limit the number of concurrent open files a user can have at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent open files are:

1. Define a user group
2. Add file_limit to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When open files are limited, the file_limit_override and file_limit_exclude options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the following format used is:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting open files with file_limit are identical to user groups used for other functions in the BetterLinux suite.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Processes Using process_limit

BetterCPU can be used to limit the number of concurrent processes a user can execute at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent processes are:

1. Define a user group
2. Add process_limit to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the format used is:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting concurrent processes with process_limit are identical to user groups used for other functions in the BetterLinux suite.

Add process_limit to main.conf

Limiting concurrent processes per user is very straightforward.

To apply a limit, once a user group is defined, add a line to apply the desired restiction to the main.conf file:

process_limit group=bad_users limit=100

This limits the users in the bad_users group to 100 concurrent processes each at any given time.

To completely exclude specific programs from having limits applied, create a program group in groups.conf with the programs in it:

prog exclude_process_limits /bin/cron,/bin/bash

Then add a process_limit_exclude line to main.conf:

process_limit_exclude group=exclude_process_limits

If a user limit should be overridden for particular programs, specify the user group as part of a process_limit_override command and specify the specific programs that should have the different limit in main.conf:

process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd

This overrides the existing limits for bad_users for instances of /usr/sbin/sshd and sets those limits to 120.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Memory Utilization Using memory_limit

BetterCPU can be used to limit the amount of RSS memory programs can use. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting RSS memory usage are:

1. Define a program or user group
2. Add memory_limit to /etc/betterlinux/cpud.conf.d/main.conf
3. Reload the BetterCPU daemon's configuration

When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is:

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example:

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. These user groups are structured as:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for RSS memory usage limiting are identical to user groups used for other functions in the BetterLinux suite.

Add memory_limit to main.conf

Limiting the RSS memory used by a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again. The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.

To define the limit, add the following line to the main.conf file:

memory_limit group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run=30

In this example, we use the sample bad_progs program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB). If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.

The final parameter (30) tells cpud to not run the script more than once every 30 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:

memory_limit_prog_exclude exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

BetterIO Configuration

Refer to the BetterIO HOWTOs section for details on configuring BetterIO.

BetterIO Concepts

The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.

In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.

From a CPU perspective, this is referred to as “Waiting on I/O”, or iowait%. If the I/O channel is not also managed, then the system will not be able to run optimally.

When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.

There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To monitor these factors, one can use the iostat command or blstat (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.

What can BetterIO tune to affect block device utilization?

At a high level, tuning is applied to I/O operations per second (iops) and bandwidth (bw). These two parameters can be tuned by applying minimum and maximum values in /etc/iothrottled/iothrottled.conf.d/rules.conf.

Those system-wide minimum and maximum values can be refined by applying user limits and program limits, which are defined in /etc/iothrottled/user_limits.conf and /etc/iothrottled/prog_limits.conf, respectively.

The limits defined can be scaled to allow the block device channel to be utilized more effectively when it is under-utilized. The auto_scale parameter defines how scaling is applied to keep both the bandwidth utilization and iops within a desired performance range.

Setting I/O Bandwidth Limits Per-User or by User-Group

Individual user or group limits can be set using the configuration file /etc/iothrottled/user_limits.conf.

This file uses a single line per entry:

[user|group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or mount points to include or exclude.

For example, the file might contain a line that reads:

testuser bw max sda,sdb,sdc 10000000

This would mean that the user testuser has their maximum bandwidth (bw) limited on devices sda, sdb, and sdc to 10000000 B/s (Bytes per Second).

The first field can be a user or group name. Groups are defined in the file /etc/betterlinux/groups.conf. If a group and a user have the same name, the limit applies to the user.

Setting I/O Bandwidth Limits by Program-Group

More granular control over individual programs’ usage of block device I/O rates and data rates can be achieved using the /etc/betterlinux/iothrottled.conf.d/prog_limits.conf file.

This file is similar to the user_limits.conf file, except that it specifies a program group name. The line format for this file is:

[program_group_name] [S|G] [bw|ops] [device/mount point list] [limit]

The fields that match up with fields in prog_limits.conf are defined the same way as in user_limits.conf.

The prognam_group_name field contains the names of the program or programs which is defined in groups.conf.

If the third field contains an “S”, the limit applies to each instance of the program started by the use or group. A “G” means the limit applies to all instances together.

For example: NOTE: In all the examples below we assume to have the group "tar_progs" defined in /etc/betterlinux/iothrottled.conf.d/groups.conf:

prog tar_progs /bin/tar,/usr/bin/tar

If our goal is to limit each individual instance of "tar" to a 5 MB/s I/O rate we would use the following line.

tar_progs S bw max inc="^/" 5000000

NOTE: all the "/bin/tar" and "/usr/bin/tar" instances will be affected by this rule if the UID that runs them is not included in any UID group. Remember that UID groups always win against PROG groups.

If our goal was to group ALL instances of tar to an aggregate bandwidth of 5 MB/s then you would change the "S" (Single process) in the example above to "G" (For grouping all tar processes together). This option is helpful when you want to limit one or more instances together. So, if you had one tar running or 50 tars the total disk I/O bandwidth would never exceed 5 MB/s.

The syntax for this scenario would be as follows:

tar_progs G bw max inc="^/" 5000000

If our goal was to limit each instance of tar to a disk I/O rate of 5 MB/s, but would only like to enforce this limit on any filesystem mounted under /home (and any sub-directories) but have no limits enforced on other drives then the syntax would be as follows.

tar_progs G bw max inc="^/home" 5000000

You can include multiple block devices (hard drives) or mount points to be included in the limits that you have defined as shown below:

tar_progs G bw max inc="/home",sdb,dm-1,inc="/var/matt",inc="/etc" 5000000

In addition, you can specific drives to be excluded from the limits that you have defined. This syntax is shown below.

tar_progs G bw max inc="/",exc="^/media|^/mnt" 5000000

For example:

tar_progs S bw sda 1000000

This applies a bandwidth limit of 1000000 bytes/s to each instance of the /bin/tar or /usr/bin/tar commands run by any user not included in a UID group. If one of these users starts two concurrent instances of the tar command, each one is limited to 1000000 b/s of block device I/O bandwidth. If this line reads:

tar_progs G bw sda 1000000

The bandwidth limit is applied to all instances of tar that the users launch. In this case, if the users launch two concurrent instances of the tar command, the total bandwidth both instances can use together is 1000000 B/s.

Below are some configuration rule examples of how to limit I/O bandwidth for certain programs.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied. The inc= and exc= directives take an extended regular expression for mount points. For example, the line:

tar_progs G bw max sda,inc="/home|/backup",exc="/mnt1|/mnt2" 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that have "/home" or "/backup" in their mount point (that would also include, for example, "/home2", "/media/home3", excluding all the mount points that have "/mnt1" or "/mnt2" (like "/mnt21", "/mnt/mnt1", ...). The values are regular expressions and they can appear anywhere within the mount point). Including ^ and $ in the regex will force a match at the begining or end.

The evaluation of these expressions always follows the order of:

• Explicit device names
• Devices mounted in the "inc" list
• Devices mounted in the "exc" list

Exclusions will ultimately override the other values in the list. Consider the following list of mounted devices:

/dev/sda mounted at /
/dev/sdb mounted at /home
/dev/sdc mounted at /media/backups
/dev/sdd mounted at /usr
/dev/sde mounted at /mnt/home
/dev/sdf mounted at /srv/www
/dev/sdg mounted at /srv/ftp
/dev/sdh mounted at /srv/home

With the following restrictions in user_limits.conf:

tar_progs G bw max sda,sdd,inc="/home",exc="/srv" 1000000

Programs in group "tar_progs" will be limited to 1 MB/s on /home and /mnt/home, but not on /srv/home, due to the exc="/srv" rule.

How do the block device inclusion/exclusion lists work?

Simple configuration of BetterIO uses device names in a list in the user_limits.conf and prog_limits.conf files. For example:

testuser bw max sda,sdb,sdc 1000000

This applies the limit of 1000000 B/s for the user testuser on devices /dev/sda, /dev/sdb, and /dev/sdc.

Modern Linux systems, however, do not always guarantee the assignment of devices, which can lead to issues with limits not being applied to the correct device.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied.

The inc= and exc=' directives take an extended regular expression for mount points. For example, the line:

testuser bw max sda,inc=”/home|/backup”,exc=”/mnt1|/mnt2” 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that are mounted at “/home” and “/backup” (and would also include, for example, “/home2”, “/media/home3” - the values are regular expressions and can appear anywhere within the mount point), but would exclude any mount point that includes “/mnt1” or “/mnt2” (including “/mnt21”, “/mnt/mnt1” - as with the “inc=” directive). Including ^ and $ in the regex will force a match at the begining or end.

Another example for this using regular expression that will apply the limit of 1MiB/s to any mounted block device except those mounted under /var or /tmp

testuser bw max inc=”^/”,exc=”^/var|^/tmp” 1000000

In this example we want to limit all the mounted block devices, except the temporary mounted devices, usually mounted under /mnt or /media.

• add the line below to /etc/betterlinux/groups.conf:

testgroup uid 500-1000

• Then add the line below to /etc/betterlinux/iothrottled/user_limits.conf:

testgroup bw max inc=”^/”,exc=”^/mnt|^/media” 1000000

The evaluation of these expressions always follows the order of:

1. Explicit device names
2. Devices mounted in the “inc” list
3. Devices mounted in the “exc” list

Exclusions will ultimately override the other values in the list.

Consider the following list of mounted devices:

/dev/sda1 mounted at /

/dev/sda2 mounted at /home

/dev/sdb1 mounted at /media/backups

/dev/sdb2 mounted at /usr

/dev/sdc1 mounted at /mnt/home

/dev/sdc2 mounted at /srv/www

/dev/sdd1 mounted at /srv/ftp

With the following restrictions in user_limits.conf:

testuser bw max sda,sdd,inc=”/home”,exc=”/srv” 1000000

The result would be:

sda would be included

sdb would be excluded

sdc would be excluded

sdd would be excluded

So the only device controlled by BetterIO in this scenario would be /dev/sda. The controls are applied per block device, not on a per-partition basis - so if a partition is excluded from control, all partitions on that block device will be excluded.

When configuring your system, it is important to take this into consideration - you do not want your swap partition to be throttled by BetterIO, but if you put swap on /dev/sda and control /dev/sda with BetterIO, you may end up impacting performance on the system in ways you may not intend to.

Examples include/exclude:

/dev/sda1 on /

/dev/sda2 on /home

/dev/sdb1 on /home2

/dev/sdc1 on /home3

/dev/sdd1 on /home4

testuser bw max inc="/home",exc="/home2"

The rexeg's are matched against the left of the mount point. So the include matches /home1, /home2, /home3, /home4 and the exclude matches /home2 only. So the result is:

/dev/sda included

/dev/sdb excluded

/dev/sdc included

/dev/sdd included

Example Exclude:

/dev/sda1 on /

/dev/sda2 on /home

/dev/sdb1 on /src

/dev/sdb2 on /usr

/dev/sdc1 on /extra

testuser bw max exc="/src"

/dev/sda included

/dev/sdb excluded

/dev/sdc included

How does auto scaling work?

The auto_scale parameter is an on/off toggle switch for enabling or disabling the auto scale feature. The auto scaling algorithm is applied to overall bandwidth utilization or iops giving the system administrator the ability provide more bandwidth to a defined set of users when it's available.

When setting the value for this parameter, an integer is entered that represents the on/off values:

0 = auto scaling off

1 = auto scaling on

Auto scaling is controlled by these parameters:

scale_pct_hi

scale_pct_low

scale_pct_periods

Additionally, the following parameters apply to auto scaling:

scale_min

scale_max

scale_up_inc

scale_down_inc

To understand how these parameters are related to each other, consider the following scenario:

With the following line in /etc/iothrottled/user_limits.conf:

testuser bw max sda 1000000

And in /etc/iothrottled/iothrottled.conf.d/rules.conf:

auto_scale 1

scale_pct_hi 80

scale_pct_low 20

scale_pct_periods 2

scale_min 25

scale_max 500

scale_up_inc 20

scale_down_inc 5

If the overall block device utilization is below 20% (scale_pct_low), then the limit applied to testuser is increased by 20% (scale_up_inc) of 1000000 (taken from user_limits.conf for this user), or 200000 B/s (Bytes per Second).

If overall block device utilization goes over 80% (scale_pct_hi), then the limit applied to testuser is decreased by 5% (scale_down_inc) of 1000000 (from the user’s limit in user_limits.conf), or 50000 B/s.

The increase/decrease in the scaling factor is tracked as well, so on the first cycle, the scaling factor is 120%. The second iteration scales up to 140%, and so on, until the limit set by scale_max is reached.

Similarly, if scaling down were to be applied instead, that scaling would drop by 5% each iteration until the scale factor reached the scale_min limit of 25%.

The result is that BetterIO attempts to keep the block device channel utilization between 20% and 80% at all times, and adjusts the utilization limits for all users in order to achieve that goal. If one were to graph the utilization over time, the goal is to have that graph look something like this:

Whenever the utilization drops below the threshold at 20% or goes above the threshold at 80%, the adjustments kick in so the utilization falls in the area between the two thresholds. The majority of the time, that device’s utilization should be between 20% and 80% with this configuration.

Those adjustments are subject to the scale_min and scale_max parameters. Having limits on the amount of scaling will prevent BetterIO from attempting to scale up without a limit when there is no I/O taking place. If it were allowed to do that, then a period with no I/O could end up with a scaling factor in excess of 1000%, and the process of scaling down would result in it taking a significant amount of time with no effective limit in place, which would make the available bandwidth the bottleneck for system performance until the scaling factor returned to a reasonable range.

How can one test to see if BetterIO is controlling devices?

A simple and quite dramatic way to see BetterIO limiting I/O is to set a lower threshold for a user that is very small. For example, create a user called testuser and log in as that user. Run the following command:

sync; dd if=/dev/zero of=temp.fil bs=1M count=10 oflag=direct; rm temp.fil

The results from that command for an unmanaged user would be similar to this screenshot:

If we then modify the /etc/iothrottled/user_limits.conf file to include the line:

testuser bw max inc=”/” 500000

and rerun the dd command above, the following output results:

The drop in throughput from 350-400 MB/s to 500-600 kB/s shows the effect of the change in bandwidth limits for this user.

Are there any other considerations to keep in mind?

With BetterIO, the limits are set on a per-block device basis. It is important to keep in mind that I/O channels are not always assigned in a 1:1 ratio with block devices. If a series of storage devices are connected to the system by a Fibre Channel connection, the channel bandwidth available for all of those devices needs to be taken into consideration across all of the devices.

If devices /dev/sda, /dev/sdb, and /dev/sdc are all on the same disk controller, and the maximum throughput of that controller is 1 Gbps, then all three disks cannot run at 1 Gbps even if the drive electronics would permit that.

If you then have 10 users limited to 100 Mbps each, then performance will effectively be running without any throttling because the disk channel will be oversubscribed if all of the users are attempting to fully utilize their allocated bandwidth to each of these three devices. At the same time, these users are still limited to 100 Mbps each and are unable to monopolize the resources.

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. Once the the system is rebooted, configure BetterIO and start the iothrottled daemon. By default, BetterIO does not throttle any users. The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).

Defining user groups

User-created groups do not exist on the system until they have been created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined. The group names defined in /etc/betterlinux/iothrottled.conf.d/groups.conf are used in the settings for setting per-group I/O limits.

To define groups:

1. Choose a group name.
2. Populate the group with UIDs, UID ranges, user names, or files containing UIDs or usernames
   Note: A group may contain one or more usernames, UIDs, or ranges.

The following examples illustrate the command syntax:

Single range:
 Syntax:  uid {group name} {uid range}

 Example: uid example-group 0-499
This example includes all UIDs from 0-499 as members of the group example-group

 Example: uid example-group 800-
This example includes all UIDs from 800 and above as members of the group example-group
 
Multiple UID ranges:
 Syntax:  uid {group name} {uid range}[,{uid range2}...]

 Example: uid example-group 0-499,600-699
This example includes all UIDs from 0-499 and from 600-699 as members of the group example-group

When planning a strategy for defining groups, use the following steps:

1. Decide how users/customers should be organized into groups.
   • This organization could be based on needs, paid service levels, or any other criteria.
   • Ideally, user UIDs ranges should be used to define users with similar requirements.
2. Edit /etc/betterlinux/iothrottled.conf.d/groups.conf.
   • For example:
     uid normal-users 500-999
     uid premium-users 1000-1999
3. Save the groups.conf file.
4. Restart the BetterIO daemon:
   service iothrottled restart

Once this parameter change is in place, you can define throttling limits that meet each group's SLA.

If the users are already in a group defined in /etc/group, it is fairly easy to use that group definition within the groups.conf file. Using the following command will extract the group definition and reformat it for inclusion in the /etc/betterlinux/iothrottled.conf.d/groups.conf file. Just replace {group_name} with the group name defined in /etc/group.

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting I/O Bandwidth Limits

Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis. To do this, edit the user_limits.conf file located in /etc/betterlinux/iothrottled.conf.d. This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/groups.conf.

The user_limits.conf file uses a single line per entry:

[group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.

For example:

example-group bw max sda,sdb,sdc 10000000

This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.

Another example:

io_standard_users bw max inc="^/",exc="^/home" 5000000

This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.

After updating the user_limits.conf file with limits applicable to the system, save the file and restart iothrottled.

Note:

• If a group and a user have the same name, the limit applies to the user.
• Only one limit per device can be set. It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.

Please see the BetterIO documentation for more configuration options for I/O bandwidth limiting features by program and inclusion/exclusion options.

Tuning I/O Scaling and Balancing

The auto_scale and seek_balance features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in user_limits.conf.

These settings are contained in /etc/betterlinux/iothrottled.conf.d/scale.conf. The configuration parameters define how these settings affect the I/O bandwidth. Details on how these settings work can be found in the user documentation for scaling and balancing.

Start the BetterIO daemon (iothrottled)

BetterIO is now configured with basic throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service iothrottled {start|stop|status|reload|restart|condrestart}
 start = starts iothrottled daemon
 stops = stops iothrottled daemon
 status = shows the status of iothrottled daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Use the following command to start the iothrottled daemon:

service iothrottled start

Start the BetterLinux stats program (blstat)

To see user processes that are being throttled by BetterIO, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterIO. Press 'i' to see the stats for BetterIO. Press '?' to display general help for the utility.

BetterMySQL Concepts

BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.

It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated. Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.

Install and Configure BetterMySQL

BetterMySQL cannot be installed unless MySQL is already on the system. The BetterMySQL HOWTO provides a step by step guid for installing and configuring BetterMySQL on Linux and Linux with cPanel.

Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the BetterCPU functional documentation. BetterMySQL’s configuration file serves only to enable BetterMySQL.

Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set. For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit:

start_throttling group=cpanel_users cpu_percent=50 seconds=3
stop_throttling group=cpanel_users cpu_percent=30 seconds=5

Same logic apply's to any BetterIO limits that have been set. In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled.

cpanel_users bw max inc="^/" 5000000

Configuring BetterMySQL for cPanel Support

BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.

Installing the BetterLinux version of MySQL for cPanel

BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL. If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script:

/etc/betterlinux/cpanel/bl-cpmysql-install

Configuring BetterMySQL for cPanel support

To provide and configure per-user query throttling of MySQL users a mapping of MySQL users to Linux users must be configured. This mapping and other configuration options are auto-configured by either the bl-cpanel-install script on first time installation or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the bl-hooks script can be used after a manual installation of BetterMySQL. Example syntax for running the bl-hooks script:

/etc/betterlinux/cpanel/bl-hooks --inint

Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.

• usersmap.conf contains the mapping of cPanel users to UIDs
• mysql.conf contains the parameter location of the usermap.conf file
• turns on BetterMySQL throttling by adding the approved_accounting parameter to /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

approved_accounting_program program=/usr/libexec/mysqld user=mysql

• Restarts the cpud daemon to enable BetterMySQL throttling.

service cpud restart

Configuring BetterMySQL for Linux Servers

BetterMySQL configuration is a manual setup for Linux servers

Installing the BetterLinux version of MySQL non-cPanel

After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command:

yum install mysql-server

Configuring BetterMySQL for non-Cpanel Linux servers

To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.

The command format is as follows:

bl_usermap_filename [usermap_file_path]

Example:

bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

Now create the MySQL mapping to Linux user mappings located in the following file:

/etc/betterlinux/mysql.conf.d/usermap.conf

The usermap file is a tab-delimited file that contains the following information:

[mysqluser]	[linux_user]	[uid]

The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.

For example, the following file will create mappings for testuser1 and testuser2. For testuser1, the uid will be determined when that user makes a query against the database.

testuser1	testuser1
testuser2	testuser2	501

The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;":

mysql> show user_map;
+-----------+-----------+------+
| From_user | Unix_user | Uid  |
+-----------+-----------+------+
| testuser2 | testuser2 |  501 |
| testuser1 | testuser1 |  500 |
+-----------+-----------+------+
2 rows in set (0.00 sec)

The approved_accounting parameter turns on the BetterMySQL throttling and is located in:

/etc/betterlinux/cpud.conf.d/main.conf

The syntax is as follows for non-cPanel servers:

approved_accounting_program program=/usr/sbin/mysqld user=mysql

Restart the cpud daemon to enable the configuration settings:

service cpud restart

Monitoring MySQL Utilization with myusertop

BetterMySQL includes the myusertop utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.

Once myusertop is running, press '?' to view the built-in help.

The output is similar to the output of the standard 'top' utility:

mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
Time: CPU 448          Busy 529         CPU/Busy 0.8477   RowsExamined:   1,476m
Qrys: Select 28,456    Update 0         Other 5,250         Adj-RowsEx:   1,251m
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
      h=Help m=MySQLUsersOn/Off                                           q=quit
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx   %CPU
root                   0     2 0.0075   0.0074   1.0060     150      151  100.00
  root                 0     2 0.0075   0.0074   1.0060     150      151  100.00
testuser2            501     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser2          501     0 0.0000   0.0000   0.0000       0        0    0.00
testuser1            500     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser1          500     0 0.0000   0.0000   0.0000       0        0    0.00

In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization. If it were divided equally between the three users, myusertop would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.

If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.

CloakFS File System Cloaking

"cloak_user" secures the file system by hiding users from each other, including others’ files, directories, and processes. A user can see only his or her own $HOME directory, with the exception files outside of home owned by that user. For example, if “sampleuser” creates “samplefile” in /tmp and then lists all files in /tmp, he will see only “samplefile” and not the /tmp files of other users on the box.

Configuration Files

The cloak_user parameter is found here:

Non-Cpanel install: /etc/betterlinux/cpud.conf.d/rules.conf

cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

Add values to the cloak_user parameter with the following syntax:

cloak_user group=user_group1,user_group2 level=1

The first value (following “group=”) in the above example is a comma-delimited list of groups (user_group1, user_group2, etc.), which you can define in groups.conf.

The second value (following “level=”) can be either 1 or 2. The default value is 1. If 1, a user can display only his or her own processes as well as processes belonging to users outside the groups named within this parameter (like root, mysql, and apache’s “nobody” user). A value of 2, however, allows a user to display only his or her own processes and nothing else, not even the processes of users outside the groups named within this parameter.

Consider a parameter with these values:

cloak_user group=user_group1 level=1

In the above, all users in user_group1 will see only their own processes and those of users not belonging to user_group1.

Next, see these values:

cloak_user group=user_group2 level=2

Here, all users in user_group2 will see only their own processes. This excludes even processes run by root, mysql, and apache, etc.

FAQ

I have a customized kernel configuration that requires I build the kernel myself.

• BetterLinux provides support for customized kernels via the Build Your Own Kernel (BYOK) feature.

BYOK Instructions

Does BetterLinux run on CentOS 5?

• BetterLinux supports kernels 2.6.32 or newer. We are currently evaluating a method for supporting CentOS 5. Some BetterLinux features depend on Linux kernel features that were introduced in version 2.6.26. Some of those features have bugs that were not fixed until version 2.6.32. Hence, we cannot support kernels prior to 2.6.26 at all, and kernels prior to 2.6.32 require backported bug fixes. CentOS 5 has kernel version 2.6.18. It is possible to build a later kernel for CentOS 5, but then some userspace programs, such as iptables, must also be updated to cope with changed kernel APIs.

What will the pricing be after the Free Until 2013 period is over?

• BetterLinux will continue to be free for non-profit use but will begin charging $9.00 per month per server for commercial use after 2013.

Will BetterLinux be compatible with cPanel/WHM and if so will that include plugins for WHM?

• BetterLinux is currently developing integration and plug-ins for cPanel and WHM. The targeted release date is end of November 2012.

How to install blstat?

blstat is not part of the product installation. Follow the steps below to download and install blstat:

yum --disableexcludes=all install perl-TermReadKey perl-JSON

This will install two perl modules blstat needs.

wget http://repo.betterlinux.com/unsupported/blstat-0.53-1.el6.noarch.rpm

Downloads the blstat rpm.

rpm -ivh blstat-0.53-1.el6.noarch.rpm

command to install blstat rpm

At the command line as root blstat is executed by running:

blstat

The initial screen is the i/o throttling stats. Pressing "c" will switch blstat to the cpu throttling screen. Pressing "i" will switch the screen back to i/o throttling. Pressing "q" will exit blstat.

Getting Started

The instructions in this quick start guide are written to work with any supported Linux distribution with a stock kernel.

Installation for cPanel Servers

In order to provide quick installation options and updates, there is a pre-built package for CentOS 6.0-6.3. Other packages will be provided shortly for other popular Linux distributions. If you need a particular pre-built package, we can make it quickly available.

Follow these steps to download and perform a BetterLinux installation on cPanel servers:

• Go to https://www.betterlinux.com and click on the "Get It Now" link.
• When the BetterLinux Trial appears in your shopping cart, click the "checkout" link.
• Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."
• You will see a "Your downloads are here!" button, which takes you to the download page, but you will not be able to download anything until you have verified your email. (see below)
• You will receive a welcome email with your account information, order confirmation, and an email verification link. The verification link will also take you to the download page.
• Login to your BetterLinux account and download the distribution of your choice tarballs and license file, then move them to your server:

betterlinux cpanel install
betterlinux install
license.key

• Unpack the tarballs.

tar –xvf betterlinux-cpanel-install-*
tar –xvf betterlinux-install-*

This will create the directory .../betterlinux-install. Move the license.key file into the betterlinux-install directory.

• Change directory to betterlinux-install, and start bl-install.

./bl-install license.key

• Once the installation is complete, reboot your server.
• cd back into the betterlinux-cpanel-install directory and run the bl-cpanel-install script:

./bl-cpanel-install

• Installation and cPanel configuration is now complete.

CPU and IO throttling limits have been auto-configured and running on your system.

Basic Configuration

To configure BetterLinux, you must modify the configuration files to customize and tune the modules to meet your needs. The BetterLinux configuration files are located in the following directory:

/etc/betterlinux

The directory structure under /etc/betterlinux includes the following three files:

• ipv4, ipv6, and license.key

It also includes the following three directories:

• cpud.conf.d, iothrottled.conf.d, and mysql.conf.d

Each directory contains its associated .conf files:

• /etc/betterlinux/cpud.conf.d: groups.conf, logging.conf, main.conf, prog_limits.conf, rules.conf, and users.conf
• /etc/betterlinux/iothrottled.conf.d: groups.conf, main.conf, scale.conf, users.conf, logging.conf, prog_limits.conf, rules.conf, and user_limits.conf
• /etc/betterlinux/mysql.conf.d: mysql.conf usermap.conf

BetterLinux is designed so that any of its parameters may be located in any file as long as the filename ends with the .conf extension. This allows the user to consolidate all the configuration settings under one .conf file. Below are two sample .conf files that consolidate the basic required parameters into a single .conf file.

Note: To use a single conf file, the user must first rename the existing .conf files so they do not have a .conf extension, create a new .conf file, then copy and paste the configuration parameters into the new .conf file. Many of the parameters, if duplicated in multiple conf files, will prevent the cpud, iothrottled, or BetterLinux MySQL daemon from starting.

BetterCPUD Configuration File

Below is a basic .conf file for the BetterCPUD module. This .conf file provides a simple configuration that requires the user to only modify the group names and UID ranges. Cut and paste the following .conf settings into a text file. Name the file with a .conf extension. Copy the file to the /etc/betterlinux/cpud.conf.d directory. After you put the .conf file in the cpud.conf.d directory, rename the other *.conf files in the directory to *.conf.bak.

#-------------------- Basic BetterCPU.conf File -------------------
#       bettercpu.conf location: /etc/betterlinux/cpud.conf.d/
#------------------------ BetterCPU Groups ------------------------
# Modify the UID range in the group settings below
uid standard_group 500-999
uid pro_group 1000-1999

#-------------------------- Jail Settings -------------------------
jail_cnt 1                                    #number of jail cores
max_jail_cnt 2                            #max number of jail cores

#-----------------------BetterCPU Throttling ----------------------
start_throttling group=standard_group cpu_percent=60,seconds=7
stop_throttling group=standard_group cpu_percent=30,seconds=7
 
start_throttling group=pro_group cpu_percent=100,seconds=5
stop_throttling group=pro_group cpu_percent=30,seconds=5

BetterIO Configuration File

Below is a basic .conf file for the BetterIO module. This will provide a simple configuration and will require the user to modify the group names and UID ranges. Cut and paste the following .conf settings into a text file and name it [choose-a-name].conf. Then move the file into /etc/betterlinux/iothrottled.conf.d. Once this is complete, rename all the other *.conf files to *.conf.bak under /etc/betterlinux/iothrottled.conf.d

#------------------- Basic BetterIO.conf File ---------------------
#     betterio.conf location: /etc/betterlinux/iothrottled.conf.d/
#------------------------ BetterIO Groups -------------------------
# Modify the UID range in the group settings below
uid io_standard_group 500-999
uid io_pro_group 1000-1999

#--------------------- IO Throttling Settings ---------------------
# The I/O bandwidth of all the tasks running as any user in the 
# group "io_standard_users" that generate IO on all the mounted 
# filesystems, exceptthose mounted under /home, are limited to 
# 5MB/s and 1500 IO operations per second.  Adjust the mount points
# for the users in the defined groups and the bandwidth and io 
# operations in bytes.
io_standard_users bw max inc="^/",exc="^/home" 5000000
io_standard_users ops max inc="^/",exc="^/home" 1500
 
io_pr_users bw max inc="^/",exc="^/home" 10000000
io_pro_users ops max inc="^/",exc="^/home" 2500
 
#------------------------ BetterIO Scaling and Balancing ------------------------
# These parameters are configured to provide optimum system io scaling and
# seek balancing 

auto_scale 1
scale_down_inc 500
scale_up_inc 500
scale_max 50000
scale_min 25
scale_pct_hi 95
scale_pct_low 90

seek_balancing 1
seek_balancing_down_pct 60
seek_balancing_up_pct 150
seek_balancing_method 90

BetterMySQL Configuration File

Below is a basic .conf file for the BetterMySQL module. This .conf file provides a simple configuration that requires the user to only modify the usermap.conf file. Rename the mysql.conf file in the /etc/betterlinux/mysqld.conf.d directory to mysql.conf.bak. Cut and paste the following .conf settings into a text file. Name the file mysql.conf. Copy the file to the /etc/betterlinux/mysql.conf.d directory.

Note: BetterMySQL throttles the UIDs/users in the BetterCPU groups.

# --------------------- Setting BetterMySQL Options ---------------
#       mysql.conf location: /etc/betterlinux/mysql.conf.d/
# --------------------- Basic BetterMySQL Settings ----------------
# "bl_throttling" enables per-user MySQL query throttling.
bl_throttling 1
# 
# "bl_usermap_filename" defines the file used for mapping MySQL
# users to system users for user blame assignment.
#  
# See the instructions for usermap.conf setup before enabling
# this option.
bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

2. The approved_accounting_program authorizes a program to apply usage blame to the correct user.

approved_accounting_program program=/usr/libexec/mysqld user=mysql

Auto-starting the Daemons

Once configured, the BetterCPU, BetterIO, and BetterMySQL daemons can be configured to automatically start. To do this, execute the following commands (on Red Hat based distributions):

chkconfig --level 235 cpud on
chkconfig --level 235 iothrottled on
chkconfig --level 235 mysqld on

Note: Some parameters configured in combination may cause one or more of the BetterLinux daemons to cause the system to not boot correctly. For example, applying very tight file or process limits to the root user may result in the system being unstable or necessary system services from starting correctly. It is strongly recommended that settings be tested before automatically starting BetterLinux service daemons automatically on system startup. Should a system become non-functional during startup, the system can be recovered by starting in single user mode (runlevel S or runlevel 1) and disabling the service that is preventing startup. Please refer to the Linux distribution's documentation for details on how to reach an appropriate single user mode.

Verifying Operation

Once the services are configured and started, use the blstat (for BetterCPU or BetterIO) or myusertop (for BetterMySQL) utilities to verify the BetterLinux daemons are functioning correctly. Start a process that uses a lot of CPU, block device I/O, or SQL queries, and monitor the process using the appropriate monitoring tool. The selected monitoring tool will indicate if throttling is taking place on the resources it monitors. For example, when throttling using BetterCPU, the blstat utility will show:

Up 0 days, 05:33:44,  5 users,  Load Average: 0.34, 0.10, 0.07
Mem:   2054572k total,   834904k used,  1219668k free,    48144 buffers
Swap:  4128760k total,        0k used,  4128760k free,   165076 buffers

Cores: 4 total, 1 currently jailed, 1 allowed in jail
Core IDs in jail: 3, Core IDs not in jail: 2,1,0

User     PID      Group      CPU Usage  RES        Type Command                 
                             percent    byte(1024)                              
testuser1 30168    bl_users   0            01 MiB   U    -bash                 
testuser1 30191    bl_users   100         360 KiB   U    jailme               

In this example, Core 3 is allocated to jail, and the testuser1 user's processes are executing in that jail core, demonstrating that the BetterCPU daemon is throttling users based on the configuration that is active.

Note: blstat is currently not installed by default. It can be downloaded from this location:

http://repo.betterlinux.com/centos/6.3/x86_64/Packages/blstat-0.52-1.el6.noarch.rpm

Installation for Linux with cPanel

For quick installation and updates, use the pre-built package for CentOS 6.0-6.3. Packages for other popular Linux distributions are coming soon. If you need a particular pre-built package, we can quickly make it available.

Basic installation instructions:

• Go to https://www.betterlinux.com and click on the "Get It Now" link.
• When the BetterLinux Trial appears in your shopping cart, click the "checkout" link.
• Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."
• You will see a "Your downloads are here!" button, which takes you to the download page, but you will not be able to download anything until you have verified your email. (see below)
• You will receive a welcome email with your account information, order confirmation, and an email verification link. The verification link will also take you to the download page.
• Login to your BetterLinux account and download the distribution of your choice, tarballs, and license file; then move them to your server. They appear as follows:

betterlinux cpanel install
betterlinux install
license.key

• Alternatively, the tarballs can be pulled down directly to the server by right clicking on the tarballs link in the downloads area and then using the wget command in combination with pasting in the tarball link. See the example below:

wget http://repo.betterlinux.com/release/betterlinux-cpanel-install-0.9.4-centos63-x86_64.tar.gz
wget http://repo.betterlinux.com/release/betterlinux-install-0.9.4-centos63-x86_64.tar.gz

• Unpack the tarballs.

tar –xvf betterlinux-cpanel-install-*
tar –xvf betterlinux-install-*

Unpacking the tarballs will create the directory .../betterlinux-install. Move the license.key file into the betterlinux-install directory.

• Change directory to betterlinux-install, and start bl-install.

./bl-install license.key

• Once the installation is complete, reboot your server.
• cd into the betterlinux-cpanel-install directory and run the bl-cpanel-install script:

./bl-cpanel-install

• Installation and cPanel configuration are now complete.

CPU and IO throttling limits have been auto-configured and are now running on your system. If you want to make throttling limit changes, please refer to our online documentation. http://docs.betterlinux.com

Upgrade From First BetterLinux Beta

To upgrade your BetterLinux beta installation on cPanel servers, see below: (This option is not for upgrading releases issued before October 15, 2012)

• Log in to your BetterLinux account and download your distribution of choice, tarballs, and license file; then move them to your server. They appear as follows:

betterlinux cpanel install
betterlinux install
license.key

• Alternatively, the tarballs can be pulled down directly to the server by right clicking on the tarballs link in the downloads area and then using the wget command in combination with pasting in the tarball link. See the example below:

wget http://repo.betterlinux.com/release/betterlinux-cpanel-install-0.9.4-centos63-x86_64.tar.gz
wget http://repo.betterlinux.com/release/betterlinux-install-0.9.4-centos63-x86_64.tar.gz

• Unpack the tarballs.

tar –xvf betterlinux-cpanel-install-*
tar –xvf betterlinux-install-*

Unpacking the tarballs will create two directories .../betterlinux-install and .../betterlinux-cpanel-install. Move the license.key file into the ../betterlinux-install directory.

• Change directory to betterlinux-install/optional, and start pre-beta-uninstall-helper.

./pre-beta-uninstall-helper

• cd into betterlinux-install, and start bl-install in conjunction with the license key:

./bl-install license.key

• Reboot the server. This will load the new BetterLinux kernel and modules.
• cd into the betterlinux-cpanel-install directory and run the bl-cpanel-install script:

./bl-cpanel-install

• Installation and cPanel configuration are now complete.

At this point, CPU and I/O throttling limits have been auto-configured and the BetterLinux controllers are running on your system. If you want to make throttling limit changes, please refer to our online documentation. http://docs.betterlinux.com

---

Removing BetterLinux New Release

To remove our installation, run the steps below: (Do not use these steps on BetterLinux versions issued before October 15, 2012)

• change directory to …/betterlinux-cpanel-install and run this command:

bl-cpanel-install –uninstall

• edit /etc/grub and select the previous kernel
• reboot
• change directory to …/betterlinux-install and run this command:

./bl-install --uninstall

BYOK - Build Your Own Kernel Installation

This feature is currently unavailable - BYOK - Build Your Own Kernel The build-your-own-kernel feature allows the user to recompile the BetterLinux modules on the client side against non officially supported kernels.

How does it work?

Each BetterLinux kernel module is implemented as a binary-only part and an open-source glue that provides a "compatibility layer" for the binary-only part across different kernel versions and configurations.

The binary-only part is provided as a non-linked object file (.o) and the open-source part as a source file (.c). When the whole module is recompiled on the client side, the compatibility layer is built as usual and linked with the binary-only part.

BYOK scripts:

In the BetterLinux tar download file, you will find the BYOK scripts. BetterLinux provides three BYOK scripts to help you apply the BetterLinux patch set against non-officially supported kernels and recompile all the required kernel modules.

Description of the BYOK scripts:

• byok-kernel-prepare: this script accepts a kernel tarball (.tar.bz2) as input, unpacks the source code, download and applies the BetterLinux patch set and prepares the environment to recompile a BetterLinux kernel.
• byok-modules-install: this script accepts a kernel source directory, download and recompile the BetterLinux kernel modules for the custom kernel build directory and install them.
• byok-userspace-install: this script configures yum to get all the required BetterLinux user-space packages and installs them.

Example usage

• First, change to the byok directory under the betterlinux-install directory

$ cd betterlinux-install/byok

• Execute the byok-kernel-prepare script and include the path to the kernel source you are compiling

$ ./byok-kernel-prepare /build/linux-3.3.3/

This applies the entire BetterLinux patch set to your kernel. When this first step is complete, you can apply your own patches on top, adjust the kernel .config, recompile and install the kernel as usual.

• Config options

Here's a list of the BetterLinux config options and their required settings:

CONFIG_UID_IO_ACCOUNTING=y - enable per uid and block device IO accounting
CONFIG_CGROUP_UID=y - enable cgroup-uid
CONFIG_CGROUP_IO_THROTTLE=y - enable core throttling infrastructure
CONFIG_CGROUP_IO_THROTTLE_BLOCKIO=y - enable blockio throttling hooks
CONFIG_CGROUP_IO_THROTTLE_NFS=y - enable nfs throttling hooks
CONFIG_CGROUP_IO_THROTTLE_PAGEIO=y - enable pageio throttling hooks
CONFIG_BLAME=y - enable socket blame core infrastructure

• Now compile your kernel

We are assuming you know how to apply any additional kernel patches you might need, update your config and compile your kernel.

• Now run byok-modules-prepare script

Once you kernel has finished compiling, run the build your own kernel modules script:

$ ./byok-modules-install [-b] /build/linux-3.3.3/

This script downloads the source RPM of all modules, recompiles them against your kernel and prepares the binary RPMs and install them in your host.

NOTE: If the option "-b" is specified, the RPMs of the BetterLinux modules will be generated and not installed on the system. This option is useful if you have a need to install the same RPMs on many different hosts (cluster, farm, etc.) and/or if you want to put them in a local repository.

• Now run the byok-userspace-install script

$ ./byok-userspace-install /path/to/license.key

This adds the BetterLinux repository to the yum configuration and installs all the BetterLinux system packages.

Location of Module RPMs and .ko files

Once you include BetterLinux in your new Kernel, you will likely want to create a different process for upgrading other systems. You can find the BetterLinux Module RPMs in this location:

RPM Modules - Change to the directory you specified when you ran the byok-modules-install script:

$ ./byok-modules-install [-b] /build/linux-3.3.3/

In the example above, the RPMs are located here: /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64

The Modules .ko files are only access via the RPMs and can to be extracted if you want to move and install them without the RPM. Change To: /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64 and run the command below to extract the .ko files from the RPM.

for f in *.rpm; do rpm2cpio $f | cpio -id; done

This command will create a new directory path and leave the .ko files in: /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64/lib/modules/3.3.7

Introduction

BetterLinux is a collection of system resource management and monitoring tools intended for hosting providers, data centers, SaaS companies, and cloud environments. With it, you can control use and allocation of CPU, memory, MySQL, device I/O bandwidth, and IP bandwidth resources. Individual users and processes that exceed set resource limits can be isolated from other system users and throttled as necessary.

Historically, process management on Linux has involved adjusting processes’ priorities and nice levels. More recently, Linux system administrators have used technologies like full virtualization, paravirtualization, Unix containers, or other similar conventions to segregate system resource usage.

These controls, while useful, are quite crude and do not address the core issue of guaranteeing and prioritizing operating system resources for critical applications across an entire system. Poorly-behaved processes can still monopolize the system and cause overall performance problems.

Recent Linux kernels have introduced the concept of control groups, or cgroups. A cgroup is used to group processes together in a hierarchical structure in order to divide resources among different process groups or subsystems. For example, one cgroup might be configured to hold all web-server related processes, with a second cgroup configured to hold system backup processes.

BetterLinux augments cgroups functionality with automatic categorization of processes based on owner or process name. This coupled with BetterLinux’s unique ability to “autoscale” resource limits based on constantly changing system loads allows you to guarantee and prioritize system resources with unprecedented confidence and precision.

BetterLinux can operate either simply or with impressive complexity as may suit your needs and interests. Each of its modules comes with configuration files wherein you can create groups of users or processes, set various limits, turn options on or off, and change logging practices, etc. For configuration changes to take effect, you must either reload the .conf changes or restart the module daemon as follows:

Service cpud restart		[Restart daemon with updated changes.]
Service cpud condrestart		[Load new .conf changes & restart daemon only if daemon is already running.]
Service cpud reload		[Load .conf file settings without restarting daemon.]

BetterLinux consists of four modules: BetterCPU, BetterIO, BetterMySQL, and BetterBandwidth. See their brief descriptions below.

BetterCPU

BetterCPU (a CPU controller) limits access to a server’s CPU resources by individual users and processess. Over-consuming processes are isolated to a “jail” where they are throttled as necessary to prevent them from hogging CPU resources. Processes operating within limits continue to work efficiently without interference.

BetterCPU Features:

• Fairly allocate CPU resources and guarantee resources for specific users/processes
• Temporarily “jail” over-consuming users/processes to prevent crashes and congestion
• Automatically reallocate unused dedicated resources to cut waste
• Trim CPU usage peaks with customized thresholds for groups and individuals
• Stop memory overuse, system lock-ups, and runaway processes
• Automatically kill zombies, hung or unauthorized processes, and memory hogs
• Multiply server density while cutting heat, power, & hardware costs; increase revenue
• Supplement set physical RAM memory limits with swap space
• Monitor CPU, user, and core behavior in real time; see detailed logs and summaries
• One configuration file can hold settings for multiple systems

BetterIO

BetterIO manages block device I/O utilization. You can create limits for individual users and groups, for individual programs, for specific devices by percent usage, and even for mount points (like NFS, iSCSI, and more). You can also limit by service time (svctime), access wait (await), or a combination of both. BetterIO can “auto-scale” system limits when resources go used to prevent waste. “Auto-scaling” responds to highly configurable conditions and use ranges in addition to recent user activity.

BetterIO Features:

• Create customized individual and group limits for users, programs, devices, and mount points
• Control seek times, I/O operations, and disk bandwidth use for a balanced system without monopolizers.
• Prevent bottlenecks and timeouts
• Eliminate waste by autoscaling limits when resources go unused
• Multiply server density
• Thoroughly monitor per-user I/O use details and totals by device of origin in real time
• Reduce need for more hardware or reductions in tenants per server
• Reduce heat and power consumption
• One configuration file can hold settings for multiple systems
• Integration with cPanel user management for web hosting providers

BetterMySQL

BetterMySQL assigns the responsibility for queries to individual Linux users instead of to the generic MySQL user to enable individualized MySQL throttling. BetterMySQL also “auto-scales” system limits when resources go used to prevent waste.

BetterMySQL Features:

• Map MySQL users to Linux users; all queries now tied to specific users for individualized throttling
• Control abusive MySQL users, processes, and threads
• Compatible even on nonstandard platforms/directories
• Automatically reallocate unused dedicated resources to cut waste
• Thoroughly monitor individual MySQL use and relevant CPU use in real time; see detailed summaries
• One configuration file can hold settings for multiple systems

BetterBandwidth

BetterBandwidth includes two control categories: Bandwidth Limiting and Connection Limiting

Bandwidth Limiting:

BetterBandwidth links traffic to specific Linux users, permitting group and individual limits without need for individual IP addresses. You can also force a user’s traffic on a shared server to originate from your main shared IP. BetterBandwidth has extensive bandwidth limiting monitoring abilities.

Connection Limiting:

BetterBandwidth ties inbound traffic to specific Linux users, even when destined for generic users, like Apache’s “nobody” user. This enables you to limit the number of inbound connections available per group member on any given port (groups are admin-defined). You can also prioritize port connections (e.g. give Apache more connections than MySQL). BetterBandwidth includes a very thorough connection limiting monitor.

BetterBandwidth Features:

Bandwidth Limiting Features:

• Link outgoing bandwidth to specific Linux users
• Limit group and individual bandwidth use without assigning individual IP addresses
• Throttle only those who exceed set limits
• Increase profits with more users, fewer support calls, and reduced need for IPV4 IP address blocks
• Specific users’ traffic on shared servers now originates from IP addresses you choose, not your main shared IP
• Have more bandwidth per user or more users on your bandwidth
• Monitor individual bandwidth use in real time with detailed info on addresses, ports, sockets, protocols, and packets; create custom summaries
• One configuration file can hold settings for multiple systems

Connection Limiting Features:

• Limit the number of inbound connections for groups of users on any port
• Link inbound traffic to specific Linux users, even when directed to Apache’s “nobody” user
• Give some ports more connections than others
• See connection type, method, user/port limits and current use, configured and in-use connections, and more
• Minimize the effect of DOS attacks and correct other less severe connection inefficiencies

Log File Mapping to Parameters

iothrottle daemon

iothrottled

Log File Name	Location	Parameters	Description
N/A	/var/log/iothrottled	enable_logging	Turns on all logging
iothrottled	/var/log/iothrottled	all parameters	All logging sent to this file
iothrottled.cnt	/var/log/iothrottled	?	cgroup creation and deletion counts are sent to this file

Note: The location and name of the logfile can be modified using the log_file or log_cnt_file in /etc/betterlinux/iothrottled/iothrottled.conf.d/logging.conf.

cpu daemon

The following is a mapping of the cpud log files and the tunable parameters that log their results in them.

cpud

Log File Name	Location	Parameters	Description
cpud	/var/log/cpud	enable_logging	Turns on all logging
cpud	/var/log/cpud	log_cpud	All cpud logging sent to this file

Note: The location and name of the logfile can be modified using the log_cpud_file in /etc/betterlinux/cpud/cpud.d/logging.conf.

cpud_governor

Log File Name	Location	Parameters	Description
cpud_governor	/var/log/cpud	log_gov	?
cpud_governor	/var/log/cpud	kill_ded_web	?
cpud_governor	/var/log/cpud	kill_ded_crond	?
cpud_governor	/var/log/cpud	kill_ded_sshd	?
cpud_governor	/var/log/cpud	kill_ded_daemon	?
cpud_governor	/var/log/cpud	kill_ded_web_busy	?
cpud_governor	/var/log/cpud	kill_ded_crond_busy	?
cpud_governor	/var/log/cpud	kill_ded_sshd_busy	?
cpud_governor	/var/log/cpud	kill_ded_daemon_busy	?
cpud_governor	/var/log/cpud	kill_non_ded_web	?
cpud_governor	/var/log/cpud	kill_non_ded_crond	?
cpud_governor	/var/log/cpud	kill_non_ded_sshd	?
cpud_governor	/var/log/cpud	kill_non_ded_daemon	?
cpud_governor	/var/log/cpud	kill_non_ded_web_busy	?
cpud_governor	/var/log/cpud	kill_non_ded_crond_busy	?
cpud_governor	/var/log/cpud	kill_non_ded_sshd_busy	?
cpud_governor	/var/log/cpud	kill_non_ded_daemon_busy	?
cpud_governor	/var/log/cpud	kill_web_io_idle_time	?
cpud_governor	/var/log/cpud	kill_ded_web_exclude	?
cpud_governor	/var/log/cpud	kill_ded_crond_exclude	?
cpud_governor	/var/log/cpud	kill_ded_sshd_exclude	?
cpud_governor	/var/log/cpud	kill_ded_daemon_include	?
cpud_governor	/var/log/cpud	kill_non_ded_web_exclude	?
cpud_governor	/var/log/cpud	kill_non_ded_crond_exclude	?
cpud_governor	/var/log/cpud	kill_non_ded_sshd_exclude	?
cpud_governor	/var/log/cpud	kill_non_ded_daemon_exclude	?

Note: The location and name of the logfile can be modified using the log_gov_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

cpud_mem_con

Log File Name	Location	Parameters	Description
cpud_mem_con	/var/log/cpud	log_memcon	?
cpud_mem_con	/var/log/cpud	max_prog_rss	?
cpud_mem_con	/var/log/cpud	max_user_rss	?
cpud_mem_con	/var/log/cpud	tmpfs_spike_limit	?
cpud_mem_con	/var/log/cpud	tmpfs_spike_val	?
cpud_mem_con	/var/log/cpud	kill_mem_free_limit_panic_pct	?
cpud_mem_con	/var/log/cpud	kill_cache_limit_panic	?
cpud_mem_con	/var/log/cpud	kill_mem_free_limit_pct	?
cpud_mem_con	/var/log/cpud	kill_cache_limit	?

Note: The location and name of the logfile can be modified using the log_memcon_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

cpud_panics

Log File Name	Location	Parameters	Description
cpud_panics	/var/log/cpud	log_panics	?
cpud_panics	/var/log/cpud	tmpfs_spike_limit	?
cpud_panics	/var/log/cpud	tmpfs_spike_val	?
cpud_panics	/var/log/cpud	kill_mem_free_limit_panic_pct	?
cpud_panics	/var/log/cpud	kill_cache_limit_panic	?
cpud_panics	/var/log/cpud	kill_mem_free_limit_pct	?
cpud_panics	/var/log/cpud	kill_cache_limit	?

Note: The location and name of the logfile can be modified using the log_panics_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

=

cpud_cpu

Log File Name	Location	Parameters	Description
cpud_cpu	/var/log/cpud	log_cpu	?
cpud_cpu	/var/log/cpud	jail_pct	?
cpud_cpu	/var/log/cpud	jail_cnt	?
cpud_cpu	/var/log/cpud	max_jail_pct	?
cpud_cpu	/var/log/cpud	max_jail_cnt	?
cpud_cpu	/var/log/cpud	kill_mem_free_limit_pct	?
cpud_cpu	/var/log/cpud	kill_cache_limit	?

Note: The location and name of the logfile can be modified using the log_cpu_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

cpud_jail

Log File Name	Location	Parameters	Description
cpud_jail	/var/log/cpud	log_jail	?
cpud_jail	/var/log/cpud	log_mem	?

Note: The location and name of the logfile can be modified using the log_jail_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

blstat

blstat is now part of the main product installation. Follow the steps below to download and install blstat if it's not part of your BetterLinux install:

If you are using cPanel, please run:

/scripts/perlinstaller Term::ReadKey JSON

Or if you are running stock CentOS6, please run:

yum --disableexcludes=all install perl-TermReadKey perl-JSON perl-Time-HiRes

This will install two perl modules blstat needs.

wget http://repo.betterlinux.com/unsupported/blstat-0.56-1.el6.noarch.rpm

Downloads the blstat rpm.

rpm -ivh blstat-0.56-1.el6.noarch.rpm

command to install blstat rpm

At the command line as root blstat is executed by running:

blstat

Hot Keys

First screen is the I/O screen.

    i - I/O throttling screen
    u - I/O status by user
    c - Cpu throttling stats
    b - Toggles the BetterBandwidth screen
    n - Network
    ? - Toggles help screen
    q - Exit's blstat

myusertop

Description

Display the MySQL users most active during the preceding 4 seconds. Show the Linux users and UIDs that "own" them. Press single "hot" keys to change the display interactively.

Hot Keys

    V - Toggle views 1 and 2
    m - Toggle the display of MySQL users under "owning" Linux users
    H - Toggle hourly/total statistics in header area (only)
    R - Reverse sort direction (ascending/descending)

    f - Invoke FLUSH USER_STATISTICS (to reset statistics to zero)

    Sort by:
    a - Adjusted rows examined                        (view 1)
    b - Busy time during last refresh interval        (view 1)
    c - CPU time during last refresh interval         (view 1)
    C - CPU time / busy time                          (view 1)
    e - Core number at end of latest query            (view 2)
    i - UID                                           (all views)
    l - Busy time of latest completed query           (view 2)
    L - CPU time of latest completed query            (view 2)
    n - Number of non-voluntary context switches in 
          latest completed query                      (view 2)
    o - "Other" commands during last refresh interval (view 2)         
    p - % of total CPU time used by user              (view 1)
    Q - Number of queries in progress                 (view 2)
    r - Rows examined                                 (view 1)
    s - Select commands during last refresh interval  (view 2 default)
    S - Start core number of latest query             (view 2)
    t - Number of threads per user                    (view 1 default)
    u - Update commands during last refresh interval  (view 2)
    U - User name                                     (all views)
    v - Number of voluntary context switches in 
          latest completed query                      (view 2)

Other:

    h,? - Display online help
    q   - Quit B<myusertop>

Column Headings

    Linux/MySQLUser - MySQL user is indented beneath Linux user name
    Thds            - Number of threads at the current time
    CPUTime         - Seconds of CPU time since previous refresh
    BusTime         - Seconds of busy (clock) time during same interval
    CPU/Bus         - CPU time divided by busy time
    RowsExam        - Number of rows "examined" since previous refresh
    AdjRowEx        - RowsExam * CPU / Bus
    %CPU            - Amount of MySQL's CPU usage used by this user
    Sel             - Number of SELECT commands since previous refresh
    Upd             - Number of UPDATE commands since previous refresh
    Oth             - Number of other commands since previous refresh
    VCxSw           - Number of voluntary context switches since previous
                        refresh
    NVCSw           - Number of non-voluntary context switches since
                        previous refresh
    SC              - Core number at start of most recent query
    EC              - Core number at end of most recent query
    LQCpu           - CPU seconds of most recent query
    LQBus           - Busy (clock) seconds of most recent query

Command-line Options

Most of the command-line options (below) can be specified interactively while myusertop is running by pressing single "hot" keys. (See above.) Many of the options can be set as defaults in the configuration file

   */etc/betterlinux/mysql.conf.d/myusertop.conf*.

Command-line options (preceded by 1 or 2 hyphens) may be abbreviated to the shortest unique prefix):

   -ve, --version

Display the version of myusertop and exit.

   -nom, --nomyuser

Do not display the MySQL users "owned" by each top Linux user. (The default is to list each Linux user's associated MySQL users, indented, beneath the Linux user.) During an interactive session, the "m" hot key toggles the display of "owned" MySQL users on and off.

   --cfg=<filename>, --config=<filename>

Override the default (/etc/betterlinux/mysql.conf.d/myusertop.conf) configuration file name with <filename>.

   -S [path], --socket=[path]

Path specifying unix domain socket to pass to the mysql command.

   -ho, --hourly

In the header summary information, show hourly statistics (instead of the totals since mysqld was started or FLUSH USER_STATISTICS reset the statistics. (If mysqld has been running for less than one hour--or if FLUSH USER_STATISTICS was invoked within the hour--values will be adjusted upward to show the hourly rate if usage continues at the same rate for the remainder of the hour.)

During an interactive session, the "H" hot key toggles the header between displaying hourly and total statistics.

Note: Regardless of whether the header displays hourly statistics or statistics since startup/flush time, the main user columns show usage during the most recent refresh interval.

   --refresh=<seconds>

Specify the refresh interval in seconds. The default refresh interval is 4 seconds.

--admin=<admin-user-name> Specify the name of a super MySQL user with authority to issue MySQL administrative commands. This option is to be used ONLY in cases where the root unix user (the one running this script) needs to specify a MySQL superuser name when starting the mysql client.

   --adminpass=<admin-user-password>

Specify the password of the MySQL user specified with the --admin option. This option is likewise used ONLY in cases where the root unix user (the one running this script) needs to specify a MySQL superuser name when starting the mysql client.

   --ascending

Sort ascending (from smallest to largest). This effectively makes myusertop show the *bottom* users. During an interactive session, the "R" hot key reverses the current sort direction. Use this option in combination with any *one* of the following sort options.

   --sortthreads

Sort users by the number of threads used by Linux users at the moment of the most recent refresh This is the default in view 1. During an interactive session, the "t" hot key will sort the number of threads.

   --sortcpu

Sort by CPU time (from highest to lowest). During an interactive session, the "c" hot key will sort by CPU time.

   --sortadjre 

Sort by adjusted rows examined (from highest users to lowest). Adjusted rows examined is the number of rows examined multipled by the ratio of CPU Time to Busy Time. During an interactive session, the "a" hot key will sort by adjusted rows examined.

   --sortbusy

Sort by busy (clock) time (from highest to lowest). During an interactive session, the "b" hot key will sort by busy time.

   --sortcob

Sort by ratio of CPU Usage to Busy Time. "CPU over Busy. During an interactive session, the "C" hot key will sort by the ratio of CPU time to busy time.

   --sortrows

Sort by number of rows examined (from highest to lowest.) During an interactive session, the "r" hot key will sort by the the number of rows examined.

   --sortcpupercent

Sort by the percentage of MySQL CPU time that the user is using. During an interactive session, the "p" hot key will sort by CPU ercentage.

   --sortselect

Sort by the number of select commands issued (from highest to lowest). During an interactive session, the "s" hot key will sort by the number of select commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortupdate

Sort by the number of update commands issued (from highest to lowest). During an interactive session, the "u" hot key will sort by the number of update commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortother

Sort by the number of other commands issued (from highest to lowest). During an interactive session, the "o" hot key will sort by the number of other commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortvcs

Sort by the number of voluntary context switches (from highest to lowest) during the most recent query. During an interactive session, the "v" hot key will sort by the number of voluntary context switches. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortnvcs

Sort by the number of nonvoluntary context switches (from highest to lowest) during the most recent query. During an interactive session, the "n" hot key will sort by the number of nonvoluntary context switches. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortqip

Sort by number of active queries in progress (from highest to lowest) for each user at the current moment. During an interactive session, the "Q" hot key will sort by the number of queries in progress. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortlqcpu

Sort by last query CPU time (from highest to lowest)--the number of seconds of CPU time used by each user's most recently completed query. During an interactive session, the "L" hot key will sort by the last query CPU time. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortlqbusy

Sort by last query busy time (from highest to lowest)--the number of seconds of busy time used by each user's most recently completed query. During an interactive session, the "l" hot key will sort by the last query busy time. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

Files

   /etc/betterlinux/mysql.conf.d/myusertop.conf

See Also

   myusertop.conf(8)