Main Page

From docs.betterlinux.com
Revision as of 15:58, 12 December 2012 by Tim (Talk | contribs)

Jump to: navigation, search

Welcome to the BetterLinux Product Documenation wiki. This is your one stop resource for everything BetterLinux - from initial setup, configuration, and fine tuning all the features. Please check here first before submitting a help desk ticket.

Contents

BetterLinux Getting Started

Introduction

BetterLinux is a collection of system resource management and monitoring tools intended for hosting providers, data centers, SaaS companies, and cloud environments. With it, you can control use and allocation of CPU, memory, MySQL, device I/O bandwidth, and IP bandwidth resources. Individual users and processes that exceed set resource limits can be isolated from other system users and throttled as necessary.

Historically, process management on Linux has involved adjusting processes’ priorities and nice levels. More recently, Linux system administrators have used technologies like full virtualization, paravirtualization, Unix containers, or other similar conventions to segregate system resource usage.

These controls, while useful, are quite crude and do not address the core issue of guaranteeing and prioritizing operating system resources for critical applications across an entire system. Poorly-behaved processes can still monopolize the system and cause overall performance problems.

Recent Linux kernels have introduced the concept of control groups, or cgroups. A cgroup is used to group processes together in a hierarchical structure in order to divide resources among different process groups or subsystems. For example, one cgroup might be configured to hold all web-server related processes, with a second cgroup configured to hold system backup processes.

BetterLinux augments cgroups functionality with automatic categorization of processes based on owner or process name. This coupled with BetterLinux’s unique ability to “autoscale” resource limits based on constantly changing system loads allows you to guarantee and prioritize system resources with unprecedented confidence and precision.

BetterLinux can operate either simply or with impressive complexity as may suit your needs and interests. Each of its modules comes with configuration files wherein you can create groups of users or processes, set various limits, turn options on or off, and change logging practices, etc. For configuration changes to take effect, you must either reload the .conf changes or restart the module daemon as follows:

Service cpud restart		[Restart daemon with updated changes.]
Service cpud condrestart		[Load new .conf changes & restart daemon only if daemon is already running.]
Service cpud reload		[Load .conf file settings without restarting daemon.]


BetterLinux consists of four modules: BetterCPU, BetterIO, BetterMySQL, and BetterBandwidth. See their brief descriptions below.


BetterCPU

BetterCPU (a CPU controller) limits access to a server’s CPU resources by individual users and processess. Over-consuming processes are isolated to a “jail” where they are throttled as necessary to prevent them from hogging CPU resources. Processes operating within limits continue to work efficiently without interference.

BetterCPU Features:
  • Fairly allocate CPU resources and guarantee resources for specific users/processes
  • Temporarily “jail” over-consuming users/processes to prevent crashes and congestion
  • Automatically reallocate unused dedicated resources to cut waste
  • Trim CPU usage peaks with customized thresholds for groups and individuals
  • Stop memory overuse, system lock-ups, and runaway processes
  • Automatically kill zombies, hung or unauthorized processes, and memory hogs
  • Multiply server density while cutting heat, power, & hardware costs; increase revenue
  • Supplement set physical RAM memory limits with swap space
  • Monitor CPU, user, and core behavior in real time; see detailed logs and summaries
  • One configuration file can hold settings for multiple systems

BetterIO

BetterIO manages block device I/O utilization. You can create limits for individual users and groups, for individual programs, for specific devices by percent usage, and even for mount points (like NFS, iSCSI, and more). You can also limit by service time (svctime), access wait (await), or a combination of both. BetterIO can “auto-scale” system limits when resources go used to prevent waste. “Auto-scaling” responds to highly configurable conditions and use ranges in addition to recent user activity.

BetterIO Features:
  • Create customized individual and group limits for users, programs, devices, and mount points
  • Control seek times, I/O operations, and disk bandwidth use for a balanced system without monopolizers.
  • Prevent bottlenecks and timeouts
  • Eliminate waste by autoscaling limits when resources go unused
  • Multiply server density
  • Thoroughly monitor per-user I/O use details and totals by device of origin in real time
  • Reduce need for more hardware or reductions in tenants per server
  • Reduce heat and power consumption
  • One configuration file can hold settings for multiple systems
  • Integration with cPanel user management for web hosting providers

BetterMySQL

BetterMySQL assigns the responsibility for queries to individual Linux users instead of to the generic MySQL user to enable individualized MySQL throttling. BetterMySQL also “auto-scales” system limits when resources go used to prevent waste.

BetterMySQL Features:
  • Map MySQL users to Linux users; all queries now tied to specific users for individualized throttling
  • Control abusive MySQL users, processes, and threads
  • Compatible even on nonstandard platforms/directories
  • Automatically reallocate unused dedicated resources to cut waste
  • Thoroughly monitor individual MySQL use and relevant CPU use in real time; see detailed summaries
  • One configuration file can hold settings for multiple systems

BetterBandwidth

BetterBandwidth includes two control categories: Bandwidth Limiting and Connection Limiting

Bandwidth Limiting:

BetterBandwidth links traffic to specific Linux users, permitting group and individual limits without need for individual IP addresses. You can also force a user’s traffic on a shared server to originate from your main shared IP. BetterBandwidth has extensive bandwidth limiting monitoring abilities.

Connection Limiting:

BetterBandwidth ties inbound traffic to specific Linux users, even when destined for generic users, like Apache’s “nobody” user. This enables you to limit the number of inbound connections available per group member on any given port (groups are admin-defined). You can also prioritize port connections (e.g. give Apache more connections than MySQL). BetterBandwidth includes a very thorough connection limiting monitor.

BetterBandwidth Features:

Bandwidth Limiting Features:
  • Link outgoing bandwidth to specific Linux users
  • Limit group and individual bandwidth use without assigning individual IP addresses
  • Throttle only those who exceed set limits
  • Increase profits with more users, fewer support calls, and reduced need for IPV4 IP address blocks
  • Specific users’ traffic on shared servers now originates from IP addresses you choose, not your main shared IP
  • Have more bandwidth per user or more users on your bandwidth
  • Monitor individual bandwidth use in real time with detailed info on addresses, ports, sockets, protocols, and packets; create custom summaries
  • One configuration file can hold settings for multiple systems
Connection Limiting Features:
  • Limit the number of inbound connections for groups of users on any port
  • Link inbound traffic to specific Linux users, even when directed to Apache’s “nobody” user
  • Give some ports more connections than others
  • See connection type, method, user/port limits and current use, configured and in-use connections, and more
  • Minimize the effect of DOS attacks and correct other less severe connection inefficiencies

Installation for Linux with cPanel


For quick installation and updates, use the pre-built package for CentOS 6.0-6.3. Packages for other popular Linux distributions are coming soon. If you need a particular pre-built package, we can quickly make it available.


Basic installation instructions:

  • Go to https://www.betterlinux.com and click on the "Get It Now" link.
  • When the BetterLinux Trial appears in your shopping cart, click the "checkout" link.
  • Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."
  • You will see a "Your downloads are here!" button, which takes you to the download page, but you will not be able to download anything until you have verified your email. (see below)
  • You will receive a welcome email with your account information, order confirmation, and an email verification link. The verification link will also take you to the download page.
  • Login to your BetterLinux account and download the distribution of your choice, tarballs, and license file; then move them to your server. They appear as follows:
betterlinux cpanel install
betterlinux install
license.key
  • Alternatively, the tarballs can be pulled down directly to the server by right clicking on the tarballs link in the downloads area and then using the wget command in combination with pasting in the tarball link. See the example below:
wget http://repo.betterlinux.com/release/betterlinux-cpanel-install-0.9.4-centos63-x86_64.tar.gz
wget http://repo.betterlinux.com/release/betterlinux-install-0.9.4-centos63-x86_64.tar.gz
  • Unpack the tarballs.
tar –xvf betterlinux-cpanel-install-*
tar –xvf betterlinux-install-*

Unpacking the tarballs will create the directory .../betterlinux-install. Move the license.key file into the betterlinux-install directory.

  • Change directory to betterlinux-install, and start bl-install.
./bl-install license.key
  • Once the installation is complete, reboot your server.
  • cd into the betterlinux-cpanel-install directory and run the bl-cpanel-install script:
./bl-cpanel-install
  • Installation and cPanel configuration are now complete.

CPU and IO throttling limits have been auto-configured and are now running on your system. If you want to make throttling limit changes, please refer to our online documentation. http://docs.betterlinux.com

Upgrade From First BetterLinux Beta

To upgrade your BetterLinux beta installation on cPanel servers, see below: (This option is not for upgrading releases issued before October 15, 2012)

  • Log in to your BetterLinux account and download your distribution of choice, tarballs, and license file; then move them to your server. They appear as follows:
betterlinux cpanel install
betterlinux install
license.key
  • Alternatively, the tarballs can be pulled down directly to the server by right clicking on the tarballs link in the downloads area and then using the wget command in combination with pasting in the tarball link. See the example below:
wget http://repo.betterlinux.com/release/betterlinux-cpanel-install-0.9.4-centos63-x86_64.tar.gz
wget http://repo.betterlinux.com/release/betterlinux-install-0.9.4-centos63-x86_64.tar.gz
  • Unpack the tarballs.
tar –xvf betterlinux-cpanel-install-*
tar –xvf betterlinux-install-*

Unpacking the tarballs will create two directories .../betterlinux-install and .../betterlinux-cpanel-install. Move the license.key file into the ../betterlinux-install directory.

  • Change directory to betterlinux-install/optional, and start pre-beta-uninstall-helper.
./pre-beta-uninstall-helper
  • cd into betterlinux-install, and start bl-install in conjunction with the license key:
./bl-install license.key
  • Reboot the server. This will load the new BetterLinux kernel and modules.
  • cd into the betterlinux-cpanel-install directory and run the bl-cpanel-install script:
./bl-cpanel-install
  • Installation and cPanel configuration are now complete.

At this point, CPU and I/O throttling limits have been auto-configured and the BetterLinux controllers are running on your system. If you want to make throttling limit changes, please refer to our online documentation. http://docs.betterlinux.com


Removing BetterLinux New Release

To remove our installation, run the steps below: (Do not use these steps on BetterLinux versions issued before October 15, 2012)

  • change directory to …/betterlinux-cpanel-install and run this command:
bl-cpanel-install –uninstall
  • edit /etc/grub and select the previous kernel
  • reboot
  • change directory to …/betterlinux-install and run this command:
./bl-install --uninstall

BYOK - Build Your Own Kernel Installation

BYOK - Build Your Own Kernel The build-your-own-kernel feature allows the user to recompile the BetterLinux modules on the client side against non officially supported kernels.

How does it work?

Each BetterLinux kernel module is implemented as a binary-only part and an open-source glue that provides a "compatibility layer" for the binary-only part across different kernel versions and configurations.

The binary-only part is provided as a non-linked object file (.o) and the open-source part as a source file (.c). When the whole module is recompiled on the client side, the compatibility layer is built as usual and linked with the binary-only part.

BYOK scripts:

In the BetterLinux tar download file, you will find the BYOK scripts. BetterLinux provides three BYOK scripts to help you apply the BetterLinux patch set against non-officially supported kernels and recompile all the required kernel modules.

Description of the BYOK scripts:

  • byok-kernel-prepare: this script accepts a kernel tarball (.tar.bz2) as input, unpacks the source code, download and applies the BetterLinux patch set and prepares the environment to recompile a BetterLinux kernel.
  • byok-modules-install: this script accepts a kernel source directory, download and recompile the BetterLinux kernel modules for the custom kernel build directory and install them.
  • byok-userspace-install: this script configures yum to get all the required BetterLinux user-space packages and installs them.


Example usage

  • First, change to the byok directory under the betterlinux-install directory
$ cd betterlinux-install/byok
  • Execute the byok-kernel-prepare script and include the path to the kernel source you are compiling
$ ./byok-kernel-prepare /build/linux-3.3.3/

This applies the entire BetterLinux patch set to your kernel. When this first step is complete, you can apply your own patches on top, adjust the kernel .config, recompile and install the kernel as usual.

  • Config options
Here's a list of the BetterLinux config options and their required settings:
CONFIG_UID_IO_ACCOUNTING=y - enable per uid and block device IO accounting
CONFIG_CGROUP_UID=y - enable cgroup-uid
CONFIG_CGROUP_IO_THROTTLE=y - enable core throttling infrastructure
CONFIG_CGROUP_IO_THROTTLE_BLOCKIO=y - enable blockio throttling hooks
CONFIG_CGROUP_IO_THROTTLE_NFS=y - enable nfs throttling hooks
CONFIG_CGROUP_IO_THROTTLE_PAGEIO=y - enable pageio throttling hooks
CONFIG_BLAME=y - enable socket blame core infrastructure
  • Now compile your kernel
We are assuming you know how to apply any additional kernel patches you might need, update your config and compile your kernel.
  • Now run byok-modules-prepare script

Once you kernel has finished compiling, run the build your own kernel modules script:

$ ./byok-modules-install [-b] /build/linux-3.3.3/

This script downloads the source RPM of all modules, recompiles them against your kernel and prepares the binary RPMs and install them in your host.

NOTE: If the option "-b" is specified, the RPMs of the BetterLinux modules will be generated and not installed on the system. This option is useful if you have a need to install the same RPMs on many different hosts (cluster, farm, etc.) and/or if you want to put them in a local repository.

  • Now run the byok-userspace-install script
$ ./byok-userspace-install /path/to/license.key

This adds the BetterLinux repository to the yum configuration and installs all the BetterLinux system packages.

Location of Module RPMs and .ko files

Once you include BetterLinux in your new Kernel, you will likely want to create a different process for upgrading other systems. You can find the BetterLinux Module RPMs in this location:

RPM Modules - Change to the directory you specified when you ran the byok-modules-install script:

$ ./byok-modules-install [-b] /build/linux-3.3.3/

In the example above, the RPMs are located here: /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64

The Modules .ko files are only access via the RPMs and can to be extracted if you want to move and install them without the RPM. Change To: /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64 and run the command below to extract the .ko files from the RPM.

for f in *.rpm; do rpm2cpio $f | cpio -id; done

This command will create a new directory path and leave the .ko files in: /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64/lib/modules/3.3.7

BetterCPU Overview

BetterCPU is designed to maximize cpu utilization for shared hosting environments and at the same time prevent one or many users from abusing the cpu utilization. BetterCPU makes it very easy to quickly identify the CPU hogs on the system and keep them under control and then decide if these users need to moved to a higher paying account that provides them with the resources they need to run their business. This is accomplished by defining BetterLinux groups of users or mapping BetterLinux groups to cPanel packages. Once the user groups are configured, then throttling limits are defined for CPU.

BetterCPU Configuration

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. By default, the non-cPanel installation (betterlinux-install) does not configure BetterCPU to throttle users. However, the cPanel installation (betterlinux-cpanel-install) script does configure and start BetterCPU throttling for all cPanel users on the server. The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).

Configuring the Jail Cores

Throttling is configured using parameters located in:

non-cPanel install: /etc/betterlinux/cpud.conf.d/rules.conf:
cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf
jail_cnt
max_jail_cnt

These settings are designed so that the same configuration file may be used on different systems with different numbers of cores. However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file. When these parameters are removed the cpud daemon uses the default settings wich is jail_cnt=1 and max_jail_cnt=total cores minus 1.

Syntax:

jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]

The jail_cnt parameter is defined using one of two methods:

  • If a fixed number of cores for jail is desired, make the assignment using jail_cnt as in this example:
jail_cnt 2
In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core. If the system is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.
  • If the number of cores for jail varies based on the total number of cores in the system, use the following format:
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. For example, using the values specified in the previous example, a 32-core system will use the jail_cnt value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails and is formatted identically to the jail_cnt parameter. By default, only the value of jail_cnt is used, but when a max_jail_cnt is specified, the jail is expanded when less than the value in jail_min_idle is available per jail core.

The jail_min_idle value is the minimum amount of idle utilization to maintain on the jail cores if jail_max_cnt has not been reached. If the average free utilization on each jail core is less than jail_min_idle, the jail is expanded until either the jail_min_idle value is maintained or jail_max_cnt is reached.

Syntax:

max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}

Example

max_jail_cnt 2:1 4:2 8:2 16:2

The result on a dual core system is no different from jail_cnt. But for 4-core machines and above, if the max_jail_cnt value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded.

max_jail_cnt 2

The result will always set the number of maximum jails to two. The maximum number of jails are set at two no matter how many cores may be on the system. The exception being, when they system has 2 cores it will set the max jail core to one.

In order for the jail to expand, add a jail_min_idle value to denote how much free utilization should be maintained in the jail:

jail_min_idle 5

BetterCPU will attempt to maintain at least 5% free CPU per jail core, and will expand the jail if max_jail_cnt has not been reached.

Defining User Groups

For more details on groups please see the User Groups section of the documentation.

The BetterLinux cPanel install script will discover all the cPanel users on the system and create a user group called "cpanel-users". The non-cPanel installation script will not create users groups, they need to be created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, or files containing UIDs or usernames, CPU usage levels can be precisely defined for different groups, and durations and conditions for starting/stopping throttling can be defined. The group names defined in /etc/betterlinux/cpud.conf.d/groups.conf are used in the settings for the start_throttling and stop_throttling parameters.

To define groups:

  1. Choose a group name
  2. Populate it with UIDs, UID ranges, user names, or files containing UIDs or usernames
    Note: A group may contain one or more usernames, UIDs, or ranges.

The following example illustrates the syntax rqeuired to define a group in the groups.conf file:

Single range:
 Syntax:  uid {group name} {uid range}

 Example: uid example-group 0-499
This example includes all UIDs from 0-499 as members of the group example-group

 Example: uid example-group 800-
This example includes all UIDs from 800 and above as members of the group example-group
 
Multiple UID ranges:
 Syntax:  uid {group name} {uid range}[,{uid range2}...]

 Example: uid example-group 0-499,600-699
This example includes all UIDs from 0-499 and from 600-699 as members of the group example-group

When planning a strategy for defining groups, use the following steps:

  1. Decide how users/customers should be organized into groups
    • This organization could be based on needs, paid service levels, or any other criteria.
    • Ideally, user UIDs ranges should be used to define users with similar requirements
  2. Edit /etc/betterlinux/cpud.conf.d/groups.conf
    • For example:
      uid normal-users 500-999
      uid premium-users 1000-1999
  3. Save the groups.conf file
  4. Restart the BetterCPU daemon:
    service cpud restart

Once this parameter change is in place, throttling limits that meet each group's SLA can be defined for each group.

If the users are already in a group defined in /etc/group, it is fairly easy to use that group definition within the groups.conf file. Use the following command to extract the group definition and reformat it for inclusion in the /etc/betterlinux/cpud.conf.d/groups.conf file. When defining the syntax, simply replace {group_name} with the name of the group defined in /etc/group.

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting CPU Core Usage Limits

Once jail_cnt and jail_cnt_max are defined and groups are set up, it is necessary to configure the throttling parameters that define when user groups are moved into and out of the jail cores. The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set throttling limits for groups defined in /etc/betterlinux/cpud.conf.d/groups.conf. The cpu_percent value sets the percentage an individual user (uid) can use of the total non-jail cores on the system.

To understand how this works, it is important to understand how CPU utilization is measured in a multi-core system. In Linux, multithreaded processes can take more than 100% utilization because overall CPU utilization is measured as 100% * number_of_cores. So in an 8 core system, if all CPUs are running at maximum utilization, the system's overall utilization is 800% (100% * 8 cores).

The values for cpu_percent in start_throttling and stop_throttling are based on a normalized utilization assigned to a single core (rather than being spread across all cores). This means that while an 8 core system can have a maximum of 800% utilization, the value used for the cpu_percent value represents the fraction of a single core that could be used. Specifying a value of 80 for cpu_percent means that on a system with no cores presently allocated to jail, when the user exceeds a value of 80% * 8 = 640% on an 8 core system, a core is allocated to the jail and the user's processes are throttled. When a core is allocated to jail, the percentage used to determine if the threshold has been exceeded is 80% * 7 = 560%. If a second core is allocated to jail, this threshold drops to 80% * 6 = 480%.

What this means is that - using the example of cpu_percent set to 80 - any distribution of utilization across all 8 cores of an 8 core system that adds up to more than 640% utilization will trigger the allocation of a core for throttling and that user's processes will be moved to that single core. The utilization could be 100% of 6 cores and 40% of a seventh core, it could be 80% utilization distributed across all 8 cores, or it could be any variation where the user's total utilization exceeds the threshold.

When a non-throttled user's total process utilization exceeds cpu_percent for seconds seconds as defined in the start_throttling parameter, the user's processes are moved into the jail cores.

When a throttled user’s total utilization drops below the cpu_percent value for seconds seconds as defined in the stop_throttling parameter, the user's processes are moved out of the jail back onto the non-jailed cores.

To define these limits:

  • Edit the users.conf located in /etc/betterlinux/cpud.conf.d.
  • Add the following parameters users.conf, with appropriate limits to the system's needs:
start_throttling group=example-group cpu_percent=50,seconds=3
stop_throttling group=example-group cpu_percent=25,seconds=10
  • Save users.conf.
  • Restart cpud:
service cpud restart

With these settings, a user in the group defined with processes that exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system. When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.

Start the BetterCPU Daemon (cpud)

BetterCPU is now configured with basic jail core and throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service cpud {start|stop|status|reload|restart|condrestart}
 start = starts cpud daemon
 stops = stops cpud daemon
 status = shows the status of cpud daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Start the cpud daemon with the following command:

service cpud start

Start the BetterLinux Stats Program (blstat)

In order to see user processes that are in the jail cores, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterI/O. Press 'c' to see the stats for BetterCPU. Press '?' to display general help for the utility.

Limiting Execution Time Using time_monitor

BetterCPU can be used to limit the amount of time programs can execute. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting execution time are:

  1. Define a program or user group
  2. Add time_monitor to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is as follows:

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example:

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. The user groups are structured as follows:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for time_monitor execution limiting are identical to user groups used for other functions in the BetterLinux suite.

Add time_monitor to main.conf

Limiting the execution time for a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again. The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.

To define the limit, add the following line to the main.conf file:

time_monitor groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer 10

In this example, we use the sample "bad_progs" program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash. If the program has run for 5 seconds or longer, /root/killer is called with the PID of the program, and the script kills the program after logging the date and time.

The final parameter (10) tells cpud to not run the script more than once every 10 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring the execution time, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:

time_monitor_prog_exclude groups=exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

Limiting Open Files Using file_limit

BetterCPU can be used to limit the number of concurrent open files a user can have at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent open files are:

  1. Define a user group
  2. Add file_limit to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When open files are limited, the file_limit_override and file_limit_exclude options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the following format used is:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting open files with file_limit are identical to user groups used for other functions in the BetterLinux suite.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Processes Using process_limit

BetterCPU can be used to limit the number of concurrent processes a user can execute at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent processes are:

  1. Define a user group
  2. Add process_limit to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the format used is:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting concurrent processes with process_limit are identical to user groups used for other functions in the BetterLinux suite.

Add process_limit to main.conf

Limiting concurrent processes per user is very straightforward.

To apply a limit, once a user group is defined, add a line to apply the desired restiction to the main.conf file:

process_limit group=bad_users limit=100

This limits the users in the bad_users group to 100 concurrent processes each at any given time.

To completely exclude specific programs from having limits applied, create a program group in groups.conf with the programs in it:

prog exclude_process_limits /bin/cron,/bin/bash

Then add a process_limit_exclude line to main.conf:

process_limit_exclude group=exclude_process_limits

If a user limit should be overridden for particular programs, specify the user group as part of a process_limit_override command and specify the specific programs that should have the different limit in main.conf:

process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd

This overrides the existing limits for bad_users for instances of /usr/sbin/sshd and sets those limits to 120.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Memory Utilization Using memory_limit

BetterCPU can be used to limit the amount of RSS memory programs can use. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting RSS memory usage are:

  1. Define a program or user group
  2. Add memory_limit to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is:

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example:

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. These user groups are structured as:

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example:

uid bad_users testuser1,501,1000-1999

User group definitions used for RSS memory usage limiting are identical to user groups used for other functions in the BetterLinux suite.

Add memory_limit to main.conf

Limiting the RSS memory used by a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again. The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.

To define the limit, add the following line to the main.conf file:

memory_limit group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run=30

In this example, we use the sample bad_progs program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB). If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.

The final parameter (30) tells cpud to not run the script more than once every 30 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:

memory_limit_prog_exclude exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

CloakFS File System Cloaking

"cloak_user" secures the file system by hiding users from each other, including others’ files, directories, and processes. A user can see only his or her own $HOME directory, with the exception files outside of home owned by that user. For example, if “sampleuser” creates “samplefile” in /tmp and then lists all files in /tmp, he will see only “samplefile” and not the /tmp files of other users on the box.

Configuration Files

The cloak_user parameter is found here:

Non-Cpanel install: /etc/betterlinux/cpud.conf.d/rules.conf
cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

Add values to the cloak_user parameter with the following syntax:

cloak_user group=user_group1,user_group2 level=1

The first value (following “group=”) in the above example is a comma-delimited list of groups (user_group1, user_group2, etc.), which you can define in groups.conf.

The second value (following “level=”) can be either 1 or 2. The default value is 1. If 1, a user can display only his or her own processes as well as processes belonging to users outside the groups named within this parameter (like root, mysql, and apache’s “nobody” user). A value of 2, however, allows a user to display only his or her own processes and nothing else, not even the processes of users outside the groups named within this parameter.

Consider a parameter with these values:

cloak_user group=user_group1 level=1

In the above, all users in user_group1 will see only their own processes and those of users not belonging to user_group1.

Next, see these values:

cloak_user group=user_group2 level=2

Here, all users in user_group2 will see only their own processes. This excludes even processes run by root, mysql, and apache, etc.

Load Average Overview

"load_average" is important for shared hosting providers using BetterLinux cpu throttling feature. Now admins can set configure the proper load average for the system.

Load Average Usage

This parameter takes a value in the format:

load_average type=all or non_jail_only normalize_load=on or off

The first switch (type=) value takes the word "all" meaning all the cores on the system or the word "non_jail_only" meaning to calculate the load average only on the non_jail_cores. The second switch (normalize_load=) value is "on" or "off". Entering a value of "off" adds up the total load of the cores to calculate the load average. Entering a value of "on" adds up the total load of the cores and divides it by the total number of cores.

For example, if this parameter is set to:

load_average type=all normalize_load=off

Will add up the load for all the cores.

load_average type=all normalize_load=on

Will add up the load for all the core (jail and non_jail) and divide the total load by the total number of cores.

  1. load_average type=non_jail_only normalize_load=off

Will add up the load for all non-jail cores.

load_average type=non_jail_only normalize_load=on

Will add up the load for all the non-jail cores and divide by the total number of non_jail cores.

BetterIO Overview

The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.

In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.

From a CPU perspective, this is referred to as “Waiting on I/O”, or iowait%. If the I/O channel is not also managed, then the system will not be able to run optimally.

When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.

There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To monitor these factors, one can use the iostat command or blstat (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.

BetterIO Configuration

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. Once the the system is rebooted, configure BetterIO and start the iothrottled daemon. By default, BetterIO does not throttle any users. The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).

Defining user groups

User-created groups do not exist on the system until they have been created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined. The group names defined in /etc/betterlinux/iothrottled.conf.d/groups.conf are used in the settings for setting per-group I/O limits.

To define groups:

  1. Choose a group name.
  2. Populate the group with UIDs, UID ranges, user names, or files containing UIDs or usernames
    Note: A group may contain one or more usernames, UIDs, or ranges.

The following examples illustrate the command syntax:

Single range:
 Syntax:  uid {group name} {uid range}

 Example: uid example-group 0-499
This example includes all UIDs from 0-499 as members of the group example-group

 Example: uid example-group 800-
This example includes all UIDs from 800 and above as members of the group example-group
 
Multiple UID ranges:
 Syntax:  uid {group name} {uid range}[,{uid range2}...]

 Example: uid example-group 0-499,600-699
This example includes all UIDs from 0-499 and from 600-699 as members of the group example-group

When planning a strategy for defining groups, use the following steps:

  1. Decide how users/customers should be organized into groups.
    • This organization could be based on needs, paid service levels, or any other criteria.
    • Ideally, user UIDs ranges should be used to define users with similar requirements.
  2. Edit /etc/betterlinux/iothrottled.conf.d/groups.conf.
    • For example:
      uid normal-users 500-999
      uid premium-users 1000-1999
  3. Save the groups.conf file.
  4. Restart the BetterIO daemon:
    service iothrottled restart

Once this parameter change is in place, you can define throttling limits that meet each group's SLA.

If the users are already in a group defined in /etc/group, it is fairly easy to use that group definition within the groups.conf file. Using the following command will extract the group definition and reformat it for inclusion in the /etc/betterlinux/iothrottled.conf.d/groups.conf file. Just replace {group_name} with the group name defined in /etc/group.

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting I/O Bandwidth Limits

Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis. To do this, edit the user_limits.conf file located in /etc/betterlinux/iothrottled.conf.d. This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/groups.conf.

The user_limits.conf file uses a single line per entry:

[group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.

For example:

example-group bw max sda,sdb,sdc 10000000

This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.

Another example:

io_standard_users bw max inc="^/",exc="^/home" 5000000

This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.

After updating the user_limits.conf file with limits applicable to the system, save the file and restart iothrottled.

Note:

  • If a group and a user have the same name, the limit applies to the user.
  • Only one limit per device can be set. It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.

Please see the BetterIO documentation for more configuration options for I/O bandwidth limiting features by program and inclusion/exclusion options.

Tuning I/O Scaling and Balancing

The auto_scale and seek_balance features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in user_limits.conf.

These settings are contained in /etc/betterlinux/iothrottled.conf.d/scale.conf. The configuration parameters define how these settings affect the I/O bandwidth. Details on how these settings work can be found in the user documentation for scaling and balancing.

Start the BetterIO daemon (iothrottled)

BetterIO is now configured with basic throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service iothrottled {start|stop|status|reload|restart|condrestart}
 start = starts iothrottled daemon
 stops = stops iothrottled daemon
 status = shows the status of iothrottled daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Use the following command to start the iothrottled daemon:

service iothrottled start

Start the BetterLinux stats program (blstat)

To see user processes that are being throttled by BetterIO, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterIO. Press 'i' to see the stats for BetterIO. Press '?' to display general help for the utility.

BetterMySQL Overview

BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.

It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated. Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.

Configuring BetterMySQL for cPanel Support

BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.

Installing the BetterLinux version of MySQL for cPanel

BetterMySQL cannot be installed unless MySQL is already on the system. BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL. If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script:

/etc/betterlinux/cpanel/bl-cpmysql-install

Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the BetterCPU functional documentation. BetterMySQL’s configuration file serves only to enable BetterMySQL.

Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set. For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit:

start_throttling group=cpanel_users cpu_percent=50 seconds=3
stop_throttling group=cpanel_users cpu_percent=30 seconds=5

Same logic apply's to any BetterIO limits that have been set. In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled.

cpanel_users bw max inc="^/" 5000000

Configuring BetterMySQL for cPanel support

To provide and configure per-user query throttling of MySQL users a mapping of MySQL users to Linux users must be configured. This mapping and other configuration options are auto-configured by either the bl-cpanel-install script on first time installation or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the bl-hooks script can be used after a manual installation of BetterMySQL. Example syntax for running the bl-hooks script:

/etc/betterlinux/cpanel/bl-hooks --inint

Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.

  • usersmap.conf contains the mapping of cPanel users to UIDs
  • mysql.conf contains the parameter location of the usermap.conf file
  • turns on BetterMySQL throttling by adding the approved_accounting parameter to /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf
approved_accounting_program program=/usr/libexec/mysqld user=mysql
  • Restarts the cpud daemon to enable BetterMySQL throttling.
service cpud restart

Configuring BetterMySQL for Linux Servers

BetterMySQL configuration is a manual setup for Linux servers

Installing the BetterLinux version of MySQL non-cPanel

After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command:

yum install mysql-server

Configuring BetterMySQL for non-Cpanel Linux servers

To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.

The command format is as follows:

bl_usermap_filename [usermap_file_path]

Example:

bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

Now create the MySQL mapping to Linux user mappings located in the following file:

/etc/betterlinux/mysql.conf.d/usermap.conf

The usermap file is a tab-delimited file that contains the following information:

[mysqluser]	[linux_user]	[uid]

The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.

For example, the following file will create mappings for testuser1 and testuser2. For testuser1, the uid will be determined when that user makes a query against the database.

testuser1	testuser1
testuser2	testuser2	501

The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;":

mysql> show user_map;
+-----------+-----------+------+
| From_user | Unix_user | Uid  |
+-----------+-----------+------+
| testuser2 | testuser2 |  501 |
| testuser1 | testuser1 |  500 |
+-----------+-----------+------+
2 rows in set (0.00 sec)

The approved_accounting parameter turns on the BetterMySQL throttling and is located in:

/etc/betterlinux/cpud.conf.d/main.conf

The syntax is as follows for non-cPanel servers:

approved_accounting_program program=/usr/sbin/mysqld user=mysql

Restart the cpud daemon to enable the configuration settings:

service cpud restart

Monitoring MySQL Utilization with myusertop

BetterMySQL includes the myusertop utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.

Once myusertop is running, press '?' to view the built-in help.

The output is similar to the output of the standard 'top' utility:

mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
Time: CPU 448          Busy 529         CPU/Busy 0.8477   RowsExamined:   1,476m
Qrys: Select 28,456    Update 0         Other 5,250         Adj-RowsEx:   1,251m
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
      h=Help m=MySQLUsersOn/Off                                           q=quit
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx   %CPU
root                   0     2 0.0075   0.0074   1.0060     150      151  100.00
  root                 0     2 0.0075   0.0074   1.0060     150      151  100.00
testuser2            501     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser2          501     0 0.0000   0.0000   0.0000       0        0    0.00
testuser1            500     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser1          500     0 0.0000   0.0000   0.0000       0        0    0.00

In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization. If it were divided equally between the three users, myusertop would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.

If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.

BetterBandwidth Overview

BetterBandwidth uses 'tc' (a built in Linux tool) to provide traffic shaping for users that have their own network interface. The tc tool on its own does not provide a mechanism for limiting bandwidth usage by uid/user on a shared network interface. BetterLinux has added this functionality into its BetterBandwidth feature. Now admins can limit bandwidth for a single user on a shared server that has a corporate internet connection.

A common problem for web hosts is that one single user with a 1Gbit connection could kill the entire TCP stack. Often the host is unaware that it happens because they use 5 min bandwidth averages to monitor bandwidth on their 1 gig or 10 gig port. But all it takes is a 30 second spike in bandwidth out of 5 mins to make customers angry. And because of the 5 minute average it looks like the peak was around 50Mbits when it easily could have been 500Mbits. Customers will blame it on the server or the hosting company.

Many hosting companies give out IPs just to be able to limit bandwidth. This is an expensive approach. The BetterLinux BetterBandwidth feature can do it without giving out dedicated IPs.

BetterLinux has an apache mod called mod_betterlinux that allows BetterBandwidth to limit bandwidth for all ports for an individual user or group of users like email, FTP, etc is all included. This is a tremendous advantage for a host. BetterBandwidth monitors and controls individual, group, and system-wide bandwidth usage in multi-tenant Linux environments. Administrators can now limit outbound bandwidth as if each user had a dedicated IP address. This lets admins consolidate multiple users onto a single IP, which innovatively addresses the fast-increasing IPV4 IP address shortage.

BetterBandwidth Configuration

BetterBandwidth limiting uses the mod_betterlinux Apache module located in in the betterlinux-core-*.rpm , as well as the linux tc method of using the uid classifier to categorize specific user processes. The betterlinux-core-*.rpm is installed by the bl-install script by default and auto-configured for cPanel systems during the betterlinux-cpanel-install script.

cPanel Bandwidth Limiting Setup

During the betterlinux-cpanel-install script installation, the user will answer Y/N to this question:

Install BetterLinux Apache throttling? [y/N]

Selecting yes, will auto-configure and load the apache module mod_betterlinux on a cPanel system.

After the installation is complete, the following parameters and arguments need to be added to the BetterLinux configuration files:

  • In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example:
interface interface-types eth0
  • In /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format:
bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to:

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

non-cPanel Bandwidth Limiting Setup

  • Depending on the distribution, edit the apache conf or create a new one under conf.d and add the following:
LoadModule betterlinux_module /usr/lib64/httpd/modules/mod_betterlinux.so
<IfModule betterlinux_module>
 Betterlinux On
</IfModule>

Then restart apache.

Using this command will check the mod_betterlinux syntax in the apache .conf file:

apachectl -t -D DUMP_MODULES
  • Configure BetterCPU (cpud) by adding or uncommenting the following parameter located in /etc/betterlinux/cpud.conf.d/main.conf
 approved_accounting_program /usr/sbin/httpd apache

Please note, the full path to the httpd program is required, and it must be the actual program and not a symlink. Also, be sure to list the user it runs as, whether that is apache, httpd, or different username.

  • In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example:
interface interface-types eth0
  • In /etc/betterlinux/cpud.conf.d/main.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format:
bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to:

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

Assigning IP addresses with the dedicated IP address mapper

With multi-homed systems, incoming traffic is directed to the IP address where DNS resolves the domain name. A system with multiple virtual hosts can accept traffic to secondary addresses which, in turn, are assigned to secondary addresses, which are assigned to the system's network adapters.

Without some modification, responses sent from a server in a multi-homed system originate from the host's primary IP address. This makes it difficult to apply bandwidth or connection limits using a server's IP address. Most often, hosting providers work around this problem by implementing some sort of full virtualization solution that provides a unique virtual instance for each client. <KGO> What is the drawback of this solution?

BetterBandwidth uses the dedicated kernel module to provide a more cost-effective, efficient solution for managing bandwidth in multi-homed environments. When used in conjunction with the BetterBandwidth dedicated daemon, the dedicated kernel module allows administrators to assign outgoing traffic to a specific secondary address which, in turn, is assigned to a specific user--all without the overhead of a fully virtualized system. (Both IPv4 and IPv6 are supported.)

To enable this functionality:

  1. Ensure the dedicated kernel module is loaded:
    modprobe dedicated
  2. Edit /etc/betterlinux/ipv4 and/or /etc/betterlinux/ipv6 to include the specific mappings.
  3. Restart dedicatedd:
    service dedicatedd restart

The IPv4 and IPv6 mapping files are in the following format. Each line contains a single entry.

username|uid: address

The address field can contain either a raw IPv4 or IPv6 address or a DNS name. If the DNS name resolves to more than one address, the kernel randomly selects one of the addresses.

Once configured, the active mappings for IPv4 addresses can be viewed using the following command:

cat /proc/sys/net/ipv4/dedicated

To view mappings for IPv6 addresses, use the following command:

cat /proc/sys/net/ipv6/dedicated

Note: The command output displays the uid rather than the username.

Frequently Asked Questions

I have a customized kernel configuration that requires I build the kernel myself.

  • BetterLinux provides support for customized kernels via the Build Your Own Kernel (BYOK) feature.

BYOK Instructions

Does BetterLinux run on CentOS 5?

  • BetterLinux supports kernels 2.6.32 or newer. We are currently evaluating a method for supporting CentOS 5. Some BetterLinux features depend on Linux kernel features that were introduced in version 2.6.26. Some of those features have bugs that were not fixed until version 2.6.32. Hence, we cannot support kernels prior to 2.6.26 at all, and kernels prior to 2.6.32 require backported bug fixes. CentOS 5 has kernel version 2.6.18. It is possible to build a later kernel for CentOS 5, but then some userspace programs, such as iptables, must also be updated to cope with changed kernel APIs.

What will the pricing be after the Free Until 2013 period is over?

  • BetterLinux will continue to be free for non-profit use but will begin charging $9.00 per month per server for commercial use after 2013.

Will BetterLinux be compatible with cPanel/WHM and if so will that include plugins for WHM?

  • BetterLinux is currently developing integration and plug-ins for cPanel and WHM. The targeted release date is end of November 2012.

How to install blstat?

blstat is not part of the product installation. Follow the steps below to download and install blstat:

yum --disableexcludes=all install perl-TermReadKey perl-JSON

This will install two perl modules blstat needs.

wget http://repo.betterlinux.com/unsupported/blstat-0.53-1.el6.noarch.rpm

Downloads the blstat rpm.

rpm -ivh blstat-0.53-1.el6.noarch.rpm

command to install blstat rpm

At the command line as root blstat is executed by running:

blstat

The initial screen is the i/o throttling stats. Pressing "c" will switch blstat to the cpu throttling screen. Pressing "i" will switch the screen back to i/o throttling. Pressing "q" will exit blstat.

System Requirements

The BetterLinux products are supported under the following system requirements:

  • Linux Kerenel 2.6.32 or newer
  • MySQL 5.1 or newer
  • CentOS 6.x or RHEL 6.x
  • CPUs with at least 2 cores

Log File Mapping to Parameters|Log Files

iothrottle daemon

iothrottled

Log File Name Location Parameters Description
N/A /var/log/iothrottled enable_logging Turns on all logging
iothrottled /var/log/iothrottled all parameters All logging sent to this file
iothrottled.cnt /var/log/iothrottled  ? cgroup creation and deletion counts are sent to this file

Note: The location and name of the logfile can be modified using the log_file or log_cnt_file in /etc/betterlinux/iothrottled/iothrottled.conf.d/logging.conf.

cpu daemon

The following is a mapping of the cpud log files and the tunable parameters that log their results in them.

cpud

Log File Name Location Parameters Description
cpud /var/log/cpud enable_logging Turns on all logging
cpud /var/log/cpud log_cpud All cpud logging sent to this file

Note: The location and name of the logfile can be modified using the log_cpud_file in /etc/betterlinux/cpud/cpud.d/logging.conf.

cpud_governor

Log File Name Location Parameters Description
cpud_governor /var/log/cpud log_gov  ?
cpud_governor /var/log/cpud kill_ded_web  ?
cpud_governor /var/log/cpud kill_ded_crond  ?
cpud_governor /var/log/cpud kill_ded_sshd  ?
cpud_governor /var/log/cpud kill_ded_daemon  ?
cpud_governor /var/log/cpud kill_ded_web_busy  ?
cpud_governor /var/log/cpud kill_ded_crond_busy  ?
cpud_governor /var/log/cpud kill_ded_sshd_busy  ?
cpud_governor /var/log/cpud kill_ded_daemon_busy  ?
cpud_governor /var/log/cpud kill_non_ded_web  ?
cpud_governor /var/log/cpud kill_non_ded_crond  ?
cpud_governor /var/log/cpud kill_non_ded_sshd  ?
cpud_governor /var/log/cpud kill_non_ded_daemon  ?
cpud_governor /var/log/cpud kill_non_ded_web_busy  ?
cpud_governor /var/log/cpud kill_non_ded_crond_busy  ?
cpud_governor /var/log/cpud kill_non_ded_sshd_busy  ?
cpud_governor /var/log/cpud kill_non_ded_daemon_busy  ?
cpud_governor /var/log/cpud kill_web_io_idle_time  ?
cpud_governor /var/log/cpud kill_ded_web_exclude  ?
cpud_governor /var/log/cpud kill_ded_crond_exclude  ?
cpud_governor /var/log/cpud kill_ded_sshd_exclude  ?
cpud_governor /var/log/cpud kill_ded_daemon_include  ?
cpud_governor /var/log/cpud kill_non_ded_web_exclude  ?
cpud_governor /var/log/cpud kill_non_ded_crond_exclude  ?
cpud_governor /var/log/cpud kill_non_ded_sshd_exclude  ?
cpud_governor /var/log/cpud kill_non_ded_daemon_exclude  ?

Note: The location and name of the logfile can be modified using the log_gov_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

cpud_mem_con

Log File Name Location Parameters Description
cpud_mem_con /var/log/cpud log_memcon  ?
cpud_mem_con /var/log/cpud max_prog_rss  ?
cpud_mem_con /var/log/cpud max_user_rss  ?
cpud_mem_con /var/log/cpud tmpfs_spike_limit  ?
cpud_mem_con /var/log/cpud tmpfs_spike_val  ?
cpud_mem_con /var/log/cpud kill_mem_free_limit_panic_pct  ?
cpud_mem_con /var/log/cpud kill_cache_limit_panic  ?
cpud_mem_con /var/log/cpud kill_mem_free_limit_pct  ?
cpud_mem_con /var/log/cpud kill_cache_limit  ?

Note: The location and name of the logfile can be modified using the log_memcon_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

cpud_panics

Log File Name Location Parameters Description
cpud_panics /var/log/cpud log_panics  ?
cpud_panics /var/log/cpud tmpfs_spike_limit  ?
cpud_panics /var/log/cpud tmpfs_spike_val  ?
cpud_panics /var/log/cpud kill_mem_free_limit_panic_pct  ?
cpud_panics /var/log/cpud kill_cache_limit_panic  ?
cpud_panics /var/log/cpud kill_mem_free_limit_pct  ?
cpud_panics /var/log/cpud kill_cache_limit  ?

Note: The location and name of the logfile can be modified using the log_panics_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

=

cpud_cpu

Log File Name Location Parameters Description
cpud_cpu /var/log/cpud log_cpu  ?
cpud_cpu /var/log/cpud jail_pct  ?
cpud_cpu /var/log/cpud jail_cnt  ?
cpud_cpu /var/log/cpud max_jail_pct  ?
cpud_cpu /var/log/cpud max_jail_cnt  ?
cpud_cpu /var/log/cpud kill_mem_free_limit_pct  ?
cpud_cpu /var/log/cpud kill_cache_limit  ?

Note: The location and name of the logfile can be modified using the log_cpu_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

cpud_jail

Log File Name Location Parameters Description
cpud_jail /var/log/cpud log_jail  ?
cpud_jail /var/log/cpud log_mem  ?

Note: The location and name of the logfile can be modified using the log_jail_file in /etc/betterlinux/cpud/cpud.conf.d/logging.conf.

Monitoring Tools

blstat

blstat is not part of the main product installation. Follow the steps below to download and install blstat:

If you are using cPanel, please run:

/scripts/perlinstaller Term::ReadKey JSON

Or if you are running stock CentOS6, please run:

yum --disableexcludes=all install perl-TermReadKey perl-JSON

This will install two perl modules blstat needs.

wget http://repo.betterlinux.com/unsupported/blstat-0.53-1.el6.noarch.rpm

Downloads the blstat rpm.

rpm -ivh blstat-0.53-1.el6.noarch.rpm

command to install blstat rpm

At the command line as root blstat is executed by running:

blstat

Hot Keys

First screen is the i/o screen.

    c - Toggle cpu throttling screen
    i - Toggle i/o throttling screen
    q - exit's blstat

myusertop

Description

Display the MySQL users most active during the preceding 4 seconds. Show the Linux users and UIDs that "own" them. Press single "hot" keys to change the display interactively.

Hot Keys

    V - Toggle views 1 and 2
    m - Toggle the display of MySQL users under "owning" Linux users
    H - Toggle hourly/total statistics in header area (only)
    R - Reverse sort direction (ascending/descending)
    f - Invoke FLUSH USER_STATISTICS (to reset statistics to zero)
    Sort by:
    a - Adjusted rows examined                        (view 1)
    b - Busy time during last refresh interval        (view 1)
    c - CPU time during last refresh interval         (view 1)
    C - CPU time / busy time                          (view 1)
    e - Core number at end of latest query            (view 2)
    i - UID                                           (all views)
    l - Busy time of latest completed query           (view 2)
    L - CPU time of latest completed query            (view 2)
    n - Number of non-voluntary context switches in 
          latest completed query                      (view 2)
    o - "Other" commands during last refresh interval (view 2)         
    p - % of total CPU time used by user              (view 1)
    Q - Number of queries in progress                 (view 2)
    r - Rows examined                                 (view 1)
    s - Select commands during last refresh interval  (view 2 default)
    S - Start core number of latest query             (view 2)
    t - Number of threads per user                    (view 1 default)
    u - Update commands during last refresh interval  (view 2)
    U - User name                                     (all views)
    v - Number of voluntary context switches in 
          latest completed query                      (view 2)

Other:

    h,? - Display online help
    q   - Quit B<myusertop>

Column Headings

    Linux/MySQLUser - MySQL user is indented beneath Linux user name
    Thds            - Number of threads at the current time
    CPUTime         - Seconds of CPU time since previous refresh
    BusTime         - Seconds of busy (clock) time during same interval
    CPU/Bus         - CPU time divided by busy time
    RowsExam        - Number of rows "examined" since previous refresh
    AdjRowEx        - RowsExam * CPU / Bus
    %CPU            - Amount of MySQL's CPU usage used by this user
    Sel             - Number of SELECT commands since previous refresh
    Upd             - Number of UPDATE commands since previous refresh
    Oth             - Number of other commands since previous refresh
    VCxSw           - Number of voluntary context switches since previous
                        refresh
    NVCSw           - Number of non-voluntary context switches since
                        previous refresh
    SC              - Core number at start of most recent query
    EC              - Core number at end of most recent query
    LQCpu           - CPU seconds of most recent query
    LQBus           - Busy (clock) seconds of most recent query

Command-line Options

Most of the command-line options (below) can be specified interactively while myusertop is running by pressing single "hot" keys. (See above.) Many of the options can be set as defaults in the configuration file

   */etc/betterlinux/mysql.conf.d/myusertop.conf*.

Command-line options (preceded by 1 or 2 hyphens) may be abbreviated to the shortest unique prefix):

   -ve, --version

Display the version of myusertop and exit.

   -nom, --nomyuser

Do not display the MySQL users "owned" by each top Linux user. (The default is to list each Linux user's associated MySQL users, indented, beneath the Linux user.) During an interactive session, the "m" hot key toggles the display of "owned" MySQL users on and off.

   --cfg=<filename>, --config=<filename>

Override the default (/etc/betterlinux/mysql.conf.d/myusertop.conf) configuration file name with <filename>.

   -S [path], --socket=[path]

Path specifying unix domain socket to pass to the mysql command.

   -ho, --hourly

In the header summary information, show hourly statistics (instead of the totals since mysqld was started or FLUSH USER_STATISTICS reset the statistics. (If mysqld has been running for less than one hour--or if FLUSH USER_STATISTICS was invoked within the hour--values will be adjusted upward to show the hourly rate if usage continues at the same rate for the remainder of the hour.)

During an interactive session, the "H" hot key toggles the header between displaying hourly and total statistics.

Note: Regardless of whether the header displays hourly statistics or statistics since startup/flush time, the main user columns show usage during the most recent refresh interval.

   --refresh=<seconds>

Specify the refresh interval in seconds. The default refresh interval is 4 seconds.

--admin=<admin-user-name> Specify the name of a super MySQL user with authority to issue MySQL administrative commands. This option is to be used ONLY in cases where the root unix user (the one running this script) needs to specify a MySQL superuser name when starting the mysql client.

   --adminpass=<admin-user-password>

Specify the password of the MySQL user specified with the --admin option. This option is likewise used ONLY in cases where the root unix user (the one running this script) needs to specify a MySQL superuser name when starting the mysql client.

   --ascending

Sort ascending (from smallest to largest). This effectively makes myusertop show the *bottom* users. During an interactive session, the "R" hot key reverses the current sort direction. Use this option in combination with any *one* of the following sort options.

   --sortthreads

Sort users by the number of threads used by Linux users at the moment of the most recent refresh This is the default in view 1. During an interactive session, the "t" hot key will sort the number of threads.

   --sortcpu

Sort by CPU time (from highest to lowest). During an interactive session, the "c" hot key will sort by CPU time.

   --sortadjre 

Sort by adjusted rows examined (from highest users to lowest). Adjusted rows examined is the number of rows examined multipled by the ratio of CPU Time to Busy Time. During an interactive session, the "a" hot key will sort by adjusted rows examined.

   --sortbusy

Sort by busy (clock) time (from highest to lowest). During an interactive session, the "b" hot key will sort by busy time.

   --sortcob

Sort by ratio of CPU Usage to Busy Time. "CPU over Busy. During an interactive session, the "C" hot key will sort by the ratio of CPU time to busy time.

   --sortrows

Sort by number of rows examined (from highest to lowest.) During an interactive session, the "r" hot key will sort by the the number of rows examined.

   --sortcpupercent

Sort by the percentage of MySQL CPU time that the user is using. During an interactive session, the "p" hot key will sort by CPU ercentage.

   --sortselect

Sort by the number of select commands issued (from highest to lowest). During an interactive session, the "s" hot key will sort by the number of select commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortupdate

Sort by the number of update commands issued (from highest to lowest). During an interactive session, the "u" hot key will sort by the number of update commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortother

Sort by the number of other commands issued (from highest to lowest). During an interactive session, the "o" hot key will sort by the number of other commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortvcs

Sort by the number of voluntary context switches (from highest to lowest) during the most recent query. During an interactive session, the "v" hot key will sort by the number of voluntary context switches. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortnvcs

Sort by the number of nonvoluntary context switches (from highest to lowest) during the most recent query. During an interactive session, the "n" hot key will sort by the number of nonvoluntary context switches. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortqip

Sort by number of active queries in progress (from highest to lowest) for each user at the current moment. During an interactive session, the "Q" hot key will sort by the number of queries in progress. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortlqcpu

Sort by last query CPU time (from highest to lowest)--the number of seconds of CPU time used by each user's most recently completed query. During an interactive session, the "L" hot key will sort by the last query CPU time. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortlqbusy

Sort by last query busy time (from highest to lowest)--the number of seconds of busy time used by each user's most recently completed query. During an interactive session, the "l" hot key will sort by the last query busy time. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

Files

   /etc/betterlinux/mysql.conf.d/myusertop.conf

See Also

   myusertop.conf(8)

Parameters

  • add_user_time
  • approved_accounting_program
  • auto_scale
  • average_type
  • bandwidth_limit
  • block_suid
  • busy_nonjail_idle
  • busy_num_R_nonjailed
  • cgroup_dir
  • cloak_user
  • connection_limit
  • cpu_average_periods
  • enable_dumps
  • enable_logging
  • enable_logging
  • enable_program_limits
  • enable_sched_dumps
  • in_buf_sz
  • interruptible_sleeps
  • iothrottled_gid
  • jail_cnt
  • jail_min_idle
  • load_average
  • log_auto_scale
  • log_cgroups
  • log_cnt_file
  • log_cpu
  • log_cpu_file
  • log_cpud
  • log_cpud_file
  • log_failure
  • log_file
  • log_flush_rate
  • log_gov
  • log_gov_file
  • log_io_limits
  • log_jail
  • log_jail_file
  • log_jail_idle
  • log_killed
  • log_mem
  • log_mem_file
  • log_memcon
  • log_memcon_file
  • log_nonjail_idle
  • log_panics
  • log_panics_file
  • log_prog
  • log_prog_file
  • log_seek_balancing
  • log_success
  • log_throttled
  • log_tids
  • log_tids_file
  • log_user
  • log_user_cnt
  • log_user_file
  • log_user_in_jail
  • max_cgroups
  • max_jail_cnt
  • memory_limit
  • memory_limit_prog_exclude
  • nonjail_min_idle
  • process_limit
  • process_limit_exclude
  • process_limit_override
  • run_priority
  • sample_periods
  • scale_down_inc
  • scale_max
  • scale_min
  • scale_pct_hi
  • scale_pct_low
  • scale_pct_periods
  • scale_up_inc
  • seek_balancing
  • seek_balancing_down_pct
  • seek_balancing_up_pct
  • sleep_time_ms
  • sleep_time_ms
  • start_throttling
  • stop_throttling
  • throttle_group
  • time_monitor
  • user_average_periods