Main Page

From docs.betterlinux.com
Jump to: navigation, search


Welcome to the BetterLinux Product Documenation wiki, your one-stop resource for all things BetterLinux. It has everything from initial setup and configuration to fine tuning all the features. Please check here first before submitting a help desk ticket.

Contents

BetterLinux Introduction

BetterLinux is a collection of tools for system resource management, monitoring, and security intended for hosting providers, data centers, SaaS companies, and cloud environments. With it, you can control use and allocation of CPU, memory, MySQL, device I/O bandwidth, and IP bandwidth resources all within a secure environment. Individual users and processes that exceed set resource limits can be isolated from other system users and throttled as necessary.

BetterLinux manages, monitors, and secures system resources for hosting providers, data centers, SaaS companies, and cloud environments. It lets you control use and allocation of CPU, memory, MySQL, device I/O bandwidth, and IP bandwidth resources all within a secure environment. When individual users and processes exceed set resource limits, you can isolate them from other users and throttle as necessary.

Historically, process management on Linux has involved adjusting processes’ priorities and nice levels. More recently, Linux system administrators have used technologies like full virtualization, paravirtualization, Unix containers, or other similar conventions to segregate system resource usage. These controls, while useful, are quite crude and do not address the core issue of guaranteeing and prioritizing operating system resources for critical applications across an entire system. Poorly-behaved processes can still monopolize the system and cause overall performance problems.

Recent Linux kernels have introduced control groups, or cgroups, which divide resources according to hierarchically defined groups of processes. For example, one cgroup might include all web-server-related processes, while another includes system backup processes. You could allocate more resources to the first.

BetterLinux augments cgroups' abilities. It automatically categorizes processes based on owner or process name. It also uniquely auto-adjusts (to scale) resources limits in response to constantly-changing system loads, which guarantees resource priority and availability with unprecedented precision.

BetterLinux operates either simply or with impressive complexity as it suits you. Each module comes with configuration files wherein you can create groups of users or processes, set various limits, turn options on or off, and change logging practices, etc. For configuration changes to take effect, you must either reload the .conf changes or restart the module daemon as follows:

BetterLinux operates either simply or with impressive complexity as it suits you. To start, create groups of users or processes (using included configuration files), set various limits, turn options on or off, and change logging practices. For configuration changes to take effect, either reload the .conf changes or restart the module daemon as follows:

Service cpud restart		[Restart daemon with updated changes.]
Service cpud condrestart	[Load new .conf changes & restart daemon only if daemon is already running.]
Service cpud reload		[Load .conf file settings without restarting daemon.]


BetterLinux consists of four modules: BetterCPU, BetterIO, BetterMySQL, BetterBandwidth, and CloakFS. See their brief descriptions below.

BetterCPU

BetterCPU (a CPU controller) limits access to a server’s CPU resources by individual users and processess. Over-consuming processes are isolated to a “jail” where they are throttled as necessary to prevent them from hogging CPU resources. Processes operating within limits are left alone.

BetterCPU Features:

  • Fairly allocate CPU resources and guarantee resources for specific users/processes
  • Temporarily “jail” over-consuming users/processes to prevent crashes and congestion
  • Automatically reallocate unused dedicated resources for better system-wide performance
  • Limit the number of processes and/or open files per user by creating customized groups and setting limits for each group. You can even create exclusions and overrides.
  • Trim CPU usage peaks with customized thresholds for groups and individuals
  • Stop programs that run too long or eat too much memory. Make custom groups, set time or memory limits, and write a custom script that runs when those limits are exceeded, with the option of creating a log entry.
  • Protect individual groups of programs with specific cpu settings.
  • Multiply server density while cutting heat, power, & hardware costs; increase revenue
  • Monitor CPU, user, and core behavior in real time; see logs, summaries, and detailed stats (accessible with hot keys)
  • One configuration file can hold settings for multiple systems

BetterIO

BetterIO manages block device I/O utilization. You can create limits for individual users and groups, for individual programs, for specific devices by percent usage, and even for mount points (like NFS, iSCSI, and more). You can also limit by service time (svctime), access wait (await), or a combination of both. BetterIO can “auto-scale” system limits when resources go used to prevent waste. “Auto-scaling” responds to highly configurable conditions and use ranges in addition to recent user activity.

BetterIO manages block device I/O utilization. You can create limits for individual users, individual programs, specific devices by percent usage, and even for mount points (like NFS, iSCSI, and more). You can also limit by service time (svctime), access wait (await), or a combination of both. BetterIO can auto-adjust (to scale) system limits when resources go used to prevent waste. “Auto-scaling” responds to highly configurable conditions and use ranges in addition to recent user activity.

BetterIO Features:

  • Create customized individual and group limits for users, programs, devices, and mount points
  • Control seek times, I/O operations, and disk bandwidth use for a balanced system without monopolizers.
  • Prevent bottlenecks and timeouts
  • Eliminate waste by autoscaling limits when resources go unused
  • Multiply server density
  • Thoroughly monitor per-user I/O use details and totals by device of origin in real time
  • Reduce need for more hardware or reductions in tenants per server
  • Reduce heat and power consumption
  • One configuration file can hold settings for multiple systems
  • Integration with cPanel user management for web hosting providers

BetterMySQL

BetterMySQL assigns the responsibility for queries to individual Linux users instead of to the generic MySQL user to enable individualized MySQL throttling. BetterMySQL also auto-adjusts (to scale) system limits when resources go used to prevent waste.

BetterMySQL Features:

  • Map MySQL users to Linux users; all queries now tied to specific users for individualized throttling
  • Control abusive MySQL users, processes, and threads
  • Compatible even on nonstandard platforms/directories
  • Automatically reallocate unused dedicated resources to cut waste
  • Thoroughly monitor individual MySQL use and relevant CPU use in real time; see detailed summaries
  • One configuration file can hold settings for multiple systems

BetterBandwidth

BetterBandwidth links traffic to specific Linux users, permitting group and individual limits without need for individual IP addresses. You can also make a user’s traffic on a shared server originate from an IP address your choose instead of your main shared IP. BetterBandwidth has extensive bandwidth limiting monitoring abilities.

Bandwidth Limiting Features:

  • Link outgoing bandwidth to specific Linux users
  • Limit group and individual bandwidth use without assigning individual IP addresses
  • Throttle only those who exceed set limits
  • Increase profits with more users, fewer support calls, and reduced need for IPV4 IP address blocks
  • Specific users’ traffic on shared servers now originates from IP addresses you choose, not your main shared IP
  • Have more bandwidth per user or more users on your bandwidth
  • Monitor individual bandwidth use in real time with detailed info on addresses, ports, sockets, protocols, and packets; create custom summaries
  • One configuration file can hold settings for multiple systems

CloakFS

CloakFS secures the file system by hiding cPanel and Linux users from each other, including others’ files, directories, and processes. A user can see only his or her own $HOME directory and files outside home that he owns. For example, if “sampleuser” creates “samplefile” in /tmp and then lists all files in /tmp, he will see only “samplefile” and not the /tmp files of other users on the box.

CloakFS Features:

  • Users cannot see or access other cPanel or Linux users’ processes, directories, and files anywhere on the system via suexec, suPHP, symlink, and ssh.
  • Even if you could access other users’ files, we prevent you from reading them or writing to them.


Install, Upgrade, Uninstall, & BYOK

System Requirements

BetterLinux is supported under the following system requirements:

  • Linux Kerenel 2.6.32 or newer
  • MySQL 5.1 or newer
  • CentOS 6.x or RHEL 6.x
  • CPUs with at least 4 cores (CPU throtting requires 4-core servers. Other BetterLinux features are supported with 2-core servers.)

We support virtual machines like KVM, VMWare, etc, but do not currently support OpenVZ.

1.1.x-x Installation on CentOS 6.x with cPanel

Easily install BetterLinux with our pre-built package for CentoOS 6.x. Packages for other popular Linux distributions are on their way.

Note: BetterLinux does not support OpenVZ/Virtuozzo systems, so please uninstall CloudLinux before you install BetterLinux.

Quick Install Steps

  • Download and unpack the BetterLinux tarball, available from logging into your account on BetterLinux.com and going to "My Downloads"
  • Get a license key by logging into your account on BetterLinux.com
  • Move the key file to the tarball directory on your server
  • Run the bl-install script
  • Reboot

Detailed install steps

  • When the BetterLinux version appears in your shopping cart, click "Checkout"
  • Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order":
  • You are now taken to the "Order Complete" page:
  • Here, you receive your order number, which you'll need in order to submit support tickets from the client area. You also receive notice that your email confirmation was sent.
  • Open your BetterLinux welcome email. It includes your email verification link and login information. Please note that your link will expire if not used quickly. The verification takes you to "Downloads" where you will be automatically logged into your account: (if you are not logged in for some reason, log in now)
  • If you are not automatically taken to "Downloads", click on "Continue to Downloads" at the bottom of the "Order Complete" page:
  • If you click on "Continue to Downloads" before verifying your email address, you will be taken to the "Email Verification page": (You will not be able to go beyond this point without verifying your email)
  • In "Downloads", click "License key" to download your license key: (you'll see what to do with it momentarily)
  • Or you can right click on "License key" and select Copy Link. The link can now be placed into a wget command in a terminal window on your server.
    • Since this link has ampersand characters, you will need to include double quotes at the beginning and end of the url.
    • Also, include the -O option at the end of the link with the file name license.key to put the downloaded info into a file called license.key:
wget "[insert the copied link]" -O license.key

Below "License key" are links to the documentation page you are reading now. (They are for those who didn't come here first)

  • At the bottom of "Downloads" is the "Installation Packages" drop-down menu. From the menu, choose your distribution, release #, and architecture. Then click "view files."
  • Your chosen tarball appears. In this case, it could look like this:
betterlinux-install-<version>-centos66-x86_64.tar.gz
  • Download the tarball by clicking "Download" next to it.
  • Move the license Key file (license.key) and tarball to your server (see below for an easy way to do this).

NOTE: Depending on your browser, the license key file may appear as either of these:

license.key
license.key.txt

The name of the license key file will not affect its function provided the name is consistently used and the file's contents are accurate.

Quick steps to get the tarball and license file onto your server (EXAMPLE ONLY):

  • Right click on the tarball link in "Downloads" and copy the URL address.
  • In any directory on your server, paste the copied link after the wget command. It looks like this:
wget "[insert the copied link]"
  • After downloading the tarball from our repository, unpack it. (This creates a long list of files as well as the directory where you will put your license file):
tar -xvf betterlinux-install-<version>-centos66-x86_64.tar.gz
  • Unpacking the tarball creates a new directory called .../betterlinux-install-<version>. Put the license key file (license.key) into this new directory.

NOTE: Instead of moving the file, you can also use an editor (like vi) to make a new file called license.key inside the directory and then copy and paste the license key text into it.

You are now ready to run the install program:

In the .../betterlinux-install-<version> directory, among other files you will see bl-install* and bl-cpanel-install*.

  • Even though you are installing BetterLinux on a cPanel box, you have to run bl-install* first. If cPanel is on your box, the installer will auto-detect it and offer to install bl-cpanel-install* for you automatically after bl-install* runs.
  • From within the .../betterlinux-install-<version> directory, run bl-install* with the license key file right after it, like this: (remember to type your license key file name correctly if it has a different name)
./bl-install license.key

Or run the installer from anywhere using the full file path:

/etc/betterlinux-install-<version>/bl-install /etc/betterlinux-install-<version>/license.key
  • When this portion of the install completes, you will see this notice:

Install1.0cPanel8.png

  • Before rebooting, you are given a chance to run the cPanel portion of the installer. Say "y" to this and hit enter. This will install cPanel configuration scripts and optionally replace MySQL with the BetterLinux version.
  • If you say no, you can run the bl-cpanel-install script after the bl-install script completes. This is done by running the following command:
./bl-cpanel-install
  • As the cPanel configuration script runs, you are now prompted to answer a series of questions. You can see these questions with explanations here: Bl-cpanel-install options and prompts.
  • After responding to all prompts and confirming your responses, you will have successfully configured BetterLinux. Cpud, iothrottled, and CloakFS will be running on the system.

Installation for Stock CentOS 6.x

Easily install BetterLinux with our pre-built package for CentoOS 6.x. Packages for other popular Linux distributions are coming soon.

Note: BetterLinux does not support OpenVZ/Virtuozzo systems, so please uninstall CloudLinux before you install BetterLinux.

Quick Install Steps

  • Download and unpack the BetterLinux tarball, available from logging into your account on BetterLinux.com and going to "My Downloads"
  • Get a license key by logging into your account on BetterLinux.com
  • Move the key file to the tarball directory on your server
  • Run the bl-install script
  • Reboot

Detailed install steps

  • When the BetterLinux version appears in your shopping cart, click "Checkout"
  • Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order":
  • You are now taken to the "Order Complete" page:
  • Here, you receive your order number, which you'll need in order to submit support tickets from the client area. You also receive notice that your email confirmation was sent.
  • Open your BetterLinux welcome email. It includes your email verification link and login information. Please note that your link will expire if not used quickly. The verification takes you to "Downloads" where you will be automatically logged into your account: (if you are not logged in for some reason, log in now)
  • If you are not automatically taken to "Downloads", click on "Continue to Downloads" at the bottom of the "Order Complete" page:
  • If you click on "Continue to Downloads" before verifying your email address, you will be taken to the "Email Verification page": (You will not be able to go beyond this point without verifying your email)
  • In "Downloads", click "License key" to download your license key: (you'll see what to do with it momentarily)
  • Or you can right click on "License key" and select Copy Link. The link can now be placed into a wget command in a terminal window on your server.
    • Since this link has ampersand characters, you will need to include double quotes at the beginning and end of the url.
    • Also, include the -O option at the end of the link with the file name license.key to put the downloaded info into a file called license.key:
wget "[insert the copied link]" -O license.key

Below "License key" are links to the documentation page you are reading now. (They are for those who didn't come here first)

  • At the bottom of "Downloads" is the "Installation Packages" drop-down menu. From the menu, choose your distribution, release #, and architecture. Then click "view files."
  • Your chosen tarball appears. In this case, it could look like this:
betterlinux-install-<version>-centos66-x86_64.tar.gz
  • Download the tarball by clicking "Download" next to it.
  • Move the license Key file (license.key) and tarball to your server (see below for an easy way to do this).

NOTE: Depending on your browser, the license key file may appear as either of these:

license.key
license.key.txt

The name of the license key file will not affect its function provided the name is consistently used and the file's contents are accurate.

Quick steps to get the tarball and license file onto your server (EXAMPLE ONLY):

  • Right click on the tarball link in "Downloads" and copy the URL address.
  • In any directory on your server, paste the copied link after the wget command. It looks like this:
wget "[insert the copied link]"
  • After downloading the tarball from our repository, unpack it. (This creates a long list of files as well as the directory where you will put your license file):
tar -xvf betterlinux-install-<version>-centos66-x86_64.tar.gz
  • Unpacking the tarball creates a new directory called .../betterlinux-install-<version>. Put the license key file (license.key) into this new directory.

NOTE: Instead of moving the file, you can also use an editor (like vi) to make a new file called license.key inside the directory and then copy and paste the license key text into it.

You are now ready to run the install program:

In the .../betterlinux-install-<version> directory, among other files you will see bl-install* and bl-cpanel-install*.

  • Even though you are installing BetterLinux on a cPanel box, you have to run bl-install* first. If cPanel is on your box, the installer will auto-detect it and offer to install bl-cpanel-install* for you automatically after bl-install* runs.
  • From within the .../betterlinux-install-<version> directory, run bl-install* with the license key file right after it, like this: (remember to type your license key file name correctly if it has a different name)
./bl-install license.key

Or run the installer from anywhere using the full file path:

/etc/betterlinux-install-<version>/bl-install /etc/betterlinux-install-<version>/license.key
  • When this portion of the install completes, you will see this notice:

Install1.0cPanel33.png

  • Reboot your system.
  • You have now successfully installed BetterLinux. Cpud, iothrottled, and CloakFS are running on the system.

To use BetterLinux MySQL throttling (BetterMySQL), you must install the BetterLinux version of MySQL:

  • First, make sure the betterlinux-mysql section is enabled in the betterlinux.repo file:
vim /etc/yum.repos.d/betterlinux.repo
  • Then run the install via yum:
yum --disableexcludes=all install betterlinux-mysql

Now enable the mod-betterlinux feature to blame httpd processes on the right users:

  • vim /usr/local/apache/conf/includes/pre_virtualhost_global.conf

add this line:

Include "/etc/betterlinux/mod_betterlinux.conf"
  • Now create the mod_betterlinux.conf file:
vim /etc/betterlinux/mod_betterlinux.conf

add the following content and save the file:

LoadModule betterlinux_module /usr/lib64/httpd/modules/mod_betterlinux.so

<IfModule betterlinux_module>
  Betterlinux On
</IfModule>
  • Create the apache_usermap configuration file:
vim /etc/betterlinux/apache_usermap.conf

add the following instructions for future reference and populate this conf file with the UID and home directory for each user you want to track and blame httpd processes:

# This file should have UID to apache homedir mappings for mod_betterlinux
# It is automatically periodically reloaded
# uid=513 home=/home/user17
# uid=514 home=/home/otheruser


Upgrade BetterLinux 1.1.x-x

Run the following command:

yum --disableexcludes=all upgrade betterlinux\*


Uninstall BetterLinux with cPanel Integration

Uninstall BetterLinux 1.1.x with cPanel

Uninstallation has three stages: 1) Replace BetterLinux's version of MySQL with cPanel's version, 2) Ensure the system boots from the proper non-BetterLinux kernel, and 3) Remove BetterLinux rpms.

Stage #1

Run the uninstall command:

/etc/betterlinux/bin/bl-cpanel-uninstall

You'll see "Uninstalling betterlinux-cpanel-mysql," followed by a series of uninstall actions, a list of what is being uninstalled, notices and reminders, and finally, "BetterMySQL uninstallation complete" and "Uninstalled BetterLinux cpanel modules."

Stage #2

(This stage ensures the system boots from a pre-BetterLinux kernel after uninstallation)

Change the default kernel from the BetterLinux kernel to your most recent non-BetterLinux kernel:

  • Open /etc/grub.conf with an editor.
  • Change the default= value from 0 to 1 (unless you've installed multiple BetterLinux kernels, in which case you may need to enter a number greater than 1. '0' refers to the current kernel, '1' to the previous kernel, '2' the kernel before that, and so on.

Each kernel number line begins with the word title. This is how to tell how many available kernels there are. All BetterLinux kernels include the letters bl somewhere in the number. Choose the number of the first title line with a kernel number not containing bl. Here is an example of a kernel number containing the letters bl outlined in red:

Uninstall17a.png

  • As root user, reboot the system.
Stage #3

(This final stage removes BetterLinux rpms)

  • After rebooting, run ./bl-uninstall from within /etc/betterlinux/bin. The uninstaller now performs a lengthy string of actions beginning with these notices: "Removing BetterLinux rpms" and "Removing any Legacy BetterLinux rpms." When they finish, you will see "Complete!"

You have now completed the uninstall.

Uninstall BetterLinux on Stock CentOS 6.x

  • Edit /etc/grub.conf and select the previously installed kernel. (for instructions, see Stage #2 in the above uninstall section for 1.1.x with cPanel)
  • reboot
  • change directory to …/betterlinux-install and run the uninstall command:
./bl-install --uninstall
  • Remove BetterLinux MySQL (bl-mysql) packages:
rpm -qa "bl-mysql*"

This lists remaining betterlinux rpms. Remove them using the rpm -e command:

rpm -e --nodeps bl-mysql-server-5.1.61-4.el6.bl0.9.5.x86_64
rpm -e --nodeps bl-mysql-libs-5.1.61-4.el6.bl0.9.5.x86_64
rpm -e --nodeps bl-mysql-5.1.61-4.el6.bl0.9.5.x86_64
  • Install stock MySQL:
yum install mysql
yum install mysql-server

BYOK: Build Your Own Kernel

BYOK lets you compile the BetterLinux kernel with customized build options and/or patches, even using the latest kernels from kernel.org. While the standard Linux kernel is sufficient for many, others have more exacting needs, requiring customization.

BYOK is principally for advanced users who know how to compile a kernel, apply patches, and even fix kernel bugs.

Basic steps:

1) Get our patch set (source code of our changes).
2) Apply our patch set to your kernel.
3) Recompile our modules.

Build options

Compiling the kernel yourself gives you choices not possible otherwise. Many under-the-hood features and behaviors can be selected, enabled, disabled, or tuned. Build options enable or disable preexisting kernel code.

Patches

For some, even customized build options are not enough. This is where patches come in. When a feature or behavior is hard-coded and no alternative selection is available, a patch can add or change actual kernel code to create new functionality or a new alternative. Some patches may require certain build options to work.

Support

Because BYOK kernels range from moderate to fully-custom kernels, which have code beyond BetterLinux control, we cannot officially support them.

BetterLinux Manager (for Panels)

cPanel WHM BetterLinux Manager Plug-In

The cPanel WHM BetterLinux Manager plug-in will allow users to set BetterLinux group limits via WHM for CPU, I/O, Bandwidth, Memory, time processes can run, and maximum process a user can run. Additionally, from the General tab, all BetterLinux services can be started and stopped.

The BetterLinux Manager Plug-In is automatically installed during an upgrade or fresh installation. Look for it at the bottom of the WHM menu under Plugins.

General

Bl-manager.general.png

CPU

Bl-manager.cpu.png

Process

Bl-manager.process.png

Memory

Bl-manager.memory.png

Time

Bl-manager.time.png

Bandwidth

Bl-manager.bandwidth.png

Disk I/O

Bl-manager.diskio.png

Panel Integration

cPanel

Overview

cPanel users can customize cpu and I/O settings either manually or automatically during install. You can even automatically create groups based on different cPanel packages so all you have to do is set limits for them.

cPanel resellers can run one setting system-wide, customize settings per cPanel package, or mix and match. We also have a WHM module for easy gui configuration.

The bl-cpanel-install script automates configuration during install, which is included in the tarball on our main repo:

http://repo.betterlinux.com/release/betterlinux-install-1.x.x-x-centos65-x86_64.tar.gz.  

Download the tarball, unpack it, and you’ll see the script in the tarball directory: ../betterlinux-cpanel-install

(For help, see 1.1.x-x Installation on CentOS 6.x with cPanel. Note also that you cannot run bl-cpanel-install until after running bl-install, a script in the same directory.)

From within the tarball directory, run betterlinux-cpanel-install:

./betterlinux-cpanel-install

This installs an auto-updated cPanel hooks script to automatically do the following:

  • Add or remove cPanel users and packages to or from BetterLinux groups.conf files.
  • Discover all cPanel packages, the users assigned to them, and create BetterLinux groups for each.

You can then assign throttling limits by group for cpu, I/O, MySQL, Apache, bandwidth, processes, RSS memory, max time for CGI, cron, and ssh.

Bl-cpanel-install options and prompts

The bl-cpanel-install script prompts for the following information: (Note: you will be able to review your selections at the end before final confirmation)

Should I replace non-BetterLinux MySQL version 5.1.68 with BetterMySQL version 5.1.68? [y/N]

Choosing "yes" removes cPanel's MySQL version and replaces it with the equivalent BetterLinux MySQL version.

Install BetterLinux Apache throttling? [y/N]

Choosing "yes" installs the mod-betterlinux Apache module. It ties Apache threads to the users who generated them, making per-user cpu and I/O throttling possible.

Limit user network bandwidth? [y/N] y
Bandwidth limit in Mb/s:

This sets the BetterBandwidth bandwidth_limit parameter, which limits the amount of outgoing traffic per user.

Limit total number of simultaneous processes a cPanel user may run? [y/N]
Process limit (recommended minimum: 50 processes):

This sets the process_limit parameter. This is a per-user limit.

Log when user exceeds RSS memory threshold? (per-process RAM limits) [y/N]
RSS (resident memory) threshold in MB:

This sets the memory_limit parameter. When the per-user limit is exceeded, a script logs the event (via syslog).

Limit run time of user CGI process? [y/N]
Maximum CGI run time in seconds (recommended min: 300 seconds; recommended max: 1800 seconds):

This sets an element in the time_monitor parameter that kills CGI scripts running longer than the entered value.

Limit run time of user cron processes? [y/N] y
Maximum cron process run time in seconds (recommended min: 1800 seconds):

This sets an element in the time_monitor parameter that kills cron jobs running longer than the entered value.

Limit run time of user ssh processes? [y/N] y
Maximum ssh process run time in seconds (recommended min: 1800 seconds):

Sets an element in the time_monitor parameter that kills ssh-executed processes running longer than the entered value.


Individual limits per package

When the above options are set, the script displays cPanel packages that have these default settings and then prompts the user with the option to define individual limits for each package.

Would you like to customize these parameters for any individual package? [y/N]

Answering yes, will display a prompt to define limits for one of the cPanel packages:

Customize package BetterHost_Level1? [y/N]

Answering yes, will walk the user through the same questions presented during the default package configuration.

After settings are selected for the desired packages, the script will asks to confirm your choices before it proceeds: (see the example confirmation below)

Confirm settings
================
 install-mysql            : 1
 install-mod_betterlinux  : 1

Package Specific Settings:

Package: cPanel-Package_Level1
================================
 bandwidth-limit          : 50mbit
 process-limit            : 70
 log-rss-limit            : 3000m
 cgi-time-limit           : 900
 cron-time-limit          : 900
 ssh-time-limit           : 900

Package: bl_default
================================
 bandwidth-limit          : 20mbit
 process-limit            : 50
 log-rss-limit            : 2000m
 cgi-time-limit           : 400
 cron-time-limit          : 400
 ssh-time-limit           : 400
Confirm and finish

Finally, confirm your settings. Once you proceed with this step, you will not be able to abort until the script is finished.

Proceed with these settings? [y/N]

In addition to your chosen settings, other settings are auto-configured.

BetterLinux is now fully configured. Cpud, iothrottled, and CloakFS are running on the system.

Note: You can always reconfigure the cPanel package settings by running the /etc/betterlinux/bin/bl-cpanel-reconfigure script.

Reconfigure cPanel Package Settings

Location: /etc/betterlinux/bin/bl-cpanel-reconfigure

Each cPanel package can be redefined by running the bl-cpanel-reconfigure script:

/etc/betterlinux/bin/bl-cpanel-reconfigure

You are prompted to reconfigure cPanel package limits:

Do you wish to reconfigure? [Y/N]

If "Y," you are prompted to configure default settings, which apply to any cPanel package without specific settings.

Would you like to re-configure your Default package? [Y/N]

If "Y," you are prompted to choose the same settings that appear while running the ../bl-cpanel-install script.

Settings for individual cPanel packages

After the default settings are defined, you are now prompted to configure settings for individual cPanel packages:

Would you like to customize these parameters for any individual package? [y/N]

Selecting yes prompts you to key in settings for each package. Answering no attaches default settings to the skipped cPanel package and continues on to the next package. When finsihed, BetterLinux hooks create new package-specific .conf files for newly-defined packages. Undefined cPanel packages keep default settings. The new cpanel-[package-name].conf files are located in:

/etc/betterlinux/cpud.conf.d

Their names contain the cPanel package name:

cpanel-[package-name].conf

The default settings are located in cpanel-allusers.conf:

/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf

Any value in a configuration file can be manually edited.

Configure BetterLinux Standard Hooks for cPanel

The bl-hooks script makes a BetterLinux group for each cPanel package and associates each package's UIDs with the new group. Each package is matched with a similarly-named text file, which is populated with that package's uids. The script is found here:

/etc/betterlinux/cpanel/bl-hooks

Text files matched with cPanel packages are found here:

/etc/betterlinux/cpanel/users/[cpanel_package_name].txt

The hooks script updates each [cpanel_package_name].txt file whenever a user is added or removed from a cPanel package. These .txt files are referenced by groups.conf files in /etc/betterlinux/cpud.conf.d and /etc/betterlinx/iothrottled.conf.d.

Hook Management

The bl-hooks script requires cPanel registration to work. This happens automatically when you run bl-cpanel-install or the bl-cpanel-reconfiguration script. But you can also manually register it using the following command (as root):

# /usr/local/cpanel/bin/manage_hooks add script /etc/betterlinux/cpanel/bl-cphooks

NOTE: This will re-create all the .conf files, so if you have settings you don't want to loose, save them first. If you forget, the <code>bl-cphooks script creates a backup copy found here:

/etc/betterlinux/cpanel/backup

To see what hooks are registered, issue this command:

# /usr/local/cpanel/bin/manage_hooks list
bl-cphooks Configuration Initialization

Running bl-cpanel-install automatically starts bl-cphooks, which sets and/or refreshes BetterLinux cPanel .conf files and [cpanel-package-name].txt files. But you can also run bl-cphooks manually with this command (as root):

# /etc/betterlinux/cpanel/bl-cphooks --init

Configuration Files

bl-cpanel-install or /etc/betterlinux/bin/bl-cpanel-reconfigure creates the following .conf files and keeps them synchronized with new or removed cPanel users, resellers, and packages:

cpud:
/etc/betterlinux/cpud.conf.d/cpanel.conf
/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf
/etc/betterlinux/cpud.conf.d/cpanel_[package].conf
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf
/etc/betterlinux/cpud.conf.d/cpanel-resellers.conf
iothrottled:
/etc/betterlinux/iothrottled.conf.d/cpanel.conf
/etc/betterlinux/iothrottled.conf.d/cpanel-groups.conf
cpanel.conf for cpu throttling and CloakFS

Location: /etc/betterlinux/cpud.conf.d

The cpanel.conf file has the settings for cloakfs, load_average, and the throttling for mysql and apache. This .conf file can be edited.

cpanel-allusers.conf for cpu throttling

The cpanel-allusers.conf file has the default limit settings for cPanel packages that were not given specific settings while running the bl-cpanel-install or the bl-cpanel-reconfigure scripts. The cpanel-allusers.conf file uses the group name "cpanel", which cannot be changed. However, any of the values for the parameters can be modified manually. For example, the cpu_throttling values of 50 or 30 for in|out_cpu_percent= can be modified as well as the values for the seconds= argument:

cpu_throttling group=cpanel in_cpu_percent=50 out_cpu_percent=30 seconds=4 type=jail# lhash:da58fb0ba0a820254b830f762e260152
cpanel_[package].conf for cpu throttling
cpanel-groups.conf for cpu throttling

Location: /etc/betterlinux/cpud.conf.d/cpanel-groups.conf

The cpanel-groups.conf file has all the cPanel packages that are auto-configured and defined as betterlinux groups. The bl-cphooks script adds all the packages setup under WHM and any new packages added after BetterLinux is installed. Custom groups can also be defined in this .conf file as long as the group UIDs are not defined in other groups.

cpanel-reseller.conf for CloakFS

Location: /etc/betterlinux/cpud.conf.d/cpanel-resellers.conf

The cpanel-reseller.conf file is created only if there are cPanel resellers. This file's purpose is to enable resellers to see their users' directories as well as resellers' users to see resellers' branding directories. Please do not edit the contents of this file.

cpanel.conf for I/O throttling

Location: /etc/betterlinux/iothrottled.conf.d

The cpanel.conf file for iothrottled contains all the limiting parameters for throttling I/O. Configure this file's default throttling parameters by running bl-cpanel-install or bl-cpanel-reconfigure. You can manually edit any auto-generated value and further customize.

cpanel-groups.conf for I/O throttling

Location: /etc/betterlinux/iothrottled.conf.d/cpanel-groups.conf

The cpanel-groups.conf file has all the cPanel packages that are auto configured and defined as betterlinux groups. The bl-cphooks script adds all packages (present or future) setup under WHM. You can also define custom groups in this .conf file if their associated UIDs are not defined in other groups.

Concepts

CPU

BetterCPU maximizes CPU and memory resources for shared hosting environments while preventing user abuse. Quickly find and control CPU hogs and determine who needs higher-paying accounts with increased resources. To get started, define user groups (match them to cPanel packages if applicable) and set throttling limits for each group.

Configure the CPU daemon through a series of configuration files found here:

/etc/betterlinux/cpud.conf.d/

BetterCPU's primary configuration file is cpanel-allusers.conf (for cPanel) and betterlinux.conf (for a stock install). These files have basic configuration parameters.

Basic Settings

Basic stock configuration begins by setting the sleep_time_ms parameter in the betterlinux.conf file (this .conf file does not exist in cPanel installations):

/etc/betterlinux/cpud.conf.d/betterlinux.conf

This is the default setting: [units in milliseconds]

sleep_time_ms 1000 

The sleep_time parameter determines time elapsed between BetterCPU’s system snapshots. CPU usage is calculated by averaging the data from these snapshots. BetterCPU then determines which users and processes are utilizing the most CPU resources and if those usages exceed permitted thresholds. If thresholds are exceeded, BetterCPU manages users and processes to limit the negative impact to overall system performance, and limit-exceeding users and processes are "jailed."

Advanced Settings

The advanced settings used to configure BetterCPU (in a Stock installation) can be added to /etc/betterlinux/cpud.conf.d/betterlinux.conf and consist of:

"add_user_time" defines the number of seconds between checks for new users to be managed by cpud.

add_user_time 3600	# [Default = 3600]

"run_priority" defines the priority for CPUd. Valid values range from -100 to 39.

run_priority 0	# [Default = 0]

"throttle_group" defines the group id that CPUd should use to set the group ownership on all cgroups it creates. Other programs that use cgroups can use a different gid to own and identify their cgroups, or they can share the gid that CPUd uses to access the cgroups created for and used by CPUd.

throttle_group 0	# [Default = 0]

"in_buf_sz" defines the size (in bytes) of the buffer used for buffering taskstats information.

in_buf_sz 1000	# [Default = 1000]

Starting and stopping BetterCPU

After the installation is complete, a reboot of your system is required to load your BetterLinux kernel and modules. Once your system is rebooted, the BetterCPU daemon will not be running by default and requires a manual start using the service command unless it's a cPanel server and the betterlinux-cpanel-install script has been run. If so, the daemons will be started after the installation. Example:

service cpud start

Restarting and stopping examples:

service cpud stop
service cpud restart

Using the condrestart will restart the daemon, but only if it's already running. Example:

service cpud condrestart:

When changes are made to the configuration files, the BetterCPU daemon must reload the new settings. This is done by either running the service restart option or by running the service reload command. Example:

service cpud reload

Understanding Jail Cores

Before limits are configured, it's important to understand BetterCPU jails. The jails are elastic by nature and only exist if they are needed by the BetterCPU controller to temporarily limit an abusive user process. Think of process jailing as analogous to law enforcement:

  • A user is a “citizen”
  • The performance thresholds are the “law”
  • BetterCPU enforces the laws

A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.

If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.

When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.

Configuration of the jail is done using the parameters:

jail_cnt
max_jail_cnt

The jail_cnt parameter is used for defining a soft or hard limit for the number of jail cores cpud will assign on demand. Using jail_cnt without max_jail_cnt defined, will become the hard limit and the system will not allocate cpu cores above the amount set in jail_cnt. When used in conjunction with max_jail_cnt, jail_cnt becomes a soft limit. When cpud needs to throttled some users processes, it will assign jail cores as needed up to the value set for jail_cnt. If there are free non_jail cores available, cpud will allocate more jail cores as needed up to the amount defined in max_jail_cnt. If there are no users that need to be throttled, then cpud will release all jail cores back to the main pool of jail cores (non-jail cores).

The default setting for cPanel systems leaves jail_cnt and max_jail_cnt undefined which uses the auto-jail-scale feature which sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1. BetterCPU (cpud) then auto-scales the jails the system needs based on demand for jail cores. However, the jail cores will never take a priority over non-jail cores.

The jail_cnt can be defined using two different methods:

  • If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jail_cnt. For example:
jail_cnt 2

The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems. If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.

  • is expressed as one or more pairs of numbers. For example:
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail; an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system. The 32 core system will be assigned 6 cores as the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails, and is written in the same manner as the jail_cnt parameter. Setting the max_jail_cnt parameter will fix the ultimate size of the jail. Fixing the jail size will determines the amount of the cpu utilization abusive users will share in the jail core(s).

example:

max_jail_cnt 4

By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.

Defining BetterCPU Throttling Limits

BetterCPU throttling limits are defined as a percentage of one core for each user. When a user exceeds this limit, he is sent to a "jail" core.

With jail cores defined, you can determine when a user is moved into and out of the jail core(s) by setting thresholds in two parameters. Here is the syntax:

cpu_throttling group={group}[,{group2}...] in_cpu_percent={integer} out_cpu_percent={integer} seconds={integer} type=jail

The above limits apply to groups of uids defined in groups.conf (stock install) or cpanel-groups.conf (cPanel install), found in /etc/betterlinx/cpud.d. The in|out_cpu_percent values set the percentage a user (uid) can use of one core on the system. The values for in|out_cpu_percent are based on one core and are not adjusted when cores are added or removed from the jail.

When a throttled user’s total utilization drops below the out_cpu_percent and seconds values, then the process is removed from the jail.

For example, consider an 8-core system with the following values configured:

cpu_throttling group=example-group in_cpu_percent=50 out_cpu_percent=40 seconds=3 type=jail

A user whose processes exceeds 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.

Throttling limits can also be applied to program groups in the same manner as groups of users.

Defining BetterCPU guaranteed CPU Limits

In addition to the jailing mechanism documented above, (type=jail), BetterCPU provides also an alternative CPU-throttling mechanism, using the option type=guaranteed.

This mechanism is based on the mainline "cpu" cgroup controller and it allows to set a proportional fair share of CPU amongst different users. It is possible to use this option also to provide QoS amongst different groups of users.

For example:

cpu_throttling group=group1 in_cpu_percent=100 type=guaranteed burst=Y
cpu_throttling group=group2 in_cpu_percent=50 type=guaranteed burst=Y

In this case group1's users will get 2x CPU bandwidth in respect to group2's users. Users in each group will also get a fair amount of CPU, independent of the amount of tasks they're running. The option in_cpu_percent can be seen as a weight to define the proportional bandwith amongst different groups. If burst is enabled, in_cpu_percent is actually a weight to determine the proportional CPU bandwidth amongst different cgroups. Otherwise (with burst=N) users will be limited to their absolute cpu_percent value.

Re-using the same example above:

cpu_throttling group=group1 in_cpu_percent=100 type=guaranteed burst=N
cpu_throttling group=group2 in_cpu_percent=50 type=guaranteed burst=N

In this case users in group1 will be able to use a maximum of 1 core at 100% over a period of 1 sec, users in group2 will be able to use a maximum of 1 core at 50% over a period of 1 sec.

NOTE: 2+ cores are required to use type=guaranteed, despite jailing (type=jail) that requires 4+ cores.

Defining Groups

Groups are automatically defined by the bl-cpanel-install script. Installations on non-cPanel systems need to be defined manually. For more information, see the main Define Groups section.

Selecting Managed Users

There are a few ways users can be selected for control by BetterLinux.

Certain parameters can be used to define a minimum and maximum UID value that define a range of identities to be managed by BetterCPU. For cPanel installation, these are found in cpanel-allusers.conf, and for stock installation, in betterlinux.conf.

There also are parameters that define UIDs or users for inclusion or exclusion from being managed by the suite.

The simplest way to define managed users is with groups.conf (stock) and cpanel-groups.conf (cPanel):

/etc/betterlinux/cpud.conf.d/groups.conf
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf

These files contain entries that define a group name and a list of users within that group. For example:

uid cpu_limit testuser,testuser1,testuser2,testuser3,502

This line adds the users in the list to the group cpu_limit. A numeric value in this line is interpreted as a UID value which is translated by the system to a user name.

How can I tell BetterCPU is managing my system?

A few tools have been mentioned already - top and htop will both show CPU utilization figures, memory utilization figures, and a breakdown of utilization by CPU core. BetterLinux comes with it's own top-like program called blstat.


If logging is enabled, an easy way to see that process are in being throttled in the jail core(s) is to monitor the file defined in log_cpud for jail expansion messages. When a user is jailed, the log will reflect that:

Cpud managing system.png

The “Throttling User” message indicates that a user’s processes are being moved into the jail. One can also see messages indicating the jail cgroup is being created as well as information about the number of running programs that user has, their UID, and other information about that user.


In htop, when a user is jailed, if the display of the CPU column (which displays the CPU number a process is running on) is enabled, it is easy to observe a user being moved into the jail. Here is a user using excessive CPU resources prior to throttling:

Cpu resources prior to jailing.png


Here is that same user after BetterCPU has jailed them for excessive CPU use:

Cpud jailed them.png

Notice how the first image shows the utilization on all 4 CPUs of the system and how separate instances of the “jailme” process is running on each CPU. After the user is jailed, all of that user’s processes have been moved to CPU 4.

User Blame Security

This setting enables BetterMySQL throttling and authorizes a program to apply usage blame to other users on the system. This is used for programs like mysql and apache, which run under one uid, to divide the usage up by the individual users of the program.

In the example below we will authorize apache and mysql to assign blame to the correct user on the system. Add one line per program / user:

approved_accounting_program program=/usr/sbin/httpd user=apache
approved_accounting_program program=/usr/libexec/mysqld user=mysql

This allows the program /usr/sbin/httpd as run by the apache user and /usr/libexec/mysqld as run by the mysqld user to determine if one of the web server or mysqld users is exceeding a configured resource constraint enforced by the BetterCPU (cpud) start and stop throttling parameters. Instances of /usr/sbin/httpd run by other users are not affected unless a similar line is included in the configuration for that user as well.

BetterCPU Logging

BetterCPU provides extensive logging functionality which allows a system administrator to see how BetterCPU is helping manage their systems. Logging configuration parameters fall into two general categories: Parameters to enable or disable specific logging functionality, and parameters to define the files messages from various components of BetterCPU are logged to.


Overall logging of BetterCPU information is controlled by the log_cpud parameter. If this parameter is set to a value of 1, then BetterCPU events are logged based on the settings of other log_* parameters. If this parameter is set to a value of 0, then no logging is done by BetterCPU.


General BetterCPU messages are logged to the file pointed to by the log_cpud_file parameter, but specific components of BetterCPU write to their own log files.


The parameter documentation provides in-depth explanations of all of the logging functionality in BetterCPU. -->

I/O

The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.

In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.

From a CPU perspective, this is referred to as “Waiting on I/O”, or iowait%. If the I/O channel is not also managed, then the system will not be able to run optimally.

When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.

There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To monitor these factors, one can use the iostat command or blstat (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.

What can BetterIO tune to affect block device utilization?

At a high level, tuning is applied to I/O operations per second (iops) and bandwidth (bw). These two parameters can be tuned by applying minimum and maximum values in /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf.

Those system-wide minimum and maximum values can be refined by applying user limits and program limits, which are defined in /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf.

The limits defined can be scaled to allow the block device channel to be utilized more effectively when it is under-utilized. The auto_scale parameter defines how scaling is applied to keep both the bandwidth utilization and iops within a desired performance range.

Basic Settings

The basic settings used to configure BetterIO can be added to /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf and consist of:

sleep_time_ms
sample_periods
enable_program_limits

IO usage is calculated using averages of samples taken over time. BetterIO profiles usage by process and by user on a regular interval to determine which processes and users are utilizing the most IO, and if those usages exceed permitted thresholds, then BetterIO manages those processes in a way that limits their ability to impact overall system performance.

"sample_periods" determines the number of samples used in calculating the average I/O channel utilization. This parameter is used in conjunction with the sleep_time_ms parameter to determine the utilization average interval. If sleep_time_ms is set to 500 and sample_periods is set to 2, then the utilization is sampled every 500 ms and that value is averaged with the sample from the previous period (i.e. the last two samples are averaged together).

Increasing the number of periods will result in a smoother average over a longer period of time, but will allow a process to run with a higher utilization of the I/O channel longer before it is detected and throttled.

"enable_program_limits" enables limiting individual programs using definitions in the bl_io.conf file.

The averages for user and IO utilization (sleep_time_ms and sample_periods, respectively) are calculated based on the number of periods (or samples) taken. The defaults for these parameters are:

sleep_time_ms 1000
sample_periods 6
enable_program_limits 1

This means snapshots are averaged every 6 periods, and each period is 1000 ms long.

Advanced Settings

"cgroup_dir" identifies the path where the cgroup block I/O controller is mounted.

cgroup_dir /cgroups_blockio	# [Default = /cgroups_blockio]

"iothrottled_gid" the gid that iothrottled should use to set the group ownership on all cgroups it creates. Other programs that use cgroups can use a different gid to own and identify their cgroups, or they can share the gid that iothrottled uses to access the cgroups created for and used by iothrottled.

iothrottled_gid 0	# [Default = 0]

"max_cgroups" defines the maximum number of cgroups allowed. A value of '0' removes the limit. When the number of cgroups exceeds max_cgroups then cgroups are deleted until the total is below max_cgroups. The least recently used cgoups are deleted first.

max_cgroups 4096 # [Default = 4096]

"remove_stale_groups_time" defines how often to call the garbage collector to remove stale groups, expressed in sleep_time_ms periods (see also /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf). A value of '0' disables it.

remove_stale_groups_time 0 # [Default = 0]

"interruptible_sleeps" determines how a process that is being throttled is put to sleep while having its I/O throttled by iothrottled.

In a normal system without iothrottled, when the I/O channel is congested, the system will put a process to sleep because it's waiting on I/O. When this occurs, the system loadavg value is increased because the system is under a load that's limited not by CPU utilization but rather by I/O performance.

When artificially restricting I/O using iothrottled, this can cause programs that trigger off of loadavg values to trigger incorrectly. This option introduces the ability to pause process execution due to iothrottled's artificial I/O channel throttling as a part of the CPU scheduler rather than due to I/O. This is done using an interruptible sleep, which means that while the process is being suspended due to I/O, the suspension is not due to an actual I/O bottleneck but due to an artificial bottleneck caused by

interruptible_sleeps 1 # [Default = 1]

"auto_wakeup" Under certain conditions I/O throttling may contribute to increasing the memory pressure of the system, especially with stacked block devices such as loopback mounted filesystem, LVM, software RAID, iSCSI, etc.

The option auto_wakup reduces the risk of page allocation failures by temporarily awakening all the tasks previously put to sleep due to the enforced I/O limits. This allows the kernel to drain some pending I/O requests and give more breathing room to the system in terms of memory utilization.

I/O limits are then enforced as normal when the memory utilization in the system returns back to a safe level.

auto_wakeup supports three different configurations:

  • auto_wakeup = 0: disable auto-wakeup
  • auto_wakeup = 1: wake up all tasks when kernel is running out of emergency memory (reduce the probability of GFP_ATOMIC allocation failures) -- This is the recommended and default setting.
  • auto_wakeup = 2: wake up all tasks when kernel is running out of emergency memory or when journal I/O is submitted (meta-data I/O)

Option 2 should be used only if the kernel reports some hung task timeout traces. They can happen when there is too much pressure on filesystem locks like intensive meta-data operations.

auto_wakeup 1 # [Default 1]

Setting I/O Bandwidth Limits Per-User or by User-Group

Individual user or group limits can be set using the configuration file /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf.

This file uses a single line per entry:

[user|group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or mount points to include or exclude.

For example, the file might contain a line that reads:

testuser bw max sda,sdb,sdc 10000000

This would mean that the user testuser has their maximum bandwidth (bw) limited on devices sda, sdb, and sdc to 10000000 B/s (Bytes per Second).

The first field can be a user or group name. Groups are defined in the file /etc/betterlinux/bl_groups_io.conf. If a group and a user have the same name, the limit applies to the user.

Setting I/O Bandwidth Limits by Program-Group

More granular control over individual programs’ usage of block device I/O rates and data rates can be achieved using the /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf file.

The line format for this file is:

[program_group_name] [S|G] [bw|ops] [device/mount point list] [limit]

The fields that match up with fields in bl_io.conf are defined the same way as in bl_io.conf.

The prognam_group_name field contains the names of the program or programs which is defined in bl_groups_io.conf.

If the third field contains an “S”, the limit applies to each instance of the program started by the use or group. A “G” means the limit applies to all instances together.

For example: NOTE: In all the examples below we assume to have the group "tar_progs" defined in /etc/betterlinux/iothrottled.conf.d/groups.conf:

prog tar_progs /bin/tar,/usr/bin/tar

If our goal is to limit each individual instance of "tar" to a 5 MB/s I/O rate we would use the following line.

tar_progs S bw max inc="^/" 5000000

NOTE: all the "/bin/tar" and "/usr/bin/tar" instances will be affected by this rule if the UID that runs them is not included in any UID group. Remember that UID groups always win against PROG groups.

If our goal was to group ALL instances of tar to an aggregate bandwidth of 5 MB/s then you would change the "S" (Single process) in the example above to "G" (For grouping all tar processes together). This option is helpful when you want to limit one or more instances together. So, if you had one tar running or 50 tars the total disk I/O bandwidth would never exceed 5 MB/s.

The syntax for this scenario would be as follows:

tar_progs G bw max inc="^/" 5000000

If our goal was to limit each instance of tar to a disk I/O rate of 5 MB/s, but would only like to enforce this limit on any filesystem mounted under /home (and any sub-directories) but have no limits enforced on other drives then the syntax would be as follows.

tar_progs G bw max inc="^/home" 5000000

You can include multiple block devices (hard drives) or mount points to be included in the limits that you have defined as shown below:

tar_progs G bw max inc="/home",sdb,dm-1,inc="/var/matt",inc="/etc" 5000000

In addition, you can specific drives to be excluded from the limits that you have defined. This syntax is shown below.

tar_progs G bw max inc="/",exc="^/media|^/mnt" 5000000

For example:

tar_progs S bw sda 1000000

This applies a bandwidth limit of 1000000 bytes/s to each instance of the /bin/tar or /usr/bin/tar commands run by any user not included in a UID group. If one of these users starts two concurrent instances of the tar command, each one is limited to 1000000 b/s of block device I/O bandwidth. If this line reads:

tar_progs G bw sda 1000000

The bandwidth limit is applied to all instances of tar that the users launch. In this case, if the users launch two concurrent instances of the tar command, the total bandwidth both instances can use together is 1000000 B/s.

Below are some configuration rule examples of how to limit I/O bandwidth for certain programs.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied. The inc= and exc= directives take an extended regular expression for mount points. For example, the line:

tar_progs G bw max sda,inc="/home|/backup",exc="/mnt1|/mnt2" 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that have "/home" or "/backup" in their mount point (that would also include, for example, "/home2", "/media/home3", excluding all the mount points that have "/mnt1" or "/mnt2" (like "/mnt21", "/mnt/mnt1", ...). The values are regular expressions and they can appear anywhere within the mount point). Including ^ and $ in the regex will force a match at the begining or end.

The evaluation of these expressions always follows the order of:

  • Explicit device names
  • Devices mounted in the "inc" list
  • Devices mounted in the "exc" list

Exclusions will ultimately override the other values in the list. Consider the following list of mounted devices:

/dev/sda mounted at /
/dev/sdb mounted at /home
/dev/sdc mounted at /media/backups
/dev/sdd mounted at /usr
/dev/sde mounted at /mnt/home
/dev/sdf mounted at /srv/www
/dev/sdg mounted at /srv/ftp
/dev/sdh mounted at /srv/home

With the following restrictions in bl_io.conf:

tar_progs G bw max sda,sdd,inc="/home",exc="/srv" 1000000

Programs in group "tar_progs" will be limited to 1 MB/s on /home and /mnt/home, but not on /srv/home, due to the exc="/srv" rule.

How do the block device inclusion/exclusion lists work?

Simple configuration of BetterIO uses device names in a list in the bl_io.conf and bl_io.conf files. For example:

testuser bw max sda,sdb,sdc 1000000

This applies the limit of 1000000 B/s for the user testuser on devices /dev/sda, /dev/sdb, and /dev/sdc.

Modern Linux systems, however, do not always guarantee the assignment of devices, which can lead to issues with limits not being applied to the correct device.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied.

The inc= and exc=' directives take an extended regular expression for mount points. For example, the line:

testuser bw max sda,inc=”/home|/backup”,exc=”/mnt1|/mnt2” 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that are mounted at “/home” and “/backup” (and would also include, for example, “/home2”, “/media/home3” - the values are regular expressions and can appear anywhere within the mount point), but would exclude any mount point that includes “/mnt1” or “/mnt2” (including “/mnt21”, “/mnt/mnt1” - as with the “inc=” directive). Including ^ and $ in the regex will force a match at the begining or end.

Another example for this using regular expression that will apply the limit of 1MiB/s to any mounted block device except those mounted under /var or /tmp

testuser bw max inc=”^/”,exc=”^/var|^/tmp” 1000000

In this example we want to limit all the mounted block devices, except the temporary mounted devices, usually mounted under /mnt or /media.

  • add the line below to /etc/betterlinux/bl_groups_io.conf:
testgroup uid 500-1000
  • Then add the line below to /etc/betterlinux/iothrottled/bl_io.conf:
testgroup bw max inc=”^/”,exc=”^/mnt|^/media” 1000000

The evaluation of these expressions always follows the order of:

  1. Explicit device names
  2. Devices mounted in the “inc” list
  3. Devices mounted in the “exc” list

Exclusions will ultimately override the other values in the list.

Consider the following list of mounted devices:

/dev/sda1 mounted at /
/dev/sda2 mounted at /home
/dev/sdb1 mounted at /media/backups
/dev/sdb2 mounted at /usr
/dev/sdc1 mounted at /mnt/home
/dev/sdc2 mounted at /srv/www
/dev/sdd1 mounted at /srv/ftp

With the following restrictions in bl_io.conf:

testuser bw max sda,sdd,inc=”/home”,exc=”/srv” 1000000

The result would be:

sda would be included
sdb would be excluded
sdc would be excluded
sdd would be excluded

So the only device controlled by BetterIO in this scenario would be /dev/sda. The controls are applied per block device, not on a per-partition basis - so if a partition is excluded from control, all partitions on that block device will be excluded.

When configuring your system, it is important to take this into consideration - you do not want your swap partition to be throttled by BetterIO, but if you put swap on /dev/sda and control /dev/sda with BetterIO, you may end up impacting performance on the system in ways you may not intend to.

Examples include/exclude:

/dev/sda1 on /
/dev/sda2 on /home
/dev/sdb1 on /home2
/dev/sdc1 on /home3
/dev/sdd1 on /home4
testuser bw max inc="/home",exc="/home2"

The rexeg's are matched against the left of the mount point. So the include matches /home1, /home2, /home3, /home4 and the exclude matches /home2 only. So the result is:

/dev/sda included
/dev/sdb excluded
/dev/sdc included
/dev/sdd included

Example Exclude:

/dev/sda1 on /
/dev/sda2 on /home
/dev/sdb1 on /src
/dev/sdb2 on /usr
/dev/sdc1 on /extra
testuser bw max exc="/src"
/dev/sda included
/dev/sdb excluded
/dev/sdc included

How does auto scaling work?

The auto_scale parameter is an on/off toggle switch for enabling or disabling the auto scale feature. The auto scaling algorithm is applied to overall bandwidth utilization or iops giving the system administrator the ability provide more bandwidth to a defined set of users when it's available. The auto_scale throttling is applied when all the users in the defined groups are going above the target disk utilization percentage defined in the seek_balancing_method option.

When setting the value for the auto_scale parameter, an integer is entered that represents the on/off values:

0 = auto scaling off
1 = auto scaling on

Auto scaling is controlled by these parameters:

scale_min - defines the minimum percentage the scaling algorithm may scale down in order to reach the target disk utilization.
scale_max - defines the maximum percentage the auto_scaling algorithm may scale up to in order to reach the targeting disk utilization.
seek_balancing_method - sets the value for the target disk utilization used by the auto_scale algorithm. This is achieved by dynamically adjusting the defined I/O limits. Setting this value close to 100 will use more of the total disk bandwidth, but will leave less room for new I/O users. Setting this parameter value under 100 will provide more breathing room for new I/O users or the non-throttled users. For example, if the target utilization is 80%, the device will has a free disk bandwidth of 20% for non-throttled users.

To understand how these parameters are related to each other, consider the following scenario:

With the following line in /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf:

testuser bw max sda 1000000

And in /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf:

auto_scale 1
scale_min 200
scale_max 50000
seek_balance_method 80

Assume in this scenario the current disk utilization is 0.001%. The seek_balance_method percentage is divided by current disk utilization for a total of 80,000 which is higher than the scale_max of 50000. The global multiplier is now set to 50,000 and all the I/O limits for the users in the defined group are multiplied by the 50,0000% and there for given 500MB/s of disk I/O.

How can one test to see if BetterIO is controlling devices?

A simple and quite dramatic way to see BetterIO limiting I/O is to set a lower threshold for a user that is very small. For example, create a user called testuser and log in as that user. Run the following command:

sync; dd if=/dev/zero of=temp.fil bs=1M count=10 oflag=direct; rm temp.fil

The results from that command for an unmanaged user would be similar to this screenshot:

Iothrottled blocking devices.png

If we then modify the /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf file to include the line:

testuser bw max inc=”/” 500000

and rerun the dd command above, the following output results:

Iothrottled dd results.png

The drop in throughput from 350-400 MB/s to 500-600 kB/s shows the effect of the change in bandwidth limits for this user.

Are there any other considerations to keep in mind?

With BetterIO, the limits are set on a per-block device basis. It is important to keep in mind that I/O channels are not always assigned in a 1:1 ratio with block devices. If a series of storage devices are connected to the system by a Fibre Channel connection, the channel bandwidth available for all of those devices needs to be taken into consideration across all of the devices.


If devices /dev/sda, /dev/sdb, and /dev/sdc are all on the same disk controller, and the maximum throughput of that controller is 1 Gbps, then all three disks cannot run at 1 Gbps even if the drive electronics would permit that.

If you then have 10 users limited to 100 Mbps each, then performance will effectively be running without any throttling because the disk channel will be oversubscribed if all of the users are attempting to fully utilize their allocated bandwidth to each of these three devices. At the same time, these users are still limited to 100 Mbps each and are unable to monopolize the resources.

MySQL

BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.

It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated. Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.

Configuring BetterMySQL for cPanel Support

BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.

Installing the BetterLinux version of MySQL for cPanel

BetterMySQL cannot be installed unless MySQL is already on the system. BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL. If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script:

/etc/betterlinux/cpanel/bl-cpmysql-install

Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared information in the Concepts and Limits sections for the other products. BetterMySQL’s configuration file serves only to enable BetterMySQL.

Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set. For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit:

cpu_throttling group=cpanel_users in_cpu_percent=50 out_cpu_percent=40 seconds=3 type=jail

Same logic apply's to any BetterIO limits that have been set. In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled.

cpanel_users bw max inc="^/" 5000000
Configuring BetterMySQL for cPanel support

Per-user query throttling of MySQL users requires mapping MySQL users to Linux users. This mapping and other configuration options are auto-configured by either the bl-cpanel-install script on first time installation, or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the bl-hooks script can be used after a manual installation of BetterMySQL. Here is the example syntax for running the bl-hooks script:

/etc/betterlinux/cpanel/bl-cphooks --init

Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.

  • usermap.conf contains the mapping of cPanel users to UIDs
  • mysql.conf contains the parameter location of the usermap.conf file
  • turns on BetterMySQL throttling by adding the approved_accounting parameter to cpanel-allusers.conf (called betterlinux.conf in a non-cPanel install). cpanel-allusers.conf is found in /etc/betterlinux/cpud.conf.d.
approved_accounting_program program=/usr/libexec/mysqld user=mysql
  • Restarts the cpud daemon to enable BetterMySQL throttling.
service cpud restart

Configuring BetterMySQL for Linux Servers

BetterMySQL configuration is a manual setup for Linux servers

Installing the BetterLinux version of MySQL non-cPanel

After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command:

yum install mysql-server
Configuring BetterMySQL for non-Cpanel Linux servers

To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.

The command format is as follows:

bl_usermap_filename [usermap_file_path]

Example:

bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

Now create the MySQL mapping to Linux user mappings located in the following file:

/etc/betterlinux/mysql.conf.d/usermap.conf

The usermap file is a tab-delimited file that contains the following information:

[mysqluser]	[linux_user]	[uid]

The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.

For example, the following file will create mappings for testuser1 and testuser2. For testuser1, the uid will be determined when that user makes a query against the database.

testuser1	testuser1
testuser2	testuser2	501

The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;":

mysql> show user_map;
+-----------+-----------+------+
| From_user | Unix_user | Uid  |
+-----------+-----------+------+
| testuser2 | testuser2 |  501 |
| testuser1 | testuser1 |  500 |
+-----------+-----------+------+
2 rows in set (0.00 sec)

The approved_accounting parameter turns on the BetterMySQL throttling and is located in betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install), found in this directory:

/etc/betterlinux/cpud.conf.d

The syntax is as follows for non-cPanel servers:

approved_accounting_program program=/usr/sbin/mysqld user=mysql

Restart the cpud daemon to enable the configuration settings:

service cpud restart

Monitoring MySQL Utilization with myusertop

BetterMySQL includes the myusertop utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.

Once myusertop is running, press '?' to view the built-in help.

The output is similar to the output of the standard 'top' utility:

mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
Time: CPU 448          Busy 529         CPU/Busy 0.8477   RowsExamined:   1,476m
Qrys: Select 28,456    Update 0         Other 5,250         Adj-RowsEx:   1,251m
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
      h=Help m=MySQLUsersOn/Off                                           q=quit
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx   %CPU
root                   0     2 0.0075   0.0074   1.0060     150      151  100.00
  root                 0     2 0.0075   0.0074   1.0060     150      151  100.00
testuser2            501     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser2          501     0 0.0000   0.0000   0.0000       0        0    0.00
testuser1            500     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser1          500     0 0.0000   0.0000   0.0000       0        0    0.00

In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization. If it were divided equally between the three users, myusertop would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.

If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.

Bandwidth

BetterBandwidth uses 'tc' (a built in Linux tool) to provide traffic shaping for users that have their own network interface. The tc tool on its own does not provide a mechanism for limiting bandwidth usage by uid/user on a shared network interface. BetterLinux has added this functionality into its BetterBandwidth feature. Now admins can limit bandwidth for a single user on a shared server that has a corporate internet connection.

A common problem for web hosts is that one single user with a 1Gbit connection could kill the entire TCP stack. Often the host is unaware that it happens because they use 5 min bandwidth averages to monitor bandwidth on their 1 gig or 10 gig port. But all it takes is a 30 second spike in bandwidth out of 5 mins to make customers angry. And because of the 5 minute average it looks like the peak was around 50Mbits when it easily could have been 500Mbits. Customers will blame it on the server or the hosting company.

Many hosting companies give out IPs just to be able to limit bandwidth. This is an expensive approach. The BetterLinux BetterBandwidth feature can do it without giving out dedicated IPs.

BetterLinux has an apache mod called mod_betterlinux that allows BetterBandwidth to limit bandwidth for all ports for an individual user or group of users like email, FTP, etc is all included. This is a tremendous advantage for a host. BetterBandwidth monitors and controls individual, group, and system-wide bandwidth usage in multi-tenant Linux environments. Administrators can now limit outbound bandwidth as if each user had a dedicated IP address. This lets admins consolidate multiple users onto a single IP, which innovatively addresses the fast-increasing IPV4 IP address shortage.

CloakFS

CloakFS now cloaks via the kernel instead of a pam module. This simpler solution makes it easy to support all user space applications and versions (apache, suexec, suphp, ssh, pam, etc.) without risking constant changes. CloakFS is also now easier to configure.

CloakFS secures web hosting systems by preventing users from seeing each other’s directories, files, and processes. It protects users from each other even if file/directory permissions are insecure. For example, cPanel user A couldn’t read/write user B’s file even with chmod 777 permission. CloakFS also auto-detects all users’ home directories so no extra configuration is required.

CloakFS prevents directory-listing scripts from running via ssh, cron jobs, Apache (suexec, suphp) and symlinks. Further, a user sees only his or her own $HOME directory and other files he or she owns. For example, if User A creates “samplefile” in /tmp and then lists all /tmp files, he will see only “samplefile,” but not other users’ /tmp files.


Limits (Configuration)

Manual Configuration Quick Start Guide

As a quick start, not all features are represented. This section shows beginners how to quickly achieve a simplified, working configuration. For a more thorough manual configuration, see BASE CONFIGURATION FILES.

You are about to create two simplified .conf files, one to control CPU-related features and another to control I/O.

  • Using any editor, create your own .conf file in /etc/betterlinux/cpud.conf.d. BetterLinux looks for instructions in any file ending in .conf found in this directory. You could call your file mycpu.conf.
vim /etc/betterlinux/cpud.conf.d/mycpu.conf
  • From the Configuration: CPU / security section, copy and paste all gray box text below into your new .conf file.
  • Now edit the line below. This says who your configuration applies to (called a BetterLinux group). Right now, the group is called "mygroup" and it includes all UIDs from 500 to 600.
uid mygroup 500-600
  • You can edit other lines' values too, but the included values are a good place to start.
  • When ready, save your new .conf file.
  • By default, /etc/betterlinux/cpud.conf.d already has .conf files. Because parameter duplication causes errors, you'll want to comment out redundant parameter instances (in files you aren't using), or change the .conf file extension on files you aren't using.

Configuration: CPU / security

Example group definition (all UIDs in range [500 ... 600])

uid mygroup 500-600

CPU limits:

Start throttling if a non-throttled UID in mygroup is using more than 50% of a core for 5 seconds. Stop throttling if a throttled UID in mygroup is using less 30% of a core for 5 seconds.

cpu_throttling group=mygroup in_cpu_percent=50 out_cpu_percent=30 seconds=5 type=jail

Memory limits:

Limit users in mygroup to 500m RSS memory usage and kill offending processes every 30 seconds. (Note: limits require 'm' or 'g' for 'megs' or 'gigs.')

memory_limit group_type=user group=mygroup rss_memory=500m program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30

Bandwidth limits:

This sets up Linux tc rules for each user in mygroup and throttles outbound bandwidth for each user to 25mbits.

bandwidth_limit group=mygroup interface_group=interface_types limit=25mbit

Security:

Hide all mygroup user files from other mygroup users (a user cannot be hidden from himself). System processes are also hidden.

cloak hide_user_group=mygroup hide_user_group_from=mygroup hide_system_processes=Y

Prevent mygroup users from running setuid executable files

block_suid group=mygroup

Configuration: I/O

Now you'll create your simplified I/O .conf file.

  • Using any editor, create your own .conf file in /etc/betterlinux/iothrottled.conf.d. BetterLinux looks for instructions in any file ending in .conf found in this directory. You could call your file myio.conf.
vim /etc/betterlinux/cpud.conf.d/myio.conf
  • From the Configuration: I/O section, copy and paste all gray box text into your new .conf file.
  • Now edit the line below. This says who your configuration applies to (called a BetterLinux group). Right now, the group is called "mygroup" and it includes all UIDs from 500 to 600.
uid mygroup 500-600
  • You can edit other lines' values too, but the included values are a good place to start.
  • When ready, save your new .conf file.
  • By default, /etc/betterlinux/iothrottled.conf.d already has .conf files. Because parameter duplication causes errors, you'll want to comment out redundant parameter instances (in files you aren't using), or change the .conf file extension (on files you aren't using).

Example group definition (all UIDs in range [500 ... 600])

uid mygroup 500-600

Apply a fixed I/O limit of 10MB/s to any user in mygroup on any mounted block device [NOTE: set auto_scale to 1 to use dynamic limit (limits are automatically adjusted in function of the available disks' bandwidth) ]

auto_scale 0
mygroup bw max inc="^/" 10000000

Start Services

  • Now that your new .conf files are ready, restart the following services to apply the new settings:
service cpud restart
service cloakfs restart
service iothrottled restart
  • Congratulations! You are now hiding the users in your BetterLinux group from each other, preventing them from running setuid executables, and are limiting their CPU and I/O use according to your settings.

Define Groups

Understanding and defining groups is a key concept that is combined with the BetterLinux throttling parameter settings. BetterLinux groups allows a system administrator to organize different types of users and programs by group. For example, a hosting company may have various customer subscription levels with each level requiring resources from the server. Having the ability to divide customers by group, allows BetterLinux limits to be set by customer group or cPanel packages.

The BetterLinux cPanel integration (bl-cpanel-install) installs custom cPanel script hooks that automate the creation and update of BetterLinux groups based on cPanel packages. Each time a new user is added/removed to/from a cPanel package the corresponding BetterLinux group will be automatically updated. The BetterLinux cPanel hook script will discover all the cPanel users on the system, create a user groups with the name of the cPanel package(s) and define these groups for both the BetterIO and BetterCPU groups.conf or cpanel-groups.conf and groups.conf or cpanel-groups.conf files.

Below is an example of the syntax the hook script would write to the configuration files groups.conf or cpanel-groups.conf for cpud and groups.conf or cpanel-groups.conf for iothrottled if there were three cPanel packages named SampleHost-Level1, 2, and 3:

 uid  SampleHost-Level1 /etc/betterlinux/cpanel/users/ SampleHost-Level1.txt
 uid  SampleHost-Level2 /etc/betterlinux/cpanel/users/ SampleHost-Level2.txt
 uid  SampleHost-Level3 /etc/betterlinux/cpanel/users/ SampleHost-Level3.txt

The BetterLinux cPanel hook script will also create uid files that match the cPanel package names (if they exist) and will contain the cPanel package corresponding UIDs. These uid file names (located in /etc/betterlinux/cpanel/users) contain the UIDs and have names that match the cPanel package names.

Example of uid files created if there were three cPanel packages named SampleHost-Level1, 2, and, 3:

/etc/betterlinux/cpanel/users/SampleHost-Level1.txt
/etc/betterlinux/cpanel/users/SampleHost-Level2.txt
/etc/betterlinux/cpanel/users/SampleHost-Level3.txt

If for any reason the cPanel packages are not in the groups .conf files, run this command to refresh them:

  • NOTE: This command will remove any .conf file customizations.
/etc/betterlinux/cpanel/bl-cphooks --init

The bl-install script script will not create users groups, they need to be created manually by the system administrator.

Groups .conf FILE LOCATIONS:

  • /etc/betterlinux/cpud.conf.d/groups.conf or cpanel-groups.conf
  • /etc/betterlinux/iothrottled.conf.d/groups.conf or cpanel-groups.conf

Group Types and Syntax

Groups can be defined by UID (Linux user), Program, Interface, and Port.

Syntax: uid NAME UID_ITEM[,UID_ITEM,...] prog NAME PROG_ITEM[,PROG_ITEM,...] port NAME PORT_ITEM[,PORT_ITEM,...] interface NAME INTERFACE_ITEM[,INTERFACE_ITEM,...]

Examples:

uid bl_users 500-1000
prog bl_progs /usr/bin/rsync,/bin/tar
interface bl_interfaces eth0
port bl_ports 1024-65545

Where:

  • uid: used to define a group of users
  • prog: used to define a group of programs
  • port: used to define a group of ports
  • interface: used to define an interface like eth0

NAME: is a custom name that identifies the group

UID_ITEM: can be one of the following:

  • USERNAME: name of a UNIX user
  • UID: a UNIX user uid
  • RANGE: an ID ranges and/or names (i.e., 1000-1999)
  • FILE: external file that contains a comma-separated list of USERNAME,
  • UID, and RANGE elements
  • COMMAND: external command to execute, the output can contain a comma/space/tab/newline separated list of USERNAME, UID, and RANGE elements

PROG_ITEM: can be one of the following:

  • PATH: absolute path of an executable file
  • COMMAND: external command to execute, the output can contain a comma/space/tab/newline separated list of PATH elements

INTERFACE_ITEM: can be one of the following:

  • INTERFACE: a name of a network interface (i.e, eth0,eth1,wlan0, ...)
  • FILE: an external that contains a comma/space/tab/newline separated list of INTERFACE items
  • COMMAND: an external command to execute, the output can contain a comma/space/tab/newline separated list of INTERFACE items

PORT_ITEM: can be one of the following:

  • PORT: a port number
  • RANGE: a port range in the form START-END (i.e, 1024-65545)
  • FILE: external file that contains a comma-separated list of PORT or RANGE elements
  • COMMAND: external command to execute, the output can contain a comma/space/tab/newline separated list of PORT and/or RANGE elements

User Groups

To create a group, choose a group name, then populate it with UIDs, UID ranges, user names, or files of UIDs or usernames, as follows:

Group with a UID range example: Syntax: uid {group name} {uid range}

Example:

uid example-group 0-499
uid example-group 800-		(this includes all UIDs from 800 and above)

Group with multiple UID ranges: Syntax: uid {group name} {uid range}[,{uid range2}...]

Example:

uid example-group 0-499,800-850

Group with UID range and a username: Syntax: uid {group name} {uid range},[,{username},...] Example:

uid example-group 0-499,nfsnobody

Group with multiple usernames: Syntax: uid {group name} {username1},[,{username2}...]

Example:

uid example-group bluser1,bluser2

Group with files of UIDs or usernames: (files must use a comma delimited list) Syntax: uid {group name} {file path}[,{file path2}...]

Example:

uid example-group /etc/passwd
uid cpanel_users /var/cpanel/users/username

Groups made from external file using regular expresion and the cut command (must be a comma delimited list). This is especially handy if you are pulling a list from the passwords file or some other file that keeps lists of created users by a program like cPanel.

Syntax: uid {group name} `cut -d: -f1 {file path}`

Example:

uid bl_local_users `cut -d: -f1 /etc/passwd`

Result: Using ":" as a delimiter, this syntax reads field one (the UID field) from the passwd file, making these UIDs members of the group "bl_local_users."

Program Groups

BetterLinux has multiple features for controlling limits for programs.

To create your program groups, choose a group name, then populate it with programs by location and name separated by a comma with no spaces, as follows:

Group with programs example: Syntax: prog {group name} {/location/program-name,/location/program-name2,/location/program-name3}

Example:

prog example-group-programs /usr/sbin/sshd,/usr/bin/rsync

Group with files of programs: (files must use a comma delimited list) Syntax: uid {group name} {file path}[,{file path2}...]

Example:

prog example-group-programs /etc/betterlinux/list-programs.txt

Port Groups

The ports group is tied to the BetterBandwidth connections feature which is currently on hold and not part of the BetterLinux release.

Interface Groups

The BetterBandwidth feature requires an interface group name for the network interface to throttle. To create an interface group, chose a group name, then populate it with the network interfaces for throttling as follows.

Group with interface examples: Syntax: interface {group name} {interface name} Example:

interface interface-example-group eth0,eth1

Group with interface example using an external file that contains the interfaces: Syntax: interface {group name} {file path},{file path2} Example:

interface interface-example-group /etc/betterlinux/interfaces.txt

Fixed Groups

The fixed groups feature is a predefined set of group names that will always or never throttle. Their names and functions are set, but they have not yet been populated with users or programs. In order to configure these groups, they need to be populated with UIDs or Programs. The concept behind fixed groups is to create a predefined group of users or programs that will aways be or never be throttled. These parameters apply to the groups.conf or cpanel-groups.conf and groups.conf or cpanel-groups.conf files for BetterCPU and BetterIO.

Configuration Location:

/etc/betterlinux/cpud.conf.d/groups.conf or cpanel-groups.conf
/etc/betterlinux/iothrottled.conf.d/groups.conf or cpanel-groups.conf

Syntax for fixed group examples:

BetterCPU:

uid ALWAYS_THROTTLE_CPU_GROUP {members}   #[Default: empty]
uid NEVER_THROTTLE_CPU_GROUP {members}   #[Default: empty]
prog ALWAYS_THROTTLE_CPU_GROUP {programs} #[Default: empty]
prog NEVER_THROTTLE_CPU_GROUP {programs}   #[Default: empty]

BetterIO:

uid NEVER_THROTTLE_GROUP {members}   #[Default: empty]
BetterCPU (cpud) Fixed Throttling Examples

Throttle UIDs BetterCPU (cpud):

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid1}[,{uid2}...]

Example:

uid ALWAYS_THROTTLE_CPU_GROUP 13,15,19 

(Always throttles UIDs 13,15,19)

Throttle UID range:

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid range}

Example:

uid ALWAYS_THROTTLE_CPU_GROUP 500-999

(Always throttles UIDs from 500-999)

Throttle usernames:

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {username1}[,{username2}...]

Example:

uid ALWAYS_THROTTLE_CPU_GROUP bl_user1,bl_user2

(Always throttles listed users)

Throttle a file of UIDs: (where the file contains a delimited list of UIDs)

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {file path}

Example:

uid ALWAYS_THROTTLE_CPU_GROUP /etc/betterlinux/file_with_a_list_of_uids

Throttled Programs:

Syntax: prog ALWAYS_THROTTLE_CPU_GROUP {program1}[,{program2}...]

Example:

prog ALWAYS_THROTTLE_CPU_GROUP program1,program2

(Always throttles programs programs1,program2)

Throttle a File of Programs: (where the file contains a delimited list of programs)

Syntax: prog ALWAYS_THROTTLE_CPU_GROUP {file path}

Example:

prog ALWAYS_THROTTLE_CPU_GROUP /file/path/list/of/programs

"NEVER_THROTTLE_CPU_GROUP" lets you list members that will never be throttled. It accepts UIDs, UID ranges, user names, or files of UIDs, programs, or files containing programs. See the above "ALWAYS_THROTTLE_CPU_GROUP" syntax and examples substituting in "NEVER_THROTTLE_CPU_GROUP".

BetterIO (iothrottled) Fixed Throttling Examples

Throttle UIDs BetterIO (iothrottled):

Syntax: uid NEVER_THROTTLE_GROUP {uid1}[,{uid2}...]

Example:

uid NEVER_THROTTLE_GROUP 13,15,19 

(Never throttles UIDs 13,15,19)

Never throttle UID range:

Syntax: uid NEVER_THROTTLE_GROUP {uid range}

Example:

uid NEVER_THROTTLE_GROUP 500-999

(Never throttles UIDs from 500-999)

Never throttle usernames:

Syntax: uid NEVER_THROTTLE_GROUP {username1}[,{username2}...]

Example:

uid NEVER_THROTTLE_GROUP bl_user1,bl_user2

(Never throttles listed users)

Never throttle a file of UIDs: (where the file contains a delimited list of UIDs)

Syntax: uid NEVER_THROTTLE_GROUP {file path}

Example:

uid NEVER_THROTTLE_GROUP /etc/betterlinux/file_with_a_list_of_uids

CPU

By default, the non-cPanel installation (bl-install) does not configure BetterCPU to throttle users. However, the cPanel installation (bl-cpanel-install) script does auto-configure and start BetterCPU throttling for all cPanel users that are assigned to a cPanel package. The default settings for each cPanel package level is 50% of one core for 4 seconds per user triggers throttling the user in the jail core(s) and 40 % for 4 seconds releases the user from the jail. Example defaults settings:

cpu_throttling group=ExampleHost_Level1 in_cpu_percent=50 out_cpu_percent=40 seconds=4 type=jail
cpu_throttling group=ExampleHost_Level2 in_cpu_percent=50 out_cpu_percent=40 seconds=4 type=jail
cpu_throttling group=ExampleHost_Level3 in_cpu_percent=50 out_cpu_percent=40 seconds=4 type=jail

The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).

Configuring the Jail Cores

Throttling is configured using parameters located in betterlinux.conf (for stock installations):

/etc/betterlinux/cpud.conf.d/betterlinux.conf
jail_cnt
max_jail_cnt

These settings are designed so that the same configuration file may be used on different systems with different numbers of cores. However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file. When these parameters are removed, the cpud daemon uses the default settings: jail_cnt=1 and max_jail_cnt=total cores minus 1.

Syntax:

jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]

The jail_cnt parameter is defined using one of two methods:

  • If a fixed number of cores for jail is desired, make the assignment using jail_cnt as in this example:
jail_cnt 2
In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core. If the system is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.
  • If the number of cores for jail varies based on the total number of cores in the system, use the following format:
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. For example, using the values specified in the previous example, a 32-core system will use the jail_cnt value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails and is formatted identically to the jail_cnt parameter. By default, only the value of jail_cnt is used, but when a max_jail_cnt is specified, the jail is expanded when less than the value in jail_min_idle is available per jail core.

The jail_min_idle value is the minimum amount of idle utilization to maintain on the jail cores if jail_max_cnt has not been reached. If the average free utilization on each jail core is less than jail_min_idle, the jail is expanded until either the jail_min_idle value is maintained or jail_max_cnt is reached.

Syntax:

max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}

Example

max_jail_cnt 2:1 4:2 8:2 16:2

The result on a dual core system is no different from jail_cnt. But for 4-core machines and above, if the max_jail_cnt value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded.

max_jail_cnt 2

The result will always set the number of maximum jails to two. The maximum number of jails are set at two no matter how many cores may be on the system. But when the system has 2 cores, it will set the max jail core to one.

In order for the jail to expand, add a jail_min_idle value to denote how much free utilization should be maintained in the jail:

jail_min_idle 5

BetterCPU will attempt to maintain at least 5% free CPU per jail core and will expand the jail if max_jail_cnt has not been reached.

Defining User Groups

Groups are automatically defined by the bl-cpanel-install script. Installations on non-cPanel systems need to be defined manually. For more details on groups please see Define Groups under the Limits section.

Setting CPU Core Usage Limits

Once jail_cnt and jail_cnt_max are defined and groups are set up, you must configure throttling parameters to define when user groups enter and exit jail core(s). Here are the relevant parameters and their syntax:

cpu_throttling group={group}[,{group2}...] in_cpu_percent={integer} out_cpu_percent={integer} seconds={integer} type=jail

These parameters set throttling limits for groups defined in groups.conf (stock install) or cpanel-groups.conf (cPanel install), which are found in /etc/betterlinux/cpud.conf.d.

The cpu_percent value determines how much of a system's cpu resources an individual user (uid) may use. This percent value reflects an amount of cpu system resources equal to 'x' percentage of a single core, not a percentage of the total system. A cpu_percent value of 80 means that a user may not use beyond the equivalent of 80% of one core. On an 8-core system with a value of 80, for example, a user may exceed his limits by the following: 1) using more than 80% of one core, 2) using more than 10% of each of 8 cores, 3) using 6% of one core, 12% of another, and 63% of another, or 4) any other combination that exceeds the equivalent of 80% of a single core.

It is also possible to have a cpu_percent value greater than 100. For example, a value of 150 would give a user the equivalent of 1.5 system cores. The maximum value is equal to a system's total possible CPU utilization minus 100 (The subtracted 100 represents one whole core reserved for essential system processes). Remember that in a multi-core Linux system, multithreaded processes can exceed 100% CPU utilization. Total possible CPU utilization = 100% * core#. So in an 8-core system, the system’s total possible CPU utilization is 800% (100% * 8 cores) if all cores run at maximum. Therefore, the maximum cpu_percent value would be 700 (800% - 100%).

Because the cpu_percent value is based on a single core, it does not matter how many cores are in the system, which means the same .conf file can be used in machines with differing numbers of cores.

The cpu_throttling parameter are found in betterlinux.conf (for stock installations) and cpanel-allusers.conf (for cPanel installations). To define the parameters’ values, edit the .conf file. After editing, remember to save the .conf file and then restart cpud with the command line entry "service cpud restart". The .conf files are found here:

/etc/betterlinux/cpud.conf.d/betterlinux.conf
/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf

To learn how to interpret the parameters and their associated values, see the following example:

cpu_throttling group=example-group in_cpu_percent=50 out_cpu_percent=40 seconds=4 type=jail

In the above, a user in example-group that maintains CPU usage beyond the equivalent of 50% of one core for at least 4 seconds will be assigned to one of the system’s jail cores. When that user’s CPU usage drops below the equivalent of 40% of a single core for at least four seconds, the user will be removed from the jail.

When a non-throttled user's usage exceeds the in_cpu_percent value (the equivalent of x% of a single core) for ‘x’ number of seconds, the user's processes are moved into the jail cores.

When a throttled user’s usage drops below the out_cpu_percent value (the equivalent of x% of a single core) for ‘x’ number of seconds, the user's processes are moved out of the jail and back into the non-jailed cores.

Start the BetterCPU Daemon (cpud)

BetterCPU is now configured with basic jail core and throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service cpud {start|stop|status|reload|restart|condrestart}
 start = starts cpud daemon
 stops = stops cpud daemon
 status = shows the status of cpud daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in
 the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon
 only if it is currently running. This option is useful in scripts because it
 does not start the daemon if it is not running.

Start the cpud daemon with the following command:

service cpud start
Start the BetterLinux Stats Program (blstat)

In order to see user processes that are in the jail cores, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterI/O. Press 'c' to see the stats for BetterCPU. Press '?' to display general help for the utility.

Limiting Program Execution Time Using time_monitor

BetterCPU can be used to limit the amount of time programs can execute. Limits can be defined using program groups or user groups, both of which are defined here:

/etc/betterlinux/cpud.conf.d/betterlinux.conf (for stock installations)
/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf (for cPanel installations) 

The basic steps for limiting execution time are:

  1. Define a program or user group
  2. Add time_monitor to one of the .conf files mentioned above.
  3. Reload the BetterCPU daemon's configuration
service cpud restart

When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.

The format of this parameter is:

  • time_monitor group_type=[user or prog] group=group1,group2 seconds=5 child_of=/parent/program program=/script/to/run wait_to_run_again_seconds=3 exclude_prog_group=group_name or exclude_user_group=group_name

BetterLinux includes two scripts that can be used with time_monitor: /etc/betterlinux/bin/kill_pid.pl (This will kill programs) /etc/betterlinux/bin/pid_running_too_long.pl (will report when a program is running longer than the number of seconds defined in time_monitor)

time_monitor auto-configuration for cPanel servers

The bl-cpanel-install script will auto-configure the time_monitor feature with the number of seconds entered here at the time of installation:

/etc/betterlinux/cpud.conf.d/betterlinux.conf (for stock install)
/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf (for cPanel install) 

In the examples below, 900 seconds was entered by the user during the bl-cpanel-install script. The user has a cPanel package called AcmeHost_Level1. Any child programs of httpd, chroottpd, and crond owned by the users in the AcmeHost_Level1 package are limited to run for 900 seconds. If they exceed the 900 seconds, kill_pid.pl script will run every 30 seconds until the program is killed. The child of programs excluded from the 900 seconds are bash, ksh93, tcsh, and zsh.:

  • time_monitor group_type=user group=AcmeHost_Level1 seconds=900 child_of=/usr/local/apache/bin/httpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list
  • time_monitor group_type=user group=AcmeHost_Level1 seconds=900 child_of=/usr/local/cpanel/bin/chroothttpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list
  • time_monitor group_type=user group=BetterHost_Level1 seconds=900 child_of=/usr/sbin/crond program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

The monitor_exclude_list group is auto-defined with the following syntax in groups.conf (stock installs) or cpanel-groups.conf (cPanel installs) (which are found in /etc/betterlinux/cpud.conf.d</code):

prog monitor_exclude_list bash,ksh93,tcsh,zsh
Add time_monitor to betterlinux.conf (Stock) or cpanel-allusers.conf (cPanel)

Limiting the execution time for a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again. The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.

For example, a script in <code>/root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.

To define the limit, add the following line to betterlinux.conf (Stock) or cpanel-allusers.conf (cPanel):

  • time_monitor group_type=prog groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer wait_to_run_again_seconds=10

In this example, we use the sample "bad_progs" program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash. If the program has run for 5 seconds or longer, /root/killer is called with the PID of the program, and the script kills the program after logging the date and time.

The final value (10) tells cpud to not run the script more than once every 10 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.

When using a user group (rather than a program group), the same concepts apply. If there are programs, the users should be allowed to run without monitoring the execution time, define a program group for those exceptions, and use the following command to globally exclude those programs from being monitored. For example:

Create a program exclude group in:

/etc/betterlinux/cpud.conf.d/groups.conf (for stock installation)
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf (for cPanel installation)

Define a Program Group:

prog monitor_exclude_list bash,ksh93,tcsh,zsh

Then define a time_monitor setting to kill any users child processes of sshd that run longer than 900 seconds, with the exception of bash,ksh93,tcsh,zsh. The /etc/betterlinux/bin/kill_pid.pl script is included in the BetterLinux installation.

  • time_monitor group_type=user group=cpanel_users seconds=900 child_of=/usr/sbin/sshd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

Limiting Open Files Using file_limit

BetterCPU can be used to limit the number of concurrent open files a user can have at one time. Limits are defined using user groups defined in groups.conf (for stock installations) or cpanel-groups.conf (cPanel installations). These files are found here:

/etc/betterlinux/cpud.conf.d/groups.conf
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf

These are the basic steps for limiting concurrent open files:

  1. Define a user group.
  2. Add file_limit to betterlinux.conf (stock install) or to cpanel-allusers.conf (cPanel install). These are found here:
/etc/betterlinux/cpud.conf.d/betterlinux.conf
/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf
  1. Reload BetterCPU's daemon configuration

The format of this parameter is:

file_limit group=GROUP,GROUP limit=[number of files to limit]

The groups are user groups defined in groups.conf (stock) or cpanel-groups.conf (cPanel).

When a user exceeds the allowed number of open files, CPUd prevents any additional files from being opened.

For example, assume cpanel-groups.conf has the following definitions:

uid bl_users 500-1000

Then the following file_limit setting is applied:

file_limit group=bl_users limit=100

For any user with a UID from 500 to 1000, if 100 files are opened by that user, then additional attempts to open files (for reading or writing) will fail.

file_limit group=bl_users limit=100	

"file_limit_exclude" excludes programs from concurrent file open limits.

As with the file_limit setting, this parameter takes a group defined in groups.conf (stock) or cpanel-groups.conf (cPanel). The group contains a list of programs to be excluded from concurrent open file limits.

file_limit_exclude group=prog_group

"file_limit_override" overrides file limits by program. Program groups are defined in groups.conf (stock) or cpanel-groups.conf (cPanel).

file_limit_override group=prog_group1,prog_group2 limit=120 child_of=/bin/cron,/usr/bin/ssh

When open files are limited, the file_limit_override and file_limit_exclude options can be applied to override the limits for certain programs and to remove the limits from others.

Limiting Processes Using process_limit

BetterCPU can be used to limit the number of concurrent processes a user can execute at one time. Limits are defined using user groups defined in groups.conf (stock install) or cpanel-groups.conf (cPanel install). Find them here:

/etc/betterlinux/cpud.conf.d/groups.conf
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf

The basic steps for limiting concurrent processes are:

  1. Define a user group.
  2. Add process_limit to betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel). These are found here:
/etc/betterlinux/cpud.conf.d/betterlinux.conf
/etc/betterlinux/cpud.conf.d/cpanel-allusers.conf
  1. Reload the BetterCPU daemon's configuration

When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.

process_limit auto-configuration for cPanel servers

The bl-cpanel-install script will auto-configure the process_limit feature by prompting for the number of processes to limit each user assigned to a cPanel package. In the example below, the administrator selected a limit of 50 processes and has a cPanel packages called AcmeHost_Level1. The auto-configuration will also define an exclude group in groups.conf (stock install) or cpanel-groups.conf (cPanel install). These are found here:

/etc/betterlinux/cpud.conf.d/groups.conf
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf

See the example:

  • prog ignored_programs_for_process_limit /usr/libexec/dovecot/imap,/usr/libexec/dovecot/pop3,/bin/bash,/bin/zsh,/bin/ksh93,/bin/tcsh,/usr/bin/screen,/usr/bin/emacs-23.1,/bin/nano,/usr/bin/alpine,/usr/bin/mysqldump,/usr/bin/rsync

Next, it uses the ignored_programs_for_process_limit group in the process_limit_exclude parameter in betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install), both found in /etc/betterlinux/cpud.conf.d. This will exclude all these programs from the process_limit parameter settings:

process_limit_exclude group=ignored_programs_for_process_limit

Then, it will configures the following process_limit parameters in betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install), both found in /etc/betterlinux/cpud.conf.d.

process_limit group=AcmneHost_Level1 limit=50
process_limit_override group=AcmeHost_Level1 limit=75 child_of=/usr/sbin/crond
process_limit_override group=AcmneHost_Level1 limit=100 child_of=/usr/sbin/sshd

The configuration above sets a process limit of 50 for each user in the AcmneHost_Level1 package, then assigns an exception limit of 75 to child process crond and 100 for sshd. Additionally, all the programs in the ignored_programs_for_process_limit will not be held to a limit.

Add process_limit parameter

Limiting concurrent processes per user is very straightforward.

To apply a limit, once a user group is defined, add a line to apply the desired restriction to betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install):

process_limit group=bad_users limit=100

This limits the users in the bad_users group to 100 concurrent processes each at any given time.

To completely exclude specific programs from having limits applied, create a program group in groups.conf (stock install) or cpanel-groups.conf (cPanel install) with the programs in it:

prog exclude_process_limits /bin/cron,/bin/bash

Then add a process_limit_exclude line to betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install):

process_limit_exclude group=exclude_process_limits

If a user limit should be overridden for particular programs, specify the user group as part of a process_limit_override command, and specify the specific programs that should have the different limit in betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install):

process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd

This overrides the existing limits for bad_users for instances of /usr/sbin/sshd and sets those limits to 120.

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Memory Utilization Using memory_limit

BetterCPU can be used to monitor the amount of RSS memory programs can use. Limits can be defined using program groups or user groups, both of which are defined in groups.conf (stock install) or cpanel-groups.conf (cPanel install), both found in /etc/betterlinux/cpud.conf.d/.

The basic steps for limiting RSS memory usage are:

  1. Define a program group or a user group.
  2. Add memory_limit to betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install), both found in /etc/betterlinux/cpud.conf.d.
  3. Reload the BetterCPU daemon's configuration

The format of this parameter is:

  • memory_limit group_type=[user or prog] group=group1,group2 rss_memory=[amount in megabytes "m" or gigabytes "g"] program=/script/to/run wait_to_run_again_seconds=m exclude_prog_group=group_name or exclude_user_group=group_name

When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.

memory_limit auto-configuration for cPanel servers

The bl-cpanel-install script will auto-configure the memory_limit feature by prompting the administrator for the amount of RSS memory to limit each user assigned to a cPanel package. In the example below, the administrator selected an RSS memory limit of 2000m (megabytes) and has a cPanel package called AcmeHost_Level1. Once the 2000m threshhold has been exceeded for any user in the AcmeHost_Level1 group, the script /etc/betterlinux/bin/pid_running_too_long.pl will run and log the event.

  • memory_limit group_type=user group=BetterHost_Level1 rss_memory=2000m program=/etc/betterlinux/bin/pid_running_too_long.pl
Add memory_limit

Limiting the RSS memory used by a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again. The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.

To define the limit, add the following line to betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install):

memory_limit group_type=prog group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run_again_seconds=30

In this example, we use the sample bad_progs program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB). If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.

The final value (30) tells cpud to not run the script more than once every 30 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored by adding them to the end of the syntax format:

exclude_prog_group=group_name or exclude_user_group=group_name

In this example, a specific program will run when a users total rss memory usage exceeds 2000m or 2 Gigs. This is done by defining the program that will run when the limits are reached. Add an 'm' or 's' to the end of the memory amount to specifying memory limits (such as 4096m for 4 Gigs).

memory_limit group_type=user group=cpanel_users rss_memory=2000m program=/etc/betterlinux/bin/pid_running_too_long.pl

Assume the following is in groups.conf (stock install) or in cpanel-groups.conf (cPanel install):

uid bl_users 500-1000
prog prog_excl /usr/bin/top,usr/bin/vmstat

Also assume the following memory_limit setting is applied:

memory_limit group_type=user group=bl_users rss_memory=1000m program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=prog_excl

Then if any user with a UID from 500 to 1000 runs a process that uses more than 1GB of memory, the kill script executes every 30 seconds (at most) with the exception of top and vmstat.

In order to activate the changes, issue the command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart


Load Average Overview

The BetterLinux cpu throttling feature moves processes exceeding limits into cpu jail cores. Using this method makes the system load average higher. In order to resolve this issue and properly calculate the load average, BetterLinux has introduced the load_average parameter. Now system administrators can configure the proper load average for the system using this parameter.

Load Average Usage

The default configuration for cPanel installations sets the load_average parameter with the recommended usage below:

load_average type=all or non_jail_only normalize_load=on or off

The first switch (type=) value takes the word "all" meaning all the cores on the system or the word "non_jail_only" meaning to calculate the load average only on the non_jail_cores. The second switch (normalize_load=) value is "on" or "off". Entering a value of "off" adds up the total load of the cores to calculate the load average. Entering a value of "on" adds up the total load of the cores and divides it by the total number of cores.

For example, if this parameter is set to:

load_average type=all normalize_load=off

Will add up the load for all the cores.

load_average type=all normalize_load=on

Will add up the load for all the core (jail and non_jail) and divide the total load by the total number of cores.

  1. load_average type=non_jail_only normalize_load=off

Will add up the load for all non-jail cores.

load_average type=non_jail_only normalize_load=on

Will add up the load for all the non-jail cores and divide by the total number of non_jail cores.

I/O

During install, the bl-cpanel-install script auto-configures I/O throttling for all users previously assigned to cPanel packages. The default limits for each cPanel package and mounted device is 10MBs of disk I/O per seconded. For example:

ExampleHost_Level1 bw max vda 10000000 ops max vda 1000
ExampleHost_Level2 bw max vda 10000000 ops max vda 1000
ExampleHost_Level3 bw max vda 10000000 ops max vda 1000

In stock Linux (non-cPanel) installs, IO throttling is not auto-configured and needs to be setup manually. The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).

Defining user groups

User groups are automatically defined by the bl-cpanel-install script. Non-cPanel install requires manual definition. For more information, see User Groups.

By creating and populating groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined.

Once the groups have been created, you can define throttling limits that meet each group's SLA.

Setting I/O Bandwidth Limits

Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis. To do this, edit the bl_io.conf file located in /etc/betterlinux/iothrottled.conf.d. This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/groups.conf.

The bl_io.conf file uses a single line per entry:

[group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.

For example:

example-group bw max sda,sdb,sdc 10000000

This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.

Another example:

io_standard_users bw max inc="^/",exc="^/home" 5000000

This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.

After updating the bl_io.conf file with limits applicable to the system, save the file and restart iothrottled.

Note:

  • If a group and a user have the same name, the limit applies to the user.
  • Only one limit per device can be set. It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.

Please see BetterIO Concepts documentation and especially BetterIO Limits documentation for more I/O bandwidth limiting and configuration options, including for programs and inclusion/exclusion options.

Tuning I/O Scaling and Balancing

The auto_scale and seek_balance features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in bl_io.conf.

These settings are contained in /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf. The configuration parameters define how these settings affect the I/O bandwidth. Details on how these settings work can be found in the user documentation for scaling.

Start the BetterIO daemon (iothrottled)

BetterIO is now configured with basic throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service iothrottled {start|stop|status|reload|restart|condrestart}
 start = starts iothrottled daemon
 stops = stops iothrottled daemon
 status = shows the status of iothrottled daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Use the following command to start the iothrottled daemon:

service iothrottled start

Start the BetterLinux stats program (blstat)

To see user processes that are being throttled by BetterIO, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterIO. Press 'i' to see the stats for BetterIO. Press '?' to display general help for the utility.

MySQL

Bandwidth

BetterBandwidth limiting uses the mod_betterlinux Apache module located in in the betterlinux-cpanel*.rpm , as well as the linux tc method of using the uid classifier to categorize specific user processes. The betterlinux-cpanel*.rpm is installed by the bl-install script by default and auto-configured for cPanel systems during the betterlinux-cpanel-install script.

Bandwidth Limiting Setup

For notes on automatic setup in cPanel, see Bl-cpanel-install options and prompts.

Manual BetterBandwidth Configuration

These steps apply to cPanel and non-cPanel servers.

  • In groups.conf (for stock install) or cpanel-groups.conf (for cPanel install), create an interface group. (The .conf file is found in /etc/betterlinux/cpud.d)

Here's an example of how to create an interface group:

interface interface-types eth0
  • In betterlinux.conf (stock install) or cpanel-allusers.conf (cPanel install), which are found in /etc/betterlinux/cpud.conf.d, add the bandwidth_limit parameter. This sets the bandwidth limit by user and interface groups and also allows the admin to pass in any "tc" command line options. The parameter takes a value in the format:
bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf (for stock install) or cpanel-groups.conf (for cPanel install). The second switch (interface_group=) value is the name of the interface group defined in groups.conf (stock) or cpanel-groups.conf (cPanel). The third switch (limit=) is the amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, assume this parameter is set to limit bandwidth by 20mbit for each user in the cpanel_users group (any defined user group can be used):

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the cpanel_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

Assigning IP addresses with the dedicated IP address mapper

With multi-homed systems, incoming traffic is directed to the IP address where DNS resolves the domain name. A system with multiple virtual hosts can accept traffic to secondary addresses which, in turn, are assigned to secondary addresses, which are assigned to the system's network adapters.

Without some modification, responses sent from a server in a multi-homed system originate from the host's primary IP address. This makes it difficult to apply bandwidth or connection limits using a server's IP address. Most often, hosting providers work around this problem by implementing some sort of full virtualization solution that provides a unique virtual instance for each client. <KGO> What is the drawback of this solution?

BetterBandwidth uses the dedicated kernel module to provide a more cost-effective, efficient solution for managing bandwidth in multi-homed environments. When used in conjunction with the BetterBandwidth dedicated daemon, the dedicated kernel module allows administrators to assign outgoing traffic to a specific secondary address which, in turn, is assigned to a specific user--all without the overhead of a fully virtualized system. (Both IPv4 and IPv6 are supported.)

To enable this functionality:

  1. Ensure the dedicated kernel module is loaded:
    modprobe dedicated
  2. Edit /etc/betterlinux/ipv4 and/or /etc/betterlinux/ipv6 to include the specific mappings.
  3. Restart dedicatedd:
    service dedicatedd restart

The IPv4 and IPv6 mapping files are in the following format. Each line contains a single entry.

username|uid: address

The address field can contain either a raw IPv4 or IPv6 address or a DNS name. If the DNS name resolves to more than one address, the kernel randomly selects one of the addresses.

Once configured, the active mappings for IPv4 addresses can be viewed using the following command:

cat /proc/sys/net/ipv4/dedicated

To view mappings for IPv6 addresses, use the following command:

cat /proc/sys/net/ipv6/dedicated

Note: The command output displays the uid rather than the username.

CloakFS & Security

Configuration Files

The cloak parameter is found here for basic Linux install:

/etc/betterlinux/cpud.conf.d/betterlinux.conf

or here for cPanel installation:

/etc/betterlinux/cpud.conf.d/cpanel.conf

The cloak parameter secures the file system by preventing users from seeing other users' files, directories, and processes--even root and system processes. Users see only their own $HOME directory and files outside home that they own. cloak also blocks scripts running via SSH, Cron Jobs, Apache (suPHP, suexec, # etc.), and symlinks.

Add values to the cloak parameter with the following syntax:

cloak hide_user_group=[group1_name,group2_name] hide_user_group_from=[group1_name,group2_name] hide_system_processes=Y

The first value ("hide_user_group=") is a comma-delimited list of groups (defined in groups.conf or cpanel-groups.conf) that will be hidden from the users in the second value (hide_user_group_from=). Users see only their own process. System processes, those run by root, apache, mysqld, etc., can also be hidden or visible with "Y" or "N". By default, system processes are hidden ("Y").

For example, if this parameter is set to:

cloak hide_user_group=user_group1 hide_user_group_from=user_group1 hide_system_processes=Y

Then all users in user_group1 are hidden from each other and cannot see system processes. Note that users cannot be hidden from themselves.


cloak_file is a parameter that allows you to hide all files in /etc/passwd/, named.conf, and /var/log. You can also write your own scripts for hiding other files and directories.

This parameter takes values in this format:

cloak_file file=[file_name] cloak_with_program=[script] cloak_file_from_group=[group1_name,group2_name]

file= accepts the filename you want to hide. It accepts only one filename. To hide multiple files, create a new parameter line for each file with accompanying values. cloak_file_from_group= accepts the groups from which you want to hide the file (groups are defined in groups.conf). cloak_with_program= accepts the full file path of the script that performs the hiding.

The cloak_file parameter comes with three pre-installed scripts for hiding etc/passwd, /etc/named.conf, and /var/log files. This is how to use each of them:

Hide /etc/passwd:

cloak_file file=/etc/passwd cloak_with_program=/etc/betterlinux/bin/etc_passwd_handler cloak_file_from_group=BetterHost_Level1,BetterHost_Level2,default,BetterHost_Level3

Hide /etc/named.conf:

cloak_file file=/etc/named.conf cloak_with_program=/etc/betterlinux/bin/etc_named_conf_handler cloak_file_from_group=BetterHost_Level1,BetterHost_Level2,default,BetterHost_Level3

Hide /var/log:

cloak_file file=/var/log cloak_with_program=/etc/betterlinux/bin/var_log_handler cloak_file_from_group=BetterHost_Level1,BetterHost_Level2,default,BetterHost_Level3

As stated above, using the same format, you can write your own scripts for hiding other files and directories.


block_suid is a parameter that uses groups defined in groups.conf to prevent users from running programs flagged as SUID root.

It uses groups defined here:

groups.conf (for stock install)
cpanel-groups.conf (for cPanel install).

This parameter takes values in this format:

block_suid group=BetterHost_Level1,BetterHost_Level2,default,BetterHost_Level3

Programs that users in the group are allowed to run can be specified by creating a program in the groups.conf file. For example, users may need to run a program like passwd or a cPanel program like jailshell. In groups.conf, specify:

exception_prog prog_group1 /usr/bin/passwd,/usr/local/cpanel/bin/jailshell

Then add the prog_group1 group name to the example syntax below:

block_suid group=user_group1,user_group2 program=prog_group1	# [Default = NONE]

After adding a group, restart cpud.

In the example above, users in user_group1 and user_group2 are restricted from all programs flagged SUID root except jailshell, which is defined in exception_prog above.

In this example, if the parameter is set to:

block_suid group=user_group1

The users in user_group1 will be blocked from running programs flagged SUID root.

cPanel Resellers

The BetterLinux cPanel hook script handles the complexities for reseller users. This feature allows the reseller to see the users they have created, but blocks the users from seeing their reseller with the exception of the resellers branding directory. The script will do the following:

  • Finds resellers and creates a group for each
  • Finds resellers users and creates a group for each user
  • Configures these groups and the cloakfs_exception parameter in the /etc/betterlinux/cpud.conf.d/resellers.conf file.

The syntax for the cloaks_exception parameter is as follows:

  cloakfs_exception unprotect_groups=[group-name] unprotect_groups=[group_name]


Start / Stop CloakFS

BetterLinux has a start/stop feature exclusively for CloakFS. If the cpud daemon is shut down, CloakFS will continue to protect the system. Here's the syntax for the CloakFS start and stop service:

The command syntax is as follows:
service cloakfs {start|stop|status|reload|restart|condrestart}
 start = starts cloakfs daemon
 stops = stops cloakfs daemon
 status = shows the status of cloakfs
 reload = reloads changes to the .conf files
 restart = restarts cloakfs
 

Start or stop cloakfs with the following commands:

service cloakfs start
service cloakfs stop

Master Parameters List

You can create and name your own .conf files containing the parameters below, as long as your file retains the “.conf” extension. You cannot mix CPU and I/O parameters in the same .conf file. And for now, any groups you create must go in the pre-existing “groups .conf” file. All CPU and I/O .conf files you create should go here, respectively:

/etc/betterlinux/cpud.conf.d
/etc/betterlinux/iothrottled.conf.d

Parameter names are hyperlinked to their associated man pages, while their feature names are hyperlinked to the documentation section most relevant to that parameter.

Parameter name Feature
add_user_time CPU
approved_accounting_program CPU/MySQL
auto_scale IO
average_type IO
auto_wakeup IO
bandwidth_limit BANDWIDTH/CPU
block_suid CPU
busy_nonjail_idle CPU
busy_num_R_nonjailed CPU
cgroup_dir IO
cloak CPU
cpu_average_periods CPU
enable_dumps CPU
enable_logging CPU
enable_logging IO
enable_program_limits IO
enable_sched_dumps CPU
in_buf_sz CPU
interruptible_sleeps IO
iothrottled_gid IO
jail_cnt CPU
jail_min_idle CPU
load_average CPU
log_auto_scale IO
log_cgroups IO
log_cnt_file IO
log_cpu CPU
log_cpu_file CPU
log_cpud CPU
log_cpud_file CPU
log_failure CPU
log_file IO
log_flush_rate CPU
log_gov CPU
log_gov_file CPU
log_io_limits IO
log_jail CPU
log_jail_file CPU
log_jail_idle CPU
log_killed CPU
log_mem CPU
log_mem_file CPU
log_memcon CPU
log_memcon_file CPU
log_nonjail_idle CPU
log_panics CPU
log_panics_file CPU
log_prog CPU
log_prog_file CPU
log_success CPU
log_throttled IO
log_tids CPU
log_tids_file CPU
log_user CPU
log_user_cnt CPU
log_user_file CPU
log_user_in_jail CPU
max_cgroups IO
max_jail_cnt CPU
memory_limit CPU
nonjail_min_idle CPU
process_limit CPU
process_limit_exclude CPU
process_limit_override CPU
run_priority CPU
sample_periods IO
scale_max IO
scale_min IO
sleep_time_ms CPU
sleep_time_ms IO
cpu_throttling CPU
throttle_group CPU
time_monitor CPU
user_average_periods CPU

Monitoring Tools

blstat

blstat automatically installs during the BetterLinux installation.

blstat is executed at the command line by running (as root) from a terminal window:

blstat

Hot Key Options

note: default screen is the i/o throttling screen.

 ‘c’     toggles to the i/o throttling statistics screen
 ‘i’     toggles to the i/o throttling statistics screen
 ‘n’     toggles to the network connections statistics screen
 ‘b’     toggles to the network bandwidth throttling statistics screen
 ‘p’     pause blstat screen
 ‘q’     exit’s blstat
 ‘h’     toggles the blstat help screen

Command Line Options

 --help | -h               show this usage statement
 -i                        start on i/o screen (default)
 -c                        start on CPU screen
 -n                        start on network connection screen
 -b                        start on network bandwidth screen

myusertop

Description

Display the MySQL users most active during the preceding 4 seconds. Show the Linux users and UIDs that "own" them. Press single "hot" keys to change the display interactively.

Hot Keys

    V - Toggle views 1 and 2
    m - Toggle the display of MySQL users under "owning" Linux users
    H - Toggle hourly/total statistics in header area (only)
    R - Reverse sort direction (ascending/descending)
    f - Invoke FLUSH USER_STATISTICS (to reset statistics to zero)
    Sort by:
    a - Adjusted rows examined                        (view 1)
    b - Busy time during last refresh interval        (view 1)
    c - CPU time during last refresh interval         (view 1)
    C - CPU time / busy time                          (view 1)
    e - Core number at end of latest query            (view 2)
    i - UID                                           (all views)
    l - Busy time of latest completed query           (view 2)
    L - CPU time of latest completed query            (view 2)
    n - Number of non-voluntary context switches in 
          latest completed query                      (view 2)
    o - "Other" commands during last refresh interval (view 2)         
    p - % of total CPU time used by user              (view 1)
    Q - Number of queries in progress                 (view 2)
    r - Rows examined                                 (view 1)
    s - Select commands during last refresh interval  (view 2 default)
    S - Start core number of latest query             (view 2)
    t - Number of threads per user                    (view 1 default)
    u - Update commands during last refresh interval  (view 2)
    U - User name                                     (all views)
    v - Number of voluntary context switches in 
          latest completed query                      (view 2)

Other:

    h,? - Display online help
    q   - Quit B<myusertop>

Column Headings

    Linux/MySQLUser - MySQL user is indented beneath Linux user name
    Thds            - Number of threads at the current time
    CPUTime         - Seconds of CPU time since previous refresh
    BusTime         - Seconds of busy (clock) time during same interval
    CPU/Bus         - CPU time divided by busy time
    RowsExam        - Number of rows "examined" since previous refresh
    AdjRowEx        - RowsExam * CPU / Bus
    %CPU            - Amount of MySQL's CPU usage used by this user
    Sel             - Number of SELECT commands since previous refresh
    Upd             - Number of UPDATE commands since previous refresh
    Oth             - Number of other commands since previous refresh
    VCxSw           - Number of voluntary context switches since previous
                        refresh
    NVCSw           - Number of non-voluntary context switches since
                        previous refresh
    SC              - Core number at start of most recent query
    EC              - Core number at end of most recent query
    LQCpu           - CPU seconds of most recent query
    LQBus           - Busy (clock) seconds of most recent query

Command-line Options

Most command-line options (below) can be run with "hot" keys while myusertop is on screen. Many options can be set as defaults in the configuration file:

   */etc/betterlinux/mysql.conf.d/myusertop.conf*

NOTE: Command-line options can be abbreviated to the shortest unique prefix (preceded by 1 or 2 hyphens).

THE LIST:

   -ve, --version

Display the version of myusertop and exit:

   -nom, --nomyuser

Prevent displaying MySQL users "owned" by each top Linux user. (By default, MySQL users appear in an indented list below each Linux user name) You can toggle this display on and off with hot key "m" while myusertop is on screen.

   --cfg=<filename>, --config=<filename>

Override the default configuration file name (/etc/betterlinux/mysql.conf.d/myusertop.conf) with a new <filename>:

   -S [path], --socket=[path]

Set the Unix domain socket path you want to pass to the MySQL command:

   -ho, --hourly 

Show hourly stats in the header summary (not just totals since MySQLd started or since FLUSH USER_STATISTICS reset the statistics). If there is less than an hour of data (system recently started or FLUSH USER_STATISTICS was used), the hourly rate is extrapolated by assuming the same usage rates for the rest of the hour. NOTE: Hot key "H" toggles this display between hourly and total stats if myusertop is on screen. And whether the header displays stats hourly or since startup/flush time, the main user columns show usage during the most recent refresh interval.

   --refresh=<seconds>

Specify the refresh interval in seconds: (The default refresh interval is 4 seconds)

   --admin=<admin-user-name>

Specify a super MySQL user authorized to issue MySQL administrative commands. Use ONLY if you (the root user running this script) must specify a MySQL superuser name when starting the MySQL client:

   --adminpass=<admin-user-password>

Choose the super MySQL user's password (the superuser created with the --admin option above). Use ONLY if you (the root user running this script) need to specify a MySQL superuser name when starting the MySQL client:

   --ascending

Sort ascending (smallest to largest). This effectively shows the *bottom* users. Hot key "R" reverses current sort direction when myusertop is on screen. Use this option combined with any *one* of the following sort options.

   --sortthreads

Sort MySQL users by how many threads their Linux users are using at the moment of the most recent refresh. This is the default in view 1. Hot key "t" sorts the number of threads when myusertop is on screen.

   --sortcpu

Sort by CPU time (highest to lowest). Hot key "c" sorts by CPU time when myusertop is on screen.

   --sortadjre 

Sort by "adjusted rows examined" (highest users to lowest). "Adjusted rows examined" is the number of rows examined multiplied by the ratio of CPU Time to Busy Time. Hot key "a" sorts by this when myusertop is on screen.

   --sortbusy

Sort by busy (clock) time (highest to lowest). Hot key "b" sorts by busy time when myusertop is on screen.

   --sortcob

Sort by ratio of CPU Usage to Busy Time (cob = "CPU over Busy"). Hot key "C" sorts by this ratio when myusertop is on screen.

   --sortrows

Sort by number of rows examined (highest to lowest.) Hot key "r" sorts by this when myusertop is on screen.

   --sortcpupercent

Sort by each user's percentage use of MySQL CPU time. Hot key "p" sorts by this when myusertop is on screen.

   --sortselect

Sort by the number of select commands issued (from highest to lowest). Hot key "s" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortupdate

Sort by the number of update commands issued (from highest to lowest). Hot key "u" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortother

Sort by the number of other commands issued (from highest to lowest). Hot key "o" sorts by the number of other commands. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortvcs

Sort by the number of voluntary context switches (from highest to lowest) during the most recent query. Hot key "v" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortnvcs

Sort by the number of non-voluntary context switches (from highest to lowest) during the most recent query. Hot key "n" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortqip

Sort by number of active queries in progress (from highest to lowest) for each user at the current moment. Hot key "Q" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortlqcpu

Sort by last query CPU time (from highest to lowest). This is the number of seconds of CPU time each user's most recently-completed query used. Hot key "L" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

   --sortlqbusy

Sort by last query busy time (from highest to lowest). This is the number of seconds of busy time used by each user's most recently-completed query. Hot key "l" sorts by this when myusertop is on screen. (This value appears in view 2, which is displayed with hot key "V" when view 1 is being displayed.)

Files

   /etc/betterlinux/mysql.conf.d/myusertop.conf

See Also

   myusertop.conf(8)

Logging

iothrottled log file names and locations

A parameter switch determines the location of each iothrottled log file. You can modify this switch if you want to write logs to a different location.

Log File Name Parameter Syntax - Location Description
iothrottled log_file /var/log/iothrottled/iothrottled Defines the filename for the iothrottled log file. This is used only if enable_logging is set to '1'.
iothrottled.cnt log_cnt_file /var/log/iothrottled/iothrottled.cnt Defines the filename to track cgroup creation/deletion counts.

iothrottled enable logging parameters

Parameter Syntax Default Description
enable_logging 1 ON Enables logging of information and errors for iothrottled. Set to '1' to enable logging, set to '0' to disable logging. The location of the log file is defined by the log_file parameter.
log_cgroups 1 ON Enables logging of cgroup creations/removals under "cgroup_dir".
log_io_limits 1 ON Enables logging of I/O limit changes.
log_auto_scale 1 ON Enables logging of auto_scale events.
log_throttled 0 OFF enables logging of throttled tasks.

Logging of throttled tasks is done in a loop that scans every /proc/pid/task/tid/io-throttle-stat entries periodically each sleep_time_ms millisecs.

cpud log file names and locations

Each of the following cpud log files has a parameter switch that determines the location of the log file. This parameter switch can be modified if the user wants to write the log files to a different location.

Log File Name Parameter Syntax - Location Description
cpud log_cpud_file /var/log/cpud/cpud This logfile includes messages about users being put in jail, the total number of users in jail, and messages about the jail being expanded or contracted.
cpud_cpu log_cpu_file /var/log/cpud/cpud_cpu defines the file used for logging summary information about cpu core usage. The information logged in this file is the number of cores that are defined as being used by the jail and non-jail, and the number of runnable processes in jail and not in jail.
cpud_governor log_gov_file /var/log/cpud/cpud_governor Defines the file used for logging messages from the cpud governor. The governor is responsible for handling runaway processes. The output includes a date/time stamp, UNIX time, signal sent, process owner UID, whether the process is a dedicated process, the process type, PID, program name, age, and command line used to start the process.
cpud_jail log_jail_file /var/log/cpud/cpud_jail Defines the file used for logging information about jail usage. The output includes a date/time stamp, UNIX time, and information about the number of cores assigned to the jail; percent utilization of the jail by user-owned processes, system-owned processes, and percent of idle jail capacity; the UID of the user with the most processes in jail and the total utilization of that user's processes; the program name with the highest single instance utilization; and the program name with the highest overall utilization.
cpud_mem_con log_memcon_file /var/log/cpud/cpud_mem_con defines the file used for logging messages from the memory controller.
cpud_mem log_mem_file /var/log/cpud/cpud_mem Defines the file used for logging statistics captured by the log_mem parameter. The output includes a date/time stamp, UNIX time, iteration counter, and then the following values: root/non-root indicator, sleep time, program name, number of running processes (rss_cnt), total RSS memory used, max RSS memory used by a single instance of the program. The groups are sorted into root/non-root collections and are sorted within each grouping in descending order of total RSS memory used.
cpud_panics log_panics_file /var/log/cpud/cpud_panics defines the file used for logging cpud panic information. This file contains information about processes killed when cpud enters panic mode.
cpud_prog log_prog_file /var/log/cpud/cpud_prog defines the file used for logging statistics captured by the log_prog parameter. The output includes a date/time stamp, UNIX time, iteration counter, and then the following values: root/non-root indicator, sleep time, program name, number of running processes (cpu_cnt), and percent utilization. The groups are sorted into root/non-root collections, with each terminated with a total CPU percentage for the group.
cpud_tids log_tids_file /var/log/cpud/cpud_tids Defines the file used for logging tid/pid counts.
cpud_user log_user_file /var/log/cpud/cpud_user defines the file used for logging statistics captured by the log_user parameter. The output includes a date/time stamp, UNIX time, iteration counter, and then the following values: sleep time, user UID, percentage of CPU utilized, execs, and the exec rate. At the end of each grouping of the top users is a summary showing the system total CPU utilization, customer total utilization, and the total execs and exec rate.

cpud enable logging parameters

Parameter Syntax Default Description
enable_logging 1 ON Enables logging of information and errors for cpud. Set to '1' to enable logging, set to '0' to disable logging. This a the global logging toggle that controls all the other logging options.
enable_dumps 1 ON Enables or disables triggering of dumps for diagnostic use. When enabled, sending a SIGUSR1 signal to cpud will trigger a dump. Sending SIGUSR2 will reset the dump statistics.
enable_sched_dumps 0 OFF Enables or disables the inclusion of process scheduler wait times when enable_dumps is set to '1'. If enable_dumps is set to '0', then this parameter has no effect.
log_cpu 1 ON Enables or disables logging of CPU status information, such as how many cores are in the jail, how many processes are in the jail, and how many processes are not in jail. This information is logged to the file defined in log_cpu_file.
log_cpud 1 ON Enables or disables all logging. When this option is enabled, other logging switches are used to fine tune what logging is enabled.
log_failure 1 ON Enables or disables logging of jail expansion failures. This information is logged to the file defined in log_cpud_file.
log_gov 1 ON Enables or disables logging of governor activities. This information is logged to the file defined in log_gov_file.
log_jail 1 ON Enables or disables logging of the number of users in jail, how long they have been in jail, and how many processes have been jailed. This information is logged to the file defined in log_jail_file.
log_killed 1 ON Enables or disables logging of information about processes logged by cpud. This information is logged to the file defined in log_memcon_file.
log_mem 0 OFF Defines the number of top memory consuming processes to log. This information is logged to the file defined in log_mem_file.
log_memcon 1 ON Enables or disables logging processes killed for using too much memory. This information is logged to the file defined in log_memcon_file.
log_panics 1 ON Enables or disables logging of killed processes when cpud is in panic mode.
log_prog 0 OFF Defines the number of jailed programs to be logged, starting from the program using the highest percentage of CPU and working down by CPU% utilization. This information is logged to the file defined in log_prog_file.
log_tids 0 OFF Enables or disables logging of how many processes and threads are running per core. This information is logged to the file defined in log_tids_file.
log_user 0 OFF Defines the number of jailed users to be logged, starting from the user using the highest percentage of CPU and working down by CPU% utilization. This information is logged to the file defined in log_user_file.
log_user_cnt 1 ON Enables or disables logging of the total number of users in jail. This information is logged to the file defined in log_cpud_file.
log_flush_rate 1 ON Defines how frequently logs are committed to disk in sleep periods. For example, if log_flush_rate is set to 2 and the sleep_time_ms is set to 500, then the log is committed to disk once a second.
log_jail_idle 10 10 Defines the idle CPU percentage in jail at which a "cpu busy" message is logged. These messages are written to the file defined in log_cpud_file.
log_nonjail_idle 10 10 Defines the idle CPU percentage not in jail at which a "cpu busy" message is logged. These messages are written to the file defined in log_cpud_file.
log_success 1 ON Enables or disables logging of information about successful jail expansion or reduction. This information is logged to the file defined in log_cpud_file.
log_user_in_jail 1 ON Enables or disables logging of users being added to jail. This information is logged to the file defined in log_cpud_file.
do_core_dump 0 OFF Enables or disables cpud leaving core files in the event that it segfaults. The core files are left in / by default.

On-Sceen Messages (Errors, Warnings, Notices)

This section provides further information about certain on-screen messages where more information might be useful.

Category Message Potential Causes
bl-licmgr
ERROR: license invalid or corrupted Changed license file, including manual modifications, attribute changes, line removal, or anything that prevents the encoded checksum from matching.
ERROR: could not validate IP address for license verification The BetterLinux license agent checked all defined network interfaces and none had a valid address. This usually happens when the network is down.
WARNING: could not update license file Filesystem errors: BetterLinux can't write to /etc/betterlinux/license.key because the filesystem is full, mounted in read-only, or corrupted.
WARNING: could not get a valid license token from license.betterlinux.com:80
  • The server was unreachable due to network or firewall issues.
  • The server explicitly returns one of these errors:
    • License key not enabled for host A.B.C.D (where A.B.C.D is an IP address)
    • License key expired
    • Invalid request (Client sent a broken request, ie., one not containing all required/expected information)
    • License key invalid (the license is not in our database)
    • License username is invalid (the username is not in our database)
bl-cphooks
Moving existing BetterLinux config files to /etc/betterlinux/cpanel/backup This happens every time you re-initialize.
Can't open apache usermap configuration file. You’re trying to run the script as someone other than root or a lock has been put on this config file.
Can't open the cPanel users directory. You’re trying to run the script as someone other than root or a lock has been put on this config file.
Can't open dbindex.db input file. You’re trying to run the script as someone other than root or a lock has been put on this config file.
Can't open /etc/betterlinux/cpanel/mysql-usermap.conf file. You’re trying to run the script as someone other than root or a lock has been put on this config file.
Can't open <package_name> UID output file /etc/betterlinux/cpanel/users/<package_name>_package.txt. You’re trying to run the script as someone other than root or a lock has been put on this config file.
bl-cpmysql-install
X X

BASE CONFIGURATION FILES:

BetterLinux

Contains a list of the configuration options for cpu throttling, memory_limit, time_monitor, process_limit, bandwidth_limit, cloaking, and turning on mysql and apache throttling. Location of the file goes in /etc/betterlinux/cpud.conf.d

Step 1:

vim /etc/betterlinux/cpud.conf.d/betterlinux.conf

copy and paste the following configuration settings and modify to your specifications:

#
# BetterLinux Base Configuration File
# Substitute the [add-group] reference with you defined groups
#

# Set maximum number of concurrent processes for any cPanel user:
process_limit group=[add-group] limit=100 

# Set maximum number of concurrent crond child processes for any cPanel user:
process_limit_override group=[add-group] limit=125 child_of=/usr/sbin/crond

# Set maximum number of sshd child processes for any cPanel user:
process_limit_override group=[add-group] limit=150 child_of=/usr/sbin/sshd

time_monitor group_type=user group=[add-group] seconds=300 child_of=/usr/local/apache/bin/httpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

time_monitor group_type=user group=[add-group] seconds=300 child_of=/usr/local/cpanel/bin/chroothttpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

time_monitor group_type=user group=[add-group] seconds=1800 child_of=/usr/sbin/crond program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

time_monitor group_type=user group=[add-group] seconds=1800 child_of=/usr/sbin/sshd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

# CPU THROTTLING TUNABLES
# This is only configured initially to throttle cpanel user accounts.
cpu_throttling group=[add-group] in_cpu_percent=50 out_cpu_percent=40 seconds=2 type=jail

# MEMORY LIMITS
# This will run a specific program when a users total rss memory usage exceeds a
# specific value.  You can define the program that is run when limits are reached.
# Make sure when specifying memory limits to add m or g on the end - Such as 4096m for
# 4 Megs
memory_limit group_type=user group=[add-group] rss_memory=500m program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=mysql_group,apache_group

# BANDWIDTH LIMITS
# This will setup Linux tc rules for each user and throttle outbound bandwidth to this limit.
bandwidth_limit group=[add-group] interface_group=interface_types limit=25mbit

# Hide all cPanel user files from other cPanel users:
cloak hide_user_group=[add-group] hide_user_group_from=[add-group] hide_system_processes=Y

# Block programs set flagged SUID
block_suid group=[add-group]

load_average type=non_jail_only normalize_load=off

# Ignore these programs when enforcing process limits
process_limit_exclude group=ignored_programs_for_process_limit

# APPROVED ACCOUNTING
# This directive approves specific programs to "blame" other users for resources
# that are consumed.  For instance, cPanel runs apache as the nobody
# user.  In order to allow the nobody user to blame someone else for
# cpu/io/bandwidth/connections, etc we have to specifically grant permission for
# that program to blame another uid/user.  The approved_account_program parameter
# is the method we use to grant this permission.
approved_accounting_program program=/usr/local/apache/bin/httpd user=nobody
approved_accounting_program program=/usr/local/cpanel/bin/chroothttpd user=nobody
approved_accounting_program program=/usr/libexec/mysqld user=mysql
approved_accounting_program program=/usr/sbin/mysqld user=mysql

BetterLinux Groups

Contains the groups that match the group names in /etc/betterlinux/cpud.conf.d/betterlinux.conf

Step 1:

vim /etc/betterlinux/cpud.conf.d/bl-group.conf

copy and paste the following configuration settings and modify to your specifications:

# These are BetterLinux groups the admin needs to define.
# Each group contains Linux user names associated with a group.
#
# Add groups with a range:
# uid [add-group] 500-1000

# Add a group that references a file containing a list of uids:
# uid [add-group] /etc/betterlinux//[group_name].txt
 
# Add a group with a list of uids:
# uid [add-group] 500,501,502,503

#LIST OF PROGRAMS WITH FULL PATH THAT ARE EXCLUDED FROM BEING COUNTED IN PROCESS LIMITS THAT ARE SET
#FOR CPANEL USERS
prog ignored_programs_for_process_limit /usr/libexec/dovecot/imap,/usr/libexec/dovecot/pop3,/bin/bash,/bin/zsh,/bin/ksh93,/bin/tcsh,/usr/bin/screen,/usr/bin/emacs-23.1,/bin/nano,/usr/bin/alpine,/usr/bin/mysqldump,/usr/bin/rsync

#LIST OF PROGRAMS TO BE EXCLUDED FROM TIME_MONITOR PARAMETER.  THESE SHOULD BE FULL PATHS AND NOT
#JUST A PROGRAM NAME TO BE SAFE.
prog monitor_exclude_list bash,ksh93,tcsh,zsh

#INTERFACE TO USE FOR BANDWIDTH THROTTLING.  IT SHOULD BE A REAL INTERFACE LIKE eth0
interface interface_types eth0

#MYSQL MEMORY_LIMIT EXCLUDES GROUP
prog mysql_group /usr/libexec/mysqld,/usr/sbin/mysqld

#APACHE MEMORY_LIMIT EXCLUDES GROUP
prog apache_group /usr/local/apache/bin/httpd,/usr/local/cpanel/bin/chroothttpd

I/O throttling

Contains the parameters for i/o throttling.

Location: /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf

Step 1:

vim /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf

copy and paste the following configuration settings and modify to your specifications:

# The settings below are a list of currently active settings. The content
# below this section contains examples of the various BetterIO parameters
#
auto_scale 1 # [Default = 1]
auto_wakeup 1    # [Default = 1]
seek_balancing_method 75 # [Default = 75]

remove_stale_groups_time 60 # [Default = 60]

# The settings below limits the users in the defined group to 10 MBs per second
# per mounted device.
#
# cpanel_package1 bw max sda 10000000 ops max sda 1000
[add-group] bw max loop0 10000000 ops max loop0 1000
[add-group] bw max vda 10000000 ops max vda 1000

#
1MB_prog S bw max inc="^/",exc="^/mnt|^/media" 1048576
5MB_prog S bw max inc="^/",exc="^/mnt|^/media" 5242880
#10MB_prog S bw max inc="^/",exc="^/mnt|^/media" 10485750
#15MB_prog S bw max inc="^/",exc="^/mnt|^/media" 15728640

I/O Throttling Groups

Contains the groups that match the group names in /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf

Step 1:

vim /etc/betterlinux/iothrottled.conf.d/betterlinux-io.conf

copy and paste the following configuration settings and modify to your specific needs:

# These are BetterLinux groups the admin needs to define.
# Each group contains Linux user names associated with a group for i/o throttling.
#
# Add groups with a range:
# uid [add-group] 500-1000

# Add a group that references a file containing a list of uids:
# uid [add-group] /etc/betterlinux/[group_name].txt
 
# Add a group with a list of uids:
# uid [add-group] 500,501,502,503

#
prog 1MB_prog du,ls,rm,tripwire,updatedb
prog 5MB_prog cp,dirsync,migratetool,rsync,scp,awstats.pl,mv,webalizer,mysqldump,cpanellogd,cpio,dd,find,gtar,gzip,pkgacct,pkgacct_runner,rar,tar,wget,zip

#ADD PROGRAMS WITH OR WITHOUT FULL PATH BELOW TO LIMIT TO 10 Megabytes per second
#prog 10MB_prog
#ADD PROGRAMS WITH OR WITHOUT FULL PATH BELOW TO LIMIT TO 15 Megabytes per second
#prog 15MB_prog

Release Notes

BetterLinux release notes generally include some or all of the following: A brief overview of each version's more significant changes, a link to the changelog and/or full changelog text, basic upgrade instructions, and other instructions when necessary.

To see all prior release notes, go here: BetterLinux Release Notes

FAQs

Is OpenVZ supported?

OpenVZ is currently not supported, however, it is something we are considering. We are unable to provide an estimate on when it will make it on the roadmap.

Does BetterLinux run on CentOS 5?

  • BetterLinux supports kernels 2.6.32 or newer. We are currently evaluating a method for supporting CentOS 5. Some BetterLinux features depend on Linux kernel features that were introduced in version 2.6.26. Some of those features have bugs that were not fixed until version 2.6.32. Hence, we cannot support kernels prior to 2.6.26 at all, and kernels prior to 2.6.32 require backported bug fixes. CentOS 5 has kernel version 2.6.18. It is possible to build a later kernel for CentOS 5, but then some userspace programs, such as iptables, must also be updated to cope with changed kernel APIs.

What will the pricing be once we begin to charge?

  • BetterLinux will remain free for non-profit use, but others will pay $9.00 per month per server.