Difference between revisions of "1.0.0"

From docs.betterlinux.com
Jump to: navigation, search
(Defining Groups)
(tim_monitor auto-configuration for cPanel servers)
 
(237 intermediate revisions by 2 users not shown)
Line 100: Line 100:
 
-->
 
-->
  
==Installation==
+
==Install, Upgrade, & Uninstall (newest to oldest)==
  
===Upgrade from BL version 0.9.6-1 with MySQL 5.1.x (cPanel and non-cPanel)===
+
===1.0.x-x Installation on CentOS 6.x with cPanel===
* If you are running Litespeed, you will need to turn it off before beginning the upgrade. Once the upgrade is complete, you can turn it back on.
+
For quick installation and updates, use the pre-built package for CentOS 6.x. Packages for other popular Linux distributions are coming soon.
  
Remove yum header files and cached packages with this command:
+
Basic installation instructions:  
$ yum clean all
+
$ wget http://repo.betterlinux.com/utilities/update-repo-files
+
$ chmod 755 update-repo-files
+
$ ./update-repo-files
+
$ yum upgrade betterlinux\*
+
  
<pre style="color:red">NOTES:
+
*Go to [https://www.betterlinux.com/ https://www.betterlinux.com] and click on the "Get It Now" link.
* If your system does not have BetterLinux MySQL installed, then use the yum command below:
+
$ yum --exclude=betterlinux-cpanel-mysql\* upgrade  betterlinux\*
+
  
* If you have excludes in your yum.conf, the dependencies may fail.  Use the following command to resolve dependencies:  
+
[[Image:Installstock1.png]]
$ yum --disableexcludes=all upgrade betterlinux/*</pre>
+
* Reboot to load the new BetterLinux kernel and modules.
+
* If this is an upgrade on a cPanel server and mysql is upgraded, we recommend you also run easy apache from WHM to recompile apache and php.
+
* If the server is running litespeed, switch from litespeed to apache and then back to litespeed, then run easy apache in whm to recompile apache and php.
+
  
===Upgrade from BL version 0.9.6-1 with MySQL 5.5.x (cPanel and non-cPanel)===
+
*When BetterLinux 1.0.1-1 appears in your shopping cart, click the "Checkout" link:
* If you are running Litespeed, you will need to turn it off before beginning the upgrade. Once the upgrade is complete, you can turn it back on.
+
* edit /etc/yum.repos.d/betterlinux-cpanel.repo and enable the [betterlinux-cpanel55-remote] section.  The [betterlinux-cpanel-remote] section should already be enabled.
+
$ yum clean all
+
$ wget http://repo.betterlinux.com/utilities/update-repo-files
+
$ chmod 755 update-repo-files
+
$ ./update-repo-files
+
$ yum upgrade betterlinux\*
+
<pre style="color:red">NOTES:
+
* If you have excludes in your yum.conf, the dependencies may fail.  Use the following command to resolve dependencies:
+
$ yum --disableexcludes=all upgrade betterlinux/*</pre>
+
* Reboot to load the new BetterLinux kernel and modules.
+
* If this is an upgrade on a cPanel server and mysql is upgraded, we recommend you also run easy apache from WHM to recompile apache and php.
+
  
===Upgrading from BetterLinux version 0.9.4 and newer===
+
[[Image:Install1.0cPanel24.png]]
* If you are running Litespeed, you will need to turn it off before beginning the upgrade. Once the upgrade is complete, you can turn it back on.
+
Download and enable your BetterLinux repos with this script:
+
    $ wget http://repo.betterlinux.com/utilities/enable-betterlinux-repos
+
    $ chmod 755 enable-betterlinux-repos
+
    $ ./enable-betterlinux-repos
+
    $ yum --disableexcludes=all upgrade betterlinux\*
+
  
Reboot to load the new BetterLinux kernel and modules. If this is an upgrade
+
*Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order":
on a cPanel server, we recommend you also run easy apache from WHM to recompile
+
apache and php.
+
  
===Installation for CentOS 6.x with cPanel===
+
[[Image:Installstock3.png]]
For quick installation and updates, use the pre-built package for CentOS 6.0-6.3. Packages for other popular Linux distributions are coming soon.
+
  
Basic installation instructions:  
+
*You are now taken to the "Order Complete" page:
*If you are running Litespeed, you will need to turn it off before beginning the install. Once the install is complete, you can turn it back on.
+
 
*Go to [https://www.betterlinux.com/ https://www.betterlinux.com] and click on the "Get It Now" link.
+
[[Image:Install1.0cPanel26.png]]
*When the BetterLinux Trial appears in your shopping cart, click the "checkout" link.
+
 
*Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."
+
*Here, you are given your order number, which you will need to submit support tickets from the client area. You are also given notice that your email confirmation is on its way.  
*You will see a "Your downloads are here!" button, which takes you to the download page, but you will not be able to download anything until you have verified your email. (see below)
+
 
*You will receive a welcome email with your account information, order confirmation, and an email verification link. The verification link will also take you to the download page.
+
*Open your BetterLinux welcome email. It includes your email verification link and login information. Please note that your link will expire if not used quickly. The verification takes you to the Downloads page where you will be automatically logged into your account: (if you are not logged in for some reason, log in now)
*Login to your BetterLinux account and download the distribution of your choice, tarballs, and license file; then move them to your server. They appear as follows:
+
 
  betterlinux cpanel install
+
[[Image:Install 1.0 cPanel1.png]]
betterlinux install
+
 
 +
*If you are not automatically taken to the downloads page, click on the "Continue to Downloads" button at the bottom of the Order Complete page:
 +
 
 +
[[Image:Installstock4.png]]
 +
 
 +
*If you click on the "Continue to Downloads" button before verifying your email address, you will be taken to the Email Verification page: (You will not be able to go beyond this point without verifying your email)
 +
 
 +
[[Image:Install1.0cPanel25.png]]
 +
 
 +
*On the Downloads page, click the "License key" link to download your license key: (you'll see what to do with it momentarily)
 +
 
 +
[[Image:Install1.0cPanel2.png]]
 +
 
 +
Below the "License key" link are links to the documentation page you are reading now. (They are for those who didn't come here first)
 +
 
 +
*At the bottom of the Downloads page is the Installation Packages drop-down menu. From the menu, choose your distribution, release #, and architecture. Then click "view files":
 +
 
 +
[[Image:Install1.0cPanel27.png]]
 +
 
 +
*You will now see your chosen tarball. In this case, it would look like this:
 +
  betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
 +
 
 +
*Download the tarball by clicking the "Download" button next to it:
 +
[[Image:Install1.0cPanel3.png]]
 +
 
 +
*Move the License Key file and tarball to your server (see below for an easy way to do this).
 +
NOTE: Depending on your browser, the license key file may appear as either of these:
 
  license.key
 
  license.key
*Alternatively, the tarballs can be pulled down directly to the server by right clicking on the tarballs link in the downloads area to copy the url address and then using the wget command in combination with pasting in the tarball link. See the '''example''' below:
+
license.key.txt
  wget http://repo.betterlinux.com/release/betterlinux-cpanel-install-0.9.*-centos6*-x86_64.tar.gz
+
The name of the license key file will not affect its function provided the name is consistently used and the file's contents are accurate.
wget http://repo.betterlinux.com/release/betterlinux-install-0.9.*-centos6*-x86_64.tar.gz
+
 
*Unpack the tarballs.
+
 
tar –xvf betterlinux-cpanel-install-*
+
<b>Here is one easy way to get the tarball and license file onto your server:</b>
tar –xvf betterlinux-install-*
+
 
Unpacking the tarballs will create the directory .../betterlinux-install. Move the license.key file into the betterlinux-install directory.
+
*First, move the tarball to your server and unpack it since it creates the directory where you will put your license file.
*Change directory to betterlinux-install, and start bl-install.
+
 
  ./bl-install license.key
+
You can pull the tarball directly to the server:
*Once the installation is complete, reboot your server.
+
*Right click on the tarball link in Downloads and copy the url address:
*cd into the betterlinux-cpanel-install directory and run the bl-cpanel-install script:
+
 
 +
[[Image:Install1.0cPanel28.png]]
 +
 
 +
*In any directory on your server, paste the copied link after the wget command. It will look like this:  
 +
  wget http://repo.betterlinux.com/release/betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
 +
[[Image:Install1.0cPanel31.png]]
 +
 
 +
*Hit return and the tarball will download from the BetterLinux repository.
 +
 
 +
*Unpack the tarball: (this will create a long list of files)
 +
tar -xvf betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
 +
[[Image:Install1.0cPanel32.png]]
 +
 
 +
 
 +
*Recall that unpacking the tarball creates a new directory in the same directory where you unpack it. It's called <code>.../betterlinux-install-1.0.1-1</code>. <b>Put the license key file (license.key) into this new directory.</b>
 +
 
 +
 
 +
NOTE: If it's easier, instead of moving the file, you can just make a new file called <code>license.key</code> and then copy and paste the license key text into it, like this:
 +
 
 +
*Change directory to .../betterlinux-install-1.0.1-1
 +
*Using vi, create a file in this directory called license.key:
 +
vi license.key
 +
 
 +
[[Image:Installstock11.png]]
 +
 
 +
*When you hit enter, a file by that name is created and opened.
 +
*Type <code>i</code>, which starts input mode (allowing you to enter text).
 +
*Copy and paste the contents of your downloaded license key file into this new file:
 +
Copy: [[Image:Installstock12.png]]
 +
 
 +
Paste: [[Image:Installstock13.png]]
 +
 
 +
*Still in vi, hit <code>escape</code> (<code>esc</code>) to exit input mode and re-enter command mode.
 +
*Type <code>:x</code> and hit enter. (this saves and exits vi)<br><br>
 +
 
 +
<b>You are now ready to run the install program:</b>
 +
 
 +
 
 +
In the <code>.../betterlinux-install-1.0.1-1</code> directory, among other files you will see <code>bl-install*</code> and <code>bl-cpanel-install*</code>:
 +
 
 +
[[Image:Install1.0cPanel6.png]]
 +
 
 +
*Even though you are installing BetterLinux on a cPanel box, you have to run <code>bl-install*</code> first. If cPanel is on your box, the installer will auto-detect it and offer to install <code>bl-cpanel-install*</code> for you automatically after <code>bl-install*</code> runs.
 +
*From within the <code>.../betterlinux-install-1.0.1-1</code> directory, run <code>bl-install*</code> with the license key file right after it, like this: (remember to type your license key file name correctly if it has a different name)
 +
  ./bl-install-1.0.1-1 license.key
 +
 
 +
*When this portion of the install completes, you will see this notice:
 +
 
 +
[[Image:Install1.0cPanel8.png]]
 +
 
 +
*Before rebooting, you are given a chance to run the cPanel portion of the installer. Say "y" to this and hit enter. This will install cPanel configuration scripts and optionally replace MySQL with the BetterLinux version.
 +
 
 +
* If you say no, you can run the bl-cpanel-install script after the bl-install script completes.  This is done by running the following command:
 
  ./bl-cpanel-install
 
  ./bl-cpanel-install
*Installation and cPanel configuration are now complete.
+
 
CPU and IO throttling limits have been auto-configured and are now running on your system. If you want to make throttling limit changes, please refer to our online documentation.  http://docs.betterlinux.com
+
*As the cPanel configuration script runs, you are now prompted to answer a series of questions. You can see these questions with explanations here: [[1.0.0#Bl-cpanel-install_options_and_prompts|Bl-cpanel-install options and prompts]].
 +
 
 +
*After responding to all prompts and confirming your responses, you will have successfully configured BetterLinux. Cpud, iothrottled, and CloakFS will be running on the system.
  
 
===Installation for Stock CentOS 6.x===
 
===Installation for Stock CentOS 6.x===
For quick installation and updates, use the pre-built package for CentOS 6.0-6.3. Packages for other popular Linux distributions are coming soon.
+
For quick installation and updates, use the pre-built package for CentOS 6.x. Packages for other popular Linux distributions are coming soon.
  
 
Basic installation instructions:  
 
Basic installation instructions:  
*If you are running Litespeed, you will need to turn it off before beginning the upgrade. Once the upgrade is complete, you can turn it back on.
+
 
 
*Go to [https://www.betterlinux.com/ https://www.betterlinux.com] and click on the "Get It Now" link:
 
*Go to [https://www.betterlinux.com/ https://www.betterlinux.com] and click on the "Get It Now" link:
 
[[Image:Installstock1.png]]
 
[[Image:Installstock1.png]]
*When the BetterLinux Trial appears in your shopping cart, click the "Start Free Trial" link:
+
*When BetterLinux 1.0.1-1 appears in your shopping cart, click the "Checkout" link:
[[Image:Installstock2.png]]
+
 
 +
[[Image:Install1.0cPanel24.png]]
 
*Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."
 
*Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."
 
[[Image:Installstock3.png]]
 
[[Image:Installstock3.png]]
*You will see your order number and a "Continue to Downloads" button, which takes you to the download page, but you will not be able to download anything until you have verified your email. (see below)
+
 
 +
*You are now taken to the "Order Complete" page:
 +
 
 +
[[Image:Install1.0cPanel26.png]]
 +
 
 +
*Here, you are given your order number, which you will need to submit support tickets from the client area. You are also given notice that your email confirmation is on its way.
 +
 
 +
*Open your BetterLinux welcome email. It includes your email verification link and login information. Please note that your link will expire if not used quickly. The verification takes you to the Downloads page where you will be automatically logged into your account: (if you are not logged in for some reason, log in now)
 +
 
 +
[[Image:Install 1.0 cPanel1.png]]
 +
 
 +
*If you are not automatically taken to the downloads page, click on the "Continue to Downloads" button at the bottom of the Order Complete page:
 +
 
 
[[Image:Installstock4.png]]
 
[[Image:Installstock4.png]]
*You will receive a welcome email with your username and an email verification link. The verification link will also take you to the Downloads page where you will be automatically logged into your account. (if you are not logged in for some reason, just log in now)
 
*Download your Trial License Key:
 
[[Image:Installstock5.png]]
 
*From the menu, choose your distribution, release #, and architecture. Then click "view files":
 
[[Image:Installstock6.png]]
 
  
Because (in this example) we have chosen CentOS 6.3 x86_64, we can download a stock 6.3 tarball or a supplemental cPanel tarball for 6.3. Choose the stock tarball (the bottom one) by clicking the "Download" button.
+
*If you click on the "Continue to Downloads" button before verifying your email address, you will be taken to the Email Verification page: (You will not be able to go beyond this point without verifying your email)
  
[[Image:Installstock7.png]]
+
[[Image:Install1.0cPanel25.png]]
  
Move your Trial License Key file and your stock BetterLinux tarball to your server (see below for an easy way to do this). In this example, the files appear as follows:
+
*On the Downloads page, click the "License key" link to download your license key: (you'll see what to do with it momentarily)
  betterlinux-install-0.9.6-3-centos63-x86_64.tar
+
 
license.key
+
[[Image:Install1.0cPanel2.png]]
 +
 
 +
Below the "License key" link are links to the documentation page you are reading now. (They are for those who didn't come here first)
 +
 
 +
*At the bottom of the Downloads page is the Installation Packages drop-down menu. From the menu, choose your distribution, release #, and architecture. Then click "view files":
 +
 
 +
[[Image:Install1.0cPanel27.png]]
 +
 
 +
*You will now see your chosen tarball. In this case, it would look like this:
 +
  betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
 +
 
 +
*Download the tarball by clicking the "Download" button next to it:
 +
[[Image:Install1.0cPanel3.png]]
 +
 
 +
*Move the License Key file and tarball to your server (see below for an easy way to do this).
 
NOTE: Depending on your browser, the license key file may appear as either of these:
 
NOTE: Depending on your browser, the license key file may appear as either of these:
 
  license.key
 
  license.key
 
  license.key.txt
 
  license.key.txt
The name of the license key file will not affect its function provided it is always written correctly.<br><br>
+
The name of the license key file will not affect its function provided the name is consistently used and the file's contents are accurate.
<b>Here is one easy way to get the tarball and license file onto your server:</b><br>
+
 
(Note: First move and unpack the tarball since it creates the directory where you will put your license file.)<br><br>
+
 
This is how to pull down the tarball directly to the server:
+
<b>Here is one easy way to get the tarball and license file onto your server:</b>
 +
 
 +
*First, move the tarball to your server and unpack it since it creates the directory where you will put your license file.
 +
 
 +
You can pull the tarball directly to the server:
 
*Right click on the tarball link in Downloads and copy the url address:
 
*Right click on the tarball link in Downloads and copy the url address:
[[Image:Installstock10.png]]
 
*In any directory, paste the copied link after the wget command, like this:
 
wget http://repo.betterlinux.com/release/betterlinux-install-0.9.6-3-centos63-x86_64.tar.gz
 
[[Image:Installstock8.png]]
 
*Unpack the tarball:
 
tar -xvf betterlinux-install-0.9.6-3-centos63-x86_64.tar.gz
 
[[Image:Installstock9.png]]
 
  
*As earlier mentioned, unpacking the tarball creates a new directory called <code>.../betterlinux-install</code>.<br><br>
+
[[Image:Install1.0cPanel28.png]]
<b>Now put the license key file (<code>license.key</code>) into this new directory (<code>.../betterlinux-install</code>)</b><br>
+
 
(Instead of actually moving the file, you can just create a file called <code>license.key</code> and then copy and paste the license key text into it.)<br><br>
+
*In any directory on your server, paste the copied link after the wget command. It will look like this:
*Change directory to <code>.../betterlinux-install</code>
+
wget http://repo.betterlinux.com/release/betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
*Using vi, create a file in this directory called <code>license.key</code>.
+
 
 +
*Hit return and the tarball will download from the BetterLinux repository.
 +
 
 +
*Unpack the tarball: (this will create a long list of files)
 +
tar -xvf betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
 +
 
 +
*Recall that unpacking the tarball creates a new directory in the same directory where you unpack it. It's called <code>.../betterlinux-install-1.0.1-1</code>. <b>Put the license key file (license.key) into this new directory.</b>
 +
 
 +
 
 +
NOTE: If it's easier, instead of moving the file, you can just make a new file called <code>license.key</code> and then copy and paste the license key text into it, like this:
 +
 
 +
*Change directory to .../betterlinux-install-1.0.1-1
 +
*Using vi, create a file in this directory called license.key:
 
  vi license.key
 
  vi license.key
[[Image:Installstock11.png]]
+
 
 
*When you hit enter, a file by that name is created and opened.
 
*When you hit enter, a file by that name is created and opened.
 
*Type <code>i</code>, which starts input mode (allowing you to enter text).
 
*Type <code>i</code>, which starts input mode (allowing you to enter text).
*Copy and paste the contents of your downloaded license key file into this new file.
+
*Copy and paste the contents of your downloaded license key file into this new file:
[[Image:Installstock12.png]]
+
Copy: [[Image:Installstock12.png]]
Copied
+
 
[[Image:Installstock13.png]]
+
Paste: [[Image:Installstock13.png]]
Pasted
+
 
 
*Still in vi, hit <code>escape</code> (<code>esc</code>) to exit input mode and re-enter command mode.
 
*Still in vi, hit <code>escape</code> (<code>esc</code>) to exit input mode and re-enter command mode.
 
*Type <code>:x</code> and hit enter. (this saves and exits vi)<br><br>
 
*Type <code>:x</code> and hit enter. (this saves and exits vi)<br><br>
  
<b>You are now ready to run the install program:</b><br><br>
+
<b>You are now ready to run the install program:</b>
*From within the <code>.../betterlinux-install</code> directory, run <code>./bl-install</code> with the license key after it like this: (remember to type your license key file name correctly if it has a different name)
+
 
  ./bl-install license.key
+
In the <code>.../betterlinux-install-1.0.1-1</code> directory, among other files you will see <code>bl-install*</code> and <code>bl-cpanel-install*</code>:
[[Image:Installstock14.png]]
+
 
* You may be asked to approve the download size. Enter <code>y</code>:
+
[[Image:Install1.0cPanel6.png]]
[[Image:Installstock15.png]]  
+
 
*Once the installation is complete, reboot your server.
+
*Because you are not using cPanel, you can disregard the cPanel scripts.
# reboot
+
*From within the <code>.../betterlinux-install-1.0.1-1</code> directory, run <code>bl-install*</code> with the license key file right after it, like this: (remember to type your license key file name correctly if it has a different name)  
[[Image:Installstock16.png]]<br><br>
+
  ./bl-install-1.0.1-1 license.key
 +
 
 +
*When this portion of the install completes, you will see this notice:
 +
 
 +
[[Image:Install1.0cPanel33.png]]
 +
 
 +
*Reboot your system.
 +
 
 +
*You have now successfully installed BetterLinux. Cpud, iothrottled, and CloakFS are running on the system.
 +
 
 +
 
  
*To throttle MySQL threads by user, install BetterMySQL with yum:
 
yum --enablerepo=betterlinux-mysql install bl-mysql
 
*You can also install it from the local repo:
 
yum --enablerepo=betterlinux-mysql-local install bl-mysql
 
*Start MySQLd with this service command:
 
service mysqld start
 
*Remember to configure CPU and IO throttling limits. Please see the configuration sections for configuration options.
 
  
 
<!--===Upgrade From First BetterLinux Beta===
 
<!--===Upgrade From First BetterLinux Beta===
Line 279: Line 361:
 
*Installation and cPanel configuration are now complete.
 
*Installation and cPanel configuration are now complete.
 
At this point, CPU and I/O throttling limits have been auto-configured and the BetterLinux controllers are running on your system. If you want to make throttling limit changes, please refer to our online documentation. http://docs.betterlinux.com-->
 
At this point, CPU and I/O throttling limits have been auto-configured and the BetterLinux controllers are running on your system. If you want to make throttling limit changes, please refer to our online documentation. http://docs.betterlinux.com-->
 +
 +
===Upgrade from BL version 0.9.6-x  to BL 1.0.x-x (cPanel and non-cPanel)===
 +
 +
$ yum clean all
 +
$ wget http://repo.betterlinux.com/utilities/pre-1.0-upgrade-tool
 +
$ chmod 755 pre-1.0-upgrade-tool
 +
$ ./pre-1.0-upgrade-tool
 +
$ yum --disableexcludes=all upgrade betterlinux/*
 +
* Reboot to load the new BetterLinux kernel and modules.  This will auto restart the cpud, iothrottled, and cloakfs services.
  
 
===Uninstall BetterLinux with cPanel Integration===
 
===Uninstall BetterLinux with cPanel Integration===
Line 300: Line 391:
 
Once you know which BetterLinux version you are running, scroll down to the specific uninstall section that applies to you.
 
Once you know which BetterLinux version you are running, scroll down to the specific uninstall section that applies to you.
  
 +
====Uninstall BetterLinux 1.0.x-x with cPanel====
 +
The uninstallation process has three stages. Among other things, stage one replaces BetterLinux's version of MySQL with cPanel's version of MySQL, stage two ensures the system boots from the proper non-BetterLinux kernel, and stage three removes BetterLinux rpms from the system.
  
====Uninstall BetterLinux 1.0.0 with cPanel====
 
The uninstallation process has two stages. Among other things, stage one replaces BetterLinux's version of MySQL with cPanel's version of MySQL.
 
  
 
<b>STAGE ONE</b> of the uninstallation:
 
<b>STAGE ONE</b> of the uninstallation:
Line 308: Line 399:
 
Run the following uninstall command from anywhere within the file system:
 
Run the following uninstall command from anywhere within the file system:
  
  /etc/betterlinux/uninstall/bl-cpanel-uninstall
+
  /etc/betterlinux/bin/bl-cpanel-uninstall
 
[[Image:Uninstall1.0.0-cPanel1.png]]
 
[[Image:Uninstall1.0.0-cPanel1.png]]
  
Line 315: Line 406:
 
[[Image:Uninstall1.0.0-cPanel2.png]]
 
[[Image:Uninstall1.0.0-cPanel2.png]]
  
After these uninstall actions finish, you will see a list of what is being uninstalled.
+
After these uninstall actions finish, you will see a list of what is being uninstalled along with several notices and reminders. Shortly thereafter, you will see this:
 +
 
 +
[[Image:Uninstall1.0.0-cPanel3.png]]
 +
 
 +
Followed by this:
 +
 
 +
[[Image:Uninstall1.0.0-cPanel4.png]]
 +
 
 +
 
 +
Begin <b>STAGE TWO</b> of uninstallation: (This stage ensures the system boots from a pre-BetterLinux kernel after uninstallation)
 +
 
 +
Change directory to <code>/etc</code>
 +
cd /etc
 +
[[Image:Uninstall9.png]]
 +
 
 +
List files within <code>/etc</code> to see the <code>grub.conf</code> file.
 +
ls -l
 +
[[Image:Uninstall2.png]]
 +
 
 +
Using your favorite editor, open the <code>grub.conf</code> file: (this example uses vi)
 +
vi grub.conf
 +
[[Image:Uninstall10.png]]
 +
 
 +
Change the default kernel from the BetterLinux kernel to your most recent non-BetterLinux kernel. You do this by changing the value on the <code>default=</code> line from <code>0</code> to <code>1</code> (unless you have installed multiple BetterLinux kernels, in which case you may need to enter a number greater than <code>1</code>. '<code>0</code>' refers to the current kernel, '<code>1</code>' to the previous kernel, '<code>2</code>' the kernel before that, and so on:
 +
 
 +
[[Image:Uninstall11.png]]
 +
 
 +
Each kernel number line is preceded by the word <code>title</code>. In this way, you can tell how many available kernels there are. All BetterLinux kernels include the letters <code>bl</code> somewhere in the number. Choose the number of the first <code>title</code> line with a kernel number not containing <code>bl</code>.
 +
Here is an example of a kernel number containing the letters <code>bl</code> outlined in red:
 +
 
 +
[[Image:Uninstall17a.png]]
 +
 
 +
To change the <code>default=</code> value, move your cursor to the <code>default=</code> position with your arrow keys (or with keys j,k,h,&l [up,down,left,right]). Replace <code>0</code> with <code>1</code> (or with another number if needed). To do this, type <code>i</code>, which begins "insert mode," allowing you change text.
 +
 
 +
[[Image:Uninstall13.png]]
 +
 
 +
To end insert mode and re-enter command mode, hit the escape key (<code>esc</code>). Then type <code>:x</code>. This command saves your changes and exits vi.
 +
 
 +
[[Image:Uninstall14.png]]
 +
 
 +
As root user, reboot the system:
 +
reboot
 +
[[Image:Uninstall15.png]]
 +
 
 +
 
 +
<b>STAGE THREE</b> (Final stage): (This removes BetterLinux rpms from the system)
 +
 
 +
After rebooting, change directory to <code>/etc/betterlinux/bin</code>:
 +
cd /etc/betterlinux/bin
 +
 
 +
Once there, run this command:
 +
./bl-uninstall
 +
 
 +
[[Image:Uninstall1.0.0-cPanel5.png]]
 +
 
 +
The uninstaller will now perform a lengthy string of actions beginning with this notice:
 +
 
 +
[[Image:Uninstall1.0.0-cPanel6.png]]
 +
 
 +
 
 +
When they finish, you will see a termination notice like this:
 +
 
 +
[[Image:Uninstall1.0.0-cPanel7.png]]
 +
 
 +
You have now completed the uninstallation of BetterLinux 1.0.0 with cPanel.
  
 
====Uninstall BetterLinux 0.9.6-x with cPanel====
 
====Uninstall BetterLinux 0.9.6-x with cPanel====
Line 499: Line 654:
  
 
This command will create a new directory path and leave the .ko files in:  /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64/lib/modules/3.3.7-->
 
This command will create a new directory path and leave the .ko files in:  /build/linux-3.3.3/modules-3.3.3/RPMS/x86_64/lib/modules/3.3.7-->
 +
 +
== cPanel Integration ==
 +
 +
=== Overview ===
 +
 +
<p><code>Bl-cpanel-install</code> makes BetterLinux configuration easier. This script comes with the BetterLinux tarball found on our main repo:</p>
 +
http://repo.betterlinux.com/release/betterlinux-install-1.x.x-x-centos64-x86_64.tar.gz
 +
 +
<p>Download the tarball, unpack it, and you’ll see the script in the tarball directory: <code>../betterlinux-cpanel-install</code></p> (for instructions on downloading, unpacking, and installing the contents of this tarball, see [[1.0.0#1.0.0-x_Installation_on_CentOS_6.x_with_cPanel|1.0.0-x Installation on CentOS 6.x with cPanel]]; note also that you cannot run <code>bl-cpanel-install</code> until after running <code>bl-install</code>, a script in the same directory).
 +
 +
<p>From within the tarball directory, run <code>betterlinux-cpanel-install</code>:</p>
 +
./betterlinux-cpanel-install
 +
 +
<p>This installs an auto-updated cPanel hooks script to automatically do the following:</p>
 +
*Add or remove cPanel users and packages to or from BetterLinux <code>groups.conf</code> files.
 +
*Discover all cPanel packages, the users assigned to them, and create BetterLinux groups for each.
 +
 +
<p>You can then assign throttling limits by group for cpu, I/O, MySQL, Apache, bandwidth, processes, RSS memory, max time for CGI, cron, and ssh.
 +
 +
=== Bl-cpanel-install options and prompts ===
 +
 +
The script prompts for the following information: (Note: you will be able to review your selections at the end before final confirmation)
 +
Should I replace non-BetterLinux MySQL version 5.1.68 with BetterMySQL version 5.1.68? [y/N]
 +
Choosing "yes" removes cPanel's MySQL version and replaces it with the equivalent BetterLinux MySQL version.
 +
 +
Install BetterLinux Apache throttling? [y/N]
 +
Choosing "yes" installs the <code>mod-betterlinux</code> Apache module. It ties Apache threads to the users who generated them, making per-user cpu and I/O throttling possible.
 +
 +
Limit user network bandwidth? [y/N] y
 +
Bandwidth limit in Mb/s:
 +
This sets the BetterBandwidth <code>bandwidth_limit</code> parameter, which limits the amount of outgoing traffic per user.
 +
 +
Limit total number of simultaneous processes a cPanel user may run? [y/N]
 +
Process limit (recommended minimum: 50 processes):
 +
This sets the <code>process_limit</code> parameter. This is a per-user limit.
 +
 +
Log when user exceeds RSS memory threshold? (per-process RAM limits) [y/N]
 +
RSS (resident memory) threshold in MB:
 +
This sets the <code>memory_limit</code> parameter. When the per-user limit is exceeded, a script logs the event (via syslog).
 +
 +
Limit run time of user CGI process? [y/N]
 +
Maximum CGI run time in seconds (recommended min: 300 seconds; recommended max: 1800 seconds):
 +
This sets an element in the <code>time_monitor</code> parameter that kills CGI scripts running longer than the entered value.
 +
 +
Limit run time of user cron processes? [y/N] y
 +
Maximum cron process run time in seconds (recommended min: 1800 seconds):
 +
This sets and element in the <code>time_monitor</code> parameter that kills cron jobs running longer than the entered value.
 +
 +
Limit run time of user ssh processes? [y/N] y
 +
Maximum ssh process run time in seconds (recommended min: 1800 seconds):
 +
Sets an element in the <code>time_monitor</code> parameter that kills ssh-executed processes running longer than the entered value.
 +
 +
After setting selection, the script asks you to confirm your choices before it proceeds: (see the example confirmation below)
 +
 +
Confirm settings
 +
================
 +
  install-mysql            : 1
 +
  install-mod_betterlinux  : 1
 +
  bandwidth-limit          : 20mbit
 +
  process-limit            : 50
 +
  log-rss-limit            : 2000m
 +
  cgi-time-limit          : 900
 +
  cron-time-limit          : 900
 +
  ssh-time-limit          : 900
 +
  network-interface        : eth0
 +
 +
You also receive this notice:
 +
Once you proceed with this step, you will not be able to abort until it is finished.
 +
Finally, confirm your settings:
 +
Proceed with these settings? [y/N]
 +
 +
In addition to your chosen settings, other settings are auto-configured.
 +
 +
BetterLinux is now fully configured. Cpud, iothrottled, and CloakFS are running on the system.
 +
 +
=== Configure BetterLinux Standard Hooks for cPanel ===
 +
 +
The <code>/etc/betterlinux/cpanel/bl-hooks</code> script is registered when <code>bl-cpanel-install</code> is run.
 +
 +
<code>Bl-cphooks</code> creates a BetterLinux group for each cPanel package. Each package is matched with a similarly-named text file, which is populated with that package's uids.
 +
 +
These text files are found here:
 +
/etc/betterlinux/cpanel/users/[cpanel_package_name].txt
 +
 +
The hooks script updates each <code>[cpanel_package_name].txt</code> file whenever a user is added or removed from a cPanel package. These .txt files are referenced by <code>groups.conf</code> files in <code>/etc/betterlinux/cpud.conf.d</code> and <code>/etc/betterlinx/iothrottled.conf.d</code>.
 +
 +
==== Hook Management ====
 +
 +
The <code>bl-cphooks</code> script is registered when <code>bl-cpanel-install</code> is run. You can also manually register it using the following command as root:
 +
# /usr/local/cpanel/bin/manage_hooks add script /etc/betterlinux/cpanel/bl-cphooks
 +
 +
To see what hooks are registered, issue the command:
 +
# /usr/local/cpanel/bin/manage_hooks list
 +
 +
To remove this hook script, issue the command:
 +
# /usr/local/cpanel/bin/manage_hooks del script /etc/betterlinux/cpanel/bl-cphooks
 +
 +
==== bl-cphooks Configuration Initialization ====
 +
 +
Once registered, <code>bl-cphooks</code> is initialized as a part of running <code>bl-cpanel-install</code>. Initialization automatically sets and/or refreshes BetterLinux cPanel <code>.conf</code> files and <code>[cpanel-package-name].txt</code> files. But initialization can also be done manually by issuing this command as root:
 +
# /etc/betterlinux/cpanel/bl-cphooks --init
 +
 +
=== Configuration Files ===
 +
<code>bl-install</code> creates the following <code>.conf</code> files and keeps them synchronized with new or removed cPanel users, resellers, and packages:
 +
/etc/betterlinux/cpud.conf.d/cpanel.conf
 +
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf
 +
/etc/betterlinux/cpud.conf.d/cpanel-resellers.conf
 +
/etc/betterlinux/iothrottled.conf.d/cpanel.conf
 +
/etc/betterlinux/iothrottled.conf.d/cpanel-groups.conf
 +
 +
==== cpanel.conf for cpu throttling ====
 +
 +
==== cpanel-groups.conf for cpu throttling ====
 +
 +
==== cpanel-reseller.conf for CloakFS ====
 +
The <code>cpanel-reseller.conf</code> file is created only if there are cPanel resellers. This file's purpose is to enable resellers to see their users' directories as well as resellers' users to see resellers' branding directories. Please do not edit the contents of this file.
 +
Location: /etc/betterlinux/cpud.conf.d/cpanel-reseller.conf
 +
 +
==== cpanel.conf for I/O throttling ====
 +
 +
==== cpanel-groups.conf for I/O throttling ====
  
 
== Defining Groups ==
 
== Defining Groups ==
 
Understanding and defining groups is a key concept that is combined with the BetterLinux throttling parameter settings.  BetterLinux groups allows a system administrator to organize different types of users and programs by group.  For example, a hosting company may have various customer subscription levels with each level requiring resources from the server.  Having the ability to divide customers by group, allows BetterLinux limits to be set by customer group or cPanel packages.
 
Understanding and defining groups is a key concept that is combined with the BetterLinux throttling parameter settings.  BetterLinux groups allows a system administrator to organize different types of users and programs by group.  For example, a hosting company may have various customer subscription levels with each level requiring resources from the server.  Having the ability to divide customers by group, allows BetterLinux limits to be set by customer group or cPanel packages.
  
The BetterLinux cPanel integration (bl-cpanel-install) includes auto-group creation and updates based on cPanel packages. Each time a new user is added or removed to/from a cPanel package the corresponding BetterLinux group will be automatically updated. The BetterLinux cPanel install script will discover all the cPanel users on the system, create a user groups with the name of the cPanel package(s) and define these groups for both the BetterIO and BetterCPU bl_groups_main.conf and bl_groups_io.conf files.
+
The BetterLinux cPanel integration (bl-cpanel-install) installs custom cPanel script hooks that automate the creation and update of BetterLinux groups based on cPanel packages. Each time a new user is added/removed to/from a cPanel package the corresponding BetterLinux group will be automatically updated. The BetterLinux cPanel hook script will discover all the cPanel users on the system, create a user groups with the name of the cPanel package(s) and define these groups for both the BetterIO and BetterCPU groups.conf or cpanel-groups.conf and groups.conf or cpanel-groups.conf files.
  
Below is an example of the syntax the hooks would write to the configuration files bl_groups_main.conf for cpud and bl_groups_io.conf for iothrottled if there were three cPanel packages named RedHost-Level1, 2, and 3:
+
Below is an example of the syntax the hook script would write to the configuration files groups.conf or cpanel-groups.conf for cpud and groups.conf or cpanel-groups.conf for iothrottled if there were three cPanel packages named SampleHost-Level1, 2, and 3:
  
   uid  RedHost-Level1 /etc/betterlinux/cpanel/users/ RedHost-Level1.txt
+
   uid  SampleHost-Level1 /etc/betterlinux/cpanel/users/ SampleHost-Level1.txt
   uid  RedHost-Level2 /etc/betterlinux/cpanel/users/ RedHost-Level2.txt
+
   uid  SampleHost-Level2 /etc/betterlinux/cpanel/users/ SampleHost-Level2.txt
   uid  RedHost-Level3 /etc/betterlinux/cpanel/users/ RedHost-Level3.txt
+
   uid  SampleHost-Level3 /etc/betterlinux/cpanel/users/ SampleHost-Level3.txt
  
The BetterLinux cPanel hooks feature will create uid files that match the cPanel package names (if they exist) and will contain the cPanel package corresponding UIDs.  These uid file names (located in /etc/betterlinux/cpanel/users) contain the UIDs and have names that match the group or cPanel package names.
+
The BetterLinux cPanel hook script will also create uid files that match the cPanel package names (if they exist) and will contain the cPanel package corresponding UIDs.  These uid file names (located in /etc/betterlinux/cpanel/users) contain the UIDs and have names that match the cPanel package names.
  
Example of uid files created if there were three cPanel packages named RedHost-Level1, 2, and, 3:
+
Example of uid files created if there were three cPanel packages named SampleHost-Level1, 2, and, 3:
  
  /etc/betterlinux/cpanel/users/RedHost-Level1-uids.txt
+
  /etc/betterlinux/cpanel/users/SampleHost-Level1.txt
  /etc/betterlinux/cpanel/users/RedHost-Level2-uids.txt
+
  /etc/betterlinux/cpanel/users/SampleHost-Level2.txt
  /etc/betterlinux/cpanel/users/RedHost-Level3-uids.txt
+
  /etc/betterlinux/cpanel/users/SampleHost-Level3.txt
  
 
If for any reason the cPanel packages are not in the groups .conf files, run this command to refresh them:
 
If for any reason the cPanel packages are not in the groups .conf files, run this command to refresh them:
 +
*NOTE: This command will remove any .conf file customizations.
  
 
  /etc/betterlinux/cpanel/bl-cphooks --init
 
  /etc/betterlinux/cpanel/bl-cphooks --init
  
The non-cPanel installation script will not create users groups, they need to be created manually by the system administrator.
+
The bl-install script script will not create users groups, they need to be created manually by the system administrator.
  
 
Groups .conf FILE LOCATIONS:
 
Groups .conf FILE LOCATIONS:
*/etc/betterlinux/cpud.conf.d/bl_groups_main.conf
+
*/etc/betterlinux/cpud.conf.d/groups.conf or cpanel-groups.conf
*/etc/betterlinux/iothrottled.conf.d/bl_groups_io.conf
+
*/etc/betterlinux/iothrottled.conf.d/groups.conf or cpanel-groups.conf
  
 
===Group Types and Syntax===
 
===Group Types and Syntax===
Line 652: Line 929:
 
   
 
   
 
===Fixed Groups===
 
===Fixed Groups===
The fixed groups feature is a predefined set of group names that will always or never throttle.  Their names and functions are set, but they have not yet been populated with users or programs. In order to configure these groups, they need to be populated with UIDs or Programs.  The concept behind fixed groups is to create a predefined group of users or programs that will aways be or never be throttled. These parameters apply to the bl_groups_main.conf and bl_groups_io.conf files for BetterCPU and BetterIO.
+
The fixed groups feature is a predefined set of group names that will always or never throttle.  Their names and functions are set, but they have not yet been populated with users or programs. In order to configure these groups, they need to be populated with UIDs or Programs.  The concept behind fixed groups is to create a predefined group of users or programs that will aways be or never be throttled. These parameters apply to the groups.conf or cpanel-groups.conf and groups.conf or cpanel-groups.conf files for BetterCPU and BetterIO.
  
 
Configuration Location:
 
Configuration Location:
  /etc/betterlinux/cpud.conf.d/bl_groups_main.conf
+
  /etc/betterlinux/cpud.conf.d/groups.conf or cpanel-groups.conf
  /etc/betterlinux/iothrottled.conf.d/bl_groups_io.conf
+
  /etc/betterlinux/iothrottled.conf.d/groups.conf or cpanel-groups.conf
  
 
Syntax for fixed group examples:
 
Syntax for fixed group examples:
Line 779: Line 1,056:
 
Configure the CPU daemon through a series of configuration files located in <code>/etc/betterlinux/cpud.conf.d/</code>
 
Configure the CPU daemon through a series of configuration files located in <code>/etc/betterlinux/cpud.conf.d/</code>
  
*<code>bl_main.conf</code>
+
[[1.0.0#cpud_Configuration_File_Names|Configuration File Names]]
*<code>bl_groups_main.conf</code>
+
 
+
  
 
BetterCPU’s primary configuration file is <code>bl_main.conf</code>. This file has basic configuration parameters.
 
BetterCPU’s primary configuration file is <code>bl_main.conf</code>. This file has basic configuration parameters.
Line 832: Line 1,107:
 
   
 
   
 
  average_type S # [Default = S]
 
  average_type S # [Default = S]
 
"busy_nonjail_idle" defines the non-jail idle value at which to enter "busy" mode.  In "busy" mode cpud is more aggressive in jailing and in the killing of old processes.
 
 
busy_nonjail_idle 2 # [Default = 2]
 
 
"busy_num_R_nonjailed" defines the non-jail run queue (R) value at which to  enter "busy" mode.  In "busy" mode cpud is more aggressive in jailing and in
 
the killing of old processes.
 
NOTE: This parameter name is case-sensitive.
 
 
busy_num_R_nonjailed 4 # [Default = 4]
 
 
"jail_min_idle" sets the minimum percent of idle cpu to maintain in the jail.
 
jail_min_idle 5 # [Default = 5]
 
 
"nonjail_min_idle" sets the minimum percent of idle cpu to maintain in the non-jail.
 
 
nonjail_min_idle 25 # [Default = 25]
 
  
 
=== Starting and stopping BetterCPU ===
 
=== Starting and stopping BetterCPU ===
Line 1,107: Line 1,365:
 
/etc/betterlinux/bin/pid_running_too_long.pl (will report when a program is running longer than the number of seconds defined in time_monitor)
 
/etc/betterlinux/bin/pid_running_too_long.pl (will report when a program is running longer than the number of seconds defined in time_monitor)
  
====tim_monitor auto-configuration for cPanel servers====
+
====time_monitor auto-configuration for cPanel servers====
 
The bl-cpanel-install script will auto-configure the time_monitor feature with the number of seconds entered at the time of installation in /etc/betterlinux/cpud.conf.d/bl_main.conf.  In the examples below, 900 seconds was entered by the user during the bl-cpanel-install script. The user has a cPanel package called AcmeHost_Level1.  Any child programs of httpd, chroottpd, and crond owned by the users in the AcmeHost_Level1 package are limited to run for 900 seconds.  If they exceed the 900 seconds, kill_pid.pl script will run every 30 seconds until the program is killed. The child of programs excluded from the 900 seconds are bash, ksh93, tcsh, and zsh.:
 
The bl-cpanel-install script will auto-configure the time_monitor feature with the number of seconds entered at the time of installation in /etc/betterlinux/cpud.conf.d/bl_main.conf.  In the examples below, 900 seconds was entered by the user during the bl-cpanel-install script. The user has a cPanel package called AcmeHost_Level1.  Any child programs of httpd, chroottpd, and crond owned by the users in the AcmeHost_Level1 package are limited to run for 900 seconds.  If they exceed the 900 seconds, kill_pid.pl script will run every 30 seconds until the program is killed. The child of programs excluded from the 900 seconds are bash, ksh93, tcsh, and zsh.:
 
*time_monitor group_type=user group=AcmeHost_Level1 seconds=900 child_of=/usr/local/apache/bin/httpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list
 
*time_monitor group_type=user group=AcmeHost_Level1 seconds=900 child_of=/usr/local/apache/bin/httpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list
Line 1,293: Line 1,551:
  
 
==CloakFS File System Cloaking==
 
==CloakFS File System Cloaking==
BetterLinux 1.0.0-x has a new and improved version of CloakFS that provides cloaking via the kernel instead of the pam module. This approach greatly simplifies the solution and eliminates the risks associated with supporting the ever changing versions of user space applications such as apache (suexec, suphp), ssh, pam, etc. Now CloakFS is easier to configure and supports all user space applications and versions.
+
BetterLinux 1.0.0-x has a new and improved version of CloakFS that cloaks via the kernel instead of a pam module. This simpler solution makes it easy to support all user space applications and versions (apache, suexec, suphp, ssh, pam, etc.) without risking constant changes. CloakFS is also now easier to configure.
  
The CloakFS feature provides an additional layer of security for web hosting companies that want to prevent their users from seeing each others directories, files, and processes. It will protect users from each other even if insecure permissions are granted on directories or files. For instance, if cPanel user A tried to read/write any file owned by cPanel user B, user A would be denied even if the permissions on the file were chmod 777. CloakFS auto detects the home directories for all cPanel users so no additional configuration is required.
+
CloakFS secures web hosting systems by preventing users from seeing each other’s directories, files, and processes. It protects users from each other even if file/directory permissions are insecure. For example, cPanel user A couldn’t read/write user B’s file even with <code>chmod 777</code> permission. CloakFS also auto-detects all users’ home directories so no extra configuration is required.
  
CloakFS will also prevent scripts that try to list directories from running via SSH, Cron Jobs, Apache (suexec, suphp) and symlinks. A user can only see his or her own $HOME directory, with the exception files outside of home owned by that user. For example, if “sampleuser” creates “samplefile” in /tmp and then lists all files in /tmp, he will see only “samplefile” and not the /tmp files of other users on the box.
+
CloakFS prevents directory-listing scripts from running via ssh, cron jobs, Apache (suexec, suphp) and symlinks. Further, a user sees only his or her own <code>$HOME</code> directory and other files he or she owns. For example, if User A creates “samplefile” in <code>/tmp</code> and then lists all <code>/tmp</code> files, he will see only “samplefile,” but not other users’ <code>/tmp</code> files.
  
 
===Configuration Files===
 
===Configuration Files===
Line 1,310: Line 1,568:
  
 
Then all users in the user_group1 will see themselves at the file system level.
 
Then all users in the user_group1 will see themselves at the file system level.
 +
 +
===cPanel Resellers===
 +
The BetterLinux cPanel hook script handles the complexities for reseller users.  This feature allows the reseller to see the users they have created, but blocks the users from seeing their reseller with the exception of the resellers branding directory.  The script will do the following:
 +
*Finds resellers and creates a group for each
 +
*Finds resellers users and creates a group for each user
 +
*Configures these groups and the cloakfs_exception parameter in the /etc/betterlinux/cpud.conf.d/resellers.conf file.
 +
 +
The syntax for the cloaks_exception parameter is as follows:
 +
  cloakfs_exception unprotect_groups=[group-name] unprotect_groups=[group_name]
 +
  
 
===Start / Stop CloakFS===
 
===Start / Stop CloakFS===
Line 1,430: Line 1,698:
 
When artificially restricting I/O using iothrottled, this can cause programs that trigger off of loadavg values to trigger incorrectly. This option introduces the ability to pause process execution due to iothrottled's artificial I/O channel throttling as a part of the CPU scheduler rather than due to I/O. This is done using an interruptible sleep, which means that while the process is being suspended due to I/O, the suspension is not due to an actual I/O bottleneck but due to an artificial bottleneck caused by
 
When artificially restricting I/O using iothrottled, this can cause programs that trigger off of loadavg values to trigger incorrectly. This option introduces the ability to pause process execution due to iothrottled's artificial I/O channel throttling as a part of the CPU scheduler rather than due to I/O. This is done using an interruptible sleep, which means that while the process is being suspended due to I/O, the suspension is not due to an actual I/O bottleneck but due to an artificial bottleneck caused by
 
  interruptible_sleeps 1 # [Default = 1]
 
  interruptible_sleeps 1 # [Default = 1]
 +
 +
"auto_wakeup" Under certain conditions I/O throttling may contribute to increasing the memory pressure of the system, especially with stacked block devices such as loopback mounted filesystem, LVM, software RAID, iSCSI, etc.
 +
 +
The option auto_wakup reduces the risk of page allocation failures by temporarily awakening all the tasks previously put to sleep due to the enforced I/O limits. This allows the kernel to drain some pending I/O requests and give more breathing room to the system in terms of memory utilization.
 +
 +
I/O limits are then enforced as normal when the memory utilization in the system returns back to a safe level.
 +
 +
auto_wakeup supports three different configurations:
 +
*auto_wakeup = 0: disable auto-wakeup
 +
*auto_wakeup = 1: wake up all tasks when kernel is running out of emergency memory (reduce the probability of GFP_ATOMIC allocation failures) -- This is the recommended and default setting.
 +
*auto_wakeup = 2: wake up all tasks when kernel is running out of emergency memory or when journal I/O is submitted (meta-data I/O)
 +
 +
Option 2 should be used only if the kernel reports some hung task timeout traces. They can happen when there is too much pressure on filesystem locks like intensive meta-data
 +
operations.
 +
 +
auto_wakeup 1 # [Default 1]
  
 
=== Setting I/O Bandwidth Limits Per-User or by User-Group ===
 
=== Setting I/O Bandwidth Limits Per-User or by User-Group ===
Line 1,946: Line 2,230:
  
 
=== Will BetterLinux be compatible with cPanel/WHM and if so will that include plugins for WHM? ===
 
=== Will BetterLinux be compatible with cPanel/WHM and if so will that include plugins for WHM? ===
*BetterLinux is currently developing integration and plug-ins for cPanel and WHM.  The targeted release date is end of November 2012.
+
*BetterLinux is now supports cPanel and we expect to release our WHM plug-in by early May.
  
 
=== How to install blstat? ===
 
=== How to install blstat? ===
Line 1,962: Line 2,246:
 
  blstat
 
  blstat
  
The initial screen is the i/o throttling stats. Pressing "c" will switch blstat to the cpu throttling screen. Pressing "i" will switch the screen back to i/o throttling. Pressing "q" will exit blstat.
+
The initial screen is the i/o throttling stats. Press "c" to switch blstat to the cpu throttling screen. Press "i" to switch the screen back to i/o throttling. Press any key to exit blstat.
  
 
==System Requirements==
 
==System Requirements==
Line 2,279: Line 2,563:
 
|-
 
|-
 
| [[Main_Page#Advanced_Settings|average_type]] || IO
 
| [[Main_Page#Advanced_Settings|average_type]] || IO
 +
|-
 +
| [[Main_Page#Auto_Wakeup|auto_wakeup]] || IO
 
|-
 
|-
 
| [[Main_Page#Manual_BetterBandwidth_Configuration|bandwidth_limit]] || BANDWIDTH/CPU
 
| [[Main_Page#Manual_BetterBandwidth_Configuration|bandwidth_limit]] || BANDWIDTH/CPU
Line 2,458: Line 2,744:
 
<div style="margin-left: 4em;">Will add up the load for all the non-jail cores and divide by the total number of non_jail cores.</div><br><br>
 
<div style="margin-left: 4em;">Will add up the load for all the non-jail cores and divide by the total number of non_jail cores.</div><br><br>
 
<div style="text-indent: 2em;"><u>Configuration</u> <u>File</u></div><br>
 
<div style="text-indent: 2em;"><u>Configuration</u> <u>File</u></div><br>
 +
 +
== cpud Configuration File Names==
 +
===cpud Stock Linux Config Files===
 +
/etc/betterlinux/cpud.conf.d
 +
*betterlinux.conf
 +
*groups.conf
 +
 +
===cpud cPanel Config Files===
 +
/etc/betterlinux/cpud.conf.d
 +
*[package-name]_package.conf
 +
*cpanel-allusers.conf
 +
*cpanel.conf
 +
*cpanel-groups.conf
 +
 +
==iothrottled Configuration File Names==
 +
===iothrottled Stock Linux Config Files===
 +
/etc/betterlinux/iothrottled.conf.d
 +
*betterlinux.conf
 +
*betterlinux-io.conf
 +
 +
===iothrottled cPanel Config Files===
 +
/etc/betterlinux/iothrottled.conf.d
 +
*cpanel.conf
 +
*cpanel-groups.conf

Latest revision as of 15:21, 7 June 2013

Welcome to the BetterLinux Product Documenation wiki. This is your one stop resource for everything BetterLinux - from initial setup, configuration, and fine tuning all the features. Please check here first before submitting a help desk ticket.

Contents

BetterLinux Introduction

BetterLinux is a collection of system resource management and monitoring tools intended for hosting providers, data centers, SaaS companies, and cloud environments. With it, you can control use and allocation of CPU, memory, MySQL, device I/O bandwidth, and IP bandwidth resources. Individual users and processes that exceed set resource limits can be isolated from other system users and throttled as necessary.

Historically, process management on Linux has involved adjusting processes’ priorities and nice levels. More recently, Linux system administrators have used technologies like full virtualization, paravirtualization, Unix containers, or other similar conventions to segregate system resource usage.

These controls, while useful, are quite crude and do not address the core issue of guaranteeing and prioritizing operating system resources for critical applications across an entire system. Poorly-behaved processes can still monopolize the system and cause overall performance problems.

Recent Linux kernels have introduced the concept of control groups, or cgroups. A cgroup is used to group processes together in a hierarchical structure in order to divide resources among different process groups or subsystems. For example, one cgroup might be configured to hold all web-server related processes, with a second cgroup configured to hold system backup processes.

BetterLinux augments cgroups functionality with automatic categorization of processes based on owner or process name. This coupled with BetterLinux’s unique ability to “autoscale” resource limits based on constantly changing system loads allows you to guarantee and prioritize system resources with unprecedented confidence and precision.

BetterLinux can operate either simply or with impressive complexity as may suit your needs and interests. Each of its modules comes with configuration files wherein you can create groups of users or processes, set various limits, turn options on or off, and change logging practices, etc. For configuration changes to take effect, you must either reload the .conf changes or restart the module daemon as follows:

Service cpud restart		[Restart daemon with updated changes.]
Service cpud condrestart		[Load new .conf changes & restart daemon only if daemon is already running.]
Service cpud reload		[Load .conf file settings without restarting daemon.]


BetterLinux consists of four modules: BetterCPU, CloakFS, BetterIO, BetterMySQL, and BetterBandwidth. See their brief descriptions below.

BetterCPU

BetterCPU (a CPU controller) limits access to a server’s CPU resources by individual users and processess. Over-consuming processes are isolated to a “jail” where they are throttled as necessary to prevent them from hogging CPU resources. Processes operating within limits continue to work efficiently without interference.

BetterCPU Features:

  • Fairly allocate CPU resources and guarantee resources for specific users/processes
  • Temporarily “jail” over-consuming users/processes to prevent crashes and congestion
  • Automatically reallocate unused dedicated resources to cut waste
  • Trim CPU usage peaks with customized thresholds for groups and individuals
  • Stop memory overuse, system lock-ups, and runaway processes
  • Automatically kill zombies, hung or unauthorized processes, and memory hogs
  • Multiply server density while cutting heat, power, & hardware costs; increase revenue
  • Supplement set physical RAM memory limits with swap space
  • Monitor CPU, user, and core behavior in real time; see detailed logs and summaries
  • One configuration file can hold settings for multiple systems

CloakFS

CloakFS secures the file system by hiding users from each other, including others’ files, directories, and processes. A user can see only his or her own $HOME directory, with the exception files outside of home owned by that user. For example, if “sampleuser” creates “samplefile” in /tmp and then lists all files in /tmp, he will see only “samplefile” and not the /tmp files of other users on the box.

CloakFS Features:

  • Users cannot access or see other users on the system
  • Optionally show part of an otherwise hidden file system to certain users
  • Admins define what is cloaked and uncloaked by user
  • Ensures only safe binaries can run.
  • Limit script execution via Blocks suexec, suPHP, symlink

BetterIO

BetterIO manages block device I/O utilization. You can create limits for individual users and groups, for individual programs, for specific devices by percent usage, and even for mount points (like NFS, iSCSI, and more). You can also limit by service time (svctime), access wait (await), or a combination of both. BetterIO can “auto-scale” system limits when resources go used to prevent waste. “Auto-scaling” responds to highly configurable conditions and use ranges in addition to recent user activity.

BetterIO Features:

  • Create customized individual and group limits for users, programs, devices, and mount points
  • Control seek times, I/O operations, and disk bandwidth use for a balanced system without monopolizers.
  • Prevent bottlenecks and timeouts
  • Eliminate waste by autoscaling limits when resources go unused
  • Multiply server density
  • Thoroughly monitor per-user I/O use details and totals by device of origin in real time
  • Reduce need for more hardware or reductions in tenants per server
  • Reduce heat and power consumption
  • One configuration file can hold settings for multiple systems
  • Integration with cPanel user management for web hosting providers

BetterMySQL

BetterMySQL assigns the responsibility for queries to individual Linux users instead of to the generic MySQL user to enable individualized MySQL throttling. BetterMySQL also “auto-scales” system limits when resources go used to prevent waste.

BetterMySQL Features:

  • Map MySQL users to Linux users; all queries now tied to specific users for individualized throttling
  • Control abusive MySQL users, processes, and threads
  • Compatible even on nonstandard platforms/directories
  • Automatically reallocate unused dedicated resources to cut waste
  • Thoroughly monitor individual MySQL use and relevant CPU use in real time; see detailed summaries
  • One configuration file can hold settings for multiple systems

BetterBandwidth

BetterBandwidth links traffic to specific Linux users, permitting group and individual limits without need for individual IP addresses. You can also force a user’s traffic on a shared server to originate from your main shared IP. BetterBandwidth has extensive bandwidth limiting monitoring abilities.


Bandwidth Limiting Features:

  • Link outgoing bandwidth to specific Linux users
  • Limit group and individual bandwidth use without assigning individual IP addresses
  • Throttle only those who exceed set limits
  • Increase profits with more users, fewer support calls, and reduced need for IPV4 IP address blocks
  • Specific users’ traffic on shared servers now originates from IP addresses you choose, not your main shared IP
  • Have more bandwidth per user or more users on your bandwidth
  • Monitor individual bandwidth use in real time with detailed info on addresses, ports, sockets, protocols, and packets; create custom summaries
  • One configuration file can hold settings for multiple systems


Install, Upgrade, & Uninstall (newest to oldest)

1.0.x-x Installation on CentOS 6.x with cPanel

For quick installation and updates, use the pre-built package for CentOS 6.x. Packages for other popular Linux distributions are coming soon.

Basic installation instructions:

Installstock1.png

  • When BetterLinux 1.0.1-1 appears in your shopping cart, click the "Checkout" link:

Install1.0cPanel24.png

  • Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order":

Installstock3.png

  • You are now taken to the "Order Complete" page:

Install1.0cPanel26.png

  • Here, you are given your order number, which you will need to submit support tickets from the client area. You are also given notice that your email confirmation is on its way.
  • Open your BetterLinux welcome email. It includes your email verification link and login information. Please note that your link will expire if not used quickly. The verification takes you to the Downloads page where you will be automatically logged into your account: (if you are not logged in for some reason, log in now)

Install 1.0 cPanel1.png

  • If you are not automatically taken to the downloads page, click on the "Continue to Downloads" button at the bottom of the Order Complete page:

Installstock4.png

  • If you click on the "Continue to Downloads" button before verifying your email address, you will be taken to the Email Verification page: (You will not be able to go beyond this point without verifying your email)

Install1.0cPanel25.png

  • On the Downloads page, click the "License key" link to download your license key: (you'll see what to do with it momentarily)

Install1.0cPanel2.png

Below the "License key" link are links to the documentation page you are reading now. (They are for those who didn't come here first)

  • At the bottom of the Downloads page is the Installation Packages drop-down menu. From the menu, choose your distribution, release #, and architecture. Then click "view files":

Install1.0cPanel27.png

  • You will now see your chosen tarball. In this case, it would look like this:
betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
  • Download the tarball by clicking the "Download" button next to it:

Install1.0cPanel3.png

  • Move the License Key file and tarball to your server (see below for an easy way to do this).

NOTE: Depending on your browser, the license key file may appear as either of these:

license.key
license.key.txt

The name of the license key file will not affect its function provided the name is consistently used and the file's contents are accurate.


Here is one easy way to get the tarball and license file onto your server:

  • First, move the tarball to your server and unpack it since it creates the directory where you will put your license file.

You can pull the tarball directly to the server:

  • Right click on the tarball link in Downloads and copy the url address:

Install1.0cPanel28.png

  • In any directory on your server, paste the copied link after the wget command. It will look like this:
wget http://repo.betterlinux.com/release/betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz

Install1.0cPanel31.png

  • Hit return and the tarball will download from the BetterLinux repository.
  • Unpack the tarball: (this will create a long list of files)
tar -xvf betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz

Install1.0cPanel32.png


  • Recall that unpacking the tarball creates a new directory in the same directory where you unpack it. It's called .../betterlinux-install-1.0.1-1. Put the license key file (license.key) into this new directory.


NOTE: If it's easier, instead of moving the file, you can just make a new file called license.key and then copy and paste the license key text into it, like this:

  • Change directory to .../betterlinux-install-1.0.1-1
  • Using vi, create a file in this directory called license.key:
vi license.key

Installstock11.png

  • When you hit enter, a file by that name is created and opened.
  • Type i, which starts input mode (allowing you to enter text).
  • Copy and paste the contents of your downloaded license key file into this new file:

Copy: Installstock12.png

Paste: Installstock13.png

  • Still in vi, hit escape (esc) to exit input mode and re-enter command mode.
  • Type :x and hit enter. (this saves and exits vi)

You are now ready to run the install program:


In the .../betterlinux-install-1.0.1-1 directory, among other files you will see bl-install* and bl-cpanel-install*:

Install1.0cPanel6.png

  • Even though you are installing BetterLinux on a cPanel box, you have to run bl-install* first. If cPanel is on your box, the installer will auto-detect it and offer to install bl-cpanel-install* for you automatically after bl-install* runs.
  • From within the .../betterlinux-install-1.0.1-1 directory, run bl-install* with the license key file right after it, like this: (remember to type your license key file name correctly if it has a different name)
./bl-install-1.0.1-1 license.key
  • When this portion of the install completes, you will see this notice:

Install1.0cPanel8.png

  • Before rebooting, you are given a chance to run the cPanel portion of the installer. Say "y" to this and hit enter. This will install cPanel configuration scripts and optionally replace MySQL with the BetterLinux version.
  • If you say no, you can run the bl-cpanel-install script after the bl-install script completes. This is done by running the following command:
./bl-cpanel-install
  • As the cPanel configuration script runs, you are now prompted to answer a series of questions. You can see these questions with explanations here: Bl-cpanel-install options and prompts.
  • After responding to all prompts and confirming your responses, you will have successfully configured BetterLinux. Cpud, iothrottled, and CloakFS will be running on the system.

Installation for Stock CentOS 6.x

For quick installation and updates, use the pre-built package for CentOS 6.x. Packages for other popular Linux distributions are coming soon.

Basic installation instructions:

Installstock1.png

  • When BetterLinux 1.0.1-1 appears in your shopping cart, click the "Checkout" link:

Install1.0cPanel24.png

  • Enter your personal information, create a password, and agree to terms of service. Then click "Complete Order."

Installstock3.png

  • You are now taken to the "Order Complete" page:

Install1.0cPanel26.png

  • Here, you are given your order number, which you will need to submit support tickets from the client area. You are also given notice that your email confirmation is on its way.
  • Open your BetterLinux welcome email. It includes your email verification link and login information. Please note that your link will expire if not used quickly. The verification takes you to the Downloads page where you will be automatically logged into your account: (if you are not logged in for some reason, log in now)

Install 1.0 cPanel1.png

  • If you are not automatically taken to the downloads page, click on the "Continue to Downloads" button at the bottom of the Order Complete page:

Installstock4.png

  • If you click on the "Continue to Downloads" button before verifying your email address, you will be taken to the Email Verification page: (You will not be able to go beyond this point without verifying your email)

Install1.0cPanel25.png

  • On the Downloads page, click the "License key" link to download your license key: (you'll see what to do with it momentarily)

Install1.0cPanel2.png

Below the "License key" link are links to the documentation page you are reading now. (They are for those who didn't come here first)

  • At the bottom of the Downloads page is the Installation Packages drop-down menu. From the menu, choose your distribution, release #, and architecture. Then click "view files":

Install1.0cPanel27.png

  • You will now see your chosen tarball. In this case, it would look like this:
betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
  • Download the tarball by clicking the "Download" button next to it:

Install1.0cPanel3.png

  • Move the License Key file and tarball to your server (see below for an easy way to do this).

NOTE: Depending on your browser, the license key file may appear as either of these:

license.key
license.key.txt

The name of the license key file will not affect its function provided the name is consistently used and the file's contents are accurate.


Here is one easy way to get the tarball and license file onto your server:

  • First, move the tarball to your server and unpack it since it creates the directory where you will put your license file.

You can pull the tarball directly to the server:

  • Right click on the tarball link in Downloads and copy the url address:

Install1.0cPanel28.png

  • In any directory on your server, paste the copied link after the wget command. It will look like this:
wget http://repo.betterlinux.com/release/betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
  • Hit return and the tarball will download from the BetterLinux repository.
  • Unpack the tarball: (this will create a long list of files)
tar -xvf betterlinux-install-1.0.1-1-centos64-x86_64.tar.gz
  • Recall that unpacking the tarball creates a new directory in the same directory where you unpack it. It's called .../betterlinux-install-1.0.1-1. Put the license key file (license.key) into this new directory.


NOTE: If it's easier, instead of moving the file, you can just make a new file called license.key and then copy and paste the license key text into it, like this:

  • Change directory to .../betterlinux-install-1.0.1-1
  • Using vi, create a file in this directory called license.key:
vi license.key
  • When you hit enter, a file by that name is created and opened.
  • Type i, which starts input mode (allowing you to enter text).
  • Copy and paste the contents of your downloaded license key file into this new file:

Copy: Installstock12.png

Paste: Installstock13.png

  • Still in vi, hit escape (esc) to exit input mode and re-enter command mode.
  • Type :x and hit enter. (this saves and exits vi)

You are now ready to run the install program:

In the .../betterlinux-install-1.0.1-1 directory, among other files you will see bl-install* and bl-cpanel-install*:

Install1.0cPanel6.png

  • Because you are not using cPanel, you can disregard the cPanel scripts.
  • From within the .../betterlinux-install-1.0.1-1 directory, run bl-install* with the license key file right after it, like this: (remember to type your license key file name correctly if it has a different name)
./bl-install-1.0.1-1 license.key
  • When this portion of the install completes, you will see this notice:

Install1.0cPanel33.png

  • Reboot your system.
  • You have now successfully installed BetterLinux. Cpud, iothrottled, and CloakFS are running on the system.



Upgrade from BL version 0.9.6-x to BL 1.0.x-x (cPanel and non-cPanel)

$ yum clean all
$ wget http://repo.betterlinux.com/utilities/pre-1.0-upgrade-tool
$ chmod 755 pre-1.0-upgrade-tool
$ ./pre-1.0-upgrade-tool
$ yum --disableexcludes=all upgrade betterlinux/*
  • Reboot to load the new BetterLinux kernel and modules. This will auto restart the cpud, iothrottled, and cloakfs services.

Uninstall BetterLinux with cPanel Integration

How to uninstall BetterLinux may depend on which version you have. Beta versions 0.9.4 and later allow you to verify which version you are running by changing directory to /etc/betterlinux, listing its folders and files, and opening the RELEASE file. Change the directory with this command:

cd /etc/betterlinux

Uninstall1.png

List the folders and files within /etc/betterlinux with this command:

ls -l

Uninstall2.png

In the resulting list, you will see a file called RELEASE in all caps. Read its contents with this command:

cat RELEASE

Uninstall3.png

You will then see which BetterLinux version you are using. It will look similar to this:

Uninstall4.png

Once you know which BetterLinux version you are running, scroll down to the specific uninstall section that applies to you.

Uninstall BetterLinux 1.0.x-x with cPanel

The uninstallation process has three stages. Among other things, stage one replaces BetterLinux's version of MySQL with cPanel's version of MySQL, stage two ensures the system boots from the proper non-BetterLinux kernel, and stage three removes BetterLinux rpms from the system.


STAGE ONE of the uninstallation:

Run the following uninstall command from anywhere within the file system:

/etc/betterlinux/bin/bl-cpanel-uninstall

Uninstall1.0.0-cPanel1.png

You will see this, followed by a long series of uninstall actions:

Uninstall1.0.0-cPanel2.png

After these uninstall actions finish, you will see a list of what is being uninstalled along with several notices and reminders. Shortly thereafter, you will see this:

Uninstall1.0.0-cPanel3.png

Followed by this:

Uninstall1.0.0-cPanel4.png


Begin STAGE TWO of uninstallation: (This stage ensures the system boots from a pre-BetterLinux kernel after uninstallation)

Change directory to /etc

cd /etc

Uninstall9.png

List files within /etc to see the grub.conf file.

ls -l

Uninstall2.png

Using your favorite editor, open the grub.conf file: (this example uses vi)

vi grub.conf

Uninstall10.png

Change the default kernel from the BetterLinux kernel to your most recent non-BetterLinux kernel. You do this by changing the value on the default= line from 0 to 1 (unless you have installed multiple BetterLinux kernels, in which case you may need to enter a number greater than 1. '0' refers to the current kernel, '1' to the previous kernel, '2' the kernel before that, and so on:

Uninstall11.png

Each kernel number line is preceded by the word title. In this way, you can tell how many available kernels there are. All BetterLinux kernels include the letters bl somewhere in the number. Choose the number of the first title line with a kernel number not containing bl. Here is an example of a kernel number containing the letters bl outlined in red:

Uninstall17a.png

To change the default= value, move your cursor to the default= position with your arrow keys (or with keys j,k,h,&l [up,down,left,right]). Replace 0 with 1 (or with another number if needed). To do this, type i, which begins "insert mode," allowing you change text.

Uninstall13.png

To end insert mode and re-enter command mode, hit the escape key (esc). Then type :x. This command saves your changes and exits vi.

Uninstall14.png

As root user, reboot the system:

reboot

Uninstall15.png


STAGE THREE (Final stage): (This removes BetterLinux rpms from the system)

After rebooting, change directory to /etc/betterlinux/bin:

cd /etc/betterlinux/bin

Once there, run this command:

./bl-uninstall

Uninstall1.0.0-cPanel5.png

The uninstaller will now perform a lengthy string of actions beginning with this notice:

Uninstall1.0.0-cPanel6.png


When they finish, you will see a termination notice like this:

Uninstall1.0.0-cPanel7.png

You have now completed the uninstallation of BetterLinux 1.0.0 with cPanel.

Uninstall BetterLinux 0.9.6-x with cPanel

The uninstallation process has two stages. Among other things, stage one replaces BetterLinux's version of MySQL with cPanel's version of MySQL.

STAGE ONE of the uninstallation:

If you are running 0.9.6-3, you have one extra uninstall step. Run this command from anywhere in the file system:

sed -i 's/RELEASE=3/RELEASE=2/' /etc/betterlinux/RELEASE

All versions (0.9.6-x) should then run the following uninstall command from anywhere within the file system:

/etc/betterlinux/scripts/bl-cpanel-install --uninstall

Uninstall5.png

You will see this, followed by a long series of uninstall actions:

Uninstall6.png

After these uninstall actions finish, you will see a list of what is being uninstalled. You are then asked if you want to keep your configuration files, which you may wish to use with a future install. Answer y or n

Uninstall7.png

You will now see that stage one of the uninstallation process is complete:

Uninstall8.png


Begin STAGE TWO of uninstallation:

Change directory to /etc

cd /etc

Uninstall9.png

List files within /etc to see the grub.conf file.

ls -l

Uninstall2.png

Using your favorite editor, open the grub.conf file: (this example uses vi)

vi grub.conf

Uninstall10.png

Change the default kernel from the BetterLinux kernel to your most recent non-BetterLinux kernel. You do this by changing the value on the default= line from 0 to 1 (unless you have installed multiple BetterLinux kernels, in which case you may need to enter a number greater than 1. '0' refers to the current kernel, '1' to the previous kernel, '2' the kernel before that, and so on:

Uninstall11.png

Each kernel number line is preceded by the word title. In this way, you can tell how many available kernels there are. All BetterLinux kernels include the letters bl somewhere in the number. Choose the number of the first title line with a kernel number not containing bl. Here is an example of a kernel number containing the letters bl outlined in red:

Uninstall17a.png

To change the default= value, move your cursor to the default= position with your arrow keys (or with keys j,k,h,&l [up,down,left,right]). Replace 0 with 1 (or with another number if needed). To do this, type i, which begins "insert mode," allowing you change text.

Uninstall13.png

To end insert mode and re-enter command mode, hit the escape key (esc). Then type :x. This command saves your changes and exits vi.

Uninstall14.png

As root user, reboot the system:

reboot

Uninstall15.png

After rebooting, change directory to /etc/betterlinux/scripts:

cd /etc/betterlinux/scripts

Once there, run this command:

./bl-install --uninstall

Uninstall16.png

The uninstaller will now perform a lengthy string of actions. When they finish, you will see a termination notice:

Uninstall18.png


Uninstall BetterLinux on Stock CentOS 6.x

  • edit /etc/grub.conf and select the previously installed kernel
  • reboot
  • change directory to …/betterlinux-install and run the uninstall command:
./bl-install --uninstall
  • Remove the BetterLinux MySQL (bl-mysql) packages:
rpm -qa "bl-mysql*"

or for 0.9.6-1 release or newer the bl-mysql packages are renamed to betterlinux-mysql-*

rpm -qa "better*"

This will list the betterlinux rpms left on the system for removal using the rpm -e command in the example below:

rpm -e --nodeps bl-mysql-server-5.1.61-4.el6.bl0.9.5.x86_64
rpm -e --nodeps bl-mysql-libs-5.1.61-4.el6.bl0.9.5.x86_64
rpm -e --nodeps bl-mysql-5.1.61-4.el6.bl0.9.5.x86_64
  • Now to install stock MySQL run the yum command:
yum install mysql
yum install mysql-server



cPanel Integration

Overview

Bl-cpanel-install makes BetterLinux configuration easier. This script comes with the BetterLinux tarball found on our main repo:

http://repo.betterlinux.com/release/betterlinux-install-1.x.x-x-centos64-x86_64.tar.gz

Download the tarball, unpack it, and you’ll see the script in the tarball directory: ../betterlinux-cpanel-install

(for instructions on downloading, unpacking, and installing the contents of this tarball, see 1.0.0-x Installation on CentOS 6.x with cPanel; note also that you cannot run bl-cpanel-install until after running bl-install, a script in the same directory).

From within the tarball directory, run betterlinux-cpanel-install:

./betterlinux-cpanel-install

This installs an auto-updated cPanel hooks script to automatically do the following:

  • Add or remove cPanel users and packages to or from BetterLinux groups.conf files.
  • Discover all cPanel packages, the users assigned to them, and create BetterLinux groups for each.

You can then assign throttling limits by group for cpu, I/O, MySQL, Apache, bandwidth, processes, RSS memory, max time for CGI, cron, and ssh.

Bl-cpanel-install options and prompts

The script prompts for the following information: (Note: you will be able to review your selections at the end before final confirmation)

Should I replace non-BetterLinux MySQL version 5.1.68 with BetterMySQL version 5.1.68? [y/N]

Choosing "yes" removes cPanel's MySQL version and replaces it with the equivalent BetterLinux MySQL version.

Install BetterLinux Apache throttling? [y/N]

Choosing "yes" installs the mod-betterlinux Apache module. It ties Apache threads to the users who generated them, making per-user cpu and I/O throttling possible.

Limit user network bandwidth? [y/N] y
Bandwidth limit in Mb/s:

This sets the BetterBandwidth bandwidth_limit parameter, which limits the amount of outgoing traffic per user.

Limit total number of simultaneous processes a cPanel user may run? [y/N]
Process limit (recommended minimum: 50 processes):

This sets the process_limit parameter. This is a per-user limit.

Log when user exceeds RSS memory threshold? (per-process RAM limits) [y/N]
RSS (resident memory) threshold in MB:

This sets the memory_limit parameter. When the per-user limit is exceeded, a script logs the event (via syslog).

Limit run time of user CGI process? [y/N]
Maximum CGI run time in seconds (recommended min: 300 seconds; recommended max: 1800 seconds):

This sets an element in the time_monitor parameter that kills CGI scripts running longer than the entered value.

Limit run time of user cron processes? [y/N] y
Maximum cron process run time in seconds (recommended min: 1800 seconds):

This sets and element in the time_monitor parameter that kills cron jobs running longer than the entered value.

Limit run time of user ssh processes? [y/N] y
Maximum ssh process run time in seconds (recommended min: 1800 seconds):

Sets an element in the time_monitor parameter that kills ssh-executed processes running longer than the entered value.

After setting selection, the script asks you to confirm your choices before it proceeds: (see the example confirmation below)

Confirm settings
================
  install-mysql            : 1
  install-mod_betterlinux  : 1
  bandwidth-limit          : 20mbit
  process-limit            : 50
  log-rss-limit            : 2000m
  cgi-time-limit           : 900
  cron-time-limit          : 900
  ssh-time-limit           : 900
  network-interface        : eth0

You also receive this notice:

Once you proceed with this step, you will not be able to abort until it is finished.

Finally, confirm your settings:

Proceed with these settings? [y/N]

In addition to your chosen settings, other settings are auto-configured.

BetterLinux is now fully configured. Cpud, iothrottled, and CloakFS are running on the system.

Configure BetterLinux Standard Hooks for cPanel

The /etc/betterlinux/cpanel/bl-hooks script is registered when bl-cpanel-install is run.

Bl-cphooks creates a BetterLinux group for each cPanel package. Each package is matched with a similarly-named text file, which is populated with that package's uids.

These text files are found here:
/etc/betterlinux/cpanel/users/[cpanel_package_name].txt

The hooks script updates each [cpanel_package_name].txt file whenever a user is added or removed from a cPanel package. These .txt files are referenced by groups.conf files in /etc/betterlinux/cpud.conf.d and /etc/betterlinx/iothrottled.conf.d.

Hook Management

The bl-cphooks script is registered when bl-cpanel-install is run. You can also manually register it using the following command as root:

# /usr/local/cpanel/bin/manage_hooks add script /etc/betterlinux/cpanel/bl-cphooks

To see what hooks are registered, issue the command:

# /usr/local/cpanel/bin/manage_hooks list

To remove this hook script, issue the command:

# /usr/local/cpanel/bin/manage_hooks del script /etc/betterlinux/cpanel/bl-cphooks

bl-cphooks Configuration Initialization

Once registered, bl-cphooks is initialized as a part of running bl-cpanel-install. Initialization automatically sets and/or refreshes BetterLinux cPanel .conf files and [cpanel-package-name].txt files. But initialization can also be done manually by issuing this command as root:

# /etc/betterlinux/cpanel/bl-cphooks --init

Configuration Files

bl-install creates the following .conf files and keeps them synchronized with new or removed cPanel users, resellers, and packages:

/etc/betterlinux/cpud.conf.d/cpanel.conf
/etc/betterlinux/cpud.conf.d/cpanel-groups.conf
/etc/betterlinux/cpud.conf.d/cpanel-resellers.conf
/etc/betterlinux/iothrottled.conf.d/cpanel.conf
/etc/betterlinux/iothrottled.conf.d/cpanel-groups.conf

cpanel.conf for cpu throttling

cpanel-groups.conf for cpu throttling

cpanel-reseller.conf for CloakFS

The cpanel-reseller.conf file is created only if there are cPanel resellers. This file's purpose is to enable resellers to see their users' directories as well as resellers' users to see resellers' branding directories. Please do not edit the contents of this file.

Location: /etc/betterlinux/cpud.conf.d/cpanel-reseller.conf

cpanel.conf for I/O throttling

cpanel-groups.conf for I/O throttling

Defining Groups

Understanding and defining groups is a key concept that is combined with the BetterLinux throttling parameter settings. BetterLinux groups allows a system administrator to organize different types of users and programs by group. For example, a hosting company may have various customer subscription levels with each level requiring resources from the server. Having the ability to divide customers by group, allows BetterLinux limits to be set by customer group or cPanel packages.

The BetterLinux cPanel integration (bl-cpanel-install) installs custom cPanel script hooks that automate the creation and update of BetterLinux groups based on cPanel packages. Each time a new user is added/removed to/from a cPanel package the corresponding BetterLinux group will be automatically updated. The BetterLinux cPanel hook script will discover all the cPanel users on the system, create a user groups with the name of the cPanel package(s) and define these groups for both the BetterIO and BetterCPU groups.conf or cpanel-groups.conf and groups.conf or cpanel-groups.conf files.

Below is an example of the syntax the hook script would write to the configuration files groups.conf or cpanel-groups.conf for cpud and groups.conf or cpanel-groups.conf for iothrottled if there were three cPanel packages named SampleHost-Level1, 2, and 3:

 uid  SampleHost-Level1 /etc/betterlinux/cpanel/users/ SampleHost-Level1.txt
 uid  SampleHost-Level2 /etc/betterlinux/cpanel/users/ SampleHost-Level2.txt
 uid  SampleHost-Level3 /etc/betterlinux/cpanel/users/ SampleHost-Level3.txt

The BetterLinux cPanel hook script will also create uid files that match the cPanel package names (if they exist) and will contain the cPanel package corresponding UIDs. These uid file names (located in /etc/betterlinux/cpanel/users) contain the UIDs and have names that match the cPanel package names.

Example of uid files created if there were three cPanel packages named SampleHost-Level1, 2, and, 3:

/etc/betterlinux/cpanel/users/SampleHost-Level1.txt
/etc/betterlinux/cpanel/users/SampleHost-Level2.txt
/etc/betterlinux/cpanel/users/SampleHost-Level3.txt

If for any reason the cPanel packages are not in the groups .conf files, run this command to refresh them:

  • NOTE: This command will remove any .conf file customizations.
/etc/betterlinux/cpanel/bl-cphooks --init

The bl-install script script will not create users groups, they need to be created manually by the system administrator.

Groups .conf FILE LOCATIONS:

  • /etc/betterlinux/cpud.conf.d/groups.conf or cpanel-groups.conf
  • /etc/betterlinux/iothrottled.conf.d/groups.conf or cpanel-groups.conf

Group Types and Syntax

Groups can be defined by UID (Linux user), Program, Interface, and Port.

Syntax: uid NAME UID_ITEM[,UID_ITEM,...] prog NAME PROG_ITEM[,PROG_ITEM,...] port NAME PORT_ITEM[,PORT_ITEM,...] interface NAME INTERFACE_ITEM[,INTERFACE_ITEM,...]

Examples:

uid bl_users 500-1000
prog bl_progs /usr/bin/rsync,/bin/tar
interface bl_interfaces eth0
port bl_ports 1024-65545

Where:

  • uid: used to define a group of users
  • prog: used to define a group of programs
  • port: used to define a group of ports
  • interface: used to define an interface like eth0

NAME: is a custom name that identifies the group

UID_ITEM: can be one of the following:

  • USERNAME: name of a UNIX user
  • UID: a UNIX user uid
  • RANGE: an ID ranges and/or names (i.e., 1000-1999)
  • FILE: external file that contains a comma-separated list of USERNAME,
  • UID, and RANGE elements
  • COMMAND: external command to execute, the output can contain a comma/space/tab/newline separated list of USERNAME, UID, and RANGE elements

PROG_ITEM: can be one of the following:

  • PATH: absolute path of an executable file
  • COMMAND: external command to execute, the output can contain a comma/space/tab/newline separated list of PATH elements

INTERFACE_ITEM: can be one of the following:

  • INTERFACE: a name of a network interface (i.e, eth0,eth1,wlan0, ...)
  • FILE: an external that contains a comma/space/tab/newline separated list of INTERFACE items
  • COMMAND: an external command to execute, the output can contain a comma/space/tab/newline separated list of INTERFACE items

PORT_ITEM: can be one of the following:

  • PORT: a port number
  • RANGE: a port range in the form START-END (i.e, 1024-65545)
  • FILE: external file that contains a comma-separated list of PORT or RANGE elements
  • COMMAND: external command to execute, the output can contain a comma/space/tab/newline separated list of PORT and/or RANGE elements

User Groups

To create a group, choose a group name, then populate it with UIDs, UID ranges, user names, or files of UIDs or usernames, as follows:

Group with a UID range example: Syntax: uid {group name} {uid range}

Example:

uid example-group 0-499
uid example-group 800-		(this includes all UIDs from 800 and above)

Group with multiple UID ranges: Syntax: uid {group name} {uid range}[,{uid range2}...]

Example:

uid example-group 0-499,800-850

Group with UID range and a username: Syntax: uid {group name} {uid range},[,{username},...] Example:

uid example-group 0-499,nfsnobody

Group with multiple usernames: Syntax: uid {group name} {username1},[,{username2}...]

Example:

uid example-group bluser1,bluser2

Group with files of UIDs or usernames: (files must use a comma delimited list) Syntax: uid {group name} {file path}[,{file path2}...]

Example:

uid example-group /etc/passwd
uid cpanel_users /var/cpanel/users/username

Groups made from external file using regular expresion and the cut command (must be a comma delimited list). This is especially handy if you are pulling a list from the passwords file or some other file that keeps lists of created users by a program like cPanel.

Syntax: uid {group name} `cut -d: -f1 {file path}`

Example:

uid bl_local_users `cut -d: -f1 /etc/passwd`

Result: Using ":" as a delimiter, this syntax reads field one (the UID field) from the passwd file, making these UIDs members of the group "bl_local_users."

Program Groups

BetterLinux has multiple features for controlling limits for programs.

To create your program groups, choose a group name, then populate it with programs by location and name separated by a comma with no spaces, as follows:

Group with programs example: Syntax: prog {group name} {/location/program-name,/location/program-name2,/location/program-name3}

Example:

prog example-group-programs /usr/sbin/sshd,/usr/bin/rsync

Group with files of programs: (files must use a comma delimited list) Syntax: uid {group name} {file path}[,{file path2}...]

Example:

prog example-group-programs /etc/betterlinux/list-programs.txt

Port Groups

The ports group is tied to the BetterBandwidth connections feature which is currently on hold and not part of the BetterLinux release.

Interface Groups

The BetterBandwidth feature requires an interface group name for the network interface to throttle. To create an interface group, chose a group name, then populate it with the network interfaces for throttling as follows.

Group with interface examples: Syntax: interface {group name} {interface name} Example:

interface interface-example-group eth0,eth1

Group with interface example using an external file that contains the interfaces: Syntax: interface {group name} {file path},{file path2} Example:

interface interface-example-group /etc/betterlinux/interfaces.txt

Fixed Groups

The fixed groups feature is a predefined set of group names that will always or never throttle. Their names and functions are set, but they have not yet been populated with users or programs. In order to configure these groups, they need to be populated with UIDs or Programs. The concept behind fixed groups is to create a predefined group of users or programs that will aways be or never be throttled. These parameters apply to the groups.conf or cpanel-groups.conf and groups.conf or cpanel-groups.conf files for BetterCPU and BetterIO.

Configuration Location:

/etc/betterlinux/cpud.conf.d/groups.conf or cpanel-groups.conf
/etc/betterlinux/iothrottled.conf.d/groups.conf or cpanel-groups.conf

Syntax for fixed group examples:

BetterCPU:

uid ALWAYS_THROTTLE_CPU_GROUP {members}   #[Default: empty]
uid NEVER_THROTTLE_CPU_GROUP {members}   #[Default: empty]
prog ALWAYS_THROTTLE_CPU_GROUP {programs} #[Default: empty]
prog NEVER_THROTTLE_CPU_GROUP {programs}   #[Default: empty]

BetterIO:

uid ALWAYS_THROTTLE_GROUP {members}   #[Default: empty]
uid NEVER_THROTTLE_GROUP {members}   #[Default: empty]
prog ALWAYS_THROTTLE_GROUP {programs} #[Default: empty]
prog NEVER_THROTTLE_GROUP {programs}   #[Default: empty]

BetterCPU (cpud) Fixed Throttling Examples

Throttle UIDs BetterCPU (cpud):

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid1}[,{uid2}...]

Example:

uid ALWAYS_THROTTLE_CPU_GROUP 13,15,19 

(Always throttles UIDs 13,15,19)

Throttle UID range:

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid range}

Example:

uid ALWAYS_THROTTLE_CPU_GROUP 500-999

(Always throttles UIDs from 500-999)

Throttle usernames:

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {username1}[,{username2}...]

Example:

uid ALWAYS_THROTTLE_CPU_GROUP bl_user1,bl_user2

(Always throttles listed users)

Throttle a file of UIDs: (where the file contains a delimited list of UIDs)

Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {file path}

Example:

uid ALWAYS_THROTTLE_CPU_GROUP /etc/passwd

Throttled Programs:

Syntax: prog ALWAYS_THROTTLE_CPU_GROUP {program1}[,{program2}...]

Example:

prog ALWAYS_THROTTLE_CPU_GROUP program1,program2

(Always throttles programs programs1,program2)

Throttle a File of Programs: (where the file contains a delimited list of programs)

Syntax: prog ALWAYS_THROTTLE_CPU_GROUP {file path}

Example:

prog ALWAYS_THROTTLE_CPU_GROUP /file/path/list/of/programs

"NEVER_THROTTLE_CPU_GROUP" lets you list members that will never be throttled. It accepts UIDs, UID ranges, user names, or files of UIDs, programs, or files containing programs. See the above "ALWAYS_THROTTLE_CPU_GROUP" syntax and examples substituting in "NEVER_THROTTLE_CPU_GROUP".

BetterIO (iothrottled) Fixed Throttling Examples

Throttle UIDs BetterIO (iothrottled):

Syntax: uid ALWAYS_THROTTLE_GROUP {uid1}[,{uid2}...]

Example:

uid ALWAYS_THROTTLE_GROUP 13,15,19 

(Always throttles UIDs 13,15,19)

Throttle UID range:

Syntax: uid ALWAYS_THROTTLE_GROUP {uid range}

Example:

uid ALWAYS_THROTTLE_GROUP 500-999

(Always throttles UIDs from 500-999)

Throttle usernames:

Syntax: uid ALWAYS_THROTTLE_GROUP {username1}[,{username2}...]

Example:

uid ALWAYS_THROTTLE_GROUP bl_user1,bl_user2

(Always throttles listed users)

Throttle a file of UIDs: (where the file contains a delimited list of UIDs)

Syntax: uid ALWAYS_THROTTLE_GROUP {file path}

Example:

uid ALWAYS_THROTTLE_GROUP /etc/passwd

Throttled Programs:

Syntax: prog ALWAYS_THROTTLE_GROUP {program1}[,{program2}...]

Example:

prog ALWAYS_THROTTLE_GROUP program1,program2

(Always throttles programs programs1,program2)

Throttle a File of Programs: (where the file contains a delimited list of programs)

Syntax: prog ALWAYS_THROTTLE_GROUP {file path}

Example:

prog ALWAYS_THROTTLE_GROUP /file/path/list/of/programs

"NEVER_THROTTLE_GROUP" lets you list members that will never be throttled. It accepts UIDs, UID ranges, user names, or files of UIDs, programs, or files containing programs. See the above "ALWAYS_THROTTLE_GROUP" syntax and examples substituting in "NEVER_THROTTLE_GROUP".

BetterCPU General Usage

BetterCPU maximizes CPU and memory resources for shared hosting environments while preventing user abuse. Quickly find and control CPU hogs and determine who needs higher-paying accounts with increased resources. To get started, define user groups (match them to cPanel packages if applicable) and set throttling limits for each group.

Configure the CPU daemon through a series of configuration files located in /etc/betterlinux/cpud.conf.d/

Configuration File Names

BetterCPU’s primary configuration file is bl_main.conf. This file has basic configuration parameters.

Basic Settings

Basic configuration begins with three parameters you can add to bl_main.conf file:

/etc/betterlinux/cpud.conf.d/bl_main.conf

These are the parameter names:

sleep_time_ms
user_average_periods
cpu_average_periods

CPU usage is calculated using averages of samples taken over time. BetterCPU profiles usage by process and by user on a regular interval to determine which processes and users are utilizing the CPU the most, and if those usages exceed permitted thresholds, then BetterCPU manages those processes in a way that limits their ability to impact overall system performance.

The sleep_time parameter is used to determine the number of milliseconds elapse between BetterCPU’s snapshots.

The averages for user and CPU utilization (user_average_periods and cpu_average_periods, respectively) are calculated based on the number of periods (or samples) taken. The defaults for these parameters are:

sleep_time_ms 1000
user_average_periods 2
cpu_average_periods 2

This means snapshots are averaged every 2 periods, and each period is 500 ms long. As a result, the averages are calculated for a 1 second basis every 500 ms.

When evaluating the cpu_average_periods and user_average_periods, if a process or user is found to be exceeding predetermined utilization thresholds, the user will be “jailed”.

Advanced Settings

The advanced settings used to configure BetterCPU can be added to /etc/betterlinux/cpud.conf.d/bl_main.conf and consist of:

"add_user_time" defines the number of seconds between checks for new users to be managed by cpud.

add_user_time 3600	# [Default = 3600]

"run_priority" defines the priority for CPUd. Valid values range from -100 to 39.

run_priority 0	# [Default = 0]

"throttle_group" defines the group id that CPUd should use to set the group ownership on all cgroups it creates. Other programs that use cgroups can use a different gid to own and identify their cgroups, or they can share the gid that CPUd uses to access the cgroups created for and used by CPUd.

throttle_group 0	# [Default = 0]

"in_buf_sz" defines the size (in bytes) of the buffer used for buffering taskstats information.

in_buf_sz 1000	# [Default = 1000]

"average_type" determines how averages of data samples are computed. If average_type is set to 'S' (Simple) then averages are the arithmetic mean of the the samples. If average_type is 'E' (Exponential) then the average is exponential in nature, meaning that newer samples are weighted more heavily in the calculation. If an exponential average is chosen then cpud's decisions will be determined by more recent system behavior, which will generally result in more rapid response to changing conditions. Choosing a simple average will make cpud respond in a way determined by longer term conditions. NOTE: This parameter's value is case-sensitive

average_type S	# [Default = S]

Starting and stopping BetterCPU

After the installation is complete, a reboot of your system is required to load your BetterLinux kernel and modules. Once your system is rebooted, the BetterCPU deamon is not running by default and requires a manual start using service unless the the server is a cPanel server and the betterlinux-cpanel-install script has been run. If so, the daemons will be started after the installation. Example:

service cpud start

Restarting and stoping examples:

service cpud stop
service cpud restart

Using the condrestart will restart the daemon but only if it's running. Example:

service cpud condrestart:

When changes are made to the configuration files, the BetterCPU daemon must reload the new settings. This is can be done by either running the service restart option or by running the service reload command. Example:

service cpud reload

Understanding Jail Cores

Before limits are configured, it's important to understand BetterCPU jails. The jails are elastic by nature and only exist if they are needed by the BetterCPU controller to temporarily limit an abusive user process. Think of process jailing as analogous to law enforcement:

  • A user is a “citizen”
  • The performance thresholds are the “law”
  • BetterCPU enforces the laws

A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.

If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.

When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.

Configuration of the jail is done using the parameters:

jail_cnt
max_jail_cnt

Commenting both of the parameters out sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1. BetterCPU then auto-scales the jails the system needs based on demand for jail cores. However, the jail cores will never take a priority over non-jail cores.

The jail_cnt can be defined using two different methods:

  • If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jai_cnt. For example:
jail_cnt 2

The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems. If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.

  • is expressed as one or more pairs of numbers. For example:
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail; an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system. The 32 core system will be assigned 6 cores as the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails, and is written in the same manner as the jail_cnt parameter. Setting the max_jail_cnt parameter will fix the ultimate size of the jail. Fixing the jail size will determines the amount of the cpu utilization abusive users will share in the jail core(s).

example:

max_jail_cnt 4

By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.

Defining BetterCPU Throttling Limits

The BetterCPU throttling limits are defined by a percentage of one core for each user.

Once the jail cores are defined, BetterCPU provide two throttling parameters that define when a user is moved into and out of the jail core(s). The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set jail throttling limits for groups of uids defined in bl_groups_main.conf. The cpu_percent sets the percentage a user (uid) can use of one core on the system. The values for cpu_percent in start_throttling and stop_throttling are based on one core and are not adjusted when cores are added or removed from the jail.

When a throttled user’s total utilization drops below the stop_throttling for cpu_percent and "seconds" settings, then the process is removed from the jail.

For example, consider an 8-core system with the following values configured:

start_throttling group=example-group cpu_percent=50,time=3
stop_throttling group=example-group cpu_percent=25,time=10

A user whose processes exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.

Defining Groups

Groups are automatically defined by the bl-cpanel-install script. Installations on non-cPanel systems need to be defined manually. For more details on groups please see the Defining Groups section of the documentation.

Selecting Managed Users

There are a few ways users can be selected for control by BetterLinux.

Parameters in the bl_main.conf file can be used to define a minimum and maximum UID value that define a range of identities to be managed by BetterCPU. There also are parameters that define UIDs or users for inclusion or exclusion from being managed by the suite.

A simpler way to define managed users is through the use of the /etc/betterlinux/cpud.conf.d/bl_groups_main.conf file. This file contains entries that define a group name and a list of users within that group. For example, the line:

uid cpu_limit testuser,testuser1,testuser2,testuser3,502

Adds the users in the list to the group cpu_limit. A numeric value in this line is interpreted as a UID value which is translated by the system to a user name.

How can I tell BetterCPU is managing my system?

A few tools have been mentioned already - top and htop will both show CPU utilization figures, memory utilization figures, and a breakdown of utilization by CPU core. BetterLinux comes with it's own top-like program called blstat.


If logging is enabled, an easy way to see that process are in being throttled in the jail core(s) is to monitor the file defined in log_cpud for jail expansion messages. When a user is jailed, the log will reflect that:

Cpud managing system.png

The “Throttling User” message indicates that a user’s processes are being moved into the jail. One can also see messages indicating the jail cgroup is being created as well as information about the number of running programs that user has, their UID, and other information about that user.


In htop, when a user is jailed, if the display of the CPU column (which displays the CPU number a process is running on) is enabled, it is easy to observe a user being moved into the jail. Here is a user using excessive CPU resources prior to throttling:

Cpu resources prior to jailing.png


Here is that same user after BetterCPU has jailed them for excessive CPU use:

Cpud jailed them.png

Notice how the first image shows the utilization on all 4 CPUs of the system and how separate instances of the “jailme” process is running on each CPU. After the user is jailed, all of that user’s processes have been moved to CPU 4.

User Blame Security

This setting enables BetterMySQL throttling and authorizes a program to apply usage blame to other users on the system. This is used for programs like mysql and apache, which run under one uid, to divide the usage up by the individual users of the program.

In the example below we will authorize apache and mysql to assign blame to the correct user on the system. Add one line per program / user:

approved_accounting_program program=/usr/sbin/httpd user=apache
approved_accounting_program program=/usr/libexec/mysqld user=mysql

This allows the program /usr/sbin/httpd as run by the apache user and /usr/libexec/mysqld as run by the mysqld user to determine if one of the web server or mysqld users is exceeding a configured resource constraint enforced by the BetterCPU (cpud) start and stop throttling parameters. Instances of /usr/sbin/httpd run by other users are not affected unless a similar line is included in the configuration for that user as well.

BetterCPU Logging

BetterCPU provides extensive logging functionality which allows a system administrator to see how BetterCPU is helping manage their systems. Logging configuration parameters fall into two general categories: Parameters to enable or disable specific logging functionality, and parameters to define the files messages from various components of BetterCPU are logged to.


Overall logging of BetterCPU information is controlled by the log_cpud parameter. If this parameter is set to a value of 1, then BetterCPU events are logged based on the settings of other log_* parameters. If this parameter is set to a value of 0, then no logging is done by BetterCPU.


General BetterCPU messages are logged to the file pointed to by the log_cpud_file parameter, but specific components of BetterCPU write to their own log files.


The parameter documentation provides in-depth explanations of all of the logging functionality in BetterCPU. -->

BetterCPU Configuration

By default, the non-cPanel installation (bl-install) does not configure BetterCPU to throttle users. However, the cPanel installation (bl-cpanel-install) script does auto-configure and start BetterCPU throttling for all cPanel users that are assigned to a cPanel package. The default settings for each cPanel package level is 50% of one core for 3 seconds per user triggers throttling the user in the jail core(s) and 30 % for 5 seconds releases the user from the jail. Example defaults settings:

start_throttling group=ExampleHost_Level1 cpu_percent=50 seconds=3
start_throttling group=ExampleHost_Level2 cpu_percent=50 seconds=3
start_throttling group=ExampleHost_Level3 cpu_percent=50 seconds=3
stop_throttling group=ExampleHost_Level1 cpu_percent=30 seconds=5
stop_throttling group=ExampleHost_Level2 cpu_percent=30 seconds=5
stop_throttling group=ExampleHost_Level3 cpu_percent=30 seconds=5

The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).

Configuring the Jail Cores

Throttling is configured using parameters located in:

/etc/betterlinux/cpud.conf.d/bl_main.conf:
jail_cnt
max_jail_cnt

These settings are designed so that the same configuration file may be used on different systems with different numbers of cores. However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file. When these parameters are removed the cpud daemon uses the default settings wich is jail_cnt=1 and max_jail_cnt=total cores minus 1.

Syntax:

jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]

The jail_cnt parameter is defined using one of two methods:

  • If a fixed number of cores for jail is desired, make the assignment using jail_cnt as in this example:
jail_cnt 2
In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core. If the system is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.
  • If the number of cores for jail varies based on the total number of cores in the system, use the following format:
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. For example, using the values specified in the previous example, a 32-core system will use the jail_cnt value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails and is formatted identically to the jail_cnt parameter. By default, only the value of jail_cnt is used, but when a max_jail_cnt is specified, the jail is expanded when less than the value in jail_min_idle is available per jail core.

The jail_min_idle value is the minimum amount of idle utilization to maintain on the jail cores if jail_max_cnt has not been reached. If the average free utilization on each jail core is less than jail_min_idle, the jail is expanded until either the jail_min_idle value is maintained or jail_max_cnt is reached.

Syntax:

max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}

Example

max_jail_cnt 2:1 4:2 8:2 16:2

The result on a dual core system is no different from jail_cnt. But for 4-core machines and above, if the max_jail_cnt value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded.

max_jail_cnt 2

The result will always set the number of maximum jails to two. The maximum number of jails are set at two no matter how many cores may be on the system. The exception being, when they system has 2 cores it will set the max jail core to one.

In order for the jail to expand, add a jail_min_idle value to denote how much free utilization should be maintained in the jail:

jail_min_idle 5

BetterCPU will attempt to maintain at least 5% free CPU per jail core, and will expand the jail if max_jail_cnt has not been reached.

Defining User Groups

Groups are automatically defined by the bl-cpanel-install script. Installations on non-cPanel systems need to be defined manually. For more details on groups please see the Defining Groups section of the documentation.

Setting CPU Core Usage Limits

Once jail_cnt and jail_cnt_max are defined and groups are set up, it is necessary to configure the throttling parameters that define when user groups are moved into and out of the jail cores. The syntax for these thresholds is defined with the following parameters:

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set throttling limits for groups defined in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf. The cpu_percent value sets the percentage an individual user (uid) can use of the total non-jail cores on the system.

To understand how this works, it is important to understand how CPU utilization is measured in a multi-core system. In Linux, multithreaded processes can take more than 100% utilization because overall CPU utilization is measured as 100% * number_of_cores. So in an 8 core system, if all CPUs are running at maximum utilization, the system's overall utilization is 800% (100% * 8 cores).

The values for cpu_percent in start_throttling and stop_throttling are based on a normalized utilization assigned to a single core (rather than being spread across all cores). This means that while an 8 core system can have a maximum of 800% utilization, the value used for the cpu_percent value represents the fraction of a single core that could be used. Specifying a value of 80 for cpu_percent means that on a system with no cores presently allocated to jail, when the user exceeds a value of 80% * 8 = 640% on an 8 core system, a core is allocated to the jail and the user's processes are throttled. When a core is allocated to jail, the percentage used to determine if the threshold has been exceeded is 80% * 7 = 560%. If a second core is allocated to jail, this threshold drops to 80% * 6 = 480%.

What this means is that - using the example of cpu_percent set to 80 - any distribution of utilization across all 8 cores of an 8 core system that adds up to more than 640% utilization will trigger the allocation of a core for throttling and that user's processes will be moved to that single core. The utilization could be 100% of 6 cores and 40% of a seventh core, it could be 80% utilization distributed across all 8 cores, or it could be any variation where the user's total utilization exceeds the threshold.

When a non-throttled user's total process utilization exceeds cpu_percent for seconds seconds as defined in the start_throttling parameter, the user's processes are moved into the jail cores.

When a throttled user’s total utilization drops below the cpu_percent value for seconds seconds as defined in the stop_throttling parameter, the user's processes are moved out of the jail back onto the non-jailed cores.

To define these limits:

  • Edit the bl_main.conf located in /etc/betterlinux/cpud.conf.d.
  • Add the following parameters bl_main.conf, with appropriate limits to the system's needs:
start_throttling group=example-group cpu_percent=50,seconds=3
stop_throttling group=example-group cpu_percent=25,seconds=10
  • Save bl_main.conf.
  • Restart cpud:
service cpud restart

With these settings, a user in the group defined with processes that exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system. When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.

Start the BetterCPU Daemon (cpud)

BetterCPU is now configured with basic jail core and throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service cpud {start|stop|status|reload|restart|condrestart}
 start = starts cpud daemon
 stops = stops cpud daemon
 status = shows the status of cpud daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in
 the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon
 only if it is currently running. This option is useful in scripts because it
 does not start the daemon if it is not running.

Start the cpud daemon with the following command:

service cpud start

Start the BetterLinux Stats Program (blstat)

In order to see user processes that are in the jail cores, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterI/O. Press 'c' to see the stats for BetterCPU. Press '?' to display general help for the utility.

Limiting Program Execution Time Using time_monitor

BetterCPU can be used to limit the amount of time programs can execute. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/bl_main.conf.

The basic steps for limiting execution time are:

  1. Define a program or user group
  2. Add time_monitor to /etc/betterlinux/cpud.conf.d/bl_main.conf
  3. Reload the BetterCPU daemon's configuration
service cpud restart

When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.

The format of this parameter is:

  • time_monitor group_type=[user or prog] group=group1,group2 seconds=5 child_of=/parent/program program=/script/to/run wait_to_run_again_seconds=3 exclude_prog_group=group_name or exclude_user_group=group_name

BetterLinux includes two scripts that can be used with time_monitor: /etc/betterlinux/bin/kill_pid.pl (This will kill programs) /etc/betterlinux/bin/pid_running_too_long.pl (will report when a program is running longer than the number of seconds defined in time_monitor)

time_monitor auto-configuration for cPanel servers

The bl-cpanel-install script will auto-configure the time_monitor feature with the number of seconds entered at the time of installation in /etc/betterlinux/cpud.conf.d/bl_main.conf. In the examples below, 900 seconds was entered by the user during the bl-cpanel-install script. The user has a cPanel package called AcmeHost_Level1. Any child programs of httpd, chroottpd, and crond owned by the users in the AcmeHost_Level1 package are limited to run for 900 seconds. If they exceed the 900 seconds, kill_pid.pl script will run every 30 seconds until the program is killed. The child of programs excluded from the 900 seconds are bash, ksh93, tcsh, and zsh.:

  • time_monitor group_type=user group=AcmeHost_Level1 seconds=900 child_of=/usr/local/apache/bin/httpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list
  • time_monitor group_type=user group=AcmeHost_Level1 seconds=900 child_of=/usr/local/cpanel/bin/chroothttpd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list
  • time_monitor group_type=user group=BetterHost_Level1 seconds=900 child_of=/usr/sbin/crond program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

The monitor_exclude_list group is auto-defined with the following syntax in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf:

prog monitor_exclude_list bash,ksh93,tcsh,zsh

Add time_monitor to bl_main.conf

Limiting the execution time for a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again. The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.

To define the limit, add the following line to bl_main.conf:

  • time_monitor group_type=prog groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer wait_to_run_again_seconds=10

In this example, we use the sample "bad_progs" program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash. If the program has run for 5 seconds or longer, /root/killer is called with the PID of the program, and the script kills the program after logging the date and time.

The final parameter (10) tells cpud to not run the script more than once every 10 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring the execution time, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored. For example:

Create a program exclude group in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf (Define a Program Group):

prog monitor_exclude_list bash,ksh93,tcsh,zsh

Then define a time monitor setting to kill any users child processes of sshd that runs longer than 900 seconds with the exception of bash,ksh93,tcsh,zsh. The /etc/betterlinux/bin/kill_pid.pl script is included in the BetterLinux installation.

  • time_monitor group_type=user group=cpanel_users seconds=900 child_of=/usr/sbin/sshd program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=monitor_exclude_list

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

Limiting Open Files Using file_limit

BetterCPU can be used to limit the number of concurrent open files a user can have at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf.

The basic steps for limiting concurrent open files are:

  1. Define a user group (Defining User Groups)
  2. Add file_limit to /etc/betterlinux/cpud.conf.d/bl_main.conf
  3. Reload the BetterCPU daemon's configuration

The format of this parameter is:

file_limit group=GROUP,GROUP limit=[number of files to limit]

The groups are user groups defined in bl_groups_main.conf

When a user exceeds the allowed number of open files, CPUd prevents any additional files from being opened.

For example, if in bl_groups_main.conf the following definitions exist:

uid bl_users 500-1000

Then the following file_limit setting is applied:

file_limit group=bl_users limit=100

For any user with a UID from 500 to 1000, if 100 files are opened by that user, then additional attempts to open files (for reading or writing) will fail.

file_limit group=bl_users limit=100	

"file_limit_exclude" excludes programs from concurrent file open limits.

As with the file_limit setting, this parameter takes a group defined in bl_groups_main.conf. The group contains a list of programs to be excluded from concurrent open file limits.

file_limit_exclude group=prog_group

"file_limit_override" overrides file limits by program. Program groups are defined in bl_groups_main.conf.

file_limit_override group=prog_group1,prog_group2 limit=120 child_of=/bin/cron,/usr/bin/ssh

When open files are limited, the file_limit_override and file_limit_exclude options can be applied to override the limits for certain programs and to remove the limits from others.

Limiting Processes Using process_limit

BetterCPU can be used to limit the number of concurrent processes a user can execute at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf.

The basic steps for limiting concurrent processes are:

  1. Define a user group (Defining User Groups)
  2. Add process_limit to /etc/betterlinux/cpud.conf.d/bl_main.conf
  3. Reload the BetterCPU daemon's configuration

When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.

process_limit auto-configuration for cPanel servers

The bl-cpanel-install script will auto-configure the process_limit feature by prompting for the number of processes to limit each user assigned to a cPanel package. In the example below, the administrator selected a limit of 50 processes and has a cPanel packages called AcmeHost_Level1. The auto-configuration will also define an exclude group in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf:

  • prog ignored_programs_for_process_limit /usr/libexec/dovecot/imap,/usr/libexec/dovecot/pop3,/bin/bash,/bin/zsh,/bin/ksh93,/bin/tcsh,/usr/bin/screen,/usr/bin/emacs-23.1,/bin/nano,/usr/bin/alpine,/usr/bin/mysqldump,/usr/bin/rsync

Next, it uses the ignored_programs_for_process_limit group in the process_limit_exclude parameter in /etc/betterlinux/cpud.conf.d/bl_main.conf. This will exclude all these programs from the process_limit parameter settings:

process_limit_exclude group=ignored_programs_for_process_limit

Then, it will configures the following process_limit parameters in /etc/betterlinux/cpud.conf.d/bl_main.conf:

process_limit group=AcmneHost_Level1 limit=50
process_limit_override group=AcmeHost_Level1 limit=75 child_of=/usr/sbin/crond
process_limit_override group=AcmneHost_Level1 limit=100 child_of=/usr/sbin/sshd

The configuration above limits sets a process of 50 for each user in the AcmneHost_Level1 package, then assigns an exception limit of 75 to child process crond and 100 for sshd. Additionally, all the programs in the ignored_programs_for_process_limit will not be held to a limit.

Add process_limit parameter

Limiting concurrent processes per user is very straightforward.

To apply a limit, once a user group is defined, add a line to apply the desired restiction to the bl_main.conf file:

process_limit group=bad_users limit=100

This limits the users in the bad_users group to 100 concurrent processes each at any given time.

To completely exclude specific programs from having limits applied, create a program group in bl_groups_main.conf with the programs in it:

prog exclude_process_limits /bin/cron,/bin/bash

Then add a process_limit_exclude line to bl_main.conf:

process_limit_exclude group=exclude_process_limits

If a user limit should be overridden for particular programs, specify the user group as part of a process_limit_override command and specify the specific programs that should have the different limit in bl_main.conf:

process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd

This overrides the existing limits for bad_users for instances of /usr/sbin/sshd and sets those limits to 120.

In order to activate the changes, issue the following command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon.

service cpud restart

Limiting Memory Utilization Using memory_limit

BetterCPU can be used to monitor the amount of RSS memory programs can use. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/bl_groups_main.conf.

The basic steps for limiting RSS memory usage are:

  1. Define a program or user group (Define a Program Group) (Defining User Groups)
  2. Add memory_limit to /etc/betterlinux/cpud.conf.d/bl_main.conf
  3. Reload the BetterCPU daemon's configuration

The format of this parameter is:

  • memory_limit group_type=[user or prog] group=group1,group2 rss_memory=[amount in megabytes "m" or gigabytes "g"] program=/script/to/run wait_to_run_again_seconds=m exclude_prog_group=group_name or exclude_user_group=group_name

When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.

memory_limit auto-configuration for cPanel servers

The bl-cpanel-install script will auto-configure the memory_limit feature by prompting the administrator for the amount of RSS memory to limit each user assigned to a cPanel package. In the example below, the administrator selected an RSS memory limit of 2000m (megabytes) and has a cPanel package called AcmeHost_Level1. Once the 2000m threshhold has been exceeded for any user in the AcmeHost_Level1 group, the script /etc/betterlinux/bin/pid_running_too_long.pl will run and log the event.

  • memory_limit group_type=user group=BetterHost_Level1 rss_memory=2000m program=/etc/betterlinux/bin/pid_running_too_long.pl

Add memory_limit

Limiting the RSS memory used by a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again. The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.

For example, a script in /root called 'killer' might contain:

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.

To define the limit, add the following line to the bl_main.conf file:

memory_limit group_type=prog group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run_again_seconds=30

In this example, we use the sample bad_progs program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB). If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.

The final parameter (30) tells cpud to not run the script more than once every 30 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored by adding them to the end of the syntax format:

exclude_prog_group=group_name or exclude_user_group=group_name

In this example, a specific program will run when a users total rss memory usage exceeds 2000m or 2 Gigs. This is done by defining the program that will run when the limits are reached. Add an 'm' or 's' to the end of the memory amount to specifying memory limits - Such as 4096m for 4 Gigs

memory_limit group_type=user group=cpanel_users rss_memory=2000m program=/etc/betterlinux/bin/pid_running_too_long.pl

In this example, if the following is in bl_groups_main.conf:

uid bl_users 500-1000
prog prog_excl /usr/bin/top,usr/bin/vmstat

and the following memory_limit setting is applied:

memory_limit group_type=user group=bl_users rss_memory=1000m program=/etc/betterlinux/bin/kill_pid.pl wait_to_run_again_seconds=30 exclude_prog_group=prog_excl

Then for any user with a UID from 500 to 1000, runs any process that uses more than 1GB of memory will have the script executed at most once every 30 seconds with the exception of top and vmstat

In order to activate the changes, issue the command:

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon:

service cpud restart

CloakFS File System Cloaking

BetterLinux 1.0.0-x has a new and improved version of CloakFS that cloaks via the kernel instead of a pam module. This simpler solution makes it easy to support all user space applications and versions (apache, suexec, suphp, ssh, pam, etc.) without risking constant changes. CloakFS is also now easier to configure.

CloakFS secures web hosting systems by preventing users from seeing each other’s directories, files, and processes. It protects users from each other even if file/directory permissions are insecure. For example, cPanel user A couldn’t read/write user B’s file even with chmod 777 permission. CloakFS also auto-detects all users’ home directories so no extra configuration is required.

CloakFS prevents directory-listing scripts from running via ssh, cron jobs, Apache (suexec, suphp) and symlinks. Further, a user sees only his or her own $HOME directory and other files he or she owns. For example, if User A creates “samplefile” in /tmp and then lists all /tmp files, he will see only “samplefile,” but not other users’ /tmp files.

Configuration Files

The cloak_user parameter is found here:

/etc/betterlinux/cpud.conf.d/bl_main.conf

Add values to the cloak_user parameter with the following syntax:

cloakfs protect_groups=[group1_name,group2_name] protect_from_groups=[group1_name,group2_name]

The first value is a comma-delimited list of groups (defined in bl_groups_main.conf) that will be protected from the users in the second value protect_from_groups. The users in these groups will be cloaked to all the users in the group, users will only see their process and processes run by root, apache, mysqld, etc. For example, if this parameter is set to:

cloakfs protect_groups=user_group1 protect_from_groups=user_group1

Then all users in the user_group1 will see themselves at the file system level.

cPanel Resellers

The BetterLinux cPanel hook script handles the complexities for reseller users. This feature allows the reseller to see the users they have created, but blocks the users from seeing their reseller with the exception of the resellers branding directory. The script will do the following:

  • Finds resellers and creates a group for each
  • Finds resellers users and creates a group for each user
  • Configures these groups and the cloakfs_exception parameter in the /etc/betterlinux/cpud.conf.d/resellers.conf file.

The syntax for the cloaks_exception parameter is as follows:

  cloakfs_exception unprotect_groups=[group-name] unprotect_groups=[group_name]


Start / Stop CloakFS

BetterLinux version 1.0.0-x has a start/stop feature exclusively for CloakFS. CloakFS versions prior to 1.0.0-x were tied to the main cpud daemon start/stop service. Now if the cpud daemon is shutdown, CloakFS will continue to protect the system. Here's the syntax for the CloakFS start and stop service:

The command syntax is as follows:
service cloakfs {start|stop|status|reload|restart|condrestart}
 start = starts cloakfs daemon
 stops = stops cloakfs daemon
 status = shows the status of cloakfs
 reload = reloads changes to the .conf files
 restart = restarts cloakfs
 

Start or stop cloakfs with the following commands:

service cloakfs start
service cloakfs stop

Load Average Overview

The BetterLinux cpu throttling feature moves processes exceeding limits into cpu jail cores. Using this method makes the system load average higher. In order to resolve this issue and properly calculate the load average, BetterLinux has introduced the load_average parameter. Now system administrators can configure the proper load average for the system using this parameter.

Load Average Usage

The default configuration for cPanel installations sets the load_average parameter with the recommended usage below:

load_average type=all or non_jail_only normalize_load=on or off

The first switch (type=) value takes the word "all" meaning all the cores on the system or the word "non_jail_only" meaning to calculate the load average only on the non_jail_cores. The second switch (normalize_load=) value is "on" or "off". Entering a value of "off" adds up the total load of the cores to calculate the load average. Entering a value of "on" adds up the total load of the cores and divides it by the total number of cores.

For example, if this parameter is set to:

load_average type=all normalize_load=off

Will add up the load for all the cores.

load_average type=all normalize_load=on

Will add up the load for all the core (jail and non_jail) and divide the total load by the total number of cores.

  1. load_average type=non_jail_only normalize_load=off

Will add up the load for all non-jail cores.

load_average type=non_jail_only normalize_load=on

Will add up the load for all the non-jail cores and divide by the total number of non_jail cores.

Block Set User ID Upon Execution

"block_suid" uses groups defined in bl_groups_main.conf to restrict users from being able to run programs that are flagged SUID root. Programs that users in the group are allowed to run can be specified by creating a program in the bl_groups_main.conf file. For example, users may need to run a program like passwd or a cPanel program like jailshell. In bl_groups_main.conf, specify:

prog prog_group1 /usr/bin/passwd,/usr/local/cpanel/bin/jailshell

Then add the prog_group1 group name to the example syntax below to /etc/betterlinux/cpud.conf.d/bl_main.conf and restart cpud:

block_suid group=user_group1,user_group2 program=prog_group1	# [Default = NONE]

In the example above, users in user_group1 and user_group2 are restricted from all programs flagged SUID root except the programs ("passwd", and cPanel "jailshell") defined in the prog_group1 programs.

In this example, if the parameter is set to:

block_suid group=user_group1

The users in user_group1 will block programs flagged SUID root from running.

BetterIO General Usage

The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.

In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.

From a CPU perspective, this is referred to as “Waiting on I/O”, or iowait%. If the I/O channel is not also managed, then the system will not be able to run optimally.

When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.

There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To monitor these factors, one can use the iostat command or blstat (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.

What can BetterIO tune to affect block device utilization?

At a high level, tuning is applied to I/O operations per second (iops) and bandwidth (bw). These two parameters can be tuned by applying minimum and maximum values in /etc/iothrottled/iothrottled.conf.d/bl_io.conf.

Those system-wide minimum and maximum values can be refined by applying user limits and program limits, which are defined in /etc/iothrottled/bl_io.conf and /etc/iothrottled/bl_io.conf, respectively.

The limits defined can be scaled to allow the block device channel to be utilized more effectively when it is under-utilized. The auto_scale parameter defines how scaling is applied to keep both the bandwidth utilization and iops within a desired performance range.

Basic Settings

The basic settings used to configure BetterIO can be added to /etc/betterlinux/iothrottled.conf.d/bl_io.conf and consist of:

sleep_time_ms
sample_periods
enable_program_limits

IO usage is calculated using averages of samples taken over time. BetterIO profiles usage by process and by user on a regular interval to determine which processes and users are utilizing the most IO, and if those usages exceed permitted thresholds, then BetterIO manages those processes in a way that limits their ability to impact overall system performance.

"sample_periods" determines the number of samples used in calculating the average I/O channel utilization. This parameter is used in conjunction with the sleep_time_ms parameter to determine the utilization average interval. If sleep_time_ms is set to 500 and sample_periods is set to 2, then the utilization is sampled every 500 ms and that value is averaged with the sample from the previous period (i.e. the last two samples are averaged together).

Increasing the number of periods will result in a smoother average over a longer period of time, but will allow a process to run with a higher utilization of the I/O channel longer before it is detected and throttled.

"enable_program_limits" enables limiting individual programs using definitions in the bl_io.conf file.

The averages for user and IO utilization (sleep_time_ms and sample_periods, respectively) are calculated based on the number of periods (or samples) taken. The defaults for these parameters are:

sleep_time_ms 1000
sample_periods 6
enable_program_limits 1

This means snapshots are averaged every 6 periods, and each period is 1000 ms long.

Advanced Settings

"cgroup_dir" identifies the path where the cgroup block I/O controller is mounted.

cgroup_dir /cgroups_blockio	# [Default = /cgroups_blockio]

"iothrottled_gid" the gid that iothrottled should use to set the group ownership on all cgroups it creates. Other programs that use cgroups can use a different gid to own and identify their cgroups, or they can share the gid that iothrottled uses to access the cgroups created for and used by iothrottled.

iothrottled_gid 0	# [Default = 0]

"max_cgroups" defines the maximum number of cgroups allowed. A value of '0' removes the limit. When the number of cgroups exceeds max_cgroups then cgroups are deleted until the total is below max_cgroups. The least recently used cgoups are deleted first.

max_cgroups 4096 # [Default = 4096]

"remove_stale_groups_time" defines how often to call the garbage collector to remove stale groups, expressed in sleep_time_ms periods (see also /etc/betterlinux/iothrottled.conf.d/bl_io.conf). A value of '0' disables it.

remove_stale_groups_time 0 # [Default = 0]

"interruptible_sleeps" determines how a process that is being throttled is put to sleep while having its I/O throttled by iothrottled.

In a normal system without iothrottled, when the I/O channel is congested, the system will put a process to sleep because it's waiting on I/O. When this occurs, the system loadavg value is increased because the system is under a load that's limited not by CPU utilization but rather by I/O performance.

When artificially restricting I/O using iothrottled, this can cause programs that trigger off of loadavg values to trigger incorrectly. This option introduces the ability to pause process execution due to iothrottled's artificial I/O channel throttling as a part of the CPU scheduler rather than due to I/O. This is done using an interruptible sleep, which means that while the process is being suspended due to I/O, the suspension is not due to an actual I/O bottleneck but due to an artificial bottleneck caused by

interruptible_sleeps 1 # [Default = 1]

"auto_wakeup" Under certain conditions I/O throttling may contribute to increasing the memory pressure of the system, especially with stacked block devices such as loopback mounted filesystem, LVM, software RAID, iSCSI, etc.

The option auto_wakup reduces the risk of page allocation failures by temporarily awakening all the tasks previously put to sleep due to the enforced I/O limits. This allows the kernel to drain some pending I/O requests and give more breathing room to the system in terms of memory utilization.

I/O limits are then enforced as normal when the memory utilization in the system returns back to a safe level.

auto_wakeup supports three different configurations:

  • auto_wakeup = 0: disable auto-wakeup
  • auto_wakeup = 1: wake up all tasks when kernel is running out of emergency memory (reduce the probability of GFP_ATOMIC allocation failures) -- This is the recommended and default setting.
  • auto_wakeup = 2: wake up all tasks when kernel is running out of emergency memory or when journal I/O is submitted (meta-data I/O)

Option 2 should be used only if the kernel reports some hung task timeout traces. They can happen when there is too much pressure on filesystem locks like intensive meta-data operations.

auto_wakeup 1 # [Default 1]

Setting I/O Bandwidth Limits Per-User or by User-Group

Individual user or group limits can be set using the configuration file /etc/iothrottled/bl_io.conf.

This file uses a single line per entry:

[user|group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or mount points to include or exclude.

For example, the file might contain a line that reads:

testuser bw max sda,sdb,sdc 10000000

This would mean that the user testuser has their maximum bandwidth (bw) limited on devices sda, sdb, and sdc to 10000000 B/s (Bytes per Second).

The first field can be a user or group name. Groups are defined in the file /etc/betterlinux/bl_groups_io.conf. If a group and a user have the same name, the limit applies to the user.

Setting I/O Bandwidth Limits by Program-Group

More granular control over individual programs’ usage of block device I/O rates and data rates can be achieved using the /etc/betterlinux/iothrottled.conf.d/bl_io.conf file.

The line format for this file is:

[program_group_name] [S|G] [bw|ops] [device/mount point list] [limit]

The fields that match up with fields in bl_io.conf are defined the same way as in bl_io.conf.

The prognam_group_name field contains the names of the program or programs which is defined in bl_groups_io.conf.

If the third field contains an “S”, the limit applies to each instance of the program started by the use or group. A “G” means the limit applies to all instances together.

For example: NOTE: In all the examples below we assume to have the group "tar_progs" defined in /etc/betterlinux/iothrottled.conf.d/bl_groups_io.conf:

prog tar_progs /bin/tar,/usr/bin/tar

If our goal is to limit each individual instance of "tar" to a 5 MB/s I/O rate we would use the following line.

tar_progs S bw max inc="^/" 5000000

NOTE: all the "/bin/tar" and "/usr/bin/tar" instances will be affected by this rule if the UID that runs them is not included in any UID group. Remember that UID groups always win against PROG groups.

If our goal was to group ALL instances of tar to an aggregate bandwidth of 5 MB/s then you would change the "S" (Single process) in the example above to "G" (For grouping all tar processes together). This option is helpful when you want to limit one or more instances together. So, if you had one tar running or 50 tars the total disk I/O bandwidth would never exceed 5 MB/s.

The syntax for this scenario would be as follows:

tar_progs G bw max inc="^/" 5000000

If our goal was to limit each instance of tar to a disk I/O rate of 5 MB/s, but would only like to enforce this limit on any filesystem mounted under /home (and any sub-directories) but have no limits enforced on other drives then the syntax would be as follows.

tar_progs G bw max inc="^/home" 5000000

You can include multiple block devices (hard drives) or mount points to be included in the limits that you have defined as shown below:

tar_progs G bw max inc="/home",sdb,dm-1,inc="/var/matt",inc="/etc" 5000000

In addition, you can specific drives to be excluded from the limits that you have defined. This syntax is shown below.

tar_progs G bw max inc="/",exc="^/media|^/mnt" 5000000

For example:

tar_progs S bw sda 1000000

This applies a bandwidth limit of 1000000 bytes/s to each instance of the /bin/tar or /usr/bin/tar commands run by any user not included in a UID group. If one of these users starts two concurrent instances of the tar command, each one is limited to 1000000 b/s of block device I/O bandwidth. If this line reads:

tar_progs G bw sda 1000000

The bandwidth limit is applied to all instances of tar that the users launch. In this case, if the users launch two concurrent instances of the tar command, the total bandwidth both instances can use together is 1000000 B/s.

Below are some configuration rule examples of how to limit I/O bandwidth for certain programs.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied. The inc= and exc= directives take an extended regular expression for mount points. For example, the line:

tar_progs G bw max sda,inc="/home|/backup",exc="/mnt1|/mnt2" 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that have "/home" or "/backup" in their mount point (that would also include, for example, "/home2", "/media/home3", excluding all the mount points that have "/mnt1" or "/mnt2" (like "/mnt21", "/mnt/mnt1", ...). The values are regular expressions and they can appear anywhere within the mount point). Including ^ and $ in the regex will force a match at the begining or end.

The evaluation of these expressions always follows the order of:

  • Explicit device names
  • Devices mounted in the "inc" list
  • Devices mounted in the "exc" list

Exclusions will ultimately override the other values in the list. Consider the following list of mounted devices:

/dev/sda mounted at /
/dev/sdb mounted at /home
/dev/sdc mounted at /media/backups
/dev/sdd mounted at /usr
/dev/sde mounted at /mnt/home
/dev/sdf mounted at /srv/www
/dev/sdg mounted at /srv/ftp
/dev/sdh mounted at /srv/home

With the following restrictions in bl_io.conf:

tar_progs G bw max sda,sdd,inc="/home",exc="/srv" 1000000

Programs in group "tar_progs" will be limited to 1 MB/s on /home and /mnt/home, but not on /srv/home, due to the exc="/srv" rule.

How do the block device inclusion/exclusion lists work?

Simple configuration of BetterIO uses device names in a list in the bl_io.conf and bl_io.conf files. For example:

testuser bw max sda,sdb,sdc 1000000

This applies the limit of 1000000 B/s for the user testuser on devices /dev/sda, /dev/sdb, and /dev/sdc.

Modern Linux systems, however, do not always guarantee the assignment of devices, which can lead to issues with limits not being applied to the correct device.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied.

The inc= and exc=' directives take an extended regular expression for mount points. For example, the line:

testuser bw max sda,inc=”/home|/backup”,exc=”/mnt1|/mnt2” 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that are mounted at “/home” and “/backup” (and would also include, for example, “/home2”, “/media/home3” - the values are regular expressions and can appear anywhere within the mount point), but would exclude any mount point that includes “/mnt1” or “/mnt2” (including “/mnt21”, “/mnt/mnt1” - as with the “inc=” directive). Including ^ and $ in the regex will force a match at the begining or end.

Another example for this using regular expression that will apply the limit of 1MiB/s to any mounted block device except those mounted under /var or /tmp

testuser bw max inc=”^/”,exc=”^/var|^/tmp” 1000000

In this example we want to limit all the mounted block devices, except the temporary mounted devices, usually mounted under /mnt or /media.

  • add the line below to /etc/betterlinux/bl_groups_io.conf:
testgroup uid 500-1000
  • Then add the line below to /etc/betterlinux/iothrottled/bl_io.conf:
testgroup bw max inc=”^/”,exc=”^/mnt|^/media” 1000000

The evaluation of these expressions always follows the order of:

  1. Explicit device names
  2. Devices mounted in the “inc” list
  3. Devices mounted in the “exc” list

Exclusions will ultimately override the other values in the list.

Consider the following list of mounted devices:

/dev/sda1 mounted at /
/dev/sda2 mounted at /home
/dev/sdb1 mounted at /media/backups
/dev/sdb2 mounted at /usr
/dev/sdc1 mounted at /mnt/home
/dev/sdc2 mounted at /srv/www
/dev/sdd1 mounted at /srv/ftp

With the following restrictions in bl_io.conf:

testuser bw max sda,sdd,inc=”/home”,exc=”/srv” 1000000

The result would be:

sda would be included
sdb would be excluded
sdc would be excluded
sdd would be excluded

So the only device controlled by BetterIO in this scenario would be /dev/sda. The controls are applied per block device, not on a per-partition basis - so if a partition is excluded from control, all partitions on that block device will be excluded.

When configuring your system, it is important to take this into consideration - you do not want your swap partition to be throttled by BetterIO, but if you put swap on /dev/sda and control /dev/sda with BetterIO, you may end up impacting performance on the system in ways you may not intend to.

Examples include/exclude:

/dev/sda1 on /
/dev/sda2 on /home
/dev/sdb1 on /home2
/dev/sdc1 on /home3
/dev/sdd1 on /home4
testuser bw max inc="/home",exc="/home2"

The rexeg's are matched against the left of the mount point. So the include matches /home1, /home2, /home3, /home4 and the exclude matches /home2 only. So the result is:

/dev/sda included
/dev/sdb excluded
/dev/sdc included
/dev/sdd included

Example Exclude:

/dev/sda1 on /
/dev/sda2 on /home
/dev/sdb1 on /src
/dev/sdb2 on /usr
/dev/sdc1 on /extra
testuser bw max exc="/src"
/dev/sda included
/dev/sdb excluded
/dev/sdc included

How does auto scaling work?

The auto_scale parameter is an on/off toggle switch for enabling or disabling the auto scale feature. The auto scaling algorithm is applied to overall bandwidth utilization or iops giving the system administrator the ability provide more bandwidth to a defined set of users when it's available. The auto_scale throttling is applied when all the users in the defined groups are going above the target disk utilization percentage defined in the seek_balancing_method option.

When setting the value for the auto_scale parameter, an integer is entered that represents the on/off values:

0 = auto scaling off
1 = auto scaling on

Auto scaling is controlled by these parameters:

scale_min - defines the minimum percentage the scaling algorithm may scale down in order to reach the target disk utilization.
scale_max - defines the maximum percentage the auto_scaling algorithm may scale up to in order to reach the targeting disk utilization.
seek_balancing_method - sets the value for the target disk utilization used by the auto_scale and seek_balancing algorithms. This is achieved by dynamically adjusting the defined I/O limits. Setting this value close to 100 will use more of the total disk bandwidth, but will leave less room for new I/O users. Setting this parameter value under 100 will provide more breathing room for new I/O users or the non-throttled users. For example, if the target utilization is 80%, the device will has a free disk bandwidth of 20% for non-throttled users.

To understand how these parameters are related to each other, consider the following scenario:

With the following line in /etc/iothrottled/bl_io.conf:

testuser bw max sda 1000000

And in /etc/iothrottled/iothrottled.conf.d/bl_io.conf:

auto_scale 1
scale_min 200
scale_max 50000
seek_balance_method 80

Assume in this scenario the current disk utilization is 0.001%. The seek_balance_method percentage is divided by current disk utilization for a total of 80,000 which is higher than the scale_max of 50000. The global multiplier is now set to 50,000 and all the I/O limits for the users in the defined group are multiplied by the 50,0000% and there for given 500MB/s of disk I/O.

How does seek balancing work?

The "seek_balancing" parameter is an on/off toggle switch for enabling or disabling the I/O seek balance feature. The seek balancing algorithm enables dynamic adjustments of a single users limits based on whether a device is congested or not. The seek balancing algorithm applies additional dynamic adjustments to the users that are doing more expensive I/O operations in respect to the other I/O users in the system.

When setting the value for the seek_balancing parameter, an integer is entered that represents the on/off values:

0 = seek_balancing off
1 = seek_balancing on

To understand how these parameters are related to each other, consider the following scenario:

With the following line in /etc/iothrottled/bl_io.conf:

testuser bw max sda 1000000

And in /etc/iothrottled/iothrottled.conf.d/bl_io.conf:

seek_balancing 1
seek_balancing_method 80

The seek_balancing feature will simply calculate the actual disk utilization along with the seek_balancing_method percentage (target amount) to find the best balance of I/O to throttle the user hogging too much of the disk bandwidth (iops or throughput).

How can one test to see if BetterIO is controlling devices?

A simple and quite dramatic way to see BetterIO limiting I/O is to set a lower threshold for a user that is very small. For example, create a user called testuser and log in as that user. Run the following command:

sync; dd if=/dev/zero of=temp.fil bs=1M count=10 oflag=direct; rm temp.fil

The results from that command for an unmanaged user would be similar to this screenshot:

Iothrottled blocking devices.png

If we then modify the /etc/iothrottled/bl_io.conf file to include the line:

testuser bw max inc=”/” 500000

and rerun the dd command above, the following output results:

Iothrottled dd results.png

The drop in throughput from 350-400 MB/s to 500-600 kB/s shows the effect of the change in bandwidth limits for this user.

Are there any other considerations to keep in mind?

With BetterIO, the limits are set on a per-block device basis. It is important to keep in mind that I/O channels are not always assigned in a 1:1 ratio with block devices. If a series of storage devices are connected to the system by a Fibre Channel connection, the channel bandwidth available for all of those devices needs to be taken into consideration across all of the devices.


If devices /dev/sda, /dev/sdb, and /dev/sdc are all on the same disk controller, and the maximum throughput of that controller is 1 Gbps, then all three disks cannot run at 1 Gbps even if the drive electronics would permit that.

If you then have 10 users limited to 100 Mbps each, then performance will effectively be running without any throttling because the disk channel will be oversubscribed if all of the users are attempting to fully utilize their allocated bandwidth to each of these three devices. At the same time, these users are still limited to 100 Mbps each and are unable to monopolize the resources.

BetterIO Configuration

Using the bl-cpanel-install script, will auto-configure I/O throttling for all users previously assigned to cPanel packages. The default limits for each cPanel package and mounted device is 10MBs of disk I/O per seconded. For example:

ExampleHost_Level1 bw max vda 10000000 ops max vda 1000
ExampleHost_Level2 bw max vda 10000000 ops max vda 1000
ExampleHost_Level3 bw max vda 10000000 ops max vda 1000

On stock Linux (non-cPanel) installs IO throttling is not auto-configured and needs to be setup manually. The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).

Defining user groups

Groups are automatically defined by the bl-cpanel-install script. Installations on non-cPanel systems need to be defined manually. For more details on groups please see the Defining User Groups section of the documentation.

By creating and populating groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined.

Once the groups have been created, you can define throttling limits that meet each group's SLA.

Setting I/O Bandwidth Limits

Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis. To do this, edit the bl_io.conf file located in /etc/betterlinux/iothrottled.conf.d. This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/bl_groups_io.conf.

The bl_io.conf file uses a single line per entry:

[group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.

For example:

example-group bw max sda,sdb,sdc 10000000

This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.

Another example:

io_standard_users bw max inc="^/",exc="^/home" 5000000

This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.

After updating the bl_io.conf file with limits applicable to the system, save the file and restart iothrottled.

Note:

  • If a group and a user have the same name, the limit applies to the user.
  • Only one limit per device can be set. It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.

Please see the BetterIO general documentation for more configuration options for I/O bandwidth limiting features by program and inclusion/exclusion options.

Tuning I/O Scaling and Balancing

The auto_scale and seek_balance features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in bl_io.conf.

These settings are contained in /etc/betterlinux/iothrottled.conf.d/bl_io.conf. The configuration parameters define how these settings affect the I/O bandwidth. Details on how these settings work can be found in the user documentation for scaling and balancing.

Start the BetterIO daemon (iothrottled)

BetterIO is now configured with basic throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows:

service iothrottled {start|stop|status|reload|restart|condrestart}
 start = starts iothrottled daemon
 stops = stops iothrottled daemon
 status = shows the status of iothrottled daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Use the following command to start the iothrottled daemon:

service iothrottled start

Start the BetterLinux stats program (blstat)

To see user processes that are being throttled by BetterIO, execute the blstat program with root privileges:

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterIO. Press 'i' to see the stats for BetterIO. Press '?' to display general help for the utility.

BetterMySQL Overview

BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.

It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated. Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.

Configuring BetterMySQL for cPanel Support

BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.

Installing the BetterLinux version of MySQL for cPanel

BetterMySQL cannot be installed unless MySQL is already on the system. BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL. If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script:

/etc/betterlinux/cpanel/bl-cpmysql-install

Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the BetterCPU general documentation. BetterMySQL’s configuration file serves only to enable BetterMySQL.

Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set. For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit:

start_throttling group=cpanel_users cpu_percent=50 seconds=3
stop_throttling group=cpanel_users cpu_percent=30 seconds=5

Same logic apply's to any BetterIO limits that have been set. In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled.

cpanel_users bw max inc="^/" 5000000

Configuring BetterMySQL for cPanel support

To provide and configure per-user query throttling of MySQL users a mapping of MySQL users to Linux users must be configured. This mapping and other configuration options are auto-configured by either the bl-cpanel-install script on first time installation or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the bl-hooks script can be used after a manual installation of BetterMySQL. Example syntax for running the bl-hooks script:

/etc/betterlinux/cpanel/bl-cphooks --init

Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.

  • usersmap.conf contains the mapping of cPanel users to UIDs
  • mysql.conf contains the parameter location of the usermap.conf file
  • turns on BetterMySQL throttling by adding the approved_accounting parameter to /etc/betterlinux/cpud.conf.d/bl_main.conf
approved_accounting_program program=/usr/libexec/mysqld user=mysql
  • Restarts the cpud daemon to enable BetterMySQL throttling.
service cpud restart

Configuring BetterMySQL for Linux Servers

BetterMySQL configuration is a manual setup for Linux servers

Installing the BetterLinux version of MySQL non-cPanel

After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command:

yum install mysql-server

Configuring BetterMySQL for non-Cpanel Linux servers

To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.

The command format is as follows:

bl_usermap_filename [usermap_file_path]

Example:

bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

Now create the MySQL mapping to Linux user mappings located in the following file:

/etc/betterlinux/mysql.conf.d/usermap.conf

The usermap file is a tab-delimited file that contains the following information:

[mysqluser]	[linux_user]	[uid]

The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.

For example, the following file will create mappings for testuser1 and testuser2. For testuser1, the uid will be determined when that user makes a query against the database.

testuser1	testuser1
testuser2	testuser2	501

The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;":

mysql> show user_map;
+-----------+-----------+------+
| From_user | Unix_user | Uid  |
+-----------+-----------+------+
| testuser2 | testuser2 |  501 |
| testuser1 | testuser1 |  500 |
+-----------+-----------+------+
2 rows in set (0.00 sec)

The approved_accounting parameter turns on the BetterMySQL throttling and is located in:

/etc/betterlinux/cpud.conf.d/bl_main.conf

The syntax is as follows for non-cPanel servers:

approved_accounting_program program=/usr/sbin/mysqld user=mysql

Restart the cpud daemon to enable the configuration settings:

service cpud restart

Monitoring MySQL Utilization with myusertop

BetterMySQL includes the myusertop utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.

Once myusertop is running, press '?' to view the built-in help.

The output is similar to the output of the standard 'top' utility:

mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
Time: CPU 448          Busy 529         CPU/Busy 0.8477   RowsExamined:   1,476m
Qrys: Select 28,456    Update 0         Other 5,250         Adj-RowsEx:   1,251m
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
      h=Help m=MySQLUsersOn/Off                                           q=quit
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx   %CPU
root                   0     2 0.0075   0.0074   1.0060     150      151  100.00
  root                 0     2 0.0075   0.0074   1.0060     150      151  100.00
testuser2            501     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser2          501     0 0.0000   0.0000   0.0000       0        0    0.00
testuser1            500     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser1          500     0 0.0000   0.0000   0.0000       0        0    0.00

In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization. If it were divided equally between the three users, myusertop would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.

If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.

BetterBandwidth Overview

BetterBandwidth uses 'tc' (a built in Linux tool) to provide traffic shaping for users that have their own network interface. The tc tool on its own does not provide a mechanism for limiting bandwidth usage by uid/user on a shared network interface. BetterLinux has added this functionality into its BetterBandwidth feature. Now admins can limit bandwidth for a single user on a shared server that has a corporate internet connection.

A common problem for web hosts is that one single user with a 1Gbit connection could kill the entire TCP stack. Often the host is unaware that it happens because they use 5 min bandwidth averages to monitor bandwidth on their 1 gig or 10 gig port. But all it takes is a 30 second spike in bandwidth out of 5 mins to make customers angry. And because of the 5 minute average it looks like the peak was around 50Mbits when it easily could have been 500Mbits. Customers will blame it on the server or the hosting company.

Many hosting companies give out IPs just to be able to limit bandwidth. This is an expensive approach. The BetterLinux BetterBandwidth feature can do it without giving out dedicated IPs.

BetterLinux has an apache mod called mod_betterlinux that allows BetterBandwidth to limit bandwidth for all ports for an individual user or group of users like email, FTP, etc is all included. This is a tremendous advantage for a host. BetterBandwidth monitors and controls individual, group, and system-wide bandwidth usage in multi-tenant Linux environments. Administrators can now limit outbound bandwidth as if each user had a dedicated IP address. This lets admins consolidate multiple users onto a single IP, which innovatively addresses the fast-increasing IPV4 IP address shortage.

BetterBandwidth Configuration

BetterBandwidth limiting uses the mod_betterlinux Apache module located in in the betterlinux-core-*.rpm , as well as the linux tc method of using the uid classifier to categorize specific user processes. The betterlinux-core-*.rpm is installed by the bl-install script by default and auto-configured for cPanel systems during the betterlinux-cpanel-install script.

Bandwidth Limiting Setup

Automatic cPanel Configuration on 0.9.6-x release

During the betterlinux-cpanel-install script installation, the user will answer Y/N to this question:

Install BetterLinux Apache throttling? [y/N]

Selecting yes, will auto-configure and load the apache module mod_betterlinux on a cPanel system.

Limit user network bandwidth? [y/N]

Selecting yes, will prompt for the Mb/s per user:

Bandwidth limit in Mb/s:

Enter the amount desired to limit each cPanel user.

Which network interface (e.g., eth0) do your cPanel user accounts use?

Enter the network interface.

Answering these questions during the install, will auto-configure the BetterBandwidth limiting feature.

Manual BetterBandwidth Configuration

These steps apply to cPanel and non-cPanel servers.

  • In /etc/betterlinux/cpud.conf.d/bl_groups_main.conf create an interface group. Here's an example:
interface interface-types eth0
  • In /etc/betterlinux/cpud.conf.d/bl_main.conf add the "bandwidth_limit" parameter which sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. The parameter takes a value in the format:
bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in bl_groups_main.conf. The second switch (interface_group=) value is the name of the interface group defined in bl_groups_main.conf. The third switch (limit=) is the amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to limit bandwidth by 20mbit for each user in the cpanel_users group (any defined user group can be used):

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the cpanel_uesrs group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon:

service cpud restart

Assigning IP addresses with the dedicated IP address mapper

With multi-homed systems, incoming traffic is directed to the IP address where DNS resolves the domain name. A system with multiple virtual hosts can accept traffic to secondary addresses which, in turn, are assigned to secondary addresses, which are assigned to the system's network adapters.

Without some modification, responses sent from a server in a multi-homed system originate from the host's primary IP address. This makes it difficult to apply bandwidth or connection limits using a server's IP address. Most often, hosting providers work around this problem by implementing some sort of full virtualization solution that provides a unique virtual instance for each client. <KGO> What is the drawback of this solution?

BetterBandwidth uses the dedicated kernel module to provide a more cost-effective, efficient solution for managing bandwidth in multi-homed environments. When used in conjunction with the BetterBandwidth dedicated daemon, the dedicated kernel module allows administrators to assign outgoing traffic to a specific secondary address which, in turn, is assigned to a specific user--all without the overhead of a fully virtualized system. (Both IPv4 and IPv6 are supported.)

To enable this functionality:

  1. Ensure the dedicated kernel module is loaded:
    modprobe dedicated
  2. Edit /etc/betterlinux/ipv4 and/or /etc/betterlinux/ipv6 to include the specific mappings.
  3. Restart dedicatedd:
    service dedicatedd restart

The IPv4 and IPv6 mapping files are in the following format. Each line contains a single entry.

username|uid: address

The address field can contain either a raw IPv4 or IPv6 address or a DNS name. If the DNS name resolves to more than one address, the kernel randomly selects one of the addresses.

Once configured, the active mappings for IPv4 addresses can be viewed using the following command:

cat /proc/sys/net/ipv4/dedicated

To view mappings for IPv6 addresses, use the following command:

cat /proc/sys/net/ipv6/dedicated

Note: The command output displays the uid rather than the username.

Frequently Asked Questions

Is OpenVZ supported?

OpenVZ is currently not supported, however, it is something we are considering. We are unable to provide an estimate on when it will make it on the roadmap.

I have a customized kernel configuration that requires I build the kernel myself.

  • BetterLinux provides support for customized kernels via the Build Your Own Kernel (BYOK) feature. This feature is currently unavailable.

Does BetterLinux run on CentOS 5?

  • BetterLinux supports kernels 2.6.32 or newer. We are currently evaluating a method for supporting CentOS 5. Some BetterLinux features depend on Linux kernel features that were introduced in version 2.6.26. Some of those features have bugs that were not fixed until version 2.6.32. Hence, we cannot support kernels prior to 2.6.26 at all, and kernels prior to 2.6.32 require backported bug fixes. CentOS 5 has kernel version 2.6.18. It is possible to build a later kernel for CentOS 5, but then some userspace programs, such as iptables, must also be updated to cope with changed kernel APIs.

What will the pricing be after the Free Until 2013 period is over?

  • BetterLinux will continue to be free for non-profit use but will begin charging $9.00 per month per server for commercial use after 2013.

Will BetterLinux be compatible with cPanel/WHM and if so will that include plugins for WHM?

  • BetterLinux is now supports cPanel and we expect to release our WHM plug-in by early May.

How to install blstat?

blstat is not part of the product installation. Follow the steps below to download and install blstat:

yum --disableexcludes=all install perl-TermReadKey perl-JSON

This will install two perl modules blstat needs.

wget http://repo.betterlinux.com/unsupported/blstat-0.53-1.el6.noarch.rpm

Downloads the blstat rpm.

rpm -ivh blstat-0.53-1.el6.noarch.rpm

command to install blstat rpm

At the command line as root blstat is executed by running:

blstat

The initial screen is the i/o throttling stats. Press "c" to switch blstat to the cpu throttling screen. Press "i" to switch the screen back to i/o throttling. Press any key to exit blstat.

System Requirements

The BetterLinux products are supported under the following system requirements:

  • Linux Kerenel 2.6.32 or newer
  • MySQL 5.1 or newer
  • CentOS 6.x or RHEL 6.x
  • CPUs with at least 2 cores

BetterLinux supports virtual machines like KVM, VMWare, etc. However, it does notcurrently support OpenVZ.

Log Files

iothrottled log file names and locations

Each of the following iothrottled log files has a parameter switch that determines the location of the log file. This parameter switch can be modified if the user wants to write the log files to a different location.

Log File Name Parameter Syntax - Location Description
iothrottled log_file /var/log/iothrottled/iothrottled Defines the filename for the iothrottled log file. This is used only if enable_logging is set to '1'.
iothrottled.cnt log_cnt_file /var/log/iothrottled/iothrottled.cnt Defines the filename to track cgroup creation/deletion counts.

iothrottled enable logging parameters

Parameter Syntax Default Description
enable_logging 1 ON Enables logging of information and errors for iothrottled. Set to '1' to enable logging, set to '0' to disable logging. The location of the log file is defined by the log_file parameter.
log_cgroups 1 ON Enables logging of cgroup creations/removals under "cgroup_dir".
log_io_limits 1 ON Enables logging of I/O limit changes.
log_auto_scale 1 ON Enables logging of auto_scale events.
log_seek_balancing 1 ON enables logging of seek_balancing events.
log_throttled 0 OFF enables logging of throttled tasks.

Logging of throttled tasks is done in a loop that scans every /proc/pid/task/tid/io-throttle-stat entries periodically each sleep_time_ms millisecs. WARNING: enabling this option may strongly impact on the overall iothrottled performance. Moreover, it will probably make your log_file huge, so it is strongly suggested to rotate the iothrottled log file via logrotate when this option is enabled.

cpud log file names and locations

Each of the following cpud log files has a parameter switch that determines the location of the log file. This parameter switch can be modified if the user wants to write the log files to a different location.

Log File Name Parameter Syntax - Location Description
cpud log_cpud_file /var/log/cpud/cpud This logfile includes messages about users being put in jail, the total number of users in jail, and messages about the jail being expanded or contracted.
cpud_cpu log_cpu_file /var/log/cpud/cpud_cpu defines the file used for logging summary information about cpu core usage. The information logged in this file is the number of cores that are defined as being used by the jail and non-jail, and the number of runnable processes in jail and not in jail.
cpud_governor log_gov_file /var/log/cpud/cpud_governor Defines the file used for logging messages from the cpud governor. The governor is responsible for handling runaway processes. The output includes a date/time stamp, UNIX time, signal sent, process owner UID, whether the process is a dedicated process, the process type, PID, program name, age, and command line used to start the process.
cpud_jail log_jail_file /var/log/cpud/cpud_jail Defines the file used for logging information about jail usage. The output includes a date/time stamp, UNIX time, and information about the number of cores assigned to the jail; percent utilization of the jail by user-owned processes, system-owned processes, and percent of idle jail capacity; the UID of the user with the most processes in jail and the total utilization of that user's processes; the program name with the highest single instance utilization; and the program name with the highest overall utilization.
cpud_mem_con log_memcon_file /var/log/cpud/cpud_mem_con defines the file used for logging messages from the memory controller.
cpud_mem log_mem_file /var/log/cpud/cpud_mem Defines the file used for logging statistics captured by the log_mem parameter. The output includes a date/time stamp, UNIX time, iteration counter, and then the following values: root/non-root indicator, sleep time, program name, number of running processes (rss_cnt), total RSS memory used, max RSS memory used by a single instance of the program. The groups are sorted into root/non-root collections and are sorted within each grouping in descending order of total RSS memory used.
cpud_panics log_panics_file /var/log/cpud/cpud_panics defines the file used for logging cpud panic information. This file contains information about processes killed when cpud enters panic mode.
cpud_prog log_prog_file /var/log/cpud/cpud_prog defines the file used for logging statistics captured by the log_prog parameter. The output includes a date/time stamp, UNIX time, iteration counter, and then the following values: root/non-root indicator, sleep time, program name, number of running processes (cpu_cnt), and percent utilization. The groups are sorted into root/non-root collections, with each terminated with a total CPU percentage for the group.
cpud_tids log_tids_file /var/log/cpud/cpud_tids Defines the file used for logging tid/pid counts.
cpud_user log_user_file /var/log/cpud/cpud_user defines the file used for logging statistics captured by the log_user parameter. The output includes a date/time stamp, UNIX time, iteration counter, and then the following values: sleep time, user UID, percentage of CPU utilized, execs, and the exec rate. At the end of each grouping of the top users is a summary showing the system total CPU utilization, customer total utilization, and the total execs and exec rate.

cpud enabe logging parameters

Parameter Syntax Default Description
enable_logging 1 ON Enables logging of information and errors for cpud. Set to '1' to enable logging, set to '0' to disable logging. This a the global logging toggle that controls all the other logging options.
enable_dumps 1 ON Enables or disables triggering of dumps for diagnostic use. When enabled, sending a SIGUSR1 signal to cpud will trigger a dump. Sending SIGUSR2 will reset the dump statistics.
enable_sched_dumps 0 OFF Enables or disables the inclusion of process scheduler wait times when enable_dumps is set to '1'. If enable_dumps is set to '0', then this parameter has no effect.
log_cpu 1 ON Enables or disables logging of CPU status information, such as how many cores are in the jail, how many processes are in the jail, and how many processes are not in jail. This information is logged to the file defined in log_cpu_file.
log_cpud 1 ON Enables or disables all logging. When this option is enabled, other logging switches are used to fine tune what logging is enabled.
log_failure 1 ON Enables or disables logging of jail expansion failures. This information is logged to the file defined in log_cpud_file.
log_gov 1 ON Enables or disables logging of governor activities. This information is logged to the file defined in log_gov_file.
log_jail 1 ON Enables or disables logging of the number of users in jail, how long they have been in jail, and how many processes have been jailed. This information is logged to the file defined in log_jail_file.
log_killed 1 ON Enables or disables logging of information about processes logged by cpud. This information is logged to the file defined in log_memcon_file.
log_mem 0 OFF Defines the number of top memory consuming processes to log. This information is logged to the file defined in log_mem_file.
log_memcon 1 ON Enables or disables logging processes killed for using too much memory. This information is logged to the file defined in log_memcon_file.
log_panics 1 ON Enables or disables logging of killed processes when cpud is in panic mode.
log_prog 0 OFF Defines the number of jailed programs to be logged, starting from the program using the highest percentage of CPU and working down by CPU% utilization. This information is logged to the file defined in log_prog_file.
log_tids 0 OFF Enables or disables logging of how many processes and threads are running per core. This information is logged to the file defined in log_tids_file.
log_user 0 OFF Defines the number of jailed users to be logged, starting from the user using the highest percentage of CPU and working down by CPU% utilization. This information is logged to the file defined in log_user_file.
log_user_cnt 1 ON Enables or disables logging of the total number of users in jail. This information is logged to the file defined in log_cpud_file.
log_flush_rate 1 ON Defines how frequently logs are committed to disk in sleep periods. For example, if log_flush_rate is set to 2 and the sleep_time_ms is set to 500, then the log is committed to disk once a second.
log_jail_idle 10 10 Defines the idle CPU percentage in jail at which a "cpu busy" message is logged. These messages are written to the file defined in log_cpud_file.
log_nonjail_idle 10 10 Defines the idle CPU percentage not in jail at which a "cpu busy" message is logged. These messages are written to the file defined in log_cpud_file.
log_success 1 ON Enables or disables logging of information about successful jail expansion or reduction. This information is logged to the file defined in log_cpud_file.
log_user_in_jail 1 ON Enables or disables logging of users being added to jail. This information is logged to the file defined in log_cpud_file.

Monitoring Tools

blstat

blstat is now part of the main product installation. If for some reason you need to install it separately, follow the steps below to download and install blstat:

If you are using cPanel, please run:

/scripts/perlinstaller Term::ReadKey JSON

Or if you are running stock CentOS 6.x, please run:

yum --disableexcludes=all install perl-TermReadKey perl-JSON

This will install two perl modules blstat needs.

wget http://repo.betterlinux.com/unsupported/blstat-0.56-1.el6.noarch.rpm

Downloads the blstat rpm.

rpm -ivh blstat-0.54-1.el6.noarch.rpm

command to install blstat rpm

At the command line as root blstat is executed by running:

blstat

Hot Keys

First screen is the i/o screen.

    c - Toggle cpu throttling screen
    i - Toggle i/o throttling screen
    q - exit's blstat

myusertop

Description

Display the MySQL users most active during the preceding 4 seconds. Show the Linux users and UIDs that "own" them. Press single "hot" keys to change the display interactively.

Hot Keys

    V - Toggle views 1 and 2
    m - Toggle the display of MySQL users under "owning" Linux users
    H - Toggle hourly/total statistics in header area (only)
    R - Reverse sort direction (ascending/descending)
    f - Invoke FLUSH USER_STATISTICS (to reset statistics to zero)
    Sort by:
    a - Adjusted rows examined                        (view 1)
    b - Busy time during last refresh interval        (view 1)
    c - CPU time during last refresh interval         (view 1)
    C - CPU time / busy time                          (view 1)
    e - Core number at end of latest query            (view 2)
    i - UID                                           (all views)
    l - Busy time of latest completed query           (view 2)
    L - CPU time of latest completed query            (view 2)
    n - Number of non-voluntary context switches in 
          latest completed query                      (view 2)
    o - "Other" commands during last refresh interval (view 2)         
    p - % of total CPU time used by user              (view 1)
    Q - Number of queries in progress                 (view 2)
    r - Rows examined                                 (view 1)
    s - Select commands during last refresh interval  (view 2 default)
    S - Start core number of latest query             (view 2)
    t - Number of threads per user                    (view 1 default)
    u - Update commands during last refresh interval  (view 2)
    U - User name                                     (all views)
    v - Number of voluntary context switches in 
          latest completed query                      (view 2)

Other:

    h,? - Display online help
    q   - Quit B<myusertop>

Column Headings

    Linux/MySQLUser - MySQL user is indented beneath Linux user name
    Thds            - Number of threads at the current time
    CPUTime         - Seconds of CPU time since previous refresh
    BusTime         - Seconds of busy (clock) time during same interval
    CPU/Bus         - CPU time divided by busy time
    RowsExam        - Number of rows "examined" since previous refresh
    AdjRowEx        - RowsExam * CPU / Bus
    %CPU            - Amount of MySQL's CPU usage used by this user
    Sel             - Number of SELECT commands since previous refresh
    Upd             - Number of UPDATE commands since previous refresh
    Oth             - Number of other commands since previous refresh
    VCxSw           - Number of voluntary context switches since previous
                        refresh
    NVCSw           - Number of non-voluntary context switches since
                        previous refresh
    SC              - Core number at start of most recent query
    EC              - Core number at end of most recent query
    LQCpu           - CPU seconds of most recent query
    LQBus           - Busy (clock) seconds of most recent query

Command-line Options

Most of the command-line options (below) can be specified interactively while myusertop is running by pressing single "hot" keys. (See above.) Many of the options can be set as defaults in the configuration file

   */etc/betterlinux/mysql.conf.d/myusertop.conf*.

Command-line options (preceded by 1 or 2 hyphens) may be abbreviated to the shortest unique prefix):

   -ve, --version

Display the version of myusertop and exit.

   -nom, --nomyuser

Do not display the MySQL users "owned" by each top Linux user. (The default is to list each Linux user's associated MySQL users, indented, beneath the Linux user.) During an interactive session, the "m" hot key toggles the display of "owned" MySQL users on and off.

   --cfg=<filename>, --config=<filename>

Override the default (/etc/betterlinux/mysql.conf.d/myusertop.conf) configuration file name with <filename>.

   -S [path], --socket=[path]

Path specifying unix domain socket to pass to the mysql command.

   -ho, --hourly

In the header summary information, show hourly statistics (instead of the totals since mysqld was started or FLUSH USER_STATISTICS reset the statistics. (If mysqld has been running for less than one hour--or if FLUSH USER_STATISTICS was invoked within the hour--values will be adjusted upward to show the hourly rate if usage continues at the same rate for the remainder of the hour.)

During an interactive session, the "H" hot key toggles the header between displaying hourly and total statistics.

Note: Regardless of whether the header displays hourly statistics or statistics since startup/flush time, the main user columns show usage during the most recent refresh interval.

   --refresh=<seconds>

Specify the refresh interval in seconds. The default refresh interval is 4 seconds.

--admin=<admin-user-name> Specify the name of a super MySQL user with authority to issue MySQL administrative commands. This option is to be used ONLY in cases where the root unix user (the one running this script) needs to specify a MySQL superuser name when starting the mysql client.

   --adminpass=<admin-user-password>

Specify the password of the MySQL user specified with the --admin option. This option is likewise used ONLY in cases where the root unix user (the one running this script) needs to specify a MySQL superuser name when starting the mysql client.

   --ascending

Sort ascending (from smallest to largest). This effectively makes myusertop show the *bottom* users. During an interactive session, the "R" hot key reverses the current sort direction. Use this option in combination with any *one* of the following sort options.

   --sortthreads

Sort users by the number of threads used by Linux users at the moment of the most recent refresh This is the default in view 1. During an interactive session, the "t" hot key will sort the number of threads.

   --sortcpu

Sort by CPU time (from highest to lowest). During an interactive session, the "c" hot key will sort by CPU time.

   --sortadjre 

Sort by adjusted rows examined (from highest users to lowest). Adjusted rows examined is the number of rows examined multipled by the ratio of CPU Time to Busy Time. During an interactive session, the "a" hot key will sort by adjusted rows examined.

   --sortbusy

Sort by busy (clock) time (from highest to lowest). During an interactive session, the "b" hot key will sort by busy time.

   --sortcob

Sort by ratio of CPU Usage to Busy Time. "CPU over Busy. During an interactive session, the "C" hot key will sort by the ratio of CPU time to busy time.

   --sortrows

Sort by number of rows examined (from highest to lowest.) During an interactive session, the "r" hot key will sort by the the number of rows examined.

   --sortcpupercent

Sort by the percentage of MySQL CPU time that the user is using. During an interactive session, the "p" hot key will sort by CPU ercentage.

   --sortselect

Sort by the number of select commands issued (from highest to lowest). During an interactive session, the "s" hot key will sort by the number of select commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortupdate

Sort by the number of update commands issued (from highest to lowest). During an interactive session, the "u" hot key will sort by the number of update commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortother

Sort by the number of other commands issued (from highest to lowest). During an interactive session, the "o" hot key will sort by the number of other commands. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortvcs

Sort by the number of voluntary context switches (from highest to lowest) during the most recent query. During an interactive session, the "v" hot key will sort by the number of voluntary context switches. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortnvcs

Sort by the number of nonvoluntary context switches (from highest to lowest) during the most recent query. During an interactive session, the "n" hot key will sort by the number of nonvoluntary context switches. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortqip

Sort by number of active queries in progress (from highest to lowest) for each user at the current moment. During an interactive session, the "Q" hot key will sort by the number of queries in progress. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortlqcpu

Sort by last query CPU time (from highest to lowest)--the number of seconds of CPU time used by each user's most recently completed query. During an interactive session, the "L" hot key will sort by the last query CPU time. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

   --sortlqbusy

Sort by last query busy time (from highest to lowest)--the number of seconds of busy time used by each user's most recently completed query. During an interactive session, the "l" hot key will sort by the last query busy time. (This value appears in view 2, which is displayed by pressing the "V" hot key when view 1 is being displayed.)

Files

   /etc/betterlinux/mysql.conf.d/myusertop.conf

See Also

   myusertop.conf(8)

Parameters

You can create and name your own .conf files containing the parameters below, as long as your file retains the “.conf” extension. You cannot mix CPU and I/O parameters in the same .conf file. And for now, any groups you create must go in the pre-existing “groups .conf” file. All CPU and I/O .conf files you create should go here, respectively:

/etc/betterlinux/cpud.conf.d
/etc/betterlinux/iothrottled.conf.d

Parameter names are hyperlinked to their associated man pages, while their feature names are hyperlinked to the documentation section most relevant to that parameter.

Parameter name Feature
add_user_time CPU
approved_accounting_program CPU/MySQL
auto_scale IO
average_type IO
auto_wakeup IO
bandwidth_limit BANDWIDTH/CPU
block_suid CPU
busy_nonjail_idle CPU
busy_num_R_nonjailed CPU
cgroup_dir IO
cloak_user CPU
cpu_average_periods CPU
enable_dumps CPU
enable_logging CPU
enable_logging IO
enable_program_limits IO
enable_sched_dumps CPU
in_buf_sz CPU
interruptible_sleeps IO
iothrottled_gid IO
jail_cnt CPU
jail_min_idle CPU
load_average CPU
log_auto_scale IO
log_cgroups IO
log_cnt_file IO
log_cpu CPU
log_cpu_file CPU
log_cpud CPU
log_cpud_file CPU
log_failure CPU
log_file IO
log_flush_rate CPU
log_gov CPU
log_gov_file CPU
log_io_limits IO
log_jail CPU
log_jail_file CPU
log_jail_idle CPU
log_killed CPU
log_mem CPU
log_mem_file CPU
log_memcon CPU
log_memcon_file CPU
log_nonjail_idle CPU
log_panics CPU
log_panics_file CPU
log_prog CPU
log_prog_file CPU
log_seek_balancing IO
log_success CPU
log_throttled IO
log_tids CPU
log_tids_file CPU
log_user CPU
log_user_cnt CPU
log_user_file CPU
log_user_in_jail CPU
max_cgroups IO
max_jail_cnt CPU
memory_limit CPU
nonjail_min_idle CPU
process_limit CPU
process_limit_exclude CPU
process_limit_override CPU
run_priority CPU
sample_periods IO
scale_max IO
scale_min IO
seek_balancing IO
sleep_time_ms CPU
sleep_time_ms IO
start_throttling CPU
stop_throttling CPU
throttle_group CPU
time_monitor CPU
user_average_periods CPU

Man Pages

CPUD.CONF(5) [FIXME: manual] CPUD.CONF(5)




NAME
cpud.conf - cpud configuration file parameters


SYNOPSIS

CPUd limits user and process access to CPU resources on a server, throttling a process down so it doesn't hog the CPU, and moving users or processes into a reserved "jail" so they don't interfere with processes that use the system responsibly.

The CPUd daemon also can kill processes that use too much memory and appear to be hung.

load_average

Provides the ability to configure the load average calculation across non-jail cpu cores.

"load_average" settings provide admins with the ability to configure the proper load average for the system. This setting affects the behavior of the load average for every application that runs on the system.

Parameter Category
Advanced

Syntax
load_average type=all or non_jail_only normalize_load=on or off.

Example
load_average type=all normalize_load=off
Will add up the load for all the cores.

load_average type=all normalize_load=on
Will add up the load for all the cores (jail and non_jail) and divide the total load by the total number of cores.

load_average type=non_jail_only normalize_load=off #[Default]
Will add up the load for all non-jail cores. This is the default.

load_average type=non_jail_only normalize_load=on
Will add up the load for all the non-jail cores and divide by the total number of non_jail cores.


Configuration File

cpud Configuration File Names

cpud Stock Linux Config Files

/etc/betterlinux/cpud.conf.d

  • betterlinux.conf
  • groups.conf

cpud cPanel Config Files

/etc/betterlinux/cpud.conf.d

  • [package-name]_package.conf
  • cpanel-allusers.conf
  • cpanel.conf
  • cpanel-groups.conf

iothrottled Configuration File Names

iothrottled Stock Linux Config Files

/etc/betterlinux/iothrottled.conf.d

  • betterlinux.conf
  • betterlinux-io.conf

iothrottled cPanel Config Files

/etc/betterlinux/iothrottled.conf.d

  • cpanel.conf
  • cpanel-groups.conf