Difference between revisions of "Old Doc Pages"

From docs.betterlinux.com
Jump to: navigation, search
Line 206: Line 206:
 
  cat /proc/sys/net/ipv6/dedicated
 
  cat /proc/sys/net/ipv6/dedicated
 
Note:  The command output displays the uid rather than the username.
 
Note:  The command output displays the uid rather than the username.
 +
 +
 +
=== BetterCPU Configuration ===
 +
Refer to the '''[[BetterCPU HOWTO Collection|BetterCPU HOWTOs]]''' section for details on configuration and setup.
 +
 +
=== BetterCPU Concepts ===
 +
BetterCPU is designed to maximize cpu utilization for shared hosting environments and at the same time prevent one or many users from abusing the cpu utilization.  BetterCPU makes it very easy to quickly identify the CPU hogs on the system and keep them under control and then decide if these users need to moved to a higher paying account that provides them with the resources they need to run their business.  This is accomplished by defining BetterLinux groups of users or mapping BetterLinux groups to cPanel packages.  Once the user groups are configured, then throttling limits are defined for CPU.
 +
 +
=== Defining jail cores ===
 +
Before limits are configured, it's important to understand BetterCPU jails.  The jails are elastic by nature and only exist if they are needed by the BetterCPU controller to temporarily limit an abusive user process.  Think of process jailing as analogous to law enforcement:
 +
 +
* A user is a “citizen”
 +
* The performance thresholds are the “law”
 +
* BetterCPU enforces the laws
 +
 +
A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.
 +
 +
If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.
 +
 +
When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.
 +
 +
Configuration of the jail is done using the parameters:
 +
 +
jail_cnt
 +
max_jail_cnt
 +
 +
Commenting both of the parameters out sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1.  BetterCPU then auto-scales the jails the system needs based on demand for jail cores.  However, the jail cores will never take a priority over non-jail cores.
 +
 +
The ''jail_cnt'' can be defined using two different methods:
 +
*If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jai_cnt.  For example:
 +
jail_cnt 2
 +
 +
The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems.  If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.
 +
 +
*is expressed as one or more pairs of numbers. For example:
 +
jail_cnt 4:1 8:2 16:2 24:6
 +
 +
The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail<nowiki>; </nowiki>an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores.  In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system.  The 32 core system will be assigned 6 cores as the jail.
 +
 +
The ''max_jail_cnt'' parameter defines the upper limit for the number of jails, and is written in the same manner as the ''jail_cnt'' parameter.  Setting the max_jail_cnt parameter will fix the ultimate size of the jail.  Fixing the jail size will determines the amount of the cpu utilization abusive users will share in the jail core(s). 
 +
 +
By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.
 +
 +
=== Defining BetterCPU Throttling Limits ===
 +
The BetterCPU throttling limits are defined by a percentage of one core for each user.
 +
 +
Once the jail cores are defined, BetterCPU provide two throttling parameters that define when a user is moved into and out of the jail core(s).  The syntax for these thresholds is defined with the following parameters:
 +
 +
start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
 +
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
 +
 +
These parameters set jail throttling limits for groups of uids defined in groups.conf or betterlinux-cpud.conf on cPanel installs.  The cpu_percent sets the percentage a user (uid) can use of one core on the system.  The values for cpu_percent in start_throttling and stop_throttling are based on one core and are not adjusted when cores are added or removed from the jail.
 +
 +
When a throttled user’s total utilization drops below the ''stop_throttling'' for ''cpu_percent'' and "seconds" settings, then the process is removed from the jail.
 +
 +
For example, consider an 8-core system with the following values configured:
 +
 +
start_throttling group=example-group cpu_percent=50,time=3
 +
stop_throttling group=example-group cpu_percent=25,time=10
 +
 +
A user whose processes exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.
 +
 +
 +
<!--The primary function of BetterCPU is to manage system CPU and memory resources. Configuration of the daemon is done through a series of configuration files located in /etc/betterlinux/cpud.conf.d/:
 +
 +
For cPanel Servers:
 +
*betterlinux-cpud.conf
 +
*groups.conf
 +
 +
For non-cPanel Servers:
 +
* main.conf
 +
* logging.conf
 +
* mysql.conf
 +
* rules.conf
 +
* users.conf
 +
* groups.conf
 +
 +
BetterCPU’s primary configuration file is the main.conf file, which contains the basic parameters used for configuring the daemon.
 +
 +
The basic settings used to configure BetterCPU consist of:
 +
 +
sleep_time_ms
 +
user_average_periods
 +
cpu_average_periods
 +
 +
CPU usage is calculated using averages of samples taken over time. BetterCPU profiles usage by process and by user on a regular interval to determine which processes and users are utilizing the CPU the most, and if those usages exceed permitted thresholds, then BetterCPU manages those processes in a way that limits their ability to impact overall system performance.
 +
 +
The ''sleep_time'' parameter is used to determine the number of milliseconds elapse between BetterCPU’s snapshots.
 +
 +
The averages for user and CPU utilization (''user_average_periods'' and ''cpu_average_periods'', respectively) are calculated based on the number of periods (or samples) taken. The defaults for these parameters are:
 +
 +
sleep_time_ms 1000
 +
user_average_periods 2
 +
cpu_average_periods 2
 +
 +
This means snapshots are averaged every 2 periods, and each period is 500 ms long. As a result, the averages are calculated for a 1 second basis every 500 ms.
 +
 +
When evaluating the cpu_average_periods and user_average_periods, if a process or user is found to be exceeding predetermined utilization thresholds, the user will be “jailed”.
 +
 +
=== Starting and stopping BetterCPU ===
 +
After the installation is complete, a reboot of your system is required to load your BetterLinux kernel and modules.  Once your system is rebooted, the BetterCPU deamon is not running by default and requires a manual start using service.
 +
Example:
 +
service cpud start
 +
 +
Restarting and stoping examples:
 +
service cpud stop
 +
service cpud restart
 +
 +
Using the condrestart will restart the daemon but only if it's running.
 +
Example:
 +
service cpud condrestart:
 +
 +
When changes are made to the configuration files, the BetterCPU daemon must reload the new settings.  This is can be done by either running the service restart option or by running the service reload command.
 +
Example:
 +
service cpud reload
 +
 +
=== Defining Groups ===
 +
Understanding and defining groups is a key concept that is often combined with the BetterCPU parameter settings.  BetterLinux groups allows a system administrator to organize different types of users and programs by group.  For example, a hosting company may have various customer subscription levels with each level requiring resources from the server.  Having the ability to divide customers by group, allows BetterLinux limits to be set by customer group.  Groups can be defined by UIDs, UID ranges, user names, or files containing lists of usernames or UIDs.
 +
 +
==== Fixed Groups ====
 +
The BetterCPU groups.conf file contains all the settings for configuring groups.  There are three types of groups, fixed, user, and program. The fixed groups are predefined group names that cannot be changed.  Their names and functions are set, but they have not yet been populated with users. In order to configure these groups, you must populate them with UIDs, UID ranges, user names, or files of UIDs or usernames.  The concept behind fixed groups is to create a predefined group of users that will aways be throttled into a jail or never be throttled into a jail.
 +
 +
=====The ALWAYS_THROTTLE_CPU_GROUP=====
 +
Lets you list members that will always be throttled. This group and the other fixed group "NEVER_THROTTLE_CPU_GROUP" are not tied to the start start and stop throttling syntax.  The purpose of the NEVER_THROTTLE_CPU_GROUP settings is to have a place for the system administrator to put known spammers or users that are consistently abusing the server.  These settings do not have cpu percent usage or time settings.  Any users put in this group are ALWAYS throttled into the jail and will never be let out.   
 +
 +
To use this group, uncomment its line below and populate the group according to the syntax and examples to follow:
 +
 +
uid ALWAYS_THROTTLE_CPU_GROUP {members} #[Default: empty]
 +
 +
Throttle UIDs:
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid1}[,{uid2}...]
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP 13,15,19 (Always throttles UIDs 13,15,19)
 +
 +
Throttle UID range:
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid range}
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP 500-999 (Always throttles UIDs from 500-999)
 +
 +
Throttle usernames:
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {username1}[,{username2}...]#
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP bluser1,bluser2 (Always throttles listed users)
 +
 +
Throttle both a UID range and usernames:
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid range}[,{username1}...]
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP 500-999,bluser1
 +
 +
Throttle a two UID ranges and a username:
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid range}[,{uid range},{username1}...]
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP 500-999,2-50,bluser1
 +
 +
Throttle a file of UIDs: (where the file contains a delimited list of UIDs)
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {file path}
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP /etc/passwd
 +
 +
Throttle a UID range, a username, and a file of usernames:
 +
(where the file contains a delimited list of usernames)
 +
 +
Syntax: uid ALWAYS_THROTTLE_CPU_GROUP {uid range}[,{username},{file path}...]
 +
Example: uid ALWAYS_THROTTLE_CPU_GROUP 500-999,bluser1,/var/cpanel/users/usernames
 +
 +
=====The NEVER_THROTTLE_CPU_GROUP=====
 +
Lets you list members that will never be throttled into a jail. This group accepts UIDs, UID ranges, user names, or files of UIDs or usernames.  The purpose of this feature is to give the system administrator management a place to put senior tech users who need to be able to do anything on the system without being throttled into the jail.
 +
 +
To use this group, uncomment its line below and populate it using syntax identical to that used with "Always_throttle_cpu_group" above. See the above syntax and examples.
 +
 +
uid NEVER_THROTTLE_CPU_GROUP {members} #[Default: empty]
 +
 +
==== User Groups ====
 +
User-created groups do not exist on the system until they have been created by the system administrator. By creating and populating your own groups with UIDs, UID ranges, usernames, or files of UIDs or usernames, you can then precisely define for your group the CPU usage levels and their durations that merit throttling, and the conditions for ending throttling.  These group names will be required to list in the syntax for the [[http://docs.betterlinux.com/wiki/BetterCPU#Setting_throttling_thresholds_for_jail_cores|start and stop throttling parameters]]
 +
 +
To create your own groups, choose a group name, then populate it with UIDs, UID ranges, user names, or files of UIDs or usernames, as follows:
 +
 +
Group with a UID range example:
 +
Syntax: uid {group name} {uid range}
 +
Example: uid example-group 0-499
 +
Example: uid example-group 800- (this includes all UIDs from 800 and above)
 +
 +
Group with multiple UID ranges:
 +
 +
Syntax: uid {group name} {uid range}[,{uid range2}...]
 +
Example: uid example-group 0-499,800-850
 +
 +
Group with UID range and a username:
 +
 +
Syntax: uid {group name} {uid range},[,{username},...]
 +
Example: uid example-group 0-499,nfsnobody
 +
 +
Group with multiple usernames:
 +
 +
Syntax: uid {group name} {username1},[,{username2}...]
 +
Example: uid example-group bluser1,bluser2
 +
 +
Group with files of UIDs or usernames: (files must use a comma delimited list)
 +
 +
Syntax: uid {group name} {file path}[,{file path2}...]
 +
Example:  uid example-group /etc/passwd
 +
Example: uid cpanel_users /var/cpanel/users/username
 +
 +
Groups made from external file using regular expresion and the cut command (must be a comma delimited list).  This is escpecially handy if you are pulling a list from the passwords file or some other file that keeps lists of created users by a program like cPanel.
 +
 +
Syntax: uid {group name} `cut -d: -f1 {file path}`
 +
uid bl_local_users `cut -d: -f1 /etc/passwd`
 +
Result: Using ":" as a delimiter, this syntax reads field one (the UID field) from the passwd file, making these UIDs members of the group "bl_local_users."
 +
 +
==== Program Groups ====
 +
BetterCPU has multiple features for controlling limits for programs.
 +
 +
To create your program groups, choose a group name, then populate it with programs by location and name seperated by a comma with no spaces, as follows:
 +
 +
Group with programs example:
 +
Syntax: prog {group name} {/location/program-name,/location/program-name2,/location/program-name3}
 +
Example: prog example-group /usr/sbin/sshd,/usr/bin/rsync
 +
 +
==== Use groups with other configuration files====
 +
User-created groups are meant to be used within other configuration files. Certain parameters within main.conf (/etc/betterlinux/BetterCPU/cpud.conf.d/) accept
 +
user-created group names as values. You can use multiple lines to setup different cpu and time usage per user group.
 +
 +
For example, since start_throttling is a main.conf parameter or ( for cPanel servers betterlinux-cpud.conf) that accepts group names, a group defined as "example-group uid 0-499" could be used as follows:
 +
 +
Syntax: start_throttling group={group} cpu_percent={integer},time={integer}
 +
Example: start_throttling group=example-group cpu_percent=40,time=3
 +
This setting will throttle any member of example-group (UIDs 0-499) exceeding 40% CPU usage lasting over three seconds.
 +
 +
You can also use multiple groups within the same parameter:
 +
 +
Syntax: start_throttling group={group}[,{group2}...] cpu_percent={integer},time={integer}
 +
Example: start_throttling group=example-group1,example-group2 cpu_percent=40,time=7
 +
This configuration will throttle any member of example-group1 or example-group2 exceeding 40% CPU usage lasting over seven seconds.
 +
 +
For completeness, here is an example of the stop_throttling parameter in main.conf, the counterpart to start_throttling:
 +
 +
Syntax: stop_throttling group={group}[,{group2}...] cpu_percent={integer},time={integer}
 +
Example: stop_throttling group=example-group cpu_percent=30,time=10
 +
This setting will stop throttling any member of example-group (UIDs 0-499) with less than 30% of one core usage lasting over ten seconds.
 +
 +
Example: stop_throttling group=example-group cpu_percent=20,time=5
 +
Any throttled member of example-group (UIDs 0-499) with less than 20% usage of one core lasting over five seconds will stop being throttled.
 +
 +
These parameter can be setup in pairs to set different cpu and time usage per group.  Each start_throttling needs to have a paired stop_throttling.  The order doesn't matter. 
 +
For example:
 +
start_throttling group=customer1-group cpu_percent=30,time=5
 +
stop_throttling group=customer1-group cpu_percent=20,time=8
 +
start_throttling group=customer2-group cpu_percent=40,time=7
 +
stop_throttling group=customer2-group cpu_percent=30,time=10
 +
 +
=== Defining jail cores ===
 +
Think of process jailing as analogous to law enforcement:
 +
 +
* A user is a “citizen”
 +
* The performance thresholds are the “law”
 +
* BetterCPU enforces the laws
 +
 +
A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.
 +
 +
If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.
 +
 +
When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.
 +
 +
Configuration of the jail is done using the parameters:
 +
 +
jail_cnt
 +
max_jail_cnt
 +
 +
Commenting both of the parameters out sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1.  BetterCPU then auto-scales the jails the system needs based on demand for jail cores.  However, the jail cores will never take a priority over non-jail cores.
 +
 +
The ''jail_cnt'' can be defined using two different methods:
 +
*If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jai_cnt.  For example:
 +
jail_cnt 2
 +
 +
The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems.  If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.
 +
 +
*is expressed as one or more pairs of numbers. For example:
 +
jail_cnt 4:1 8:2 16:2 24:6
 +
 +
The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail<nowiki>; </nowiki>an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores.  In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system.  The 32 core system will be assigned 6 cores as the jail.
 +
 +
The ''max_jail_cnt'' parameter defines the upper limit for the number of jails, and is written in the same manner as the ''jail_cnt'' parameter.
 +
 +
By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.
 +
 +
=== Setting throttling thresholds for jail cores ===
 +
 +
Once the jail cores are defined, BetterCPU provide two throttling parameters that define when a user is moved into and out of the jail core(s).  The syntax for these thresholds is defined with the following parameters:
 +
 +
start_throttling group={group}[,{group2}...] cpu_percent={integer},time={integer}
 +
stop_throttling group={group}[,{group2}...] cpu_percent={integer},time={integer}
 +
 +
These parameters set jail throttling limits for groups of uids defined in groups.conf.  The cpu_percent sets the percentage a user (uid) can use of the total cores on the system.  In an 8-core system, the maximum utilization is 800% (100% * 8 cores). The values for cpu_percent in start_throttling and stop_throttling are based on one core and are not adjusted when cores are added or removed from the jail.
 +
 +
When a throttled user’s total utilization drops below the ''stop_throttling'' for ''cpu_percent'' and "time" settings, then the process is removed from the jail.
 +
 +
For example, consider an 8-core system with the following values configured:
 +
 +
start_throttling group=example-group cpu_percent=50,time=3
 +
stop_throttling group=example-group cpu_percent=25,time=10
 +
 +
A user whose processes exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.
 +
 +
When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.
 +
 +
Those utilization figures can be divided between the cores in any combination - the ''start_throttling'' and ''stop_throttling'' parameters define an average utilization per core. If, in the example above, a process was utilizing 25% of one core and 30% of another core, the aggregate utilization exceeds the 50% of one core threshold set by the "cpu_percent" value and would therefore be throttled into the jail.
 +
 +
=== Managing Memory Usage ===
 +
Using the ''memory_limit_*'' configuration parameters, a system administrator can declare that for users managed by BetterCPU, there is a maximum amount of Resident Set Size (RSS) memory that a users process can use.
 +
 +
It is important to note that a program can use more than the defined value of ''memory_limit'' by having swap space available. RSS memory refers to physical RAM in the machine, and programs consuming it are consuming “Resident” memory - defined as “Code+Data” but not including swap space.
 +
 +
The memory_limit_* parameters are useful for setting memory limits by program.  Below is the syntax for these parameters:
 +
 +
memory_limit group=GROUP,GROUP rss_memory=2048(megabytes) program=/path/to/script/to/run
 +
memory_limit_prog_exclude GROUP
 +
 +
==== Memory limit by program ====
 +
 +
In the example below, we will define a 1G memory limit for a program group with rsync in it.  If rsync hits the limit, a script of your choosing will be executed to run.  In this example, we will use a bash script that will run kill -9 once the memory limit is met.
 +
* Define the following group in groups.conf:
 +
 +
prog mem_hogs /usr/bin/rsync
 +
 +
* In this file (main.conf), configure memory_limit:
 +
 +
memory_limit group=mem_hogs rss_memory=1000 program=/home/user_name/bin/script_name_kill
 +
 +
==== Memory limit by user (uid) example ====
 +
 +
In the next example, we will configure a memory_limit for any program a user may execute:
 +
 +
* In groups.conf setup a group with a UID range:
 +
 +
uid mem_user_limits 500-1000
 +
 +
* Use the following syntax in this file to limit by the group we just defined.  Once the 1G limit is met by any program executed within the UID range, it will be killed by our kill script.
 +
 +
memory_limit group=mem_user_limits rss_memory=1000 program=/home/user_name/bin/script_name_kill
 +
 +
==== Memory limit programs to exclude example ====
 +
The memory_limit_prog_exclude parameter is used to define a group(s) of programs to exclude.  This can only be used in the example we used above, when limiting the memory for any program executed by a UID/User in the defined group range.
 +
 +
In this exclude example, we will define a program group in groups.conf to exclude /user/bin/mysqld from the settings defined in memory_limit.
 +
* In groups.conf setup a group of programs to exclude:
 +
 +
prog mem_exclude_progs /usr/bin/mysqld
 +
 +
* And then include this group name in the exclusion parameter:
 +
 +
memory_limit_prog_exclude mem_exclude_progs
 +
 +
=== Limiting process time to run ===
 +
As a system administrator managing a multi-tenant linux environment, you may find it helpful to be able to control how long a users process can run.  With the process time monitor feature you can define what programs like apache, tar, sshd, rsync, etc. you want to monitor and execute a script against.  The time_limit feature allows you do to configure a specific program(s) to watch and then trigger with a time limit when to run a script of your choosing.  Once the time limit for those processes is breached, BetterCPU will run the script you have defined. The script could be a bash script that kills the program, restarts it, or logs it, etc.  Additionally, there is an companion exclusion parameter (time_monitor_prog_exclude) that allows you go exclude certain programsThe Syntax for time_monitor_* is as follows:
 +
 +
time_monitor group=GROUP,GROUP seconds=time(seconds), child_of= /parent/program,  program=/location/script/to/run wait_to_run=time(seconds to rerun script if until it's affective)
 +
time_monitor_prog_exclude GROUP,GROUP
 +
 +
These parameters can be defined by a group containing a program(s) in  groups.conf.
 +
 +
==== Configure time_monitor example ====
 +
* First define a program group in groups.conf:
 +
 +
prog bad_progs /usr/bin/vmstat,/usr/bin/top
 +
 +
prog = the switch to define programs
 +
bad_progs = name of the group
 +
/user/bin/vmstat,user/bin/top = location and name of program(s) to watch
 +
 +
* Then use bad_progs group with the time_monitor setting.  This syntax will kill top or vmstat after ten seconds when running via sshd and if it didn't kill them all it will rerun every 30 seconds:
 +
 +
time_monitor group=bad_progs seconds=10 child_of=/user/sbin/sshd program=/home/user/script_name_kill wait_to_run=30
 +
 +
==== Amount of time a program will run example ====
 +
In the next example, we are configuring time_monitor to watch the amount of time any program will run by UID/User:
 +
 +
* In groups.conf setup a group with a UID range:
 +
 +
uid bl_users 500-1000
 +
 +
* Then we use the following syntax in this file for watching the programs of the UIDs in bl_users and executing a kill script for any that run longer than ten seconds.
 +
 +
time_monitor group=bl_users seconds=10 child_of=/user/sbin/sshd program=/home/user/script_name_kill
 +
 +
==== Program groups exclusion example ====
 +
The time_monitor_prog_exclude parameter is used to define a group(s) of programs to exclude.  This can only be used in the example we used above, when limiting the time any program will run by UID
 +
range.
 +
 +
In this exclude example, we will define a program group in groups.conf to exclude /usr/bin/rm and /user/bin/rsync from the settings defined in time_monitor.
 +
* In groups.conf setup a group of programs to exclude:
 +
 +
prog exclude_progs /usr/bin/rm,/usr/bin/rsync
 +
 +
* And then include this group name in the exclusion parameter:
 +
 +
time_monitor_prog_exclude group=exclude_progs
 +
 +
=== Limiting total open processes by user ===
 +
 +
Controlling the number of processes a user can run will allow the system administrator to better allocate resources per user.  This is especially helpful in a hosting or cloud environment where users pay for a share of the same system resources.  Customers are not allowed to concurrently run more than the number programs defined using the BetterCPU process_limit settings.  The user will be unable to start the 101 program if their process limit is set to 100.
 +
 +
==== Limiting open process by group of users example ====
 +
The process_limit setting limits the total number of open processes per user.  The example below sets the number of open processes at 100 for all UIDs defined in group1 or group2.
 +
 +
* In groups.conf setup a group of users by uid range for group1 or group2:
 +
 +
uid group1 501-700
 +
uid group2 800-1000
 +
 +
process_limit group=group1,group2 limit=100
 +
 +
==== Limiting open process overrides example ====
 +
The process_limit_override sets exceptions for process limits by program. The example below sets an extra process limit of 120 for /bin/cron, and /usr/bin/ssh.  This setting is applied to the UIDs contained in Group1 and Group2.
 +
 +
process_limit_override group=group1,group2 limit=120 child_of=/bin/cron,/usr/bin/ssh
 +
 +
==== Example limiting open processes exclusion ====
 +
The process_limit_exclude parameter is for setting excludes from the settings in process_limit. The example below shows how /bin/ssd will be ignored from the limits defined in process_limit.  Additional programs can be added to a group by using a comma with no space between the programs.
 +
 +
* In groups.conf setup a program group of programs :
 +
 +
prog ignored_programs /bin/ssh
 +
 +
process_limit_exclude group=ignored_programs
 +
 +
=== Selecting Managed Users ===
 +
There are a few ways users can be selected for control by the BetterLinux software suite.
 +
 +
 +
Parameters in the users.conf file can be used to define a minimum and maximum UID value that define a range of identities to be managed by BetterCPU.
 +
There also are parameters that define UIDs or users for inclusion or exclusion from being managed by the suite. 
 +
 +
A simpler way to define managed users is through the use of the /etc/betterlinux/groups.conf file. This file contains entries  that define a group name and a list of users within that group. For example, the line:
 +
 +
uid cpu_limit testuser,testuser1,testuser2,testuser3,502
 +
 +
Adds the users in the list to the group ''cpu_limit''. A numeric value in this line is interpreted as a UID value which is translated by the system to a user name.
 +
 +
=== How can I tell BetterCPU is managing my system? ===
 +
A few tools have been mentioned already - top and htop will both show CPU utilization figures, memory utilization figures, and a breakdown of utilization by CPU core.  BetterLinux comes with it's own top-like program called [http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat blstat].
 +
 +
 +
If logging is enabled, an easy way to see that process are in being throttled in the jail core(s) is to monitor the file defined in ''log_cpud'' for jail expansion messages. When a user is jailed, the log will reflect that:
 +
 +
[[Image:Cpud_managing_system.png‎]]
 +
 +
The “Throttling User” message indicates that a user’s processes are being moved into the jail. One can also see messages indicating the jail cgroup is being created as well as information about the number of running programs that user has, their UID, and other information about that user.
 +
 +
 +
In htop, when a user is jailed, if the display of the CPU column (which displays the CPU number a process is running on) is enabled, it is easy to observe a user being moved into the jail. Here is a user using excessive CPU resources prior to throttling:
 +
 +
[[Image:Cpu_resources_prior_to_jailing.png‎ ]]
 +
 +
 +
Here is that same user after BetterCPU has jailed them for excessive CPU use:
 +
 +
[[Image:Cpud_jailed_them.png‎]]
 +
 +
Notice how the first image shows the utilization on all 4 CPUs of the system and how separate instances of the “jailme” process is running on each CPU. After the user is jailed, all of that user’s processes have been moved to CPU 4.
 +
 +
===User Blame Security===
 +
This setting enables BetterMySQL throttling and authorizes a program to apply usage blame to other users on the system.  This is used for programs like mysql and apache, which run under one uid, to divide the usage up by the individual users of the program.
 +
 +
In the example below we will authorize apache and mysql to assign blame to the correct user on the system.  Add one line per program / user:
 +
approved_accounting_program program=/usr/sbin/httpd user=apache
 +
approved_accounting_program program=/usr/libexec/mysqld user=mysql
 +
 +
This allows the program /usr/sbin/httpd as run by the apache user and /usr/libexec/mysqld as run by the mysqld user to determine if one of the web server or mysqld users is exceeding a configured resource constraint enforced by the BetterCPU (cpud) start and stop throttling parameters.  Instances of /usr/sbin/httpd run by other users are not affected unless a similar line is included in the configuration for that user as well.
 +
 +
=== BetterCPU Logging ===
 +
BetterCPU provides extensive logging functionality which allows a system administrator to see how BetterCPU is helping manage their systems. Logging configuration parameters fall into two general categories: Parameters to enable or disable specific logging functionality, and parameters to define the files messages from various components of BetterCPU are logged to.
 +
 +
 +
Overall logging of BetterCPU information is controlled by the ''log_cpud'' parameter. If this parameter is set to a value of ''1'', then BetterCPU events are logged based on the settings of other ''log_*'' parameters. If this parameter is set to a value of ''0'', then no logging is done by BetterCPU.
 +
 +
 +
General BetterCPU messages are logged to the file pointed to by the ''log_cpud_file'' parameter, but specific components of BetterCPU write to their own log files.
 +
 +
 +
The parameter documentation provides in-depth explanations of all of the logging functionality in BetterCPU. -->
 +
 +
 +
=== Throttling Users ===
 +
Once [[installation]] is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. By default, the non-cPanel installation (betterlinux-install) does not configure BetterCPU to throttle users.  However, the cPanel installation (betterlinux-cpanel-install) script does configure and start BetterCPU throttling for all cPanel users on the server. The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).
 +
 +
==== Configuring the Jail Cores====
 +
Throttling is configured using parameters located in:
 +
non-cPanel install: /etc/betterlinux/cpud.conf.d/rules.conf:
 +
cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf
 +
 +
jail_cnt
 +
max_jail_cnt
 +
 +
These settings are designed so that the same configuration file may be used on different systems with different numbers of cores.  However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file.  When these parameters are removed, the cpud daemon uses the default settings wich is jail_cnt=1 and max_jail_cnt=total cores minus 1 and dynamically auto-scales up or down the number of jail cores needed for the system.
 +
 +
Syntax:
 +
jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]
 +
 +
The ''jail_cnt'' parameter is defined using one of two methods:
 +
* If a fixed number of cores for jail is desired, make the assignment using ''jail_cnt'' as in this example:
 +
jail_cnt 2
 +
 +
:In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core.  If the system  is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.
 +
* If the number of cores for jail varies based on the total number of cores in the system, use the following format:
 +
jail_cnt 4:1 8:2 16:2 24:6
 +
 +
The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in ''jail_cnt'', BetterCPU will assign the next lowest value you have defined as the jail cores.  For example, using the values specified in the previous example, a 32-core system will use the ''jail_cnt'' value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.
 +
 +
The ''max_jail_cnt'' parameter defines the upper limit for the number of jails and is formatted identically to the ''jail_cnt'' parameter.  By default, only the value of ''jail_cnt'' is used, but when a ''max_jail_cnt'' is specified, the jail is expanded when less than the value in ''jail_min_idle'' is available per jail core.
 +
 +
The ''jail_min_idle'' value is the minimum amount of idle utilization to maintain on the jail cores if ''jail_max_cnt'' has not been reached.  If the average free utilization on each jail core is less than ''jail_min_idle'', the jail is expanded until either the ''jail_min_idle'' value is maintained or ''jail_max_cnt'' is reached.
 +
 +
Syntax:
 +
max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}
 +
Example
 +
max_jail_cnt 2:1 4:2 8:2 16:2
 +
The result on a dual core system is no different from ''jail_cnt''. But for 4-core machines and above, if the ''max_jail_cnt'' value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded.
 +
max_jail_cnt 2
 +
The result will always set the number of maximum jails to two.  The maximum number of jails are set at two no matter how many cores may be on the system.  The exception being, when they system has 2 cores it will set the max jail core to one.
 +
 +
In order for the jail to expand, add a ''jail_min_idle'' value to denote how much free utilization should be maintained in the jail:
 +
jail_min_idle 5
 +
 +
BetterCPU will attempt to maintain at least 5% free CPU per jail core, and will expand the jail if ''max_jail_cnt'' has not been reached.
 +
 +
==== Defining User Groups ====
 +
For more details on groups please see the [[Main_Page#Defining_Groups|Defining User Groups]] section of the documentation.
 +
 +
The BetterLinux cPanel install script will discover all the cPanel users on the system and create a user group called "cpanel-users" that contains a list of all the users assigned to a cPanel package.
 +
 +
The bl-install script will not create user groups, they need to be created by the system administrator. By creating and populating  groups with UIDs, UID ranges, usernames, or files containing UIDs or usernames, CPU usage levels can be precisely defined for different groups, and durations and conditions for starting/stopping throttling can be defined.  The group names defined in /etc/betterlinux/cpud.conf.d/groups.conf are used in the settings for the [[Main_Page#Setting_CPU_Core_Usage_Limits|''start_throttling'' and ''stop_throttling'' parameters]].
 +
 +
[[Main_Page#Defining_Groups|Defining User Groups]]
 +
 +
When planning a strategy for defining groups, use the following steps:
 +
 +
# Decide how users/customers should be organized into groups or cPanel packages. This organization could be based on needs, paid service levels, or any other criteria.
 +
# Ideally, user UIDs ranges or a file that contains a list of UIDs should be used to define users with similar requirements.  The BetterLinux cPanel hooks feature will automatically create a uid text file with a name that matches the cPanel package and will add newly created cPanel users to the file.
 +
# Edit /etc/betterlinux/cpud.conf.d/groups.conf or the BetterLinux cPanel hooks feature will do this automatically.
 +
# For example:<br /> uid package-normal-users /etc/betterlinux/cpanel/package-normal-users.txt<br /> uid package-premium-users /etc/betterlinux/cpanel/package-premium-users.txt
 +
# Save the groups.conf file
 +
# Restart the BetterCPU daemon:<br /> service cpud restart
 +
 +
Once this parameter change is in place, throttling limits that meet each group's SLA can be defined for each group.
 +
 +
For non-cPanel servers, the users will likely already be in a group defined in /etc/group. If this is the case, it is fairly easy to use that group definition within the groups.conf file.  Use the following command to extract the group definition and reformat it for inclusion in the /etc/betterlinux/cpud.conf.d/groups.conf file.  When defining the syntax, simply replace {group_name} with the name of the group defined in /etc/group.
 +
echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"
 +
 +
==== Setting CPU Core Usage Limits ====
 +
Once ''jail_cnt'' and ''jail_cnt_max'' are defined and groups are set up, it is necessary to configure the throttling parameters that define when user groups are moved into and out of the jail cores.  The syntax for these thresholds is defined with the following parameters:
 +
start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
 +
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
 +
 +
These parameters set throttling limits for groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.  The cpu_percent value sets the percentage an individual user (uid) can use of the total non-jail cores on the system.
 +
 +
To understand how this works, it is important to understand how CPU utilization is measured in a multi-core system.  In Linux, multithreaded processes can take more than 100% utilization because overall CPU utilization is measured as 100% * number_of_cores.  So in an 8 core system, if all CPUs are running at maximum utilization, the system's overall utilization is 800% (100% * 8 cores).
 +
 +
The values for '''cpu_percent''' in ''start_throttling'' and ''stop_throttling'' are based on a normalized utilization assigned to a single core (rather than being spread across all cores).  This means that while an 8 core system can have a maximum of 800% utilization, the value used for the cpu_percent value represents the fraction of a single core that could be used.  Specifying a value of 80 for '''cpu_percent''' means that on a system with no cores presently allocated to jail, when the user exceeds a value of 80% * 8 = 640% on an 8 core system, a core is allocated to the jail and the user's processes are throttled.  When a core is allocated to jail, the percentage used to determine if the threshold has been exceeded is 80% * 7 = 560%.  If a second core is allocated to jail, this threshold drops to 80% * 6 = 480%.
 +
 +
What this means is that - using the example of cpu_percent set to 80 - any distribution of utilization across all 8 cores of an 8 core system that adds up to more than 640% utilization will trigger the allocation of a core for throttling and that user's processes will be moved to that single core.  The utilization could be 100% of 6 cores and 40% of a seventh core, it could be 80% utilization distributed across all 8 cores, or it could be any variation where the user's total utilization exceeds the threshold.
 +
 +
When a non-throttled user's total process utilization exceeds '''cpu_percent''' for '''seconds''' seconds as defined in the ''start_throttling'' parameter, the user's processes are moved into the jail cores.
 +
 +
When a throttled user’s total utilization drops below the '''cpu_percent''' value for '''seconds''' seconds as defined in the ''stop_throttling'' parameter, the user's processes are moved out of the jail back onto the non-jailed cores.
 +
 +
To define these limits:
 +
*Edit the users.conf located in /etc/betterlinux/cpud.conf.d.
 +
*Add the following parameters users.conf, with appropriate limits to the system's needs:
 +
start_throttling group=example-group cpu_percent=50,seconds=3
 +
stop_throttling group=example-group cpu_percent=25,seconds=10
 +
*Save users.conf.
 +
*Restart cpud:
 +
service cpud restart
 +
 +
With these settings, a user in the group defined with processes that exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system. When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.
 +
 +
==== Start the BetterCPU Daemon (cpud) ====
 +
BetterCPU is now configured with basic jail core and throttling settings.  On CentOS, use the '''service''' command to start/stop the service.
 +
 +
The command syntax is as follows:
 +
service cpud {start|stop|status|reload|restart|condrestart}
 +
  start = starts cpud daemon
 +
  stops = stops cpud daemon
 +
  status = shows the status of cpud daemon
 +
  reload = reloads changes to the .conf files without restarting the daemon
 +
  restart = restarts the daemon, loading the configuration as defined in the config files
 +
  condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
 +
  This option is useful in scripts because it does not start the daemon if it is not running.
 +
 +
Start the cpud daemon with the following command:
 +
service cpud start
 +
 +
==== Start the BetterLinux Stats Program ([http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat blstat]) ====
 +
In order to see user processes that are in the jail cores, execute the [http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat '''blstat'''] program with root privileges:
 +
blstat
 +
 +
[http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat blstat] presents a top-like user interface that includes information specific to BetterCPU and BetterI/O.  Press 'c' to see the stats for BetterCPU.  Press '?' to display general help for the utility.
 +
 +
=== Limiting Execution Time Using time_monitor ===
 +
BetterCPU can be used to limit the amount of time programs can execute.  Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.
 +
 +
The basic steps for limiting execution time are:
 +
 +
# Define a program or user group
 +
# Add ''time_monitor'' to /etc/betterlinux/cpud.conf.d/main.conf
 +
# Reload the BetterCPU daemon's configuration
 +
 +
When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.
 +
 +
==== Define a Program or User Group ====
 +
When defining a program group, the format used in groups.conf is as follows:
 +
prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]
 +
 +
For example:
 +
prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem
 +
 +
User groups may also be defined. The user groups are structured as follows:
 +
uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]
 +
 +
For example:
 +
uid bad_users testuser1,501,1000-1999
 +
 +
User group definitions used for ''time_monitor'' execution limiting are identical to user groups used for other functions in the BetterLinux suite.
 +
 +
==== Add ''time_monitor'' to main.conf ====
 +
Limiting the execution time for a program group is very straightforward.  A script or executable program is called to perform an operation on the program being executed.  In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again.  The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.
 +
 +
For example, a script in /root called 'killer' might contain:
 +
#!/bin/bash
 +
echo date >> /root/killer.log
 +
kill -9 %1
 +
 +
This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.
 +
 +
To define the limit, add the following line to the main.conf file:
 +
time_monitor groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer 10
 +
 +
In this example, we use the sample "bad_progs" program group defined above.  Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash.  If the program has run for 5 seconds or longer, '''/root/killer''' is called with the PID of the program, and the script kills the program after logging the date and time.
 +
 +
The final parameter (10) tells cpud to not run the script more than once every 10 seconds.  If the '''/root/killer''' script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.
 +
 +
When using a user group (rather than a program group), the same concepts apply.  If there are programs the users should be allowed to run without monitoring the execution time, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:
 +
time_monitor_prog_exclude groups=exclude_group
 +
 +
==== Reload the BetterCPU Configuration ====
 +
In order to activate the changes, issue the following command:
 +
service cpud reload
 +
This reloads the configuration without restarting cpud.  The following command can also be used, but will stop and restart the cpud daemon:
 +
service cpud restart
 +
 +
=== Limiting Open Files Using ''file_limit'' ===
 +
BetterCPU can be used to limit the number of concurrent open files a user can have at one time.  Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.
 +
 +
The basic steps for limiting concurrent open files are:
 +
 +
# Define a user group
 +
# Add ''file_limit'' to /etc/betterlinux/cpud.conf.d/main.conf
 +
# Reload the BetterCPU daemon's configuration
 +
 +
When open files are limited, the '''file_limit_override''' and '''file_limit_exclude''' options can be applied to override the limits for certain programs and to remove the limits from others.
 +
 +
==== Define a Program or User Group ====
 +
When defining user groups, the following format used is:
 +
uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]
 +
 +
For example:
 +
uid bad_users testuser1,501,1000-1999
 +
 +
User group definitions used for limiting open files with ''file_limit'' are identical to user groups used for other functions in the BetterLinux suite.
 +
 +
==== Reload the BetterCPU Configuration ====
 +
In order to activate the changes, issue the following command:
 +
service cpud reload
 +
This reloads the configuration without restarting cpud.  The following command can also be used, but will stop and restart the cpud daemon.
 +
service cpud restart
 +
 +
=== Limiting Processes Using ''process_limit'' ===
 +
BetterCPU can be used to limit the number of concurrent processes a user can execute at one time.  Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.
 +
 +
The basic steps for limiting concurrent processes are:
 +
 +
# Define a user group
 +
# Add ''process_limit'' to /etc/betterlinux/cpud.conf.d/main.conf
 +
# Reload the BetterCPU daemon's configuration
 +
 +
When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.
 +
 +
==== Define a Program or User Group ====
 +
When defining user groups, the format used is:
 +
uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]
 +
 +
For example:
 +
uid bad_users testuser1,501,1000-1999
 +
 +
User group definitions used for limiting concurrent processes with ''process_limit'' are identical to user groups used for other functions in the BetterLinux suite.
 +
 +
==== Add ''process_limit'' to main.conf ====
 +
Limiting concurrent processes per user is very straightforward. 
 +
 +
To apply a limit, once a user group is defined, add a line to apply the desired restiction to the main.conf file:
 +
process_limit group=bad_users limit=100
 +
 +
This limits the users in the '''bad_users''' group to 100 concurrent processes each at any given time.
 +
 +
To completely exclude specific programs from having limits applied, create a program group in groups.conf with the programs in it:
 +
prog exclude_process_limits /bin/cron,/bin/bash
 +
 +
Then add a ''process_limit_exclude'' line to main.conf:
 +
process_limit_exclude group=exclude_process_limits
 +
 +
If a user limit should be overridden for particular programs, specify the user group as part of a ''process_limit_override'' command and specify the specific programs that should have the different limit in main.conf:
 +
process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd
 +
 +
This overrides the existing limits for '''bad_users''' for instances of /usr/sbin/sshd and sets those limits to 120.
 +
 +
==== Reload the BetterCPU Configuration ====
 +
In order to activate the changes, issue the following command:
 +
service cpud reload
 +
This reloads the configuration without restarting cpud.  The following command can also be used, but will stop and restart the cpud daemon.
 +
service cpud restart
 +
 +
=== Limiting Memory Utilization Using ''memory_limit'' ===
 +
BetterCPU can be used to limit the amount of RSS memory programs can use.  Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.
 +
 +
The basic steps for limiting RSS memory usage are:
 +
 +
# Define a program or user group
 +
# Add ''memory_limit'' to /etc/betterlinux/cpud.conf.d/main.conf
 +
# Reload the BetterCPU daemon's configuration
 +
 +
When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.
 +
 +
==== Define a Program or User Group ====
 +
When defining a program group, the format used in groups.conf is:
 +
prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]
 +
 +
For example:
 +
prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem
 +
 +
User groups may also be defined. These user groups are structured as:
 +
uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]
 +
 +
For example:
 +
uid bad_users testuser1,501,1000-1999
 +
 +
User group definitions used for RSS memory usage limiting are identical to user groups used for other functions in the BetterLinux suite.
 +
 +
==== Add ''memory_limit'' to main.conf ====
 +
Limiting the RSS memory used by a program group is very straightforward.  A script or executable program is called to perform an operation on the program being executed.  In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again.  The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.
 +
 +
For example, a script in /root called 'killer' might contain:
 +
#!/bin/bash
 +
echo date >> /root/killer.log
 +
kill -9 %1
 +
 +
This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.
 +
 +
To define the limit, add the following line to the main.conf file:
 +
memory_limit group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run=30
 +
 +
In this example, we use the sample '''bad_progs''' program group defined above.  Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB).  If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.
 +
 +
The final parameter (30) tells cpud to not run the script more than once every 30 seconds.  If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.
 +
 +
When using a user group (rather than a program group), the same concepts apply.  If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored:
 +
memory_limit_prog_exclude exclude_group
 +
 +
==== Reload the BetterCPU Configuration ====
 +
In order to activate the changes, issue the command:
 +
service cpud reload
 +
This reloads the configuration without restarting cpud.  The following command can also be used, but will stop and restart the cpud daemon:
 +
service cpud restart
 +
 +
 +
=== BetterIO Configuration ===
 +
Refer to the '''[[BetterIO HOWTO Collection|BetterIO HOWTOs]]''' section for details on configuring BetterIO.
 +
 +
=== BetterIO Concepts ===
 +
The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.
 +
 +
In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.
 +
 +
From a CPU perspective, this is referred to as “Waiting on I/O”, or ''iowait%''. If the I/O channel is not also managed, then the system will not be able to run optimally.
 +
 +
When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.
 +
 +
There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To [[Monitoring Tools|monitor]] these factors, one can use the ''iostat'' command or [http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat blstat] (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.
 +
 +
=== What can BetterIO tune to affect block device utilization? ===
 +
At a high level, tuning is applied to I/O operations per second (iops) and bandwidth (bw). These two parameters can be tuned by applying minimum and maximum values in /etc/iothrottled/iothrottled.conf.d/rules.conf.
 +
 +
Those system-wide minimum and maximum values can be refined by applying user limits and program limits, which are defined in ''/etc/iothrottled/user_limits.conf'' and ''/etc/iothrottled/prog_limits.conf'', respectively.
 +
 +
The limits defined can be scaled to allow the block device channel to be utilized more effectively when it is under-utilized. The ''auto_scale'' parameter defines how scaling is applied to keep both the bandwidth utilization and iops within a desired performance range.
 +
 +
=== Setting I/O Bandwidth Limits Per-User or by User-Group ===
 +
Individual user or group limits can be set using the configuration file /etc/iothrottled/user_limits.conf.
 +
 +
This file uses a single line per entry:
 +
 +
<nowiki>[user|group] [bw|ops] [max|min] [device/mount point] [limit]</nowiki>
 +
 +
The fourth parameter in this line can be a combination of block device names, or mount points to include or exclude.
 +
 +
For example, the file might contain a line that reads:
 +
 +
testuser bw max sda,sdb,sdc 10000000
 +
 +
This would mean that the user ''testuser'' has their maximum bandwidth (bw) limited on devices sda, sdb, and sdc to 10000000 B/s (Bytes per Second).
 +
 +
The first field can be a user or group name. Groups are defined in the file ''/etc/betterlinux/groups.conf''. If a group and a user have the same name, the limit applies to the user.
 +
 +
=== Setting I/O Bandwidth Limits by Program-Group ===
 +
 +
More granular control over individual programs’ usage of block device I/O rates and data rates can be achieved using the /etc/betterlinux/iothrottled.conf.d/prog_limits.conf file.
 +
 +
This file is similar to the user_limits.conf file, except that it specifies a program group name. The line format for this file is:
 +
 +
[program_group_name] [S|G] [bw|ops] [device/mount point list] [limit]
 +
 +
The fields that match up with fields in prog_limits.conf are defined the same way as in user_limits.conf.
 +
 +
The ''prognam_group_name'' field contains the names of the program or programs which is defined in groups.conf.
 +
 +
If the third field contains an “S”, the limit applies to each instance of the program started by the use or group. A “G” means the limit applies to all instances together.
 +
 +
For example:
 +
NOTE: In all the examples below we assume to have the group "tar_progs" defined in /etc/betterlinux/iothrottled.conf.d/groups.conf:
 +
prog tar_progs /bin/tar,/usr/bin/tar
 +
If our goal is to limit each individual instance of "tar" to a 5 MB/s I/O rate we would use the following line.
 +
tar_progs S bw max inc="^/" 5000000
 +
NOTE: all the "/bin/tar" and "/usr/bin/tar" instances will be affected by this rule if the UID that runs them is not included in any UID group. Remember that UID groups always win against PROG groups.
 +
 +
If our goal was to group ALL instances of tar to an aggregate bandwidth of 5 MB/s then you would change the "S" (Single process) in the example above to "G" (For grouping all tar processes together).  This option is helpful when you want to limit one or more instances together.  So, if you had one tar running or 50 tars the total disk I/O bandwidth would never exceed 5 MB/s.
 +
 +
The syntax for this scenario would be as follows:
 +
tar_progs G bw max inc="^/" 5000000
 +
 +
If our goal was to limit each instance of tar to a disk I/O rate of 5 MB/s, but would only like to enforce this limit on any filesystem mounted under /home (and any sub-directories) but have no limits enforced on other drives then the syntax would be as follows.
 +
tar_progs G bw max inc="^/home" 5000000
 +
You can include multiple block devices (hard drives) or mount points to be included in the limits that you have defined as shown below:
 +
tar_progs G bw max inc="/home",sdb,dm-1,inc="/var/matt",inc="/etc" 5000000
 +
 +
In addition, you can specific drives to be excluded from the limits that you have defined. This syntax is shown below.
 +
tar_progs G bw max inc="/",exc="^/media|^/mnt" 5000000
 +
 +
For example:
 +
tar_progs S bw sda 1000000
 +
This applies a bandwidth limit of 1000000 bytes/s to each instance of the /bin/tar or /usr/bin/tar commands run by any user not included in a UID group. If one of these users starts two concurrent instances of the tar command, each one is limited to 1000000 b/s of block device I/O bandwidth. If this line reads:
 +
tar_progs G bw sda 1000000
 +
 +
The bandwidth limit is applied to all instances of tar that the users launch. In this case, if the users launch two concurrent instances of the tar command, the total bandwidth both instances can use together is 1000000 B/s.
 +
 +
Below are some configuration rule examples of how to limit I/O bandwidth for certain programs.
 +
 +
To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied. The inc= and exc= directives take an extended regular expression for mount points. For example, the line:
 +
tar_progs G bw max sda,inc="/home|/backup",exc="/mnt1|/mnt2" 1000000
 +
 +
Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that have "/home" or "/backup" in their mount point (that would also include, for example, "/home2", "/media/home3", excluding all the mount points that have "/mnt1" or "/mnt2" (like "/mnt21", "/mnt/mnt1", ...). The values are regular expressions and they can appear anywhere within the mount point). Including ^ and $ in the regex will force a match at the begining or end.
 +
 +
The evaluation of these expressions always follows the order of:
 +
*Explicit device names
 +
*Devices mounted in the "inc" list
 +
*Devices mounted in the "exc" list
 +
 +
Exclusions will ultimately override the other values in the list.
 +
Consider the following list of mounted devices:
 +
/dev/sda mounted at /
 +
/dev/sdb mounted at /home
 +
/dev/sdc mounted at /media/backups
 +
/dev/sdd mounted at /usr
 +
/dev/sde mounted at /mnt/home
 +
/dev/sdf mounted at /srv/www
 +
/dev/sdg mounted at /srv/ftp
 +
/dev/sdh mounted at /srv/home
 +
 +
With the following restrictions in user_limits.conf:
 +
tar_progs G bw max sda,sdd,inc="/home",exc="/srv" 1000000
 +
Programs in group "tar_progs" will be limited to 1 MB/s on /home and /mnt/home, but not on /srv/home, due to the exc="/srv" rule.
 +
 +
=== How do the block device inclusion/exclusion lists work? ===
 +
Simple configuration of BetterIO uses device names in a list in the user_limits.conf and prog_limits.conf files. For example:
 +
 +
testuser bw max sda,sdb,sdc 1000000
 +
 +
This applies the limit of 1000000 B/s for the user ''testuser'' on devices /dev/sda, /dev/sdb, and /dev/sdc.
 +
 +
Modern Linux systems, however, do not always guarantee the assignment of devices, which can lead to issues with limits not being applied to the correct device.
 +
 +
To ensure limits are applied to the correct device, the use of ''inc='' and ''exc='' directives along with (or instead of) device names provides greater flexibility in defining where limits are applied.
 +
 +
The ''inc='' and ''exc=' directives take an extended regular expression for mount points. For example, the line:
 +
 +
testuser bw max sda,inc=”/home|/backup”,exc=”/mnt1|/mnt2” 1000000
 +
 +
Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that are mounted at “/home” and “/backup” (and would also include, for example, “/home2”, “/media/home3” - the values are regular expressions and can appear anywhere within the mount point), but would exclude any mount point that includes “/mnt1” or “/mnt2” (including “/mnt21”, “/mnt/mnt1” - as with the “inc=” directive). Including ^ and $ in the regex will force a match at the begining or end.
 +
 +
Another example for this using regular expression that will apply the limit of 1MiB/s to any mounted block device except those mounted under /var or /tmp
 +
 +
testuser bw max inc=”^/”,exc=”^/var|^/tmp” 1000000
 +
 +
In this example we want to limit all the mounted block devices, except the temporary mounted devices, usually mounted under /mnt or /media.
 +
* add the line below to /etc/betterlinux/groups.conf:
 +
testgroup uid 500-1000
 +
* Then add the line below to /etc/betterlinux/iothrottled/user_limits.conf:
 +
testgroup bw max inc=”^/”,exc=”^/mnt|^/media” 1000000
 +
 +
The evaluation of these expressions always follows the order of:
 +
 +
# Explicit device names
 +
# Devices mounted in the “inc” list
 +
# Devices mounted in the “exc” list
 +
 +
Exclusions will ultimately override the other values in the list.
 +
 +
Consider the following list of mounted devices:
 +
 +
:/dev/sda1 mounted at /
 +
:/dev/sda2 mounted at /home
 +
:/dev/sdb1 mounted at /media/backups
 +
:/dev/sdb2 mounted at /usr
 +
:/dev/sdc1 mounted at /mnt/home
 +
:/dev/sdc2 mounted at /srv/www
 +
:/dev/sdd1 mounted at /srv/ftp
 +
 +
With the following restrictions in user_limits.conf:
 +
 +
testuser bw max sda,sdd,inc=”/home”,exc=”/srv” 1000000
 +
 +
The result would be:
 +
 +
:sda would be included
 +
:sdb would be excluded
 +
:sdc would be excluded
 +
:sdd would be excluded
 +
 +
So the only device controlled by BetterIO in this scenario would be /dev/sda. The controls are applied per block device, not on a per-partition basis - so if a partition is excluded from control, all partitions on that block device will be excluded.
 +
 +
When configuring your system, it is important to take this into consideration - you do not want your swap partition to be throttled by BetterIO, but if you put swap on /dev/sda and control /dev/sda with BetterIO, you may end up impacting performance on the system in ways you may not intend to.
 +
 +
Examples include/exclude:
 +
 +
:/dev/sda1 on /
 +
:/dev/sda2 on /home
 +
:/dev/sdb1 on /home2
 +
:/dev/sdc1 on /home3
 +
:/dev/sdd1 on /home4
 +
 +
testuser bw max inc="/home",exc="/home2"
 +
 +
The rexeg's are matched against the left of the mount point.  So the include matches /home1, /home2, /home3, /home4 and the exclude matches /home2 only. So the result is:
 +
 +
:/dev/sda included
 +
:/dev/sdb excluded
 +
:/dev/sdc included
 +
:/dev/sdd included
 +
 +
Example Exclude:
 +
 +
:/dev/sda1 on /
 +
:/dev/sda2 on /home
 +
:/dev/sdb1 on /src
 +
:/dev/sdb2 on /usr
 +
:/dev/sdc1 on /extra
 +
 +
testuser bw max exc="/src"
 +
 +
:/dev/sda included
 +
:/dev/sdb excluded
 +
:/dev/sdc included
 +
 +
=== How does auto scaling work? ===
 +
The ''auto_scale ''parameter is an on/off toggle switch for enabling or disabling the auto scale feature. The auto scaling algorithm is applied to overall bandwidth utilization or iops giving the system administrator the ability provide more bandwidth to a defined set of users when it's available.
 +
 +
When setting the value for this parameter, an integer is entered that represents the on/off values:
 +
 +
:0 = auto scaling off
 +
:1 = auto scaling on
 +
 +
Auto scaling is controlled by these parameters:
 +
 +
:scale_pct_hi
 +
:scale_pct_low
 +
:scale_pct_periods
 +
 +
Additionally, the following parameters apply to auto scaling:
 +
 +
:scale_min
 +
:scale_max
 +
:scale_up_inc
 +
:scale_down_inc
 +
 +
To understand how these parameters are related to each other, consider the following scenario:
 +
 +
With the following line in /etc/iothrottled/user_limits.conf:
 +
 +
testuser bw max sda 1000000
 +
 +
And in /etc/iothrottled/iothrottled.conf.d/rules.conf:
 +
 +
:auto_scale 1
 +
:scale_pct_hi 80
 +
:scale_pct_low 20
 +
:scale_pct_periods 2
 +
:scale_min 25
 +
:scale_max 500
 +
:scale_up_inc 20
 +
:scale_down_inc 5
 +
 +
If the overall block device utilization is below 20% (''scale_pct_low''), then the limit applied to ''testuser'' is increased by 20% (''scale_up_inc'') of 1000000 (taken from user_limits.conf for this user), or 200000 B/s (Bytes per Second).
 +
 +
If overall block device utilization goes over 80% (''scale_pct_hi''), then the limit applied to ''testuser'' is decreased by 5% (''scale_down_inc'') of 1000000 (from the user’s limit in user_limits.conf), or 50000 B/s.
 +
 +
The increase/decrease in the scaling factor is tracked as well, so on the first cycle, the scaling factor is 120%. The second iteration scales up to 140%, and so on, until the limit set by ''scale_max'' is reached.
 +
 +
Similarly, if scaling down were to be applied instead, that scaling would drop by 5% each iteration until the scale factor reached the ''scale_min'' limit of 25%.
 +
 +
The result is that BetterIO attempts to keep the block device channel utilization between 20% and 80% at all times, and adjusts the utilization limits for all users in order to achieve that goal. If one were to graph the utilization over time, the goal is to have that graph look something like this:
 +
 +
[[Image:Iothrottled_graph.png‎]]
 +
 +
 +
Whenever the utilization drops below the threshold at 20% or goes above the threshold at 80%, the adjustments kick in so the utilization falls in the area between the two thresholds. The majority of the time, that device’s utilization should be between 20% and 80% with this configuration.
 +
 +
Those adjustments are subject to the ''scale_min'' and ''scale_max'' parameters. Having limits on the amount of scaling will prevent BetterIO from attempting to scale up without a limit when there is no I/O taking place. If it were allowed to do that, then a period with no I/O could end up with a scaling factor in excess of 1000%, and the process of scaling down would result in it taking a significant amount of time with no effective limit in place, which would make the available bandwidth the bottleneck for system performance until the scaling factor returned to a reasonable range.
 +
 +
=== How can one test to see if BetterIO is controlling devices? ===
 +
A simple and quite dramatic way to see BetterIO limiting I/O is to set a lower threshold for a user that is very small. For example, create a user called ''testuser'' and log in as that user. Run the following command:
 +
 +
sync; dd if=/dev/zero of=temp.fil bs=1M count=10 oflag=direct; rm temp.fil
 +
 +
The results from that command for an unmanaged user would be similar to this screenshot:
 +
 +
[[Image:Iothrottled_blocking_devices.png]]
 +
 +
If we then modify the /etc/iothrottled/user_limits.conf file to include the line:
 +
 +
testuser bw max inc=”/” 500000
 +
 +
and rerun the ''dd'' command above, the following output results:
 +
 +
[[Image:Iothrottled_dd_results.png‎]]
 +
 +
The drop in throughput from 350-400 MB/s to 500-600 kB/s shows the effect of the change in bandwidth limits for this user.
 +
 +
=== Are there any other considerations to keep in mind? ===
 +
 +
With BetterIO, the limits are set on a per-block device basis. It is important to keep in mind that I/O channels are not always assigned in a 1:1 ratio with block devices. If a series of storage devices are connected to the system by a Fibre Channel connection, the channel bandwidth available for all of those devices needs to be taken into consideration across all of the devices.
 +
 +
 +
If devices /dev/sda, /dev/sdb, and /dev/sdc are all on the same disk controller, and the maximum throughput of that controller is 1 Gbps, then all three disks cannot run at 1 Gbps even if the drive electronics would permit that.
 +
 +
If you then have 10 users limited to 100 Mbps each, then performance will effectively be running without any throttling because the disk channel will be oversubscribed if all of the users are attempting to fully utilize their allocated bandwidth to each of these three devices. At the same time, these users are still limited to 100 Mbps each and are unable to monopolize the resources.
 +
 +
 +
Once [[installation]] is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules.  Once the the system is rebooted, configure BetterIO and start the iothrottled daemon.  By default, BetterIO does not throttle any users.  The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).
 +
 +
=== Defining user groups ===
 +
User-created groups do not exist on the system until they have been created by the system administrator. By creating and populating  groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined.  The group names defined in /etc/betterlinux/iothrottled.conf.d/groups.conf are used in the settings for [[BetterIO#Setting_I.2FO_Bandwidth_Limits_Per-User_or_Per-Group|setting per-group I/O limits]].
 +
 +
To define groups:
 +
# Choose a group name.
 +
# Populate the group with UIDs, UID ranges, user names, or files containing UIDs or usernames<br />'''Note:''' A group may contain one or more usernames, UIDs, or ranges.
 +
 +
The following examples illustrate the command syntax:
 +
Single range:
 +
  Syntax:  uid {group name} {uid range}
 +
 +
  Example: uid example-group 0-499
 +
This example includes all UIDs from 0-499 as members of the group ''example-group''
 +
 +
  Example: uid example-group 800-
 +
This example includes all UIDs from 800 and above as members of the group ''example-group''
 +
 
 +
Multiple UID ranges:
 +
  Syntax:  uid {group name} {uid range}[,{uid range2}...]
 +
 +
  Example: uid example-group 0-499,600-699
 +
This example includes all UIDs from 0-499 and from 600-699 as members of the group ''example-group''
 +
 +
When planning a strategy for defining groups, use the following steps:
 +
 +
# Decide how users/customers should be organized into groups.
 +
#* This organization could be based on needs, paid service levels, or any other criteria.
 +
#* Ideally, user UIDs ranges should be used to define users with similar requirements.
 +
# Edit /etc/betterlinux/iothrottled.conf.d/groups.conf.
 +
#* For example:<br /> uid normal-users 500-999<br /> uid premium-users 1000-1999
 +
# Save the groups.conf file.
 +
# Restart the BetterIO daemon:<br /> service iothrottled restart
 +
 +
Once this parameter change is in place, you can define throttling limits that meet each group's SLA.
 +
 +
If the users are already in a group defined in /etc/group, it is fairly easy to use that group definition within the groups.conf file.  Using the following command will extract the group definition and reformat it for inclusion in the /etc/betterlinux/iothrottled.conf.d/groups.conf file.  Just replace {group_name} with the group name defined in /etc/group.
 +
echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"
 +
 +
=== Setting I/O Bandwidth Limits ===
 +
Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis.  To do this, edit the user_limits.conf file located in /etc/betterlinux/iothrottled.conf.d.  This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/groups.conf.
 +
 +
The user_limits.conf file uses a single line per entry:
 +
[group] [bw|ops] [max|min] [device/mount point] [limit]
 +
 +
The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.
 +
 +
For example:
 +
example-group bw max sda,sdb,sdc 10000000
 +
 +
This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.
 +
 +
Another example:
 +
io_standard_users bw max inc="^/",exc="^/home" 5000000
 +
 +
This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.
 +
 +
After updating the user_limits.conf file with limits applicable to the system, save the file and restart iothrottled.
 +
 +
'''Note:'''
 +
*If a group and a user have the same name, the limit applies to the user.
 +
*Only one limit per device can be set.  It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.
 +
 +
Please see the [[BetterIO|BetterIO documentation]] for more configuration options for I/O bandwidth limiting features by program and inclusion/exclusion options.
 +
 +
=== Tuning I/O Scaling and Balancing ===
 +
The ''auto_scale'' and ''seek_balance'' features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in user_limits.conf.
 +
 +
These settings are contained in /etc/betterlinux/iothrottled.conf.d/scale.conf. The configuration parameters define how these settings affect the I/O bandwidth.  Details on how these settings work can be found in the user documentation for [[BetterIO#How_does_auto_scaling_work.3F|scaling]] and [[BetterIO#Balancing_I.2FO_seek_performance|balancing]].
 +
 +
=== Start the BetterIO daemon (iothrottled) ===
 +
BetterIO is now configured with basic throttling settings.  On CentOS, use the '''service''' command to start/stop the service.
 +
 +
The command syntax is as follows:
 +
service iothrottled {start|stop|status|reload|restart|condrestart}
 +
  start = starts iothrottled daemon
 +
  stops = stops iothrottled daemon
 +
  status = shows the status of iothrottled daemon
 +
  reload = reloads changes to the .conf files without restarting the daemon
 +
  restart = restarts the daemon, loading the configuration as defined in the config files
 +
  condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
 +
  This option is useful in scripts because it does not start the daemon if it is not running.
 +
 +
Use the following command to start the iothrottled daemon:
 +
service iothrottled start
 +
 +
=== Start the BetterLinux stats program ([http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat blstat]) ===
 +
To see user processes that are being throttled by BetterIO, execute the [http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat '''blstat'''] program with root privileges:
 +
blstat
 +
 +
[http://docs.betterlinux.com/wiki/Monitoring_Tools#blstat blstat] presents a top-like user interface that includes information specific to BetterCPU and BetterIO.  Press 'i' to see the stats for BetterIO.  Press '?' to display general help for the utility.
 +
 +
=== BetterMySQL Concepts ===
 +
BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.
 +
 +
It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated.  Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.
 +
 +
=== Install and Configure BetterMySQL ===
 +
BetterMySQL cannot be installed unless MySQL is already on the system. The '''[[BetterMySQL HOWTO Collection|BetterMySQL HOWTO]]''' provides a step by step guid for installing and configuring BetterMySQL on Linux and Linux with cPanel.
 +
 +
Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the '''[[BetterCPU|BetterCPU functional documentation]]'''. BetterMySQL’s configuration file serves only to enable BetterMySQL.
 +
 +
Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set.  For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit:
 +
start_throttling group=cpanel_users cpu_percent=50 seconds=3
 +
stop_throttling group=cpanel_users cpu_percent=30 seconds=5
 +
 +
Same logic apply's to any BetterIO limits that have been set.  In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled.
 +
cpanel_users bw max inc="^/" 5000000
 +
 +
<!-- === Configure BetterMySQL ===
 +
Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the [[BetterCPU|BetterCPU functional documentation]]. BetterMySQL’s configuration file serves only to enable BetterMySQL. To enable it, edit /etc/betterlinux/mysql.conf and set the parameter as follows:
 +
 +
bl_throttling 1 #[Default = 0; {1 = on, 0 = off}]
 +
 +
If bl_throttling has been enabled while mysqld is running, BetterMySQL will not run until mysqld has been restarted (service mysqld restart). Regardless the value in the bl_throttling parameter, and without restarting, you can temporarily activate BetterMySQL from a mysql prompt as follows:
 +
 +
mysql> set global bl_throttling=on {This syntax accepts these equivalent values: on; true; 1 (OR THEIR OPPOSITES) off; false; 0;}
 +
 +
This temporary functionality allows administrators to globally test BetterMySQL without committing to a lasting change in the mysql.conf file.
 +
 +
The approved_accounting setting enables BetterMySQL throttling and authorizes a program to apply usage blame to other users on the system.  This is used for programs like mysql and apache, which run under one uid, to divide the usage up by the individual users of the program.
 +
 +
Use the following syntax to authorize mysql to assign blame to the correct user on the system.  Make certain the correct username for mysqld is defined:
 +
 +
approved_accounting_program program=/usr/libexec/mysqld user=mysql
 +
 +
This allows the program /usr/libexec/mysqld as run by the mysqld user to determine if one mysqld users is exceeding a configured resource constraint enforced by the BetterCPU (cpud) start and stop throttling parameters.
 +
 +
=== Automate CPU User Mapping ===
 +
User Mapper is a Linux utility accompanying BetterMySQL that gives you, for the first time, the Linux user names and UIDs of MySQL users. Linux attributes all MySQL processes to one user, while MySQL gives every database a unique user name, ignoring that single users have multiple databases. Better information is needed for accurate, individualized throttling. To solve this, User Mapper creates a file associating MySQL user names with Linux user names and their UIDs. All MySQL processes can now be assigned to those responsible for them.
 +
 +
Although User Mapper currently supports only cPanel users, we can easily incorporate functionality for other control panels and directory services. Tell us what you need. Also note that if an ISP does not know which of its Linux users correspond to which MySQL users, there will be no way to create a user map file and you will not be able to use BetterMySQL.
 +
 +
==== Start User Mapper ====
 +
User Mapper installs automatically alongside BetterMySQL and requires it to run. The installation is found in /usr/sbin. To start User Mapper, first start BetterMySQL at the command line (service mysqld start). Then as root, type “myusermapper” without any options. This will create a map for each MySQL user name. For more detail on how this is done, run the help option (--help). After the map is created, the following notice will appear: “Usermap entries are in /etc/betterlinux/mysql/usermap.conf.” The directory and/or file name should not be changed. Once the map is created and BetterMySQL is enabled, the map file is monitored in real-time for changes.
 +
 +
=== User Mapper Options ===
 +
Access User Mapper’s complete options by running “myusermapper --help.” Below are discussed only two of the more helpful options listed there. For further details, see the help file.
 +
 +
==== Add a single entry to the user map file ====
 +
As described above, running myusermapper without other commands generates a user map by considering the name of each MySQL user. But you can also append a single user’s mapping entry to the file like this:
 +
 +
Short syntax: myusermapper -m <mysql user> -l <Linux user> -u <UID>
 +
Long syntax: myusermapper --myuser=<mysql user> --lxuser=<Linux user> --uid=<UID>
 +
Note:         Only the addition of UID above is optional, as BetterMySQL can derive the UID at runtime.
 +
 +
Short example: myusermapper -m fred_flinstone -l fred -u 522
 +
Long example: myusermapper --myuser=fred_flinstone --lxuser=fred --uid=522
 +
Result:        These add a single line to the user map file that looks like this:  fred_flinstone<TAB>fred<TAB>522
 +
 +
==== Delete outdated user map entries ====
 +
By default, User Mapper stores no-longer-mapped MySQL users from the pre-existing file at the end of the updated file. This aids record-keeping and other special uses. You can prevent this with the “--noappendold” option, which effectively deletes outdated user map entries.
 +
 +
Syntax: myusermapper --noappendold
 +
 +
=== Create Your Own User Map ===
 +
 +
==== When to Create Your Own User Map ====
 +
 +
BetterMySQL requires a user map file to work. If all of the following conditions exist, you will not be able to automate your user map creation. You will then need to create your own user map file:
 +
 +
#You do not use cPanel or a panel with similar naming conventions.
 +
#Your MySQL user names are not the same as your Linux user names.
 +
#Your MySQL user names are not based on your Linux user names.
 +
 +
==== How to Create Your Own User Map ====
 +
 +
To create your own BetterMySQL-usable user map file, open your favorite editor, get the mapping information, and enter the MySQL user names and Linux user names as follows:
 +
 +
#For ease, use mapping information from an ISP-exported file.
 +
#The user map file must be in .tsv format (tab-separated values). Exported .csv files (comma-separated values) are easily converted to .tvs format.
 +
#Include only one mapping per line.
 +
#BetterMySQL finds UIDs as needed even if they do not appear in the user map file. The file may contain lines in either or both of the following formats:
 +
#:{MySQL user name}<TAB>{Linux user name}
 +
#:{MySQL user name}<TAB>{Linux user name}<TAB>{UID}
 +
#Example file: (UIDs are included here, but are not necessary)
 +
#:myuser 10    lxuser10    521
 +
#:myuser 11    lxuser11    522
 +
#:myuser 12    lxuser12    523
 +
#To allow BetterMySQL to read the .tsv file, copy the information into usermap.conf, found here: /etc/betterlinux/mysql/usermap.conf.
 +
#To see if BetterMySQL implemented your user map file, look at /etc/betterlinux/mysql.conf. After the parameter “bl_usermap_file” you should see: /etc/betterlinux/mysql/usermap.conf.
 +
#For further verification, from the mysql prompt, enter “show user_map;”
 +
 +
Here is an example of what you might see:
 +
 +
+------------+---------------+------+
 +
| From_user  | Linux_user    | Uid  |
 +
+------------+---------------+------+
 +
| myuser44  | lxuser44      |  555 |
 +
| myuser0    | lxuser0      |  511 |
 +
| myuser15  | lxuser15      |  526 |
 +
| myuser1    | lxuser1      |  512 |
 +
| myuser17  | lxuser17      |  528 |
 +
| myuser18  | lxuser18      |  529 |
 +
| myuser33  | lxuser33      |  544 |
 +
…
 +
…
 +
+------------+---------------+------+
 +
50 rows in set (0.00 sec)
 +
 +
=== How is a MySQL user process/thred “jailed”? ===
 +
The BetterMySQL add on extends all the functionality of BetterCPU to MySQL database user process and threads. The process for enabling the BetterMySQL functionality is simple.  Edit /etc/betterlinux/mysql.conf and set the following parameter:
 +
bl_throttling 1
 +
Usage: setting bl_throttling to a 1 = on and 0 = off.
 +
 +
Please see the [[BetterCPU| BetterCPU functional documentation]] for more details on how to configure jailing.
 +
-->
 +
 +
 +
=== Configuring BetterMySQL for cPanel Support===
 +
BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.
 +
 +
==== Installing the BetterLinux version of MySQL for cPanel ====
 +
BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL.  If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script:
 +
/etc/betterlinux/cpanel/bl-cpmysql-install
 +
 +
==== Configuring BetterMySQL for cPanel support ====
 +
To provide and configure per-user query throttling of MySQL users a mapping of MySQL users to Linux users must be configured.  This mapping and other configuration options are auto-configured by either the '''bl-cpanel-install''' script on first time installation or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the '''bl-hooks''' script can be used after a manual installation of BetterMySQL.  Example syntax for running the bl-hooks script:
 +
/etc/betterlinux/cpanel/bl-hooks --inint
 +
 +
Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.
 +
* usersmap.conf contains the mapping of cPanel users to UIDs
 +
* mysql.conf contains the parameter location of the usermap.conf file
 +
* turns on BetterMySQL throttling by adding the approved_accounting parameter to /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf
 +
approved_accounting_program program=/usr/libexec/mysqld user=mysql
 +
* Restarts the cpud daemon to enable BetterMySQL throttling.
 +
service cpud restart
 +
 +
=== Configuring BetterMySQL for Linux Servers===
 +
BetterMySQL configuration is a manual setup for Linux servers
 +
 +
==== Installing the BetterLinux version of MySQL non-cPanel ====
 +
After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command:
 +
yum install mysql-server
 +
 +
==== Configuring BetterMySQL for non-Cpanel Linux servers ====
 +
To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.
 +
 +
The command format is as follows:
 +
bl_usermap_filename [usermap_file_path]
 +
Example:
 +
bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf
 +
 +
Now create the MySQL mapping to Linux user mappings located in the following file:
 +
/etc/betterlinux/mysql.conf.d/usermap.conf
 +
 +
The usermap file is a tab-delimited file that contains the following information:
 +
[mysqluser] [linux_user] [uid]
 +
 +
The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.
 +
 +
For example, the following file will create mappings for testuser1 and testuser2.  For testuser1, the uid will be determined when that user makes a query against the database.
 +
testuser1 testuser1
 +
testuser2 testuser2 501
 +
 +
The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;":
 +
mysql> show user_map;
 +
+-----------+-----------+------+
 +
| From_user | Unix_user | Uid  |
 +
+-----------+-----------+------+
 +
| testuser2 | testuser2 |  501 |
 +
| testuser1 | testuser1 |  500 |
 +
+-----------+-----------+------+
 +
2 rows in set (0.00 sec)
 +
 +
The approved_accounting parameter turns on the BetterMySQL throttling and is located in:
 +
/etc/betterlinux/cpud.conf.d/main.conf
 +
The syntax is as follows for non-cPanel servers:
 +
approved_accounting_program program=/usr/sbin/mysqld user=mysql
 +
 +
Restart the cpud daemon to enable the configuration settings:
 +
service cpud restart
 +
 +
=== [[Monitoring Tools|Monitoring]] MySQL Utilization with [http://docs.betterlinux.com/wiki/Monitoring_Tools#myusertop myusertop] ===
 +
BetterMySQL includes the [http://docs.betterlinux.com/wiki/Monitoring_Tools#myusertop myusertop] utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.
 +
 +
Once [http://docs.betterlinux.com/wiki/Monitoring_Tools#myusertop myusertop] is running, press '?' to view the built-in help.
 +
 +
The output is similar to the output of the standard 'top' utility:
 +
 +
mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
 +
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
 +
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
 +
Time: CPU 448          Busy 529        CPU/Busy 0.8477  RowsExamined:  1,476m
 +
Qrys: Select 28,456    Update 0        Other 5,250        Adj-RowsEx:  1,251m
 +
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
 +
      h=Help m=MySQLUsersOn/Off                                          q=quit
 +
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx  %CPU
 +
root                  0    2 0.0075  0.0074  1.0060    150      151  100.00
 +
  root                0    2 0.0075  0.0074  1.0060    150      151  100.00
 +
testuser2            501    0 0.0000  0.0000  0.0000      0        0    0.00
 +
  testuser2          501    0 0.0000  0.0000  0.0000      0        0    0.00
 +
testuser1            500    0 0.0000  0.0000  0.0000      0        0    0.00
 +
  testuser1          500    0 0.0000  0.0000  0.0000      0        0    0.00
 +
 +
In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization.  If it were divided equally between the three users, [http://docs.betterlinux.com/wiki/Monitoring_Tools#myusertop myusertop] would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.
 +
 +
If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.

Revision as of 15:01, 15 April 2013

Contents

BetterBandwidth Concepts

BetterBandwidth uses 'tc' (a built in Linux tool) to provide traffic shaping for users that have their own network interface. The tc tool on its own does not provide a mechanism for limiting bandwidth usage by uid/user on a shared network interface. BetterLinux has added this functionality into its BetterBandwidth feature. Now admins can limit bandwidth for a single user on a shared server that has a corporate internet connection.

A common problem for web hosts is that one single user with a 1Gbit connection could kill the entire TCP stack. Often the host is unaware that it happens because they use 5 min bandwidth averages to monitor bandwidth on their 1 gig or 10 gig port. But all it takes is a 30 second spike in bandwidth out of 5 mins to make customers angry. And because of the 5 minute average it looks like the peak was around 50Mbits when it easily could have been 500Mbits. Customers will blame it on the server or the hosting company.

Many hosting companies give out IPs just to be able to limit bandwidth. This is an expensive approach. The BetterLinux BetterBandwidth feature can do it without giving out dedicated IPs.

BetterLinux has an apache mod called mod_betterlinux that allows BetterBandwidth to limit bandwidth for all ports for an individual user or group of users like email, FTP, etc is all included. This is a tremendous advantage for a host. BetterBandwidth monitors and controls individual, group, and system-wide bandwidth usage in multi-tenant Linux environments. Administrators can now limit outbound bandwidth as if each user had a dedicated IP address. This lets admins consolidate multiple users onto a single IP, which innovatively addresses the fast-increasing IPV4 IP address shortage.

Configure BetterBandwidth

Follow the BetterBandwidth HOWTO guide for configuring bandwidth limiting

Bandwidth limiting

BetterBandwidth limiting uses the mod_betterlinux Apache module located in in the betterlinux-core-*.rpm , as well as the linux tc method of using the uid classifier to categorize specific user processes. The betterlinux-core-*.rpm is installed by the bl-install script by default and auto-configured for cPanel systems during the betterlinux-cpanel-install script.

cPanel Bandwidth Limiting Setup

During the betterlinux-cpanel-install script installation, the user will answer Y/N to this question: 

Install BetterLinux Apache throttling? [y/N]

Selecting yes, will auto-configure and load the apache module mod_betterlinux on a cPanel system.

After the installation is complete, the following parameters and arguments need to be added to the BetterLinux configuration files:

  • In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example: 
interface interface-types eth0
  • In /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format: 
bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to: 

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon: 

service cpud restart

non-cPanel Bandwidth Limiting Setup

  • Depending on the distribution, edit the apache conf or create a new one under conf.d and add the following: 
LoadModule betterlinux_module /usr/lib64/httpd/modules/mod_betterlinux.so
<IfModule betterlinux_module>
 Betterlinux On
</IfModule>

Then restart apache.

Using this command will check the mod_betterlinux syntax in the apache .conf file: 

apachectl -t -D DUMP_MODULES
  • Configure BetterCPU (cpud) by adding or uncommenting the following parameter located in /etc/betterlinux/cpud.conf.d/main.conf 
 approved_accounting_program /usr/sbin/httpd apache

Please note, the full path to the httpd program is required, and it must be the actual program and not a symlink. Also, be sure to list the user it runs as, whether that is apache, httpd, or different username.

  • In /etc/betterlinux/cpud.conf.d/groups.conf the create an interface group. Here's an example: 
interface interface-types eth0
  • In /etc/betterlinux/cpud.conf.d/main.conf add the "bandwidth_limit" parameter wich sets the bandwidth limit by user and interface groups. This parameter also allows the admin to pass in any "tc" command line options. This parameter takes a value in the format: 
bandwidth_limit group=GROUP interface_group=INTERFACE_GROUP limit=20mbit

The first switch (group=) value is the name of the user group created in groups.conf. The second switch (interface_group=) value is the name of the interface group defined in groups.conf. The third switch (limit=) is amount of bandwidth allowed for each user defined in the user group. The fifth and "OPTIONAL" switch (tc_command=) takes any tc command line values like this: tc_command=-burst=25mbit -ceil=30mbit

For example, if this parameter is set to: 

bandwidth_limit group=cpanel_users interface_group=interface-types limit=20mbit

The result will be 20 megabits bandwidth limiting for the users defined in the bandwidth_users group and for the interfaces defined in the interface_type group.

To enable these settings, restart the BetterCPU daemon: 

service cpud restart


Assigning IP addresses with the dedicated IP address mapper

With multi-homed systems, incoming traffic is directed to the IP address where DNS resolves the domain name. A system with multiple virtual hosts can accept traffic to secondary addresses which, in turn, are assigned to secondary addresses, which are assigned to the system's network adapters.

Without some modification, responses sent from a server in a multi-homed system originate from the host's primary IP address. This makes it difficult to apply bandwidth or connection limits using a server's IP address. Most often, hosting providers work around this problem by implementing some sort of full virtualization solution that provides a unique virtual instance for each client. <KGO> What is the drawback of this solution?

BetterBandwidth uses the dedicated kernel module to provide a more cost-effective, efficient solution for managing bandwidth in multi-homed environments. When used in conjunction with the BetterBandwidth dedicated daemon, the dedicated kernel module allows administrators to assign outgoing traffic to a specific secondary address which, in turn, is assigned to a specific user--all without the overhead of a fully virtualized system. (Both IPv4 and IPv6 are supported.)

To enable this functionality:

  1. Ensure the dedicated kernel module is loaded:
    modprobe dedicated
  2. Edit /etc/betterlinux/ipv4 and/or /etc/betterlinux/ipv6 to include the specific mappings.
  3. Restart dedicatedd:
    service dedicatedd restart

The IPv4 and IPv6 mapping files are in the following format. Each line contains a single entry. 

username|uid: address

The address field can contain either a raw IPv4 or IPv6 address or a DNS name. If the DNS name resolves to more than one address, the kernel randomly selects one of the addresses.

Once configured, the active mappings for IPv4 addresses can be viewed using the following command: 

cat /proc/sys/net/ipv4/dedicated

To view mappings for IPv6 addresses, use the following command: 

cat /proc/sys/net/ipv6/dedicated

Note: The command output displays the uid rather than the username.


BetterCPU Configuration

Refer to the BetterCPU HOWTOs section for details on configuration and setup.

BetterCPU Concepts

BetterCPU is designed to maximize cpu utilization for shared hosting environments and at the same time prevent one or many users from abusing the cpu utilization. BetterCPU makes it very easy to quickly identify the CPU hogs on the system and keep them under control and then decide if these users need to moved to a higher paying account that provides them with the resources they need to run their business. This is accomplished by defining BetterLinux groups of users or mapping BetterLinux groups to cPanel packages. Once the user groups are configured, then throttling limits are defined for CPU.

Defining jail cores

Before limits are configured, it's important to understand BetterCPU jails. The jails are elastic by nature and only exist if they are needed by the BetterCPU controller to temporarily limit an abusive user process. Think of process jailing as analogous to law enforcement:

  • A user is a “citizen”
  • The performance thresholds are the “law”
  • BetterCPU enforces the laws

A system managed by BetterCPU having multiple processing cores and/or CPUs has a finite number of resources available for CPU utilization. If all of the users on the system are well-behaved, then all the cores are available to be shared between all of the processes on the system.

If users start misbehaving (ie, taking up too many CPU resources), then cores are removed from the pool used by all users and assigned “jail” status. Misbehaving users are moved to the cores assigned as the jail, separating them from others running on the system.

When a misbehaving user stops operating outside of the “law”, it is released from jail and again shares resources with the rest of the processes on the system.

Configuration of the jail is done using the parameters: 

jail_cnt
max_jail_cnt

Commenting both of the parameters out sets the default setting of jail_cnt to 1 core and max_jail_cnt to total cores minus 1. BetterCPU then auto-scales the jails the system needs based on demand for jail cores. However, the jail cores will never take a priority over non-jail cores.

The jail_cnt can be defined using two different methods:

  • If you know that you always want a fixed number of cores to be jails, then you can assign the number of cores to the jai_cnt. For example: 
jail_cnt 2

The value of 2 will always assign 2 cores for the jails no matter how many cores you have on your systems. If one of your servers is a two core system, BetterCPU will assign a value of 1 knowing that it can't assign all the cores as a jail.

  • is expressed as one or more pairs of numbers. For example: 
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system, and the number of cores to use for the jail. In this example, a 4-core system uses 1 core for the jail; an 8-core system uses 2 cores for the jail, as does a 16-core system. A 24-core system uses 6 cores for jail. If for some reason, you leave out a machine with cores that don't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. In the example above, a 32 core machine will pick up the jail core value from the 24:6 setting since it's the closest total core value to a 32 core system. The 32 core system will be assigned 6 cores as the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails, and is written in the same manner as the jail_cnt parameter. Setting the max_jail_cnt parameter will fix the ultimate size of the jail. Fixing the jail size will determines the amount of the cpu utilization abusive users will share in the jail core(s).

By allowing multiple values to be defined for these parameters in the configuration file, the configuration file can be used on multiple systems without modification. BetterCPU will select the appropriate value based on the configuration of the system.

Defining BetterCPU Throttling Limits

The BetterCPU throttling limits are defined by a percentage of one core for each user.

Once the jail cores are defined, BetterCPU provide two throttling parameters that define when a user is moved into and out of the jail core(s). The syntax for these thresholds is defined with the following parameters: 

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set jail throttling limits for groups of uids defined in groups.conf or betterlinux-cpud.conf on cPanel installs. The cpu_percent sets the percentage a user (uid) can use of one core on the system. The values for cpu_percent in start_throttling and stop_throttling are based on one core and are not adjusted when cores are added or removed from the jail.

When a throttled user’s total utilization drops below the stop_throttling for cpu_percent and "seconds" settings, then the process is removed from the jail.

For example, consider an 8-core system with the following values configured: 

start_throttling group=example-group cpu_percent=50,time=3
stop_throttling group=example-group cpu_percent=25,time=10

A user whose processes exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system.



Throttling Users

Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. By default, the non-cPanel installation (betterlinux-install) does not configure BetterCPU to throttle users. However, the cPanel installation (betterlinux-cpanel-install) script does configure and start BetterCPU throttling for all cPanel users on the server. The following steps cover setting up jail cores, defining the user groups, setting CPU core usage limits and starting the BetterCPU daemon (cpud).

Configuring the Jail Cores

Throttling is configured using parameters located in: 

non-cPanel install: /etc/betterlinux/cpud.conf.d/rules.conf:
cPanel install: /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf

jail_cnt
max_jail_cnt

These settings are designed so that the same configuration file may be used on different systems with different numbers of cores. However, the cPanel installation script removes jail_cnt and max_jail_cnt from the configuration file. When these parameters are removed, the cpud daemon uses the default settings wich is jail_cnt=1 and max_jail_cnt=total cores minus 1 and dynamically auto-scales up or down the number of jail cores needed for the system.

Syntax: 

jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...]

The jail_cnt parameter is defined using one of two methods:

  • If a fixed number of cores for jail is desired, make the assignment using jail_cnt as in this example: 
jail_cnt 2
In this example, 2 cores will be assigned for the jail as long as at least one core is free as a non-jail core. If the system is a dual core system, BetterCPU automatically assigns a value of 1 to leave one core free for non-throttled processes.
  • If the number of cores for jail varies based on the total number of cores in the system, use the following format: 
jail_cnt 4:1 8:2 16:2 24:6

The pairs represent the total number of cores in the system and the minimum number of cores to use for the jail. In this example, 4-core systems use 1 core for the jail, 8-core and 16-core systems use 2 cores for the jail, and 24-core systems use 6 cores for jail. If the system has a number of cores that doesn't match the values in jail_cnt, BetterCPU will assign the next lowest value you have defined as the jail cores. For example, using the values specified in the previous example, a 32-core system will use the jail_cnt value based on the 24:6 setting because it is the closest total core value to a 32 core system; consequently, the system will be assigned 6 cores for the jail.

The max_jail_cnt parameter defines the upper limit for the number of jails and is formatted identically to the jail_cnt parameter. By default, only the value of jail_cnt is used, but when a max_jail_cnt is specified, the jail is expanded when less than the value in jail_min_idle is available per jail core.

The jail_min_idle value is the minimum amount of idle utilization to maintain on the jail cores if jail_max_cnt has not been reached. If the average free utilization on each jail core is less than jail_min_idle, the jail is expanded until either the jail_min_idle value is maintained or jail_max_cnt is reached.

Syntax: 

max_jail_cnt {total cores}:{jail cores} [{total cores}:{jail cores} ...] or max_jail_cnt {number of jail cores}

Example 

max_jail_cnt 2:1 4:2 8:2 16:2

The result on a dual core system is no different from jail_cnt. But for 4-core machines and above, if the max_jail_cnt value has not been reached and more non-jail cores are available, then an additional jail core is added if the jail needs to be expanded. 

max_jail_cnt 2

The result will always set the number of maximum jails to two. The maximum number of jails are set at two no matter how many cores may be on the system. The exception being, when they system has 2 cores it will set the max jail core to one.

In order for the jail to expand, add a jail_min_idle value to denote how much free utilization should be maintained in the jail: 

jail_min_idle 5

BetterCPU will attempt to maintain at least 5% free CPU per jail core, and will expand the jail if max_jail_cnt has not been reached.

Defining User Groups

For more details on groups please see the Defining User Groups section of the documentation.

The BetterLinux cPanel install script will discover all the cPanel users on the system and create a user group called "cpanel-users" that contains a list of all the users assigned to a cPanel package.

The bl-install script will not create user groups, they need to be created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, or files containing UIDs or usernames, CPU usage levels can be precisely defined for different groups, and durations and conditions for starting/stopping throttling can be defined. The group names defined in /etc/betterlinux/cpud.conf.d/groups.conf are used in the settings for the start_throttling and stop_throttling parameters.

Defining User Groups

When planning a strategy for defining groups, use the following steps:

  1. Decide how users/customers should be organized into groups or cPanel packages. This organization could be based on needs, paid service levels, or any other criteria.
  2. Ideally, user UIDs ranges or a file that contains a list of UIDs should be used to define users with similar requirements. The BetterLinux cPanel hooks feature will automatically create a uid text file with a name that matches the cPanel package and will add newly created cPanel users to the file.
  3. Edit /etc/betterlinux/cpud.conf.d/groups.conf or the BetterLinux cPanel hooks feature will do this automatically.
  4. For example:
    uid package-normal-users /etc/betterlinux/cpanel/package-normal-users.txt
    uid package-premium-users /etc/betterlinux/cpanel/package-premium-users.txt
  5. Save the groups.conf file
  6. Restart the BetterCPU daemon:
    service cpud restart

Once this parameter change is in place, throttling limits that meet each group's SLA can be defined for each group.

For non-cPanel servers, the users will likely already be in a group defined in /etc/group. If this is the case, it is fairly easy to use that group definition within the groups.conf file. Use the following command to extract the group definition and reformat it for inclusion in the /etc/betterlinux/cpud.conf.d/groups.conf file. When defining the syntax, simply replace {group_name} with the name of the group defined in /etc/group. 

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting CPU Core Usage Limits

Once jail_cnt and jail_cnt_max are defined and groups are set up, it is necessary to configure the throttling parameters that define when user groups are moved into and out of the jail cores. The syntax for these thresholds is defined with the following parameters: 

start_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}
stop_throttling group={group}[,{group2}...] cpu_percent={integer},seconds={integer}

These parameters set throttling limits for groups defined in /etc/betterlinux/cpud.conf.d/groups.conf. The cpu_percent value sets the percentage an individual user (uid) can use of the total non-jail cores on the system.

To understand how this works, it is important to understand how CPU utilization is measured in a multi-core system. In Linux, multithreaded processes can take more than 100% utilization because overall CPU utilization is measured as 100% * number_of_cores. So in an 8 core system, if all CPUs are running at maximum utilization, the system's overall utilization is 800% (100% * 8 cores).

The values for cpu_percent in start_throttling and stop_throttling are based on a normalized utilization assigned to a single core (rather than being spread across all cores). This means that while an 8 core system can have a maximum of 800% utilization, the value used for the cpu_percent value represents the fraction of a single core that could be used. Specifying a value of 80 for cpu_percent means that on a system with no cores presently allocated to jail, when the user exceeds a value of 80% * 8 = 640% on an 8 core system, a core is allocated to the jail and the user's processes are throttled. When a core is allocated to jail, the percentage used to determine if the threshold has been exceeded is 80% * 7 = 560%. If a second core is allocated to jail, this threshold drops to 80% * 6 = 480%.

What this means is that - using the example of cpu_percent set to 80 - any distribution of utilization across all 8 cores of an 8 core system that adds up to more than 640% utilization will trigger the allocation of a core for throttling and that user's processes will be moved to that single core. The utilization could be 100% of 6 cores and 40% of a seventh core, it could be 80% utilization distributed across all 8 cores, or it could be any variation where the user's total utilization exceeds the threshold.

When a non-throttled user's total process utilization exceeds cpu_percent for seconds seconds as defined in the start_throttling parameter, the user's processes are moved into the jail cores.

When a throttled user’s total utilization drops below the cpu_percent value for seconds seconds as defined in the stop_throttling parameter, the user's processes are moved out of the jail back onto the non-jailed cores.

To define these limits:

  • Edit the users.conf located in /etc/betterlinux/cpud.conf.d.
  • Add the following parameters users.conf, with appropriate limits to the system's needs: 
start_throttling group=example-group cpu_percent=50,seconds=3
stop_throttling group=example-group cpu_percent=25,seconds=10
  • Save users.conf.
  • Restart cpud: 
service cpud restart

With these settings, a user in the group defined with processes that exceed 50% utilization of one core for 3 seconds will be assigned to one of the jail cores on the system. When the user’s processes drops below 25% utilization of one core for ten seconds, the user will be removed from the jail.

Start the BetterCPU Daemon (cpud)

BetterCPU is now configured with basic jail core and throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows: 

service cpud {start|stop|status|reload|restart|condrestart}
 start = starts cpud daemon
 stops = stops cpud daemon
 status = shows the status of cpud daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Start the cpud daemon with the following command: 

service cpud start

Start the BetterLinux Stats Program (blstat)

In order to see user processes that are in the jail cores, execute the blstat program with root privileges: 

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterI/O. Press 'c' to see the stats for BetterCPU. Press '?' to display general help for the utility.

Limiting Execution Time Using time_monitor

BetterCPU can be used to limit the amount of time programs can execute. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting execution time are:

  1. Define a program or user group
  2. Add time_monitor to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When user groups are used to limit execution, an exception program group can be used to exclude programs from execution limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is as follows: 

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example: 

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. The user groups are structured as follows: 

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example: 

uid bad_users testuser1,501,1000-1999

User group definitions used for time_monitor execution limiting are identical to user groups used for other functions in the BetterLinux suite.

Add time_monitor to main.conf

Limiting the execution time for a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, the script or program will be called again. The script or program is called with a single parameter; that is the PID of the process that has exceeded the time limit.

For example, a script in /root called 'killer' might contain: 

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the execution time limit.

To define the limit, add the following line to the main.conf file: 

time_monitor groups=bad_progs seconds=5 child_of=/bin/bash program=/root/killer 10

In this example, we use the sample "bad_progs" program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for execution times of 5 seconds or more with a parent process that is /bin/bash. If the program has run for 5 seconds or longer, /root/killer is called with the PID of the program, and the script kills the program after logging the date and time.

The final parameter (10) tells cpud to not run the script more than once every 10 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until at least 10 seconds had passed from the previous execution.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring the execution time, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored: 

time_monitor_prog_exclude groups=exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command: 

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon: 

service cpud restart

Limiting Open Files Using file_limit

BetterCPU can be used to limit the number of concurrent open files a user can have at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent open files are:

  1. Define a user group
  2. Add file_limit to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When open files are limited, the file_limit_override and file_limit_exclude options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the following format used is: 

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example: 

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting open files with file_limit are identical to user groups used for other functions in the BetterLinux suite.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command: 

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon. 

service cpud restart

Limiting Processes Using process_limit

BetterCPU can be used to limit the number of concurrent processes a user can execute at one time. Limits are defined using user groups defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting concurrent processes are:

  1. Define a user group
  2. Add process_limit to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When concurrent processes are limited, options can be applied to override the limits for certain programs and to remove the limits from others.

Define a Program or User Group

When defining user groups, the format used is: 

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example: 

uid bad_users testuser1,501,1000-1999

User group definitions used for limiting concurrent processes with process_limit are identical to user groups used for other functions in the BetterLinux suite.

Add process_limit to main.conf

Limiting concurrent processes per user is very straightforward.

To apply a limit, once a user group is defined, add a line to apply the desired restiction to the main.conf file: 

process_limit group=bad_users limit=100

This limits the users in the bad_users group to 100 concurrent processes each at any given time.

To completely exclude specific programs from having limits applied, create a program group in groups.conf with the programs in it: 

prog exclude_process_limits /bin/cron,/bin/bash

Then add a process_limit_exclude line to main.conf: 

process_limit_exclude group=exclude_process_limits

If a user limit should be overridden for particular programs, specify the user group as part of a process_limit_override command and specify the specific programs that should have the different limit in main.conf: 

process_limit_override group=bad_users limit=120 child_of=/usr/sbin/sshd

This overrides the existing limits for bad_users for instances of /usr/sbin/sshd and sets those limits to 120.

Reload the BetterCPU Configuration

In order to activate the changes, issue the following command: 

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon. 

service cpud restart

Limiting Memory Utilization Using memory_limit

BetterCPU can be used to limit the amount of RSS memory programs can use. Limits can be defined using program groups or user groups, both of which are defined in /etc/betterlinux/cpud.conf.d/groups.conf.

The basic steps for limiting RSS memory usage are:

  1. Define a program or user group
  2. Add memory_limit to /etc/betterlinux/cpud.conf.d/main.conf
  3. Reload the BetterCPU daemon's configuration

When user groups are used to limit memory, an exception program group can be used to exclude programs from memory limits.

Define a Program or User Group

When defining a program group, the format used in groups.conf is: 

prog {group_name} {full_path_and_executable}[,{full_path_and_executable}...]

For example: 

prog bad_progs /usr/local/bin/jailme,/usr/local/bin/eatmem

User groups may also be defined. These user groups are structured as: 

uid {group_name} {username|uid|uid_range|filepath}[,{username|uid|uid_range|filepath}...]

For example: 

uid bad_users testuser1,501,1000-1999

User group definitions used for RSS memory usage limiting are identical to user groups used for other functions in the BetterLinux suite.

Add memory_limit to main.conf

Limiting the RSS memory used by a program group is very straightforward. A script or executable program is called to perform an operation on the program being executed. In practical terms, the script does not need to terminate the program or restart it, but on subsequent checks, if the memory usage still exceeds the allowed limit, the script or program will be called again. The script or program is called with a single parameter; that is, the PID of the process that has exceeded the memory limit.

For example, a script in /root called 'killer' might contain: 

#!/bin/bash
echo date >> /root/killer.log
kill -9 %1

This script will write the output of the date command to /root/killer.log, then send a SIGKILL signal to the PID that has exceeded the memory limit.

To define the limit, add the following line to the main.conf file: 

memory_limit group=bad_progs rss_memory=5000 program=script/root/killer wait_to_run=30

In this example, we use the sample bad_progs program group defined above. Programs in that list (/usr/local/bin/jailme and /usr/local/bin/eatmem) are checked for RSS memory usage of 5 GB (5000 MB). If the program exceeds the allowed RSS memory usage, /root/killer is called with the PID of the program passed on the command-line, and the script kills the program after logging the date and time.

The final parameter (30) tells cpud to not run the script more than once every 30 seconds. If the /root/killer script did not kill the program but instead did something else (such as restarting a service or modifying and activating a configuration change), the script would not be called again until 30 seconds had passed from the previous execution, assuming the memory utilization had not dropped below the specified threshold.

When using a user group (rather than a program group), the same concepts apply. If there are programs the users should be allowed to run without monitoring RSS memory usage, define a program group for those exceptions and use the following command to globally exclude those programs from being monitored: 

memory_limit_prog_exclude exclude_group

Reload the BetterCPU Configuration

In order to activate the changes, issue the command: 

service cpud reload

This reloads the configuration without restarting cpud. The following command can also be used, but will stop and restart the cpud daemon: 

service cpud restart


BetterIO Configuration

Refer to the BetterIO HOWTOs section for details on configuring BetterIO.

BetterIO Concepts

The BetterIO controller applies restrictions at the block IO layer (above the IO scheduler) to the communications channel between the CPU and the block devices on a system.

In evaluating overall system performance, BetterCPU takes care of managing CPU and memory resources, but when a process is waiting on I/O, that process may be incorrectly reflecting how much CPU it would be using if it were not waiting for the disk channel to respond to a request for resources.

From a CPU perspective, this is referred to as “Waiting on I/O”, or iowait%. If the I/O channel is not also managed, then the system will not be able to run optimally.

When talking about I/O performance, there are a few different aspects to it. The user of an application is primarily concerned with how long it takes to retrieve information from the storage device. At a high level, slow retrieval of data is the most obvious indicator of system performance. The slower data retrieval is, the more likely the user is to complain about performance.

There are a number of factors that one must take into consideration when optimizing I/O performance so the end user experiences good performance. To monitor these factors, one can use the iostat command or blstat (BetterLinux stat program) that is built specifically for the BetterControl Suite of products.

What can BetterIO tune to affect block device utilization?

At a high level, tuning is applied to I/O operations per second (iops) and bandwidth (bw). These two parameters can be tuned by applying minimum and maximum values in /etc/iothrottled/iothrottled.conf.d/rules.conf.

Those system-wide minimum and maximum values can be refined by applying user limits and program limits, which are defined in /etc/iothrottled/user_limits.conf and /etc/iothrottled/prog_limits.conf, respectively.

The limits defined can be scaled to allow the block device channel to be utilized more effectively when it is under-utilized. The auto_scale parameter defines how scaling is applied to keep both the bandwidth utilization and iops within a desired performance range.

Setting I/O Bandwidth Limits Per-User or by User-Group

Individual user or group limits can be set using the configuration file /etc/iothrottled/user_limits.conf.

This file uses a single line per entry: 

[user|group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or mount points to include or exclude.

For example, the file might contain a line that reads: 

testuser bw max sda,sdb,sdc 10000000

This would mean that the user testuser has their maximum bandwidth (bw) limited on devices sda, sdb, and sdc to 10000000 B/s (Bytes per Second).

The first field can be a user or group name. Groups are defined in the file /etc/betterlinux/groups.conf. If a group and a user have the same name, the limit applies to the user.

Setting I/O Bandwidth Limits by Program-Group

More granular control over individual programs’ usage of block device I/O rates and data rates can be achieved using the /etc/betterlinux/iothrottled.conf.d/prog_limits.conf file.

This file is similar to the user_limits.conf file, except that it specifies a program group name. The line format for this file is: 

[program_group_name] [S|G] [bw|ops] [device/mount point list] [limit]

The fields that match up with fields in prog_limits.conf are defined the same way as in user_limits.conf.

The prognam_group_name field contains the names of the program or programs which is defined in groups.conf.

If the third field contains an “S”, the limit applies to each instance of the program started by the use or group. A “G” means the limit applies to all instances together.

For example: NOTE: In all the examples below we assume to have the group "tar_progs" defined in /etc/betterlinux/iothrottled.conf.d/groups.conf: 

prog tar_progs /bin/tar,/usr/bin/tar

If our goal is to limit each individual instance of "tar" to a 5 MB/s I/O rate we would use the following line. 

tar_progs S bw max inc="^/" 5000000

NOTE: all the "/bin/tar" and "/usr/bin/tar" instances will be affected by this rule if the UID that runs them is not included in any UID group. Remember that UID groups always win against PROG groups.

If our goal was to group ALL instances of tar to an aggregate bandwidth of 5 MB/s then you would change the "S" (Single process) in the example above to "G" (For grouping all tar processes together). This option is helpful when you want to limit one or more instances together. So, if you had one tar running or 50 tars the total disk I/O bandwidth would never exceed 5 MB/s.

The syntax for this scenario would be as follows: 

tar_progs G bw max inc="^/" 5000000

If our goal was to limit each instance of tar to a disk I/O rate of 5 MB/s, but would only like to enforce this limit on any filesystem mounted under /home (and any sub-directories) but have no limits enforced on other drives then the syntax would be as follows. 

tar_progs G bw max inc="^/home" 5000000

You can include multiple block devices (hard drives) or mount points to be included in the limits that you have defined as shown below: 

tar_progs G bw max inc="/home",sdb,dm-1,inc="/var/matt",inc="/etc" 5000000

In addition, you can specific drives to be excluded from the limits that you have defined. This syntax is shown below. 

tar_progs G bw max inc="/",exc="^/media|^/mnt" 5000000

For example: 

tar_progs S bw sda 1000000

This applies a bandwidth limit of 1000000 bytes/s to each instance of the /bin/tar or /usr/bin/tar commands run by any user not included in a UID group. If one of these users starts two concurrent instances of the tar command, each one is limited to 1000000 b/s of block device I/O bandwidth. If this line reads: 

tar_progs G bw sda 1000000

The bandwidth limit is applied to all instances of tar that the users launch. In this case, if the users launch two concurrent instances of the tar command, the total bandwidth both instances can use together is 1000000 B/s.

Below are some configuration rule examples of how to limit I/O bandwidth for certain programs.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied. The inc= and exc= directives take an extended regular expression for mount points. For example, the line: 

tar_progs G bw max sda,inc="/home|/backup",exc="/mnt1|/mnt2" 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that have "/home" or "/backup" in their mount point (that would also include, for example, "/home2", "/media/home3", excluding all the mount points that have "/mnt1" or "/mnt2" (like "/mnt21", "/mnt/mnt1", ...). The values are regular expressions and they can appear anywhere within the mount point). Including ^ and $ in the regex will force a match at the begining or end.

The evaluation of these expressions always follows the order of:

  • Explicit device names
  • Devices mounted in the "inc" list
  • Devices mounted in the "exc" list

Exclusions will ultimately override the other values in the list. Consider the following list of mounted devices: 

/dev/sda mounted at /
/dev/sdb mounted at /home
/dev/sdc mounted at /media/backups
/dev/sdd mounted at /usr
/dev/sde mounted at /mnt/home
/dev/sdf mounted at /srv/www
/dev/sdg mounted at /srv/ftp
/dev/sdh mounted at /srv/home

With the following restrictions in user_limits.conf: 

tar_progs G bw max sda,sdd,inc="/home",exc="/srv" 1000000

Programs in group "tar_progs" will be limited to 1 MB/s on /home and /mnt/home, but not on /srv/home, due to the exc="/srv" rule.

How do the block device inclusion/exclusion lists work?

Simple configuration of BetterIO uses device names in a list in the user_limits.conf and prog_limits.conf files. For example: 

testuser bw max sda,sdb,sdc 1000000

This applies the limit of 1000000 B/s for the user testuser on devices /dev/sda, /dev/sdb, and /dev/sdc.

Modern Linux systems, however, do not always guarantee the assignment of devices, which can lead to issues with limits not being applied to the correct device.

To ensure limits are applied to the correct device, the use of inc= and exc= directives along with (or instead of) device names provides greater flexibility in defining where limits are applied.

The inc= and exc=' directives take an extended regular expression for mount points. For example, the line: 

testuser bw max sda,inc=”/home|/backup”,exc=”/mnt1|/mnt2” 1000000

Applies the limit of 1000000 B/s throughput to device /dev/sda, the devices that are mounted at “/home” and “/backup” (and would also include, for example, “/home2”, “/media/home3” - the values are regular expressions and can appear anywhere within the mount point), but would exclude any mount point that includes “/mnt1” or “/mnt2” (including “/mnt21”, “/mnt/mnt1” - as with the “inc=” directive). Including ^ and $ in the regex will force a match at the begining or end.

Another example for this using regular expression that will apply the limit of 1MiB/s to any mounted block device except those mounted under /var or /tmp 

testuser bw max inc=”^/”,exc=”^/var|^/tmp” 1000000

In this example we want to limit all the mounted block devices, except the temporary mounted devices, usually mounted under /mnt or /media.

  • add the line below to /etc/betterlinux/groups.conf: 
testgroup uid 500-1000
  • Then add the line below to /etc/betterlinux/iothrottled/user_limits.conf: 
testgroup bw max inc=”^/”,exc=”^/mnt|^/media” 1000000

The evaluation of these expressions always follows the order of:

  1. Explicit device names
  2. Devices mounted in the “inc” list
  3. Devices mounted in the “exc” list

Exclusions will ultimately override the other values in the list.

Consider the following list of mounted devices:

/dev/sda1 mounted at /
/dev/sda2 mounted at /home
/dev/sdb1 mounted at /media/backups
/dev/sdb2 mounted at /usr
/dev/sdc1 mounted at /mnt/home
/dev/sdc2 mounted at /srv/www
/dev/sdd1 mounted at /srv/ftp

With the following restrictions in user_limits.conf: 

testuser bw max sda,sdd,inc=”/home”,exc=”/srv” 1000000

The result would be:

sda would be included
sdb would be excluded
sdc would be excluded
sdd would be excluded

So the only device controlled by BetterIO in this scenario would be /dev/sda. The controls are applied per block device, not on a per-partition basis - so if a partition is excluded from control, all partitions on that block device will be excluded.

When configuring your system, it is important to take this into consideration - you do not want your swap partition to be throttled by BetterIO, but if you put swap on /dev/sda and control /dev/sda with BetterIO, you may end up impacting performance on the system in ways you may not intend to.

Examples include/exclude:

/dev/sda1 on /
/dev/sda2 on /home
/dev/sdb1 on /home2
/dev/sdc1 on /home3
/dev/sdd1 on /home4 
testuser bw max inc="/home",exc="/home2"

The rexeg's are matched against the left of the mount point. So the include matches /home1, /home2, /home3, /home4 and the exclude matches /home2 only. So the result is:

/dev/sda included
/dev/sdb excluded
/dev/sdc included
/dev/sdd included

Example Exclude:

/dev/sda1 on /
/dev/sda2 on /home
/dev/sdb1 on /src
/dev/sdb2 on /usr
/dev/sdc1 on /extra 
testuser bw max exc="/src"
/dev/sda included
/dev/sdb excluded
/dev/sdc included

How does auto scaling work?

The auto_scale parameter is an on/off toggle switch for enabling or disabling the auto scale feature. The auto scaling algorithm is applied to overall bandwidth utilization or iops giving the system administrator the ability provide more bandwidth to a defined set of users when it's available.

When setting the value for this parameter, an integer is entered that represents the on/off values:

0 = auto scaling off
1 = auto scaling on

Auto scaling is controlled by these parameters:

scale_pct_hi
scale_pct_low
scale_pct_periods

Additionally, the following parameters apply to auto scaling:

scale_min
scale_max
scale_up_inc
scale_down_inc

To understand how these parameters are related to each other, consider the following scenario:

With the following line in /etc/iothrottled/user_limits.conf: 

testuser bw max sda 1000000

And in /etc/iothrottled/iothrottled.conf.d/rules.conf:

auto_scale 1
scale_pct_hi 80
scale_pct_low 20
scale_pct_periods 2
scale_min 25
scale_max 500
scale_up_inc 20
scale_down_inc 5

If the overall block device utilization is below 20% (scale_pct_low), then the limit applied to testuser is increased by 20% (scale_up_inc) of 1000000 (taken from user_limits.conf for this user), or 200000 B/s (Bytes per Second).

If overall block device utilization goes over 80% (scale_pct_hi), then the limit applied to testuser is decreased by 5% (scale_down_inc) of 1000000 (from the user’s limit in user_limits.conf), or 50000 B/s.

The increase/decrease in the scaling factor is tracked as well, so on the first cycle, the scaling factor is 120%. The second iteration scales up to 140%, and so on, until the limit set by scale_max is reached.

Similarly, if scaling down were to be applied instead, that scaling would drop by 5% each iteration until the scale factor reached the scale_min limit of 25%.

The result is that BetterIO attempts to keep the block device channel utilization between 20% and 80% at all times, and adjusts the utilization limits for all users in order to achieve that goal. If one were to graph the utilization over time, the goal is to have that graph look something like this:

Iothrottled graph.png


Whenever the utilization drops below the threshold at 20% or goes above the threshold at 80%, the adjustments kick in so the utilization falls in the area between the two thresholds. The majority of the time, that device’s utilization should be between 20% and 80% with this configuration.

Those adjustments are subject to the scale_min and scale_max parameters. Having limits on the amount of scaling will prevent BetterIO from attempting to scale up without a limit when there is no I/O taking place. If it were allowed to do that, then a period with no I/O could end up with a scaling factor in excess of 1000%, and the process of scaling down would result in it taking a significant amount of time with no effective limit in place, which would make the available bandwidth the bottleneck for system performance until the scaling factor returned to a reasonable range.

How can one test to see if BetterIO is controlling devices?

A simple and quite dramatic way to see BetterIO limiting I/O is to set a lower threshold for a user that is very small. For example, create a user called testuser and log in as that user. Run the following command: 

sync; dd if=/dev/zero of=temp.fil bs=1M count=10 oflag=direct; rm temp.fil

The results from that command for an unmanaged user would be similar to this screenshot:

Iothrottled blocking devices.png

If we then modify the /etc/iothrottled/user_limits.conf file to include the line: 

testuser bw max inc=”/” 500000

and rerun the dd command above, the following output results:

Iothrottled dd results.png

The drop in throughput from 350-400 MB/s to 500-600 kB/s shows the effect of the change in bandwidth limits for this user.

Are there any other considerations to keep in mind?

With BetterIO, the limits are set on a per-block device basis. It is important to keep in mind that I/O channels are not always assigned in a 1:1 ratio with block devices. If a series of storage devices are connected to the system by a Fibre Channel connection, the channel bandwidth available for all of those devices needs to be taken into consideration across all of the devices.


If devices /dev/sda, /dev/sdb, and /dev/sdc are all on the same disk controller, and the maximum throughput of that controller is 1 Gbps, then all three disks cannot run at 1 Gbps even if the drive electronics would permit that.

If you then have 10 users limited to 100 Mbps each, then performance will effectively be running without any throttling because the disk channel will be oversubscribed if all of the users are attempting to fully utilize their allocated bandwidth to each of these three devices. At the same time, these users are still limited to 100 Mbps each and are unable to monopolize the resources.


Once installation is complete, reboot the system to automatically load the BetterLinux kernel and the BetterLinux kernel modules. Once the the system is rebooted, configure BetterIO and start the iothrottled daemon. By default, BetterIO does not throttle any users. The following sections outline how to define user groups, set I/O limits, tune I/O scaling and balancing, and start the BetterIO daemon (iothrottled).

Defining user groups

User-created groups do not exist on the system until they have been created by the system administrator. By creating and populating groups with UIDs, UID ranges, usernames, external commands, or files containing UIDs or usernames, I/O bandwidth usage levels can be precisely defined for different groups. Durations and conditions for starting/stopping throttling can also be defined. The group names defined in /etc/betterlinux/iothrottled.conf.d/groups.conf are used in the settings for setting per-group I/O limits.

To define groups:

  1. Choose a group name.
  2. Populate the group with UIDs, UID ranges, user names, or files containing UIDs or usernames
    Note: A group may contain one or more usernames, UIDs, or ranges.

The following examples illustrate the command syntax: 

Single range:
 Syntax:  uid {group name} {uid range}

 Example: uid example-group 0-499
This example includes all UIDs from 0-499 as members of the group example-group

 Example: uid example-group 800-
This example includes all UIDs from 800 and above as members of the group example-group
 
Multiple UID ranges:
 Syntax:  uid {group name} {uid range}[,{uid range2}...]

 Example: uid example-group 0-499,600-699
This example includes all UIDs from 0-499 and from 600-699 as members of the group example-group

When planning a strategy for defining groups, use the following steps:

  1. Decide how users/customers should be organized into groups.
    • This organization could be based on needs, paid service levels, or any other criteria.
    • Ideally, user UIDs ranges should be used to define users with similar requirements.
  2. Edit /etc/betterlinux/iothrottled.conf.d/groups.conf.
    • For example:
      uid normal-users 500-999
      uid premium-users 1000-1999
  3. Save the groups.conf file.
  4. Restart the BetterIO daemon:
    service iothrottled restart

Once this parameter change is in place, you can define throttling limits that meet each group's SLA.

If the users are already in a group defined in /etc/group, it is fairly easy to use that group definition within the groups.conf file. Using the following command will extract the group definition and reformat it for inclusion in the /etc/betterlinux/iothrottled.conf.d/groups.conf file. Just replace {group_name} with the group name defined in /etc/group. 

echo "uid `grep ^{group_name} /etc/group | cut -d: --output-delimiter=" " -f 1,4`"

Setting I/O Bandwidth Limits

Once groups have been defined, I/O bandwidth rate per second and operations per second can be limited on a per-device basis. To do this, edit the user_limits.conf file located in /etc/betterlinux/iothrottled.conf.d. This file is used to limit individual users or groups defined in /etc/betterlinux/iothrottled.conf.d/groups.conf.

The user_limits.conf file uses a single line per entry: 

[group] [bw|ops] [max|min] [device/mount point] [limit]

The fourth parameter in this line can be a combination of block device names, or a series of perl-compatible regular expressions (PCRE) that define mount points to include or exclude.

For example: 

example-group bw max sda,sdb,sdc 10000000

This example limits the maximum bandwidth to be used by users in example-group for devices /dev/sda, /dev/sdb, and /dev/sdc to 10 MBps.

Another example: 

io_standard_users bw max inc="^/",exc="^/home" 5000000

This example limits maximum bandwidth to be used by the users in the io_standard_users group for all directories except /home to 5 MBps.

After updating the user_limits.conf file with limits applicable to the system, save the file and restart iothrottled.

Note:

  • If a group and a user have the same name, the limit applies to the user.
  • Only one limit per device can be set. It is not possible to set different limits on multiple partitions on the same device or on different mount points when they are all defined on the same device.

Please see the BetterIO documentation for more configuration options for I/O bandwidth limiting features by program and inclusion/exclusion options.

Tuning I/O Scaling and Balancing

The auto_scale and seek_balance features are enabled by default. These algorithms apply to overall bandwidth utilization or I/O operations per second (iops). They give the system administrator the ability provide more bandwidth above the limits set for users for which I/O bandwidth limiting is configured in user_limits.conf.

These settings are contained in /etc/betterlinux/iothrottled.conf.d/scale.conf. The configuration parameters define how these settings affect the I/O bandwidth. Details on how these settings work can be found in the user documentation for scaling and balancing.

Start the BetterIO daemon (iothrottled)

BetterIO is now configured with basic throttling settings. On CentOS, use the service command to start/stop the service.

The command syntax is as follows: 

service iothrottled {start|stop|status|reload|restart|condrestart}
 start = starts iothrottled daemon
 stops = stops iothrottled daemon
 status = shows the status of iothrottled daemon
 reload = reloads changes to the .conf files without restarting the daemon
 restart = restarts the daemon, loading the configuration as defined in the config files
 condrestart = The condrestart (conditional restart) option restarts the daemon only if it is currently running
  This option is useful in scripts because it does not start the daemon if it is not running.

Use the following command to start the iothrottled daemon: 

service iothrottled start

Start the BetterLinux stats program (blstat)

To see user processes that are being throttled by BetterIO, execute the blstat program with root privileges: 

blstat

blstat presents a top-like user interface that includes information specific to BetterCPU and BetterIO. Press 'i' to see the stats for BetterIO. Press '?' to display general help for the utility.

BetterMySQL Concepts

BetterMySQL primarily extends BetterCPU and BetterIO's throttling features to abusive MySQL users, processes, and threads, which are common in multi-tenant Linux environments. Web hosting providers, for example, house many customers per server, several of whom use MySQL-intensive web applications. All too often, these employ poorly-written MySQL statements, some of which perform full-table scans without indexes. Such endless and consumptive queries unfairly stifle performance for all system tenants. BetterMySQL solves this problem: It assigns responsibility for queries to individual Linux users instead of to the generic MySQL user. This enables BetterMySQL to work with BetterCPU, BetterIO, and BetterBandwidth to identify, isolate, and throttle specific threshold-exceeding MySQL threads, processes, and users regarding CPU usage, bandwidth to the disk, and network bandwidth.

It is possible to automate deactivations for MySQL using Google's user patches. The problem with this method is the user process is deactivated. The second a user is deactivated they going to be very unhappy customers. This is not an ideal approach for a hosting company, primarily because the cost per customer acquisition is very high and likely to go up in cost not down. With BetterMySQL a MySQL user is never deactivated. Instead, their queries are slowed down a bit and only slow enough to make sure the system is responsive for everyone else on the system. BetterMySQL can determine the difference between cache hits or real I/O hits to the disk and limit I/O for a specific MySQL queries.

Install and Configure BetterMySQL

BetterMySQL cannot be installed unless MySQL is already on the system. The BetterMySQL HOWTO provides a step by step guid for installing and configuring BetterMySQL on Linux and Linux with cPanel.

Because BetterMySQL extends BetterCPU and BetterIO throttling functionality to MySQL, you will find BetterMySQL’s shared configuration options outlined in the BetterCPU functional documentation. BetterMySQL’s configuration file serves only to enable BetterMySQL.

Once BetterMySQL is installed and configured any cpu throttling parameters that are set will also throttling MySQL process / threads that exceed the cpu throttling limits that have been set. For example the cpu throttling syntax below will also include throttling for users MySQL processes that exceed the cpu limit: 

start_throttling group=cpanel_users cpu_percent=50 seconds=3
stop_throttling group=cpanel_users cpu_percent=30 seconds=5

Same logic apply's to any BetterIO limits that have been set. In this example, any users in the cpanel_users group that owns MySQL processes generating I/O that exceeds the device I/O limit of 5Mbs will be throttled. 

cpanel_users bw max inc="^/" 5000000


Configuring BetterMySQL for cPanel Support

BetterMySQL configuration is auto-configured for Linux servers with cPanel installed.

Installing the BetterLinux version of MySQL for cPanel

BetterMySQL installation for cPanel servers is done by running the bl-cpanel-install script and by answering yes to install BetterMySQL. If this step was skipped during the initial installation, BetterMySQL can be installed by running the following script: 

/etc/betterlinux/cpanel/bl-cpmysql-install

Configuring BetterMySQL for cPanel support

To provide and configure per-user query throttling of MySQL users a mapping of MySQL users to Linux users must be configured. This mapping and other configuration options are auto-configured by either the bl-cpanel-install script on first time installation or if the option to install BetterMySQL was skipped during the bl-cpanel-install, the bl-hooks script can be used after a manual installation of BetterMySQL. Example syntax for running the bl-hooks script: 

/etc/betterlinux/cpanel/bl-hooks --inint

Either of these scripts will create and configure the following files and setup the hooks to add / remove users to the usermap.conf file as they are added and removed via the cPanel WHM interface.

  • usersmap.conf contains the mapping of cPanel users to UIDs
  • mysql.conf contains the parameter location of the usermap.conf file
  • turns on BetterMySQL throttling by adding the approved_accounting parameter to /etc/betterlinux/cpud.conf.d/betterlinux-cpud.conf 
approved_accounting_program program=/usr/libexec/mysqld user=mysql
  • Restarts the cpud daemon to enable BetterMySQL throttling. 
service cpud restart

Configuring BetterMySQL for Linux Servers

BetterMySQL configuration is a manual setup for Linux servers

Installing the BetterLinux version of MySQL non-cPanel

After running the bl-install script, and the BetterLinux repository is enabled, BetterMySQL can be installed by executing the following command: 

yum install mysql-server

Configuring BetterMySQL for non-Cpanel Linux servers

To create the mapping file and define it's location, modify the /etc/betterlinux/mysql.conf.d/mysql.conf file and the parameter bl_usermap_filename.

The command format is as follows: 

bl_usermap_filename [usermap_file_path]

Example: 

bl_usermap_filename /etc/betterlinux/mysql.conf.d/usermap.conf

Now create the MySQL mapping to Linux user mappings located in the following file: 

/etc/betterlinux/mysql.conf.d/usermap.conf

The usermap file is a tab-delimited file that contains the following information: 

[mysqluser]	[linux_user]	[uid]

The UID value is optional; if not provided, MySQL populates this value in its internal user_map table when the user connects to the MySQL server to perform a database operation.

For example, the following file will create mappings for testuser1 and testuser2. For testuser1, the uid will be determined when that user makes a query against the database. 

testuser1	testuser1
testuser2	testuser2	501

The current mappings can be viewed by connecting to MySQL with a MySQL client program and executing the command "SHOW USER_MAP;": 

mysql> show user_map;
+-----------+-----------+------+
| From_user | Unix_user | Uid  |
+-----------+-----------+------+
| testuser2 | testuser2 |  501 |
| testuser1 | testuser1 |  500 |
+-----------+-----------+------+
2 rows in set (0.00 sec)

The approved_accounting parameter turns on the BetterMySQL throttling and is located in: 

/etc/betterlinux/cpud.conf.d/main.conf

The syntax is as follows for non-cPanel servers: 

approved_accounting_program program=/usr/sbin/mysqld user=mysql

Restart the cpud daemon to enable the configuration settings: 

service cpud restart

Monitoring MySQL Utilization with myusertop

BetterMySQL includes the myusertop utility which shows the MySQL-Linux user mappings and the current utilization per user within MySQL.

Once myusertop is running, press '?' to view the built-in help.

The output is similar to the output of the standard 'top' utility: 

mysqld(1) up 0 days, 05:54:48, started: Thu Jul 12 03:12:28 2012.
UID mappings from /etc/betterlinux/mysql.conf.d/usermap.conf
Totals since Startup/Flush_Stats (Summary, all users):          V=Toggle View: 1
Time: CPU 448          Busy 529         CPU/Busy 0.8477   RowsExamined:   1,476m
Qrys: Select 28,456    Update 0         Other 5,250         Adj-RowsEx:   1,251m
Sort: U=User i=UID t=Thds c=CpuTime b=BusTime C=C/B r=RowsEx a=AdjREx p=%CPU
      h=Help m=MySQLUsersOn/Off                                           q=quit
Linux/MySQLUser      UID  Thds CPUTime  BusTime CPU/Bus RowsExam AdjRowEx   %CPU
root                   0     2 0.0075   0.0074   1.0060     150      151  100.00
  root                 0     2 0.0075   0.0074   1.0060     150      151  100.00
testuser2            501     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser2          501     0 0.0000   0.0000   0.0000       0        0    0.00
testuser1            500     0 0.0000   0.0000   0.0000       0        0    0.00
  testuser1          500     0 0.0000   0.0000   0.0000       0        0    0.00

In this example, note that the %CPU column reflects the percentage of MySQL's current utilization so if MySQL is using 10% of the processor, then the numbers reflect root as using 100% of the 10% of total CPU utilization. If it were divided equally between the three users, myusertop would show 33.33% utilization for each user, but actual CPU utilization would be 33.33% of 10%, or 3.33% overall CPU utilization.

If the mapping is toggled off using the 'm' key, then the %CPU column should always add up to 100%.

Retrieved from "http://docs.betterlinux.com/w/index.php?title=Old_Doc_Pages&oldid=1625"