memmon Overview

memmon is a supervisor “event listener” which may be subscribed to a concrete TICK_x event. When memmon receives a TICK_x event (TICK_60 is recommended, indicating activity every 60 seconds), memmon checks that a configurable list of programs (or all programs running under supervisor) are not exceeding a configurable amount of memory (resident segment size, or RSS). If one or more of these processes is consuming more than the amount of memory that memmon believes it should, memmon will restart the process. memmon can be configured to send an email notification when it restarts a process.

memmon is known to work on Linux and Mac OS X, but has not been tested on other operating systems (it relies on ps output and command-line switches).

memmon is incapable of monitoring the process status of processes which are not supervisord child processes. Without the –cumulative option, only the RSS of immediate children of the supervisord process will be considered.

memmon is a “console script” installed when you install superlance. Although memmon is an executable program, it isn’t useful as a general-purpose script: it must be run as a supervisor event listener to do anything useful.

memmon uses Supervisor’s XML-RPC interface. Your supervisord.conf file must have a valid [unix_http_server] or [inet_http_server] section, and must have an [rpcinterface:supervisor] section. If you are able to control your supervisord instance with supervisorctl, you have already met these requirements.

Command-Line Syntax

$ memmon [-c] [-p processname=byte_size] [-g groupname=byte_size] \
         [-a byte_size] [-s sendmail] [-m email_address] \
         [-u email_uptime_limit] [-n memmon_name]
-h, --help

Show program help.

-c, --cumulative

Check against cumulative RSS. When calculating a process’ RSS, also consider its child processes. With this option memmon will sum up the RSS of the process to be monitored and all its children.

-p <name/size pair>, --program=<name/size pair>

A name/size pair, e.g. “foo=1MB”. The name represents the supervisor program name that you would like memmon to monitor; the size represents the number of bytes (suffix-multiplied using “KB”, “MB” or “GB”) that should be considered “too much”.

This option can be provided more than once to have memmon monitor more than one program.

Programs can be specified using a “namespec”, to disambiguate same-named programs in different groups, e.g. foo:bar represents the program bar in the foo group.

-g <name/size pair>, --groupname=<name/size pair>

A groupname/size pair, e.g. “group=1MB”. The name represents the supervisor group name that you would like memmon to monitor; the size represents the number of bytes (suffix-multiplied using “KB”, “MB” or “GB”) that should be considered “too much”.

Multiple -g options can be provided to have memmon monitor more than one group. If any process in this group exceeds the maximum, it will be restarted.

-a <size>, --any=<size>

A size (suffix-multiplied using “KB”, “MB” or “GB”) that should be considered “too much”. If any program running as a child of supervisor exceeds this maximum, it will be restarted. E.g. 100MB.

-s <command>, --sendmail=<command>

A command that will send mail if passed the email body (including the headers). Defaults to /usr/sbin/sendmail -t -i.

Note

Specifying this option doesn’t cause memmon to send mail by itself: see the -m / --email option.

-m <email address>, --email=<email address>

An email address to which to send email when a process is restarted. By default, memmon will not send any mail unless an email address is specified.

-u <email uptime limit>, --uptime=<email uptime limit>

Only send an email in case the restarted process’ uptime (in seconds) is below this limit. (Useful to only get notified if a processes gets restarted too frequently)

Uptime is given in seconds (suffix-multiplied using “m” for minutes, “h” for hours or “d” for days)

-n <memmon name>, --name=<memmon name>

An optional name that identifies this memmon process. If given, the email subject will start with memmon [<memmon name>]: instead of memmon: In case you run multiple supervisors on a single host that control different processes with the same name (eg zopeinstance1) you can use this option to indicate which project the restarted instance belongs to.

Configuring memmon Into the Supervisor Config

An [eventlistener:x] section must be placed in supervisord.conf in order for memmon to do its work. See the “Events” chapter in the Supervisor manual for more information about event listeners.

If the [unix_http_server] or [inet_http_server] has been configured to use authentication, add the environment variables SUPERVISOR_USERNAME and SUPERVISOR_PASSWORD in the [eventlistener:x] section as shown in Example Configuration 5.

The following examples assume that memmon is on your system PATH.

Example Configuration 1

This configuration causes memmon to restart any process which is a child of supervisord consuming more than 200MB of RSS, and will send mail to bob@example.com when it restarts a process using the default sendmail command.

[eventlistener:memmon]
command=memmon -a 200MB -m bob@example.com
events=TICK_60

Example Configuration 2

This configuration causes memmon to restart any process with the supervisor program name foo consuming more than 200MB of RSS, and will send mail to bob@example.com when it restarts a process using the default sendmail command.

[eventlistener:memmon]
command=memmon -p foo=200MB -m bob@example.com
events=TICK_60

Example Configuration 3

This configuration causes memmon to restart any process in the process group “bar” consuming more than 200MB of RSS, and will send mail to bob@example.com when it restarts a process using the default sendmail command.

[eventlistener:memmon]
command=memmon -g bar=200MB -m bob@example.com
events=TICK_60

Example Configuration 4

This configuration causes memmon to restart any process meeting the same requirements as in Example Configuration 2 with one difference:

The email will only be sent if the process’ uptime is less or equal than 2 days (172800 seconds)

[eventlistener:memmon]
command=memmon -p foo=200MB -m bob@example.com -u 2d
events=TICK_60

Example Configuration 5 (Authentication)

This configuration is the same as the one in Example Configuration 1 with the only difference being that the [unix_http_server] or [inet_http_server] has been configured to use authentication.

[eventlistener:memmon]
command=memmon -a 200MB -m bob@example.com
environment=SUPERVISOR_USERNAME="<username>",SUPERVISOR_PASSWORD="<password>"
events=TICK_60