Feature Focus: Monitored Server Scripts

Using Monitored Server Scripts

FusionReactor Enterprise Scripting enhances FusionReactor's Enterprise Monitoring to trigger configurable self-healing scripts when a server's responsiveness status changes.

This feature can be used to perform any task which can be launched from a platform script.

Examples might include:

• Interacting with SNMP or enterprise monitoring/reporting systems
• Performing automated restarts of affected instances
• Sending custom email messages
• Writing custom log messages

Since FusionReactor Enterprise Scripting is able to launch any platform executable (shell script,executable binary etc.), it may be tailored to virtually any environment.

What Are Enterprise Scripts?

FusionReactor (Enterprise Edition) can trigger a script when a server becomes unresponsive, and when it becomes responsive again. This mechanism might allow you to pro-actively attempt to restart a failed server or instance, integrate FusionReactor into an existing monitoring environment or provide custom logging or reporting. Any program which is runnable on your FusionReactor monitoring system may be used an Enterprise Script target.

On Windows systems, anything that is runnable as a binary, or can be run from the command prompt, for instance:

• Binary programs
• Batch files

On Unix and Unix-based systems, this includes everything you can start from a command line, for instance:

• Binary programs
• Shell scripts (including Bash, Ruby, Python and Perl)
• Java programs (when launched from an appropriate shell script)

How Do I Configure an Enterprise Script?

Enterprise scripts are configured by editing the Script of the server's property page, accessible in the Enterprise -> Manage Servers page, then clicking on the Modify icon of the required server.

You should take care to ensure the full path and filename to the script are correct. Also note that it is recommended to only configure one script for a server. If the same server has multiple scripts configured (for example you have entered the server into the dashboard in multiple groups) or you monitor the same server from different machines, then more than one script may be launched at the same time if the server has problems. The scripts may interfere with each other especially if they are trying to restart the same instance.

Enterprise Scripts are run whenever the Enterprise Monitor detects that a monitored instance has changed state:

• an instance which was previously available is no longer providing Enterprise data
• an instance which was previously unavailable has begun to provide Enterprise data

Local Monitoring: Monitoring Instances on the same computer as the Enterprise Monitor

Running all FusionReactor instances on the same computer makes it easy to develop scripts for use with Crash Protection.

Two instances run on the same computer. One of the instances acts as an Enterprise Monitor (Watchdog) for the other, monitored server. The watchdog server has added the monitored server to the list of managed servers in the Enterprise Dashboard of FusionReactor. A script for the monitored server is configured that will be executed by the watchdog server if the monitored server is unavailable or becomes available again. The script is used to restart the monitored server automatically after it became unavailable.

However, if the watchdog server itself becomes unavailable, the server is not longer monitored and can not be restarted automatically any more. To get around this the monitored server could also monitor the watchdog server which is described in the next section.

One instance/server is created for the task of Enterprise Monitor to monitor the other Operational servers. Every operational server is added to the Enterprise Monitor's list of managed servers in the Enterprise Dashboard of FusionReactor. For every server a script is configured that will be executed when the server becomes unavailable (or available again). The script is used to restart the operational server automatically if it becomes unavailable. One of the operational servers is also configured to monitor the Enterprise Monitor, and a script is configured to restart the Enterprise Monitor should it become unavailable. If this way the Enterprise Monitor watches over all of the operational servers, and one of the operational Servers watches other the Enterprise Monitor. Alternatively a second Enterprise monitor could be configured with the task of watching over the first Enterprise monitor if you do not want operational servers to perform monitoring tasks.

Distributed Monitoring

Running, monitoring and restarting FusionReactor instances in a distributed environment requires remote connections between the different machines. If a monitored server becomes unavailable this is monitored by a different machine on the network. This machine then calls a script which has to connect to the remote machine and restart the remote server/instance. Depending on the operating system the participating machines use, this can be done with (SSH) or some similar technology.

Simple Watchdog

Two servers run on different machines that have a network connection. One of the servers acts as the Enterprise Monitor (watchdog) for the other, monitored server. The watchdog server has the monitored server entered in it's list of managed servers in the Enterprise Dashboard of FusionReactor. A script for the monitored server is configured that will be executed by the watchdog server if the monitored server becomes unavailable (or available again). The script is used to login to the remote computer and restart the server/instance automatically after it becomes unavailable. This approach can have the same drawbacks as mentioned in the Local Server Simple Watchdog section before.

Cross Monitoring

All servers run on different machines that can reach each other over the network. One instance/server is created for the task of Enterprise Monitor to monitor the other Operational servers. Every operational server is added to the Enterprise Monitor's list of managed servers in the Enterprise Dashboard of FusionReactor. For every server a script is configured that will be executed when the server becomes unavailable (or available again). The script is used to login to the remote computer and restart the operational server automatically if it becomes unavailable. One of the operational servers is also configured to monitor the Enterprise Monitor, and a script is configured to login to the Enterprise Monitor computer and restart the Enterprise Monitor instance should it become unavailable. If this way the Enterprise Monitor watches over all of the operational servers, and one of the operational Servers watches other the Enterprise Monitor. Alternatively a second Enterprise monitor could be configured with the task of watching over the first Enterprise monitor if you do not want operational servers to perform monitoring tasks.

Summary

FusionReactor Crash Protection Scripts are powerful functionality that can lead to an increase of server availability in local as well as distributed server environments. You must however be aware of that a script that is executed by a Crash Protection rule is executed by the server instance that is monitoring the problem server and not by the problem server itself. If this is not the same computer, remote scripting has to be used to react to the situation in an appropriate way. Also note, that you should not use scripting in a single instance (self-monitoring) scenario, because the script may not reliably start.