Opsview Knowledge Center

Notification Methods

Overview of Notification Methods found in Opsview Monitor

What are Notification Methods?

Notification Methods are the medium through which you are notified regarding problems detected by the Opsview Monitor software.

By default, Opsview Monitor ships with 11 Notification Methods:

  • Email ' Sends an email to the email address specified.
  • RSS ' Publishes the problems as an RSS feed on a per User basis. Access to the RSS feed requires authentication so that only the correct Notifications are accessed. This uses basic authentication as RSS readers only support basis authentication.
  • Push for IOS ' Sends push Notifications to the Username/password specified. This same Username and password must be entered on the mobile app, and must be a valid Opsview.com Username/password.
  • Push for Android - Sends push Notifications to the User. The method needs a valid Opsview.com username and password to register your Opsview Monitor for receiving push Notifications. Mobile app users do not need to have an Opsview.com account for push to send to their devices.
  • AQL ' A web-based SMS gateway. Requires a valid AQL Username/password with account credit. Opsview Monitor tells the AQL Gateway the message and the mobile phone / cell number via HTTPS, and AQL will then send an SMS/Text message to the number provided.
  • SMS via Modem ' Sends an SMS via the physical modem attached to the Opsview Master/Slave server which must be connected to a POTS network.
  • VictorOps -Sends a Notification to VictorOps, a SaaS notification routing platform.
  • PagerDuty -Sends a Notification to PagerDuty, a SaaS notification routing platform.
  • Twilio - Sends a Notification to VictorOps, a SaaS notification routing platform.
  • Slack ' Sends alerts into a Slack room, i.e. #support.
  • Hipchat - Sends alerts into a Hipchat room, i.e. support.

Some of these notifications require modification, i.e. the Push methods require a Username and password, Slack requires a Webhook URL and so forth. The modification of the Notification Methods is done via 'Settings > Notification Methods', as shown below:

Configuring Notification Methods

This section will cover the configuration and edit modal window for each of the Notification Methods, and outline what is required in order to get the Notification Method working correctly.

Default Notification Methods within Opsview Monitor

Default Notification Methods within Opsview Monitor

Notification Method 'Run On' option

On a system with slaves configured, each notification method method has a 'Run On' option. This options allows you to specify whether the notification method is run on:

  • Master
  • Slave

Some methods, such as 'Service Desk Connector' must be set to Master.

Note: If you select Slave and the host that wants to send a notification is monitored by the master server, then the notification will be sent from the master server.

In the diagram above, the monitoring server for host H2 is the master server M. Host H1's monitoring server is the slave S1 and host H3 is monitored by slave S2. Host H4's monitoring server is slave S3.

This is important to decide because it could be possible that results are lost between the slave and the master in a network failure. However, there may be firewall restrictions to running a notification from a slave.

Note: The advantage of running notifications from a slave is that it is autonomous of the master. However, the slave may not do parent/child suppression if the parent of a host is on a different slave - this is particularly true if you are using a slave cluster. You have to send notifications from the master if you want the full suppression facilities as only the master has a complete view of all your hosts.

Note: If a service is monitored by a slave but the notification method is set to the master, then UNKNOWN states will not be notified on. This is because in a distributed environment, if a slave fails, the master will set the state of all services on that slave into an UNKNOWN state. This filtering means that you will not get lots of notifications for a single issue when a slave has failed.

Note: Notifications that are sent from a slave are logged on the slave, but are not logged on the Opsview master.

Email

The email Notification Method requires minimal configuration. The 'Active:' checkbox tells Opsview Monitor whether to list Email as a valid Notification Method when you are creating Notification Profiles for yourself or other Users.

'Run on:' tells Opsview Monitor to send alerts either via the master or via the slave servers (this can be useful if you have different configurations for postfix, sendmail, etc on your slave servers than your master).

Command: This is the script to execute. It is assumed that this script is stored in /usr/local/nagios/libexec/notifications. Add any command arguments as required if modifying the Notification Method .

User variables: 'EMAIL' is the name of a Variable that will be passed to the notify_by_email script and its value is the destination E-mail address for the Notification.

The 'EMAIL' field value is defined within the User edit window (Settings > Users and Roles > Users), as shown below:

The 'notify_by_email' command will use the configuration defined on the local server; be it the master or the slave server. The local email server configuration can be one of many including:

  • Postfix
  • Sendmail
  • Zimbra
  • .. more.

Simply ensure that the local mail server is configured correctly and can successfully deliver emails to email addresses. Generally configuring the local email server as a simple relay Host is sufficient, i.e. setting 'relayHost = mail.company.com' for example.

To change the 'sender address' for email notifications, simply modify /etc/mailname for the domain section (i.e. right of '@'). To modify the User/account section, i.e. 'opsview@', follow the specific guides for your email server of choice. In postfix, you can adjust this using the 'sender_canonical_maps' configuration parameter. For a guide see the Ubuntu Forums post here: http://ubuntuforums.org/showthread.php?t=38429.

The email template is written in Template Toolkit and exists in /usr/local/nagios/libexec/notifications/com.opsview.notificationmethods.email.tt.

You can customise this file, but be aware that it will be overwritten as part of any future upgrades.

RSS

Opsview Monitor comes with a Notification via RSS Notification Method. This runs only on the master server.

Notifications are written to an RSS file for each contact. The RSS feed is at http://opsview.example.com/atom.

Access to the RSS feed requires authentication so that only the correct notification file is accessed. This uses basic authentication as most RSS readers only support basic authentication.

Push for iOS

To enable push Notifications:

  1. You must include your valid Opsview.com Username (not email address) and password in the app, and in the Opsview Monitor 'Push Notifications for iOS Mobile' settings fields, like the screen.
  2. Your iOS device must be able to do a one-time connect to your Opsview Monitor Master server in order to retrieve it's UUID for registration of the device on push.opsview.com.
  3. Your iOS device must be able to connect to push.opsview.com via HTTPS.

To configure the 'Push for iOS' Notification Method, you only need to enter two items of information; the Opsview.com Username and password. After clicking 'Test Connection', you should see one of the two messages below:

Correct Opsview.com Username and password

Correct Opsview.com Username and password

Incorrect Opsview.com Username and password

Incorrect Opsview.com Username and password

If the Opsview Monitor system does not have direct access to the internet, you can specify a Proxy in the text box labelled 'Proxy:' and re-test the connection. Once the connection succeeds, that is the configuration completed.

Push for Android

To enable Push Notifications:

  1. You must set a valid Opsview.com Username and password in the app in order for it to be able to register itself with Opsview Monitor's push servers. You also need a valid Opsview.com username and password in the Opsview Monitor 'Push Notifications for Android Mobile' settings fields (pictured above) so that the Opsview Monitor server can send outgoing push notifications. These two Opsview.com accounts do not need to be the same and you can use a generic account for your Opsview Monitor notification method.
  2. Your Android device must be able to do a one-time connect to your Opsview Master server in order to retrieve it's UUID for registration of the device on push.opsview.com.
  3. Your Android device must be able to connect to push.opsview.com via HTTPS.

To configure the 'Push for Android' Notification Method, you only need to enter two items of information; the Opsview.com Username and password. After clicking 'Test Connection', you should see one of the two messages below:

Correct Opsview.com Username and password

Correct Opsview.com Username and password

Incorrect Opsview.com Username and password

Incorrect Opsview.com Username and password

If the Opsview Monitor system does not have direct access to the internet, you can specify a Proxy in the text box labelled 'Proxy:' and re-test the connection. Once the connection succeeds, that is the configuration completed.

AQL

AQL provide a simple way of transmitting SMS alerts from Opsview. This service works from most countries; all you need is an AQL account.

Sign up for prepaid credits at aql.com with the code 'opsera-1234' to receive discounted rates and 50 free credits.

AQL Username: This is the Username for the AQL system. Click on Check Credit to see how many credits you still have from AQL.

AQL Password: This is your AQL password.

AQL's solution requires that the Opsview server has connectivity to AQL over HTTP/80. AQL's messaging servers are:

  • gw1.sms2email.com
  • gw11.sms2email.com
  • gw2.sms2email.com
  • gw22.sms2email.com

These servers are used in a round-robin fashion, so your firewall must allow connection to each one.

Finally, If you use a web proxy server, enter the proxy information within the 'Proxy:' field, in the format: http://User:password@Host:port/

SMS Notification Module

This Notification Method is used for sending SMS messages via directly attached GSM modems.

For information installing and configuring the SMS module, see Section SMS Module

VictorOps

VictorOps integration is available out of the box with Opsview Monitor. Like PagerDuty, VictorOps is a message platform where Users can send various messages to VictorOps, and create rules within the VictorOps application to dispatch alerts via various methods such as SMS, Email, etc to individuals or groups of Users. Users need to setup an account on VictorOps and follow the steps below to activate it on the Opsview system.

  • Activate the method by setting it to 'Active', like the screen above.
  • Go to http://www.victorops.com and sign up, if you do not already have an account.
  • Once logged in, navigate to 'Integrations' and create a new REST Endpoint.
  • Next, navigate to 'Settings > Schedules' and add a new Team, in order to get a Routing key for your Users:
  • Once the team is added scroll down until you see the team name and its schedule and click on 'Setup Routing':
  • Once you have clicked on 'Setup Routing', scroll down to the bottom of the page where you can view and add routing keys:
  • Now that we have the API key (as seen above) and the routing key (as seen above), we can now configure Opsview Monitor. First, add the API Key (from the step above, the big ugly bit after '/alert/') to the Notification Method , as shown below:
  • Next, simply add the routing key to the Users who you wish to receive alerts. To do this, simply edit the User via 'Settings > Users and Roles >' and open the User edit modal window. Within the window, click on the 'Notifications' tab and you will be able to see the 'VictorOps Routing Key:' field as below. If you do not see this field, it is likely you have not set the Notification Method to 'Active' (see the first bullet).

And that's all the configuration that is required. In order to send notifications into VictorOps, a Notification Profile that uses the 'VictorOps' Notification Method must be configured. This is covered in Section '4.11.3. Notification Profiles'.

Note: VictorOps only works with Nagios notifications; BSM notifications are not currently supported.
Once the Notification Profile is configured, Notifications should start arriving in VictorOps, as below:

If you have configured your team schedules correctly, you should start receiving emails in your inbox:

Please note: This Guide was written August 2015. The VictorOps.com User interface may have changed since; therefore, please let us know if any screens are out of date.

PagerDuty

PagerDuty is now available out of the box with Opsview Monitor. PagerDuty is a message platform where Users can send various messages to PagerDuty, and create rules within the PagerDuty application to dispatch notifications via various methods such as SMS, Email, etc to individuals or groups of Users. To use PagerDuty with Opsview Monitor, Users need to setup an account on PagerDuty and follow the steps below to activate it on the Opsview system.

  • Activate the method by setting it to 'Active', like the screen above.
  • Go to http://www.pagerduty.com and sign up, if you do not already have an account.
  • Either as part of the sign up, or once already logged in, add a new 'Opsview' service:

Sign up ' 'Connect your first service: Opsview'

Logged in ' 'Configuration > Services'

  • Once the service is added, the 'API Key' will be visible as per the above and below screens:
  • Next, you need to add the above API key to Opsview Monitor. This key is added on a per User basis. To do this, simply edit the User via 'Settings > Users and Roles >' and open the User edit modal window. Within the window, click on the 'Notifications' tab and you will be able to see the 'Pagerduty Api Key' as shown below:

And that's all the configuration that is required. In order to send notifications into PagerDuty, a Notification Profile that uses the 'PagerDuty' Notification Method must be configured. This is covered in Section '4.11.3. Notification Profiles'.

Once the Notification Profile is configured, notifications should start arriving in PagerDuty, as below:

If you have configured your email address correctly, you should start receiving emails in your inbox:

Please note: This Guide was written August 2015. The PagerDuty.com User interface may have changed since, therefore please let us know if any screens are out of date.

Twilio

Twilio integration is available out of the box with Opsview Monitor. Twilio is a messaging platform that allows Users to send all alerts from Opsview into Twilio, and create rules that can notify specified Users using specified methods - including voice calls! To configure Twilio integration, Users need to setup an account on Twilio and follow the steps below to activate it on the Opsview system.

  • Activate the method by setting it to 'Active', as per the screen above.
  • Go to http://www.twilio.com and sign up, if you do not already have an account.
  • As part of the sign up, you will need to 'Get your Twilio number'. This is the phone number that voice calls and SMS are sent FROM, i.e. the caller.

Like the screen above, we can see the three items of required information; the Account SID, The Auth token and the Twilio number.

  • Now that we have the three parts of information; the Account SID, the Auth Token and the Twilio Number, we can configure the Notification Method:
  • Now that the Notification Method is configured correctly, we need to provide the telephone number to which notifications will be sent. The phone numbers are added on a per User basis, and must be verified from Twilio first (i.e. they are confirmed 'real' phone numbers). To add the phone numbers to Users, simply edit the User via 'Settings > Users and Roles >' and open the User edit modal window. Within the window, click on the 'Notifications' tab and you will be able to see the 'Pager' field as shown below:

And that's all the configuration that is required. In order to send notifications into Twilio, a Notification Profile that uses the 'Twilio' Notification Method must be configured. This is covered in Section Notification Profiles

Once the Notification Profile is configured, notifications should start arriving in Twilio, as shown below:

'SMS and MMS' logs

Individual log, showing SMS message to the given phone number

If you have configured your phone number correctly, you should start receiving text messages:

Please note: This Guide was written August 2015. The Twilio.com User interface may have changed since, therefore please let us know if any screens are out of date.

Slack

Slack integration is available out of the box with Opsview Monitor. Slack is a modern day pseudo 'chat room', that allows teams of Users to collaborate over the internet. The value-add over a chatroom is that Slack allows 'inbound integrations' from various tools such as Opsview Monitor. This means that an 'Operations team' can have a chat room, where alerts from Opsview are sent - they can discuss and view Notifications in the same forum - as opposed to a standalone email. To configure this integration, Users need to setup an account on Slack and follow the steps below to activate it on the Opsview system.

  • Activate the method by setting it to 'Active', as per the screen above.
  • Go to http://www.slack.com and sign up, if you do not already have an account. it will take you through the basic setup process of creating teams and rooms.
  • Once the setup is finished, log in and click on Integrations on the left hand side. Then click on 'Make your own' which will take you down to the bottom of the screen.
  • After clicking 'Or, make your own!' click on the 'View button next to 'Incoming WebHooks':
  • Within the 'Incoming WebHooks' window, select the channel you wish to send Opsview Monitor Notifications into. In our example, we are going to send Notifications into the #support slack channel:
  • Once you have selected a channel, simply click on the 'Add Incoming WebHooks Integration' and your new WebHook is created:
  • Now that we have our WebHook, we can configure the Slack Notification Method :

The 'WebHook URL' is visible in the screen above, and the 'Channel' was set during the creation of the WebHook; i.e. #support.

And that's all the configuration that is required. In order to send notifications into Slack, a Notification Profile that uses the 'Slack' Notification Method must be configured. This is covered in Section 'Notification Profiles'.

Once the Notification Profile is configured, Notifications should start arriving in the #support Slack room, as below:

Atlassian Hipchat

Atlassian Hipchat integration is available out of the box with Opsview Monitor. Like Slack, Hipchat is a modern day pseudo 'chat room', that allows teams of Users to collaborate over the internet. The value-add over a chatroom is that Hipchat allows 'inbound integrations' from various tools such as Opsview Monitor. This means that an 'Operations team' can have a chat room, where alerts from Opsview are sent - they can discuss and view notifications in the same forum - as opposed to a standalone email. To configure this integration, Users need to setup an account on HipChat and follow the steps below to activate it on the Opsview system.

  • Activate the method by setting it to 'Active', as per the screenshot above.
  • Go to http://www.hipchat.com and sign up, if you do not already have an account.
  • Once logged in, create a Hipchat room via the 'Web App':
  • Give the room a name as below:
  • Once the room has been created, click on the 'Hipchat logo' in the top left ' this will load the Hipchat configuration section. Click on 'Rooms' and you will be presented with a list of all the Hipchat rooms:
  • Click on the newly created 'opsview-alerts' room, and make a note of the value in the 'API ID' field:
  • Click on the 'Tokens' menu item on the left hand side:

This will load a window with a box as below:

  • Enter a label, i.e. 'Opsview Monitor', set the scopes to 'Send Notification' and click create. This will create a new token as below:
  • Now that we have the room notification token and the room's 'API ID', we can configure the Hipchat Notification Method :

Note: The 'Authentication Token' is where you should enter the 'Room Notification Token'.

And that's all the configuration that is required. In order to send notifications into Hipchat, a Notification Profile that uses the 'Hipchat' Notification Method must be configured. This is covered in Section '4.11.3. Notification Profiles'.

Once the Notification Profile is configured, notifications should start arriving in the opsview-alerts Hipchat room, as below:

Adding / Writing Your Own Notification Method

You may want to use an existing Nagios Core notification script in Opsview Monitor. To add this:

  • Check the Notification script for any dependencies. This may require additional software to be installed on the Opsview Monitor master (and slaves).
  • Test the Notification script. Opsview Monitor provides a way of simulating the environment variables that Nagios Core will set at notification time. If you intend to run this notification on slaves, you should test on those servers too:

su - nagios /usr/local/nagios/utils/test_notifications Hostproblem /path/to/notificationscript [other parameters]

  • Put your script into /usr/local/nagios/libexec/notifications on the Opsview master server, as the nagios user.
  • If appropriate, run send2slaves -p to distribute to all slaves.
  • Create a new Notification Method (Settings > Notification Methods >), and set the appropriate configuration (see below):
  • Each User will then have to select this Notification Method to one of their Notification Profiles.
  • Reload Opsview to make the changes live.
  • Cause a failure to test that the notification is sent.

Writing Your Own Notification Script **
Your notification script should be written to expect any input regarding the notification alert from Nagios Core via environment variables - see the Macro Availability Chart at http://nagios.sourceforge.net/docs/3_0/macros.html.

The only environment variables available for Notification Methods are:

  • NAGIOS_CONTACTALIAS
  • NAGIOS_CONTACTEMAIL
  • NAGIOS_CONTACTGROUPLIST
  • NAGIOS_CONTACTNAME
  • NAGIOS_CONTACTPAGER
  • NAGIOS_HOSTACKAUTHOR
  • NAGIOS_HOSTACKCOMMENT
  • NAGIOS_HOSTADDRESS
  • NAGIOS_HOSTALIAS
  • NAGIOS_HOSTATTEMPT
  • NAGIOS_HOSTDOWNTIME
  • NAGIOS_HOSTDURATION
  • NAGIOS_HOSTGROUPALIAS
  • NAGIOS_HOSTGROUPNAME
  • NAGIOS_HOSTNAME
  • NAGIOS_HOSTNOTIFICATIONNUMBER
  • NAGIOS_HOSTOUTPUT
  • NAGIOS_HOSTPROBLEMID
  • NAGIOS_HOSTSTATE
  • NAGIOS_HOSTSTATEID
  • NAGIOS_HOSTSTATETYPE
  • NAGIOS_LASTHOSTCHECK
  • NAGIOS_LASTHOSTDOWN
  • NAGIOS_LASTHOSTPROBLEMID
  • NAGIOS_LASTHOSTSTATE
  • NAGIOS_LASTHOSTSTATECHANGE
  • NAGIOS_LASTHOSTUNREACHABLE
  • NAGIOS_LASTHOSTUP
  • NAGIOS_LASTSERVICECHECK
  • NAGIOS_LASTSERVICECRITICAL
  • NAGIOS_LASTSERVICEOK
  • NAGIOS_LASTSERVICEPROBLEMID
  • NAGIOS_LASTSERVICESTATE
  • NAGIOS_LASTSERVICESTATECHANGE
  • NAGIOS_LASTSERVICEWARNING
  • NAGIOS_LASTSTATECHANGE
  • NAGIOS_LONGDATETIME
  • NAGIOS_LONGHOSTOUTPUT
  • NAGIOS_LONGSERVICEOUTPUT
  • NAGIOS_NOTIFICATIONAUTHOR
  • NAGIOS_NOTIFICATIONCOMMENT
  • NAGIOS_NOTIFICATIONNUMBER
  • NAGIOS_NOTIFICATIONTYPE
  • NAGIOS_SERVICEACKAUTHOR
  • NAGIOS_SERVICEACKCOMMENT
  • NAGIOS_SERVICEATTEMPT
  • NAGIOS_SERVICEDESC
  • NAGIOS_SERVICEDOWNTIME
  • NAGIOS_SERVICEDURATION
  • NAGIOS_SERVICENOTES
  • NAGIOS_SERVICENOTIFICATIONNUMBER
  • NAGIOS_SERVICEOUTPUT
  • NAGIOS_SERVICEPROBLEMID
  • NAGIOS_SERVICESTATE
  • NAGIOS_SERVICESTATEID
  • NAGIOS_SERVICESTATETYPE
  • NAGIOS_SHORTDATETIME
  • NAGIOS_TIMET
  • OPSVIEW_KEYWORDS

For example, in perl, reference theHOSTNAME variables using:

print "$ENV{NAGIOS_HOSTNAME}\n";

In bash, use:

echo $NAGIOS_HOSTNAME

A useful trick in bash to see all available environment variables is to
add at the top of a Notification Method :

{ date; echo "Called with $@"; env; echo; } >> /tmp/env.txt

To distinguish between a service alert and a Host alert, do something like (in perl):

if ( $ENV{NAGIOS_SERVICEDESC} ) {
# This is a service alert
} else {
# This is a Host alert
}

Global settings should be stored either in the script or via command arguments.

OUTPUT

The environment Variables SERVICEOUTPUT, HOSTOUTPUT, LONGSERVICEOUTPUT, LONGHOSTOUTPUT hold the output from a plugin. The first two hold only the first line of the output, while the remainder is in the LONG variables.
Note that the left and right angle brackets (<>) are stripped from the output.

CONTACTGROUPLIST

This environment variable holds all the contact groups that this Host/service belongs to, in a comma-separated list. Contact groups will be of the form:

  • HostgroupX_servicegroupY
  • kZ_NAME

You can use this to get the list of keyword names that the Host/Service is related to by:
@keywords = grep { s/k\d+_// } (split ",", $ENV{NAGIOS_CONTACTGROUPLIST})

Testing Notification Scripts

As mentioned above, notification scripts can be tested on both the master and slaves by using:

su - nagios /usr/local/nagios/utils/test_notifications Hostproblem /path/to/notificationscript [other parameters]

Some scripts may required extra configuration to test them. For example, to test the VictorOps notification method, ensure the access key is set in the UI and then use the following:

export NAGIOS__CONTACTVICTOROPS_ROUTING_KEY="<routing_key>"
/usr/local/nagios/utils/test_notifications hostproblem /usr/local/nagios/libexec/notifications/notify_by_victorops
where the routing key is set as per your VictorOps routing configuration.

This particular notification method logs to /var/log/opsview/opsviewd.log, but you need to check within each script where logging is sent (in many cases script output is captured in /usr/local/nagios/var/nagios.log).

Notification Methods

Overview of Notification Methods found in Opsview Monitor