Opsview Knowledge Center

NetAudit

An overview of NetAudit in Opsview Monitor.

NetAudit

NetAudit is a section of the Network Analyzer product, which allows you to configure automated backups of network device's configuration files (config files) on a Host by Host basis.

The Netaudit Module provides network auditing capabilities, integrating with the RANCID network infrastructure management tool. As per the RANCID official website:

"RANCID monitors a router's (or more generally a device's) configuration, including software and hardware (cards, serial numbers, etc) and uses CVS (Concurrent Version System) or Subversion to maintain history of changes. "

For installation instructions, please see section Network Analyzer

Note: This is an additional product that does not come with a standard subscription. For further information, please contact your Account Manager and ask about the 'Network Analyzer' product.

Overview

NetAudit is based on the RANCID project, and allows you to automate the backing up of network device configuration files to a centralized location; Opsview Monitor, on a regular basis.

This provides you a central repository for your network device configuration files, meaning if you need quick access to them, e.g. in the event that the network device has failed and you are replacing it/a vendor engineer is on-site replacing it, then you can quickly access and send them the configuration file and have the device back up and running within minutes. If you do not have a backup, then the configuration file will have to be written by hand, meaning greater periods of outage and impact to the customer.

NetAudit works on a host by host basis, and allows you to specify the vendor type (i.e. Cisco) and authentication credentials needed in order to log in and view the network devices configuration. With these details, Opsview Monitor will then periodically log in and make a copy of the configuration and store it centrally. This allows you to then compare the config files, i.e. what has changed, has someone added a setting they shouldn't have, etc.

Configuring a Host

In order for Opsview Monitor to back-up the configuration file of a Host, you must first enable the 'NetAudit' functionality on the Host(s) and enter the relevant credentials.

In the example shown below, we are backing up the configuration file of a Cisco router:

First, you must ensure 'Use NetAudit' is checked. If this field is not checked, the other fields will be hidden.

Once enabling NetAudit, you must select the router vendor (i.e. Cisco, HP, etc) and also the connection method (Telnet/SSH). After selecting the connection method, you will need to enter the valid authentication details - i.e. username and password(s).

Finally, enable 'Use autoenable' if your network device has the option configured.

Click on 'Test connection' to view the output of the log in attempt. If you encounter failures here, please re-confirm the authentication details before re-testing.

Note, if you are using autoenable you will need to enter the enable password immediately after the password in 'Password:'. For example, if your telnet password is 'letmein' and your enable password is 'enpass', then in the Password field you would enter:

letmein enpass

Finally, click 'Submit' and your changes are saved. Afterwards, submit a reload and your network configuration will be backed up and displayed within 'Monitoring > NetAudit'.

Note, if you encounter any problems please review Troubleshooting

Analysis

To view the configuration of a NetAudit-enabled Host, simply click on 'NetAudit' within the Monitoring tab of the overlay menu, as shown below:

Once loaded, you will be presented with a screen as shown below:

You should click on 'rancid', which will display all of the Hosts who have NetAudit enabled (and are thus having their configuration files backed up):

In the image above, we can see there is just one host that is NetAudit enabled and having its configuration file backed up. You can click on the 'Cisco2611' part (the hostname) to view the configuration file, as shown below:

This will display the latest 'whole' configuration file for the host. To view historical versions, use the arrows as listed below, and to compare with the previous configuration file (to see what has changed), click on 'Compare' as below:

Troubleshooting

Log Files

If you are encountering any issues with NetAudit, please view the output of the log files located at:

/var/opt/opsview/activemq/log - log files from file2activemq
/var/opt/opsview/activemq/log/opsview-activemq-scripts - contains log entries from consumers
/opt/opsview/activemq/data - log files from activemq daemon
These log files can provide valuable information into what the problem may be.

ActiveMQ

NetAudit requires a working and configured copy of ActiveMQ. If you have not installed and configured ActiveMQ, follow the guide located at "Opsview Messaging System".

You should have ActiveMQ and ActiveMQ Consumer processes running. To ensure they are up and running, run the commands:

/etc/init.d/opsview-activemq-consumers restart /etc/init.d/opsview-activemq restart

I've added a new host, but there isn't an SVN entry created

When a reload occurs, Opsview Monitor will generate a list of all the Hosts that have RANCID enabled. However, if the websvn repository doesn't show the Host, then maybe the message has not been passed through. Check the timestamp of the following directory:
ls -l /var/opt/opsview/activemq/spool/queue/rancid.master.events/
If this has not been updated at the time of the reload, then this could be a permissions issue. See the next troubleshooting entry, below.

Messages don't appear to be transferred

Check the various logs for activemq: /var/opt/opsview/activemq/log/opsview-activemq-scripts.log
Permissions could be an issue.
Check that the nagios user is a member of the opsview group. If this has changed, you will need to restart most daemons to get them to pick up the new permissions.

The flow of data is:

  • rancid collection is invoked by nagios user's 4 hourly cron job
  • this collection puts checksum information into /var/opt/opsview/rancid/checksums
  • the timestamps on the checksums tell you that routers have been discovered
  • a message is placed into the activemq's spool are /var/opt/opsview/activemq/spool/queue/rancid.master.events
  • file2activemq picks up this file and places it into ActiveMQ
  • ActiveMQ will route the message to the rancid master which then processes it, using the consume_rancid_events script
  • the script will update the files on the rancid master in /var/opt/opsview/rancid/svn and run an svn commit to commit the changes to svn

It has been seen that ActiveMQ could have a problem with registering the consumer at the rancid collector. Run the check_rancid_queues plugin to check that ActiveMQ is sending messages correctly to its destinations. You may need to restart ActiveMQ on the master and the slave.

Where's the rancid configuration files?

/usr/local/nagios/etc/plugins/rancid

Have router configuration files been updated?

Look in /var/opt/opsview/rancid/checksums. This gives the latest checksums for the router configurations.
To reduce traffic, only configurations that have changed will be sent to the Rancid Master. To force sending data back to Rancid master, remove the checksum files. Then the file will be sent to the Rancid Master, but if there is no difference in svn, then there will not be a new check-in.

Has the router configuration reached SVN?

Look in /var/opt/opsview/rancid/svn. If the file here contains information, then this is what should be in subversion. You can run an svn status to check compared with the subversion repository.

SVN shows the file, but websvn shows a blank file

This could be due to enscript. In /opt/opsview/repository/include/config.php, uncomment this line:
$config->useEnscript(); If websvn now shows the router configuration, then there is probably an issue with enscript.

Where are the latest files?

On the RANCID master, in /var/opt/opsview/rancid/svn will be all the latest versions of the RANCID router output files.

Testing configuration

To test the RANCID configuration for a specific Host, use the following on the monitoring server:
su - nagios cd /opt/opsview/rancid/bin export CLOGINRC=/usr/local/nagios/etc/plugins/rancid/cloginrc ./clogin -t 20 <hostname>

  • clogin may need to be changed to a device specific login script. See /opt/opsview/rancid/bin/check_rancid_connection for the device type to script table lookup

This should give you a terminal session. You may need to type exit to come out.
If this works but it doesn't from the Opsview RANCID tab, it could be a tty setting.

Troubleshooting RANCID tab test connection

This simulates running the code to test the RANCID connection with credentials.
Create a temporary file with this data:

  • add password 192.168.13.2 {terminal} {password} add method 192.168.13.2 telnet Change 192.168.13.2 with the hostname. Change terminal to the password, with a 2nd value for Cisco devices. Change telnet to ssh if applicable.
  • ssh {slave} 'cat /tmp/tempfile | /opt/opsview/rancid/bin/check_rancid_connection -t {vendor} {hostname}'

There maybe issues with tty, as this ssh does not have a tty assigned (see the e30login script for setting tty settings within the expect script).

Testing Collection

On the appropriate Opsview server:
su - nagios export CLOGINRC=/usr/local/nagios/etc/plugins/rancid/cloginrc . /etc/opt/opsview/rancid/rancid.conf mkdir /tmp/directory cd /tmp/directory rancid -d -l {hostname}

  • Make sure the cloginrc file has the authentication information
  • rancid may need to be switched to a different name (eg, arancid) depending on the type of device - see /opt/opsview/rancid/bin/check_rancid_connection for the device type to script table lookup

You should get a file in this temporary directory which is the router configuration as it will be pushed into SVN.

Walking through a collection

Running rancid -d -l {hostname} will show the command being run. You can run clogin {hostname} and do the commands listed in turn to see what output comes up. This may help if you are having specific issues in the collection of data.

WebSVN

If you get 'repos 1' listed as a repository, check /etc/websvn/svn_deb_conf.inc to remove this. Not sure why this occurs.

check_rancid_status says: 'Some routers not updated

This plugin checks for routers which have not been updated within the last 10 hours. You may get errors due to:

  • credentials changing on the device
  • the device being unavailable

This has also been seen for Hosts that are monitored by a slave system which is deactivated. This will be fixed in a future version of Opsview RANCID.

NetAudit

An overview of NetAudit in Opsview Monitor.