Migrating Opsview 6 to New Hardware

This describes how to migrate an existing Opsview 6 system to a new Opsview 6

Overview

In this section, we offer step-by-step instructions providing you with specific guidance to successfully migrate Opsview Monitor to a different hardware platform.

❗️

If you have a distributed environment then you should disable the collector (slave) devices on the old Opsview Monitor installation to avoid any contention between the Orchestrator (master) servers.

Also, If you are migrating to a new architecture, you should read through these steps in this document as it will guide you through how you should export your data.

The data can be migrated to a system running the same or a later version of Opsview Monitor, but you cannot migrate to an older version.

Please note, there will be an outage to the Opsview Monitor service during the migration.

Assumptions:

oldMaster = Your current working Opsview 6 install (source)
newMaster = The location you want to migrate Opsview 6 to (destination)

That your current install (oldMaster) has all services installed on the same server, ie Database, Reporting, Netflow etc.

If any collectors are defined on the oldMaster then these will be migrated to work with the newMaster.

Pre-requisites:

Prerequisites

Installation

All commands to be run as root unless otherwise stated.
Run the following command to update your OS packages, setup Opsview Monitor repositories on your server and install the opsview-deploy package [newMaster]:

📘

Specific version installation

To install a specific version, please use /6.4 in the curl command for example and not /6

  • if you do only specific /6 then the latest available version of Opsview 6 for your Operating System will be installed
curl -sLo- https://deploy.opsview.com/6.4 | sudo bash -s -- -A boot

Copy files from [oldMaster] to [newMaster]. In this example scp will be used to transfer files.

scp /opt/opsview/deploy/etc/user*.yml <newMaster>:/opt/opsview/deploy/etc/
scp /opt/opsview/var/machine.ref <newMaster>:/opt/opsview/var/machine.ref

Deployment

To deploy Opsview on the new infrastructure [newMaster]:

❗️

WARNING

You must ensure that any IP/hostnames addresses referenced in the files you are moving and re-using have been updated before the next steps or they will overwrite and break your current Opsview system

  • For example, references to opsview_database_backend_nodes within /opt/opsview/deploy/etc/user* files that are not 127.0.0.1. Other examples are variables such as your software key (if mentioned) and certificates being up-to-date / correct.
cd /opt/opsview/deploy/

# configure the base hosts
./bin/opsview-deploy lib/playbooks/setup-hosts.yml

# install and configure the core infrastructure (database, datastore, messagequeue, etc)
./bin/opsview-deploy lib/playbooks/setup-infrastructure.yml

# install core files for the orchestrator
./bin/opsview-deploy lib/playbooks/orchestrator-install.yml

Copy files from [oldMaster] to [newMaster]

scp /opt/opsview/coreutils/etc/opsview.conf <newMaster>:/opt/opsview/coreutils/etc/opsview.conf
scp /opt/opsview/webapp/opsview_web_local.yml <newMaster>:/opt/opsview/webapp/opsview_web_local.yml

Restart all services [newMaster]

/opt/opsview/watchdog/bin/opsview-monit restart all

Install and configure Opsview [newMaster]

./bin/opsview-deploy lib/playbooks/setup-opsview.yml

At this point you should now have a working Opsview 6 server.

Log into the [newMaster] Opsview UI and carry out a successful Reload/Apply Changes

Migrating Config and Data

This section explains how to migrate config and data, such as databases, reporting, netflow etc.

Stop all services on [oldMaster] including services on any collectors

cd /opt/opsview/deploy
source bin/rc.ansible
ansible opsview_all -m opsview_watchdog -a "name=all state=stopped"
 
# confirm with
ansible opsview_all -m shell -a "/opt/opsview/watchdog/bin/opsview-monit summary -B"

Stop all services on [newMaster]

/opt/opsview/watchdog/bin/opsview-monit stop all

Datastore (optional)

The datastore information is not essential for a successful migration.

To be able to access the datastore from the newMaster, create migration.cfg on [oldMaster] by running these commands.

cat <<EOF | install -o root -g opsview -m 640 /dev/fd/0 /opt/opsview/loadbalancer/etc/migration.cfg
listen datastore-migration
bind 0.0.0.0:15989
mode tcp
timeout client 3h
timeout server 3h
option clitcpka
server          datastore-migration-balance 127.0.0.1:15984 check inter 5s
EOF

Start loadbalancer and datastore on [oldMaster] and [newMaster]

/opt/opsview/watchdog/bin/opsview-monit start opsview-loadbalancer
/opt/opsview/watchdog/bin/opsview-monit start opsview-datastore

Delete datastore databases on [newMaster]

DS_PASS=`grep opsview_datastore_password /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'`

for DB in opsview-master opsview-collector opsview-logs; do
curl -u opsview:$DS_PASS -X DELETE http://127.0.0.1:15984/$DB;
done

Replicate the oldMaster datastore databases onto the [newMaster]
Note: populate OLDMASTER with the oldMaster's IP.

DS_PASS=`grep opsview_datastore_password /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'`
OLDMASTER='<add IP of oldMaster>'
 
for DB in opsview-master opsview-collector opsview-logs; do
cat <<EOF | curl -u opsview:$DS_PASS -d @- -H "Content-Type: application/json" -X POST http://127.0.0.1:15984/_replicate;
{
"_id": "migrate-collector",
"source": {
"url": "http://opsview:[email protected]$OLDMASTER:15989/$DB"
},
"target": {
"url": "http://opsview:[email protected]:15984/$DB"
},
"create_target": true,
"continuous": false
}
EOF
done

If an error is seen for 'opsview-logs', ignore this as it means you are not using this.

The datastores from oldMaster have now been replicated onto the newMaster.

Opsview MySQL Databases

To create a full database export, run the below command as root making sure to include any extra databases you may have (for example, include jasperserver if it exists). [oldMaster]

mysqldump -u root -p`grep database_root /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'` --add-drop-database --opt --databases opsview runtime odw dashboard notifications | gzip -c > databases.sql.gz

Copy the exported db file over to the [newMaster] and import it.

gunzip -c databases.sql.gz | mysql -u root -p`grep database_root /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'`

Migrating Configuration Files

You should migrate any configuration files that you may have customized to your new server, such as those listed here.

  • /opt/opsview/coreutils/etc/sw.key
  • /opt/opsview/coreutils/etc/sw/*
  • /opt/opsview/coreutils/etc/map.local
  • /opt/opsview/webapp/docroot/static/stylesheets/custom.css
  • Apache configuration files

Migrate Timeseries Data (RRD)

Export your graphing data by running the following command on your [oldMaster]

/opt/opsview/coreutils/installer/rrd_converter -y export

This will produce file /tmp/rrd_converter.tar.gz. Copy this over to your [newMaster] into the same location.

On the [newMaster] run the following commands to import the graphing data

cd /tmp
/opt/opsview/watchdog/bin/opsview-monit summary | grep timeseries | awk '{print $1}' | while read TSSVC; do /opt/opsview/watchdog/bin/opsview-monit stop $TSSVC; done
rm -rf /opt/opsview/timeseriesrrd/var/data/*
sudo -u opsview -i -- bash -c 'export PATH=$PATH:/opt/opsview/local/bin ; /opt/opsview/timeseriesrrd/installer/migrate-uoms.pl /opt/opsview/timeseriesrrd/var/data/'
/opt/opsview/coreutils/installer/rrd_converter -y import /tmp/rrd_converter.tar.gz
chown -R opsview:opsview /opt/opsview/timeseriesrrd/var/data
sudo -u opsview -i -- bash -c 'export PATH=$PATH:/opt/opsview/local/bin ; /opt/opsview/timeseriesrrd/installer/migrate-uoms.pl /opt/opsview/timeseriesrrd/var/data/'

Migrate Timeseries Data (InFluxDB)

To migrate the InfluxDB graphing data the new Opsview install must already be running the same InfluxDB version as the source.

Backup the InfluxDB database [oldMaster]

influxd backup -portable /tmp/influxdb_backup
cd /tmp
tar -zcvf influxdb_back.tar.gz influxdb_backup/

Backup the Opsview Timeseries InfluxDB Metadata [oldMaster]

cd /opt/opsview/timeseriesinfluxdb/var/data
tar -zcvf timeseriesinfluxdb_back.tar.gz +metadata*

Transfer the tar.gz files over to the new Opsview install (/tmp). [newMaster]
On the new install, drop the InfluxDB Opsview database and install restore the migrated data

curl -i -XPOST http://127.0.0.1:8086/query --data-urlencode "q=DROP DATABASE opsview"
tar -zxvf /tmp/influxdb_back.tar.gz
influxd restore -portable /tmp/influxdb_back/

Restore the Opsview Timeseries InfluxDB metadata [newMaster]

tar -zxvf /tmp/timeseriesinfluxdb_back.tar.gz /opt/opsview/timeseriesinfluxdb/var/data/

Final Migration Steps

On the [newMaster] start all services

/opt/opsview/watchdog/bin/opsview-monit start all

Deactivate all the collectors via the UI
Configuration --> Monitoring Collectors --> Clusters Tab --> Select Collector, uncheck 'Activated'

Master Host IP/Hostname

The main monitoring host will have the oldMaster's name. Correct the IP/hostname for the master host in the UI and, if necessary, the Host Title too.

If the masters Host Title was changed then to not lose any historic graphing data for the master host, carry out the following steps:-

Download script rrdmerge [newMaster]

wget https://bitbucket.org/flyingcircus/rrdmerge/raw/b05d69bfac6484c461047aee34fb5dbb72a071e0/rrdmerge -O /tmp/rrdmerge

Copy script rrd_merge_renamed_hosts onto your system [newMaster], say /tmp/rrd_merge_renamed_hosts

#! /bin/bash

#
# 1. location of rrdmerge
# 2. location of rrd datadir
# 3. name of original host name
# 4. name of new host name
#

if [ "$#" -ne 4 ]; then
        if [ "$#" -lt 4 ]; then
                echo "Not enough arguments"
        elif [ "$#" -gt 4 ]; then
                echo "Too many arguments"
        fi

        echo "Req: <rrdmerge-location> <rrd datadir-location> <Original hostname> <New hostname>"
        echo "Hint: rrd datadir can be located by running command 'grep data_dir /opt/opsview/timeseriesrrd/etc/*.yaml'"
        
exit 0
fi


rrdmerge=$1
rrd_datadir=$2
old_hostname=$3
new_hostname=$4


new_host_root=$rrd_datadir/$new_hostname
old_host_root=$rrd_datadir/$old_hostname

cd $new_host_root


for file in `find . -name value.rrd`
do
        rrd_path=`echo $file | awk -F 'value.rrd' '{print substr($1,3); }'`
        if [ -f $old_host_root/$rrd_path'value.rrd' ]; then
                # merge these files
                echo "File $old_host_root/$rrd_path'value.rrd' exists."
                # old - new - new-file
                $rrdmerge $old_host_root/$rrd_path'value.rrd' $new_host_root/$rrd_path'value.rrd' $new_host_root/$rrd_path'value.rrd.new'
                ret_code=$?
                if [ $ret_code -eq 0 ]; then
#                        mv $old_host_root/$rrd_path'value.rrd' $old_host_root/$rrd_path'value.rrd.old'
                        mv $new_host_root/$rrd_path'value.rrd' $new_host_root/$rrd_path'value.rrd.orig'
                        mv $new_host_root/$rrd_path'value.rrd.new' $new_host_root/$rrd_path'value.rrd'
                        chown opsview.opsview $new_host_root/$rrd_path'value.rrd'
                fi
        else
                # ignore these ones. Nothing to Merge
                echo "File $old_host_root/$rrd_path'value.rrd' does not exist - Ignoring!!"
        fi
done

Change ownership & permissions

chown opsview.opsview /tmp/rrd_merge_renamed_hosts /tmp/rrdmerge
chmod +x /tmp/rrd_merge_renamed_hosts /tmp/rrdmerge

Now run the following command [newMaster].
Substitute and with the host names that have been used.

/tmp/rrd_merge_renamed_hosts /tmp/rrdmerge /opt/opsview/timeseriesrrd/var/data/ <oldmaster> <newmaster>

Restart Timeseries Services [newMaster]

/opt/opsview/watchdog/bin/opsview-monit summary | grep timeseries | awk '{print $1}' | while read TSSVC; do /opt/opsview/watchdog/bin/opsview-monit start $TSSVC; done

Master Host Variables

Also in the edit screen for the master host, correct the following variable Override Node settings located in the Variables tab.
OPSVIEW_DATASTORE_SETTINGS
OPSVIEW_MESSAGEQUEUE_CREDENTIALS

Now carry out an Opsview 'Apply Changes', this should be successful.

Collectors

Copy all the collectors configuration from the oldMasters /opt/opsview/deploy/etc/opsview_deploy.yml file to the newMasters opsview_deploy.yml file.
Make sure the root users ssh public keys have been passed over to any collectors, otherwise the following command will fail.

Now run deploy to tell the newMaster about the collectors

cd /opt/opsview/deploy
./bin/opsview-deploy lib/playbooks/setup-everything.yml

Within the UI, re-activate the collectors
Configuration --> Monitoring Collectors --> Clusters Tab --> Select Collector, check 'Activated'

Now carry out an Opsview 'Apply Changes', this should be successful.

At this point you should have successfully migrated your oldMaster to your newMaster host and all systems should now be fully working.

Restart all services on the Orchestrator and Collectors [newMaster]

cd /opt/opsview/deploy
source bin/rc.ansible
ansible opsview_all -m opsview_watchdog -a "name=all state=restarted"

# confirm with
ansible opsview_all -m shell -a "/opt/opsview/watchdog/bin/opsview-monit summary -B"

Modules

NetAudit

On the [oldMaster] as the Opsview user

cd /opt/opsview/netaudit/var/repository/
tar -cvf /tmp/netaudit.tar.gz --gzip rancid/
scp /tmp/netaudit.tar.gz [email protected]:/tmp

On the [newMaster] as the Opsview user

cd /opt/opsview/netaudit/var/repository/
rm -fr rancid
tar -xvf /tmp/netaudit.tar.gz
cd /opt/opsview/netaudit/var
rm -fr svn
svn checkout file:///opt/opsview/netaudit/var/repository/rancid svn

Test by looking at the history of the NetAudit hosts.
Test when a change is made on the router.

Reporting Module

On [newMaster] as root user, stop Reporting Module:

/opt/opsview/watchdog/bin/opsview-monit stop opsview-reportingmodule

On [oldMaster], take a backup of your jasperserver database and transfer to [newMaster] server:

mysqldump -u root -p`grep opsview_database_root_password /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'` --add-drop-database --extended-insert --opt --databases jasperserver| gzip -c > /tmp/reporting.sql.gz
scp /tmp/reporting.sql.gz [email protected]:/tmp

On [newMaster], restore the database:

echo "drop database jasperserver" | mysql -u root -p`grep opsview_database_root_password /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'`
( echo "SET FOREIGN_KEY_CHECKS=0;"; zcat /tmp/reporting.sql.gz ) | mysql -u root -p`grep opsview_database_root_password /opt/opsview/deploy/etc/user_secrets.yml |awk '{print $2}'`

On [newMaster] as root user, run the upgrade and start the Reporting Module:

/opt/opsview/jasper/installer/postinstall_root
/opt/opsview/watchdog/bin/opsview-monit start opsview-reportingmodule

In the Reporting Module UI, you will need to reconfigure the ODW datasource connection, as it should now point to newdbserver.

Test with a few reports.

Network Analyzer

For the master, on [oldMaster] as root user, run:

cd /opt/opsview/flowcollector/var/data/
tar -cvf /tmp/netflow.tar.gz .
scp /tmp/netflow.tar.gz [email protected]:/tmp

On [newMaster] as root user, run:

cd /opt/opsview/flowcollector/var/data/
tar -xvf /tmp/netflow.tar.gz
chown -R opsview.opsview .

Network devices will need to be reconfigured to send their Flow data to the new master and/or collectors.

Service Desk Connector

Copy from the [oldMaster] the appropriate service desk connector yml file to the [newMaster]

scp /opt/opsview/servicedeskconnector/etc/config.d/*.yml [email protected]:/opt/opsview/servicedeskconnector/etc/config.d/

Restart the Service Desk Connector on the [newMaster]

/opt/opsview/watchdog/bin/opsview-monit restart opsview-servicedeskconnector

SNMP Trap MIBS

If you have any specific MIBs for translating incoming SNMP Traps, these need to exist on the new master.

On [oldMaster] as the opsview user copy over mibs files to the [newMaster]

# copy over any MIBs and subdirectories (excluding symlinks)
cd /opt/opsview/snmptraps/var/load/
find -maxdepth 1 -mindepth 1 -not -type l -print0 |  tar --null --files-from - -cvf /tmp/custom-mibs.tar.gz --gzip
scp /tmp/custom-mibs.tar.gz [email protected]:/tmp

On the [newMaster] unpack the mibs

cd /opt/opsview/snmptraps/var/load/
tar -xvf /tmp/custom-mibs.tar.gz
# now become the root user and run the following command
/opt/opsview/watchdog/bin/opsview-monit restart opsview-snmptrapscollector

Test by sending a trap to the master from a host that it is in its cluster and check that it arrives as a result for the host in the Navigator screen. You can also add the "SNMP Trap - Alert on any trap" service check to the host if it has not got any trap handlers. With the service check added to the host, you can use SNMP Tracing to capture and read any trap that is getting sent from that host.

SNMP Polling MIBS

If you have any specific MIBs for translating OIDs for check_snmp plugin executions, these need to exist in the /usr/share/snmp/mibs/ or /usr/share/mibs/ location for the orchestrator to use in the newMaster. All OIDs specified in opsview of the form of "::" need to get translated during an opsview reload into their number form using the standard MIBs in /usr/share/snmp/mibs and /usr/share/mibs to translate them. You should ensure that all your MIBs are transferred from the old folders to the newMaster.

On [oldMaster] as the root user:

# copy over any /usr/share MIBs and subdirectories (excluding symlinks)
cd /usr/share/snmp/mibs     #[DEBIAN,UBUNTU]
find -maxdepth 1 -mindepth 1 -not -type l -print0 |  tar --null --files-from - -cvf /tmp/share-snmp-mibs.tar.gz --gzip
scp /tmp/share-snmp-mibs.tar.gz [email protected]:/tmp
cd /usr/share/mibs     #[CENTOS,RHEL]
find -maxdepth 1 -mindepth 1 -not -type l -print0 |  tar --null --files-from - -cvf /tmp/share-mibs.tar.gz --gzip
scp /tmp/share-mibs.tar.gz [email protected]:/tmp

# copy over any custom MIBs and subdirectories (excluding symlinks)
cd /opt/opsview/snmptraps/var/load/
find -maxdepth 1 -mindepth 1 -not -type l -print0 |  tar --null --files-from - -cvf /tmp/custom-mibs.tar.gz --gzip
scp /tmp/custom-mibs.tar.gz [email protected]:/tmp

On [newmaster] as the root user:

# install mib package for Debian/Ubuntu
apt-get install snmp-mibs-downloader

# Note: At this point you should also install any other proprietary MIB packages necessary for translating MIBs used for SNMP Polling in your system
e.g. apt-get install {{your-MIB-Packages}}
e.g. yum install {{your-MIB-Packages}}

# unpack and copy the extra MIBs
cd /usr/share/snmp/mibs     #[DEBIAN,UBUNTU]
tar -xvf /tmp/share-snmp-mibs.tar.gz
mkdir opsview && cd opsview && tar -xvf /tmp/custom-mibs.tar.gz
cd /usr/share/mibs     #[CENTOS,RHEL]
tar -xvf /tmp/share-mibs.tar.gz
mkdir opsview && cd opsview && tar -xvf /tmp/custom-mibs.tar.gz