Overview

Opsview Web App provides the perl Catalyst web application server process, which includes the REST API functionality. Includes NGINX configuration to proxy HTTP requests to the application server.

This has a dependency on the opsview-web-app-ui package (front end application code), which in turn has a dependency on the opsview-web-app-ui-external package (third party front end libraries).

Altogether, this provides the Opsview web application.

This must be run on the Opsview Orchestrator server.

Dependencies

Requires opsview-core-utils

Installation

This component is deployed on the Master server when installating Opsview Monitor and must not be moved to another server.

Configuration

Opsview Web uses the file /opt/opsview/webapp/opsview_web_local.yml as its main configuration file, with defaults provided in opsview_web.yml (do not change this defaults file - it will be reset on every upgrade). The file opsview_web_local.yml is managed by Opsview Deploy - if there are any configuration settings you want to persist, you must add to Deploy's user_vars.yml.

Note: The format of the opsview_web_local.yml file is very significant - you must use spaces and not tab characters for indentation. Incorrect changes in this file can leave your system broken.

Note: Changes to these files require a restart of Opsview Web.

Web Authentication

Opsview uses authticket to authenticate to the web application.

On an install, a randomly generated secret will be used and will be encrypted.

On an upgrade, if you have got the old default shared secret (shared-secret-please-change), a new secret will be generated and encrypted. Otherwise, no changes will occur.

Account Lockout

You can configure the throttling parameters for the lockout policy. Amend the opsview_core_web_local_overrides parameter in /opt/opsview/deploy/etc/user_vars.yml:

opsview_core_web_local_overrides:
  Controller::Root:
    login_throttling:
      ip_limit: 50
      ip_window: 3600
      user_limit: 10
      user_window: 3600
      uid_only: 1

The parameters are:

  • ip_limit - number of failed login attempts based on IP before blocking occurs. Default: 50
  • ip_window - amount of seconds backwards in time to check for failed IP attempts. Default: 1 hour
  • user_limit - number of failed login attempts for a user before blocking occurs. Default: 10
  • user_window - amount of seconds backwards in time to check for failed user attempts. Default: 1 hour
  • uid_only - if 1, will use only the userid to identify the user, else will use a userid-IP combination. Default: 1 - this is the most secure option. Utilises user_limit and user_window parameters

Note on preventing denial of service vs. preventing brute force attack

You may decide to change uid_only to 0 if you want to ensure that the legitimate users never get locked out of the system during a brute-force attack on their account, while still achieving some level of protection from primitive attacks. Since the legitimate user is connecting from a different IP to the attacker they can still login; this is less secure because a brute-force attack can still happen if the attacker has enough unique IP addresses attempting to authenticate. Regardless of this parameter, the blocking after 50 failed attempts from a single IP still happens according to the ip_limit and ip_window.

  • Errors received may be "IP address temporarily blocked. Try again later or contact your administrator"

Recovery from brute-force attack or mass-lockout

If you need to reset the lockout controls in an emergency, you can clear all the stored attempted lockouts by running the following command within the opsview database:

delete from opsview.login_failures;

This will clear all the current login failures.

Authtkt Ignore IP

When generating the authtkt key, the browser IP address is considered as part of the algorithm. However, in some scenarios, you may not want this - for instance, if you have multiple proxies in front of Opsview. You can ignore the IP address (internally, it will set the IP to 0.0.0.0).

Add to the opsview_web_local.yml file:

Controller::Root:
  authtkt_ignoreip: 1

Note: if you change this, you will also need to update the Nginx configuration file so that the following is set:

auth_tkt_ignore_ip on;

Starman Server Processes

Opsview Web uses Starman via Catalyst to server dynamic pages. It is possible to increase the number of Starman server processes to improve web responses, at the cost of using more memory.

You can estimate the maximum amount of memory used by Starman with the following calculation:

max_servers x 280MB

The default value is 10 for max_servers.

To increase the value, add to the opsview_web_local.yml file:

ProductionEngine:
 max_servers: 20

The smallest setting you can use is:


ProductionEngine:
  max_servers: 3
  min_servers: 2
  max_spare_servers: 1
  min_spare_servers: 1

Login Timeout

You can modify the following to change the time that the interface automatically logs you out:

session:
 expires: 86400

Default Thresholds for Host Interfaces

You can add the following lines to change the default host interface thresholds:

host_interfaces:
  default_throughput_critical: 55%
  default_throughput_warning: ""
  default_errors_critical: 20
  default_discards_critical: 5

SNMPWalk timeouts

Configure the number of seconds before SNMPWalk's time out. The default is 1800 seconds (or 30 minutes).

snmpwalk_timeout: 1800

License Messages

If your license for Opsview Monitor is coming up for renewal, there will be a banner that is displayed in the top of all Opsview screens.

You can change the access required to see this message by changing the configuration file to:

Controller::Root:
  show_license_messages: ADMINACCESS

By default, show_license_messages is set to 1 which means all users will see it. If set to 0, this disables all license messages. Otherwise, it expects a string to mean which particular access level is required to display the banner - this can be a comma-separated list and thus any one of those accesses will have the banner displayed.

Note: If you disable banner messages, ensure the Service Check Opsview License Checks is running and that notifications work, so that you still get warnings about an upcoming expiry.

Collector Host Templates

When a collector is registered, an Opsview host is created with a list of host templates. This list is defined in the yml file as:

Controller::REST::Config::Collector:
  collector_registration_host_templates:
  - Opsview - Component - Agent
  - Opsview - Component - DataStore
  - Opsview - Component - Executor
  - Opsview - Component - Load Balancer
  - Opsview - Component - Machine Stats
  - Opsview - Component - MessageQueue
  - Opsview - Component - Results Forwarder
  - Opsview - Component - Results Sender
  - Opsview - Component - Scheduler
  - Opsview - Component - SNMP Traps
  - Opsview - Component - SNMP Traps Collector
  - Opsview - Component - Watchdog
  - Network - Base

If you are changing this list, ensure the names are correct, otherwise an error will be raised when the host template is not found.

Minimum Password Strength

The minimum password strength is configured in the yml file using:

# Valid values from 1 to 5
Controller::Settings:
  min_password_strength: 3
# List of words not allowed in passwords
  disallowed_passwords:
    - opsview
# Number of historical passwords for validation
  max_password_history: 4

The max_password_history values are:

SettingMeaning
0Turn off all historical password checking
1Check the current password is not the same as the new password
2Check the current password and one previous password are not the same as the new password
3Check the current password and the two previous passwords are not the same as the new password
4... etc ...

The min_password_strength values are:

LevelPassword Strength
1Very Weak
2Weak
3Medium
4Strong
5Very Strong

If the new password does not match or improve upon this level then it cannot be saved.

Thesse settings are maintained by Opsview Deploy for future upgrades

SSL Certificate

The SSL server certificate and key can be configured for the webserver. If they do not exist then a self-signed certificate will be automatically created.

By default the certificate and key are located at:

  • /opt/opsview/webapp/etc/ssl/server.crt
  • /opt/opsview/webapp/etc/ssl/server.key

The paths to the certificate and key can be overridden as the 'root' user in the /opt/opsview/deploy/etc/user_vars.yml file:

opsview_webserver_ssl_cert: /opt/opsview/webapp/etc/ssl/server.crt
opsview_webserver_ssl_key: /opt/opsview/webapp/etc/ssl/server.key

After changing these variables you must run the following command:

/opt/opsview/deploy/bin/opsview-deploy /opt/opsview/deploy/lib/playbooks/orchestrator-install.yml

This will make the necessary adjustments to opsview-webserver automatically. If the certificate does not exist then a self-signed certificate will be created.

SSL Certificate (.pem)

If your webserver certificate and key are actually combined into a single .pem file instead of separate .crt and .key files, then you must point the webserver to it's location by using the variables in the /opt/opsview/deploy/etc/user_vars.yml file:

opsview_webserver_ssl_cert: /opt/opsview/webapp/etc/ssl/server.pem
opsview_webserver_ssl_key: /opt/opsview/webapp/etc/ssl/server.pem

After changing these variables you must run the following command:

/opt/opsview/deploy/bin/opsview-deploy /opt/opsview/deploy/lib/playbooks/orchestrator-install.yml

Note: Your certificate provider may provide the certificate in .pem format, and the key in a separate file. The key will need to be added to the top of the .pem file before adding this to Opsview, if this is the case.

Handling Intermediate CA certificates in NGINX (SSL Certificates)

If you find that your homepage/web-page isn't loading up after upgrading from Apache you may find that you need to perform this configuration.

NGINX doesn't have an equivalent configuration to Apache's "SSLCertificateChainFile" for specifying seperate intermediate CA files. Instead, if your Opsview webserver's SSL certificate is signed by an intermediate CA, then you must include the intermediate CA certificate in the same file as the webserver's SSL certificate.

You can roll up your intermediate CA .crts and webserver .crt file in the following way:

cd /opt/opsview/webapp/etc/ssl/  #default location for server.crt. Your path may be different
cp -p server.crt server.crt.backup

# Replace filepaths below with paths to your certificates
cat /path/to/your/SERVER_SSL.crt  /path/to/your/INTERMEDIATE_CA_SSL.crt  [/path/to/your/INTERMEDIATE_CA_SSL_2.crt] ... > server.crt

# Restart the webserver to load the new certificates
/opt/opsview/watchdog/opsview-monit restart opsview-webserver

Switching to HTTP from HTTPS

By default Opsview Monitor uses HTTPS for all connections. If you want to use HTTP then follow these steps:

As the 'root' user, update the /opt/opsview/deploy/etc/user_vars.yml file and add or amend the following parameter:

opsview_webserver_use_ssl: False

Then run:

/opt/opsview/deploy/bin/opsview-deploy /opt/opsview/deploy/lib/playbooks/orchestrator-install.yml

This will make the necessary adjustments to opsview-webserver.

Be aware that Opsview Monitor now uses HSTS by default. If users have already connected to the web interface over HTTPS then they may have issues connecting using HTTP.

Login Session Limits

By default the maximum number of concurrent sessions is set to 10; this limit applies to both the UI and REST API sessions combined.

To change the default value, you can add the following lines to /opt/opsview/deploy/etc/user_vars.yml:

opsview_session_manager_config_overrides:
  max_sessions: 30

and then run:

/opt/opsview/deploy/bin/opsview-deploy /opt/opsview/deploy/lib/playbooks/session-manager-install.yml

This will make the necessary adjustments to the configuration.

Webserver Logging

By default Opsview Monitor logs all web access to /var/log/opsview/web_access.log and error messages are sent to /var/log/opsview/web_error.log.

Log Rotation

As part of the opsview-webserver package, Opsview will configure logrotate by placing the configuration into /etc/logrotate.d/opsview-webserver.

The default is to rotate logs every day for 7 days.

Note: As part of an upgrade, this configuration file will be overwritten.

Custom Log Files

The default log files can be overridden as the root user by adding logging configuration in a .conf file in the /opt/opsview/webserver/etc/opsview-site-customisations.d/ directory. For example:

File: /opt/opsview/webserver/etc/opsview-site-customisations.d/50-logging.conf

error_log /var/log/mylogdir/error.log notice;
access_log /var/log/mylogdir/access.log combined;

Ensure that the logging directory exists, then restart the web server by running:

/opt/opsview/watchdog/bin/opsview-monit restart opsview-webserver

Note that these log files will grow while the webserver is running. It is advisable to add a policy to rotate the log files on a regular basis - see /etc/logrotate.d/opsview-webserver for an example. Be aware that /etc/logrotate.d/opsview-webserver will be overwritten during an upgrade.

Using syslog

Opsview Monitor web accesses and errors can be sent to syslog, for example:

File: /opt/opsview/webserver/etc/opsview-site-customisations.d/50-logging.conf

error_log syslog:server=unix:/dev/log,facility=local6,tag=opsview_webserver notice;
access_log syslog:server=unix:/dev/log,facility=local6,tag=opsview_webserver combined;

Then restart the web server.

Disable Logging

Alternatively, to disable web logging in Opsview Monitor:

File: /opt/opsview/webserver/etc/opsview-site-customisations.d/50-logging.conf

error_log off;
access_log off;

Then restart the web server.

Further information

See also the NGINX Docs : Configuring Logging

Re-Homing

See Customization

Customisations

To extend the NGINX configuration, you have two possibilities:

  • For server wide changes, drop your configuration into /opt/opsview/webserver/etc/conf.d/
  • For the Opsview Monitor site changes, drop your configuration into /opt/opsview/webserver/etc/opsview-site-customisations.d/; Configuration files must have a file extension of ".conf". Files are loaded in alphabetical order. The directory /opt/opsview/webserver/etc/opsview-site-customisations.d retain all the customisation and will not be touched (overwritten) upon a re-run of opsview-deploy. Therefore not being overwritten on any future upgrades.

Note: You should not change /opt/opsview/webserver/etc/conf.d/opsview.conf as this is managed by Opsview Deploy and will be changed on upgrades.

Note: Make sure your NGINX configuration is valid, especially when upgrading Opsview Monitor, as an invalid configuration will mean that opsview-webserver will not be able to restart and may cause an upgrade to fail.

Example of this would be adding the NGINX Status Page, shown below:

NGINX Status Page

There is no possibility to enable the status page through opsview-deploy ansible playbook, so creating a .conf file as mentioned above is the way to do this.

You may enable it manually by running the following commands:

vim /opt/opsview/webserver/etc/opsview-site-customisations.d/status.conf 

location /nginx_status {
auth_jwt_enabled off;
stub_status on;
access_log off;
}

Then restart the opsview-webserver (which is nginx)

/opt/opsview/watchdog/bin/opsview-monit restart opsview-webserver

Other Configurable Items

The following settings can be modified within `/opt/opsview/deploy/etc/user_vars.yml for use with Deploy

#
# Authtkt's ignore IP feature. Needed for double proxying
#
opsview_webserver_auth_tkt_ignore_ip: False
#
# Disable configuration tasks for opsview-webserver.
#
# This is useful if you need to fine-tune the opsview-webserver configuration
# (e.g. SSO, alternative ports, SSL options, etc) with options not supported by
# opsview-deploy.
#
opsview_webserver_disable_config: False
#
# Stop and disable Apache2/HTTPD to prevent conflicts with opsview-webserver.
#
opsview_webserver_disable_apache: True
#
# Timeout for connections to upstream proxies.
#
opsview_webserver_proxy_connect_timeout: 30s
#
# Timeout for sending packets to upstream proxies.
#
opsview_webserver_proxy_send_timeout: 30s
#
# Timeout for reading packets from upstream proxies.
#
opsview_webserver_proxy_read_timeout: 86400s

When the changes have been made, run:

/opt/opsview/deploy/bin/opsview-deploy /opt/opsview/deploy/lib/playbooks/orchestrator-install.yml

This will make the necessary adjustments to opsview-webserver.

Troubleshooting

opsview-webserver says "Initializing" all the time

Opsview Watchdog will try to restart NGINX if there are any issues, but if there is an invalid configuration, watchdog will continually retry and mark the service as Initializing. Check the syslog output. For example:

Jul 15 13:26:14 ubuntu18-nginx opsview-monit[32647]: 'opsview-webserver' failed to start (exit status 1) -- '/opt/opsview/webserver/bin/start.sh': nginx: [emerg] invalid number of arguments in "add_header" directive in /opt/opsview/webserver/etc/conf.d/opsview.conf:20

This means there is an invalid configuration in opsview.conf at line 20. Resolve the issue and check watchdog again.

Management

Configuration

DPKGs

Watchdog service files are now managed by the package manager, removing the package would leave the watchdog service file behind with a .save extension. Purging the package will remove it. The package manager managed config files are as follows:

  • /opt/opsview/watchdog/etc/services/opsview-web.conf

RPMs

Watchdog service files are now managed by the package manager. When removing the package, if the files listed have been modified by the user before; they will be left behind after rpmsave extension has been added to them. When upgrading the package, if the files listed have been modifed by the user before; the modified file on the disk will remain and the new config file from the upgrading package will be written to the disk with the rpmnew extension. Users will have to:

  • Manually restore the file with the rpmsave extension.
  • Rename the rpmnew to the config file name if they want to use the new config for the package:
    /opt/opsview/watchdog/etc/services/opsview-web.conf

Service Administration

Start, stop, and restart Opsview Web using Watchdog:

opsview_watchdog opsview-web <stop|start|restart>

Logging

Logging is configured in /opt/opsview/webapp/etc/Log4perl.conf, which by default will forward to syslog. Note, this file is overwritten on an upgrade.

Double Proxying

You may want to have a second, and secured, proxy located in front of your Opsview Monitor server. See Customization