Exporting Results

Overview

Easily and securely export high volumes of event data in real time to Splunk and other SIEM/analytics platforms

Highly-scalable Opsview Monitor is a solution for monitoring, aggregating, visualizing, alerting on, and drilling into event data from across arbitrarily-large, complex, multi-location, premise- and cloud-based enterprise IT estates. Enough so that it can be the "single pane of glass" that IT operations needs to work efficiently.

Opsview Monitor can also work as a take-off point -- providing data to other analytics, Security Information and Event Management (SIEM), bulk storage and other systems. The Opsview Monitor Results Exporter provides a complete, easy-to-use toolkit for extracting, filtering, and reformatting raw data directly from Opsview Monitor's message queue, and forwarding it to Splunk (SaaS or local) analytics, Enterprise Security, or to a host of other SIEM platform via syslog or HTTP.

Getting Started

To use the Results Exporter, you first need to Install the Results Exporter Component and then configure an output.

Currently supported outputs and information on how to configure them:

• Syslog Outputs - send information to a local or remote syslog server
• File Outputs - send information to a file on the local filesystem
• HTTP Outputs - send information over HTTP to a remote server - includes predefined supported types:
  • Splunk Event Collector - over unverified SSL.
  • Certified Splunk Event Collector - over verified SSL.

Syslog Output

Results can be exported to a local or remote syslog server by configuring a syslog output, under the outputs section of your configuration file /opt/opsview/resultsexporter/etc/resultsexporter.yaml, e.g:

outputs:
    syslog:
        my_syslog_output:
            protocol: udp
            host: 192.168.1.1
            port: 514
            log_facility: user
            log_level: info
            log_format: '[opsview-resultsexporter] %(message)s'
            log_date_format: '%Y-%m-%d %H:%M:%S'

Configuring a Syslog Output

The following options may be specified for your syslog output:

Required

None - if declared with no options, will log to /dev/log with all default settings.

Options

Log Date Format Strings

Log Format Strings

File Outputs

Results can be exported to a file on the local system by configuring a file output, under the outputs section of your configuration file /opt/opsview/resultsexporter/etc/resultsexporter.yaml, e.g:

outputs:
    file:
        my_results_export_file:
            path: '/var/log/results_export.log'

🚧

If your filename is results_export.log do not call the file output method results_export_file: (i.e. removing the suffix and appending _file to the name) as this may cause an error Error: Incorrect padding., preventing processing.

Configuring a File Output

The following options can be specified for your file output:

Required

Optional

HTTP Outputs

Results can be exported via HTTP to an external service by configuring a HTTP output, under the outputs section of your configuration file /opt/opsview/resultsexporter/etc/resultsexporter.yaml, e.g:

outputs:
    http:
        my_http_output:
            type: custom
            endpoint: 'http://www.mywebsite.com/resultscollector'
            headers:
                Username: 'username'
                Password: 'pass'

Configuring a Custom HTTP Output

Required

Optional

Configuring a Predefined HTTP Output

The following options can be specified for an HTTP output with a predefined type: spunk, splunk-cert

splunk

Results can be exported via unverified HTTPS to Splunk by configuring a HTTP output with type: splunk under the http section of your configuration file outputs within /opt/opsview/resultsexporter/etc/resultsexporter.yaml, e.g:

outputs:
    http:
        my_splunk_output:
            type: splunk
            parameters:
                host: '192.168.1.1'
                port: 8088
                token: '103a4f2f-023f-0cff-f9d7-3413d52b4b2b'

The following parameters are required for an HTTP Splunk output:

splunk-cert

Results can be exported via HTTPS to Splunk (using a client certificate) by configuring a HTTP output with type splunk-cert under the http section of your configuration file outputs within /opt/opsview/resultsexporter/etc/resultsexporter.yaml, e.g:

outputs:
    http:
        my_splunk_output:
            type: splunk-cert
            parameters:
                host: '192.168.1.1'
                port: 8088
                token: '103a4f2f-023f-0cff-f9d7-3413d52b4b2b'
                ca_certs: '/mycerts/ca.crt'
                keyfile: '/mycerts/client.key'
                certfile: '/mycerts/client.crt'

The following parameters are required for an HTTP Splunk output:

Note: If your client certificate and key are both within the same .pem file then you can simply list that file path for both certfile and keyfile.

Field Mapping

The Results Exporter allows you to transform the messages as they are exported, by specifying exactly which message fields should be present in the exported result, so you can remove details from the messages you do not want or need:

Within /opt/opsview/resultsexporter/etc/resultsexporter.yaml:

resultsexporter:
    outputs:
        syslog:
            fields:
                - hostname
                - servicecheckname
                - stdout

Specifying Fields

The fields of messages being exported are fully configurable. To select the fields that should be included in exported messages, list the keys under the fields section of an output. For example to include the hostname, servicecheckname, current_state and stdout fields, add them to the fields section:

resultsexporter:
    outputs:
        file:
            message_backup:
                fields:
                    - hostname
                    - servicecheckname
                    - current_state
                    - stdout

Simple Examples

# display the host name and the host state of each result:
fields:
    - hostname
    - host_state

# display the servicecheck name, the current state of the servicecheck, and the stdout message of each result
fields:
    - servicecheckname
    - current_state
    - stdout

Mapping and renaming fields

Users can also specify custom keys which are given values based on a mapping. For example to retrieve the host_state as a new field with name host_state_string, and value UP or DOWN instead of 0 or 1:

resultsexporter:

    fields:    
        - host_state_string:   
            host_state:
                0: "UP"            
                1: "DOWN"

In this example, the value of host_state determines behaviour as below:

A default value can also be specified. For example, in the example below, if the value of the hostname field does not match 'web-server' or 'email-server' , the value will be set to 'AllCompany'. If a default value is not specified and the key does not match any of the keys provided, the field will be omitted from the output.

resultsexporter:
    fields:
        - department:
            hostname:
                web-server: "Engineering"
                email-server: "BS"
            default: "AllCompany"            # default value declared here and used if no match for source name
        - check_state
        - stdout

This example results in behaviour as below:

Fields can also be added where the value is always constant.

resultsexporter:
    fields:
        - department:
            default: "AllCompany"            # default value is always used as there is no source name
        - check_state
        - stdout

This example results in behaviour as below:

Mapped values can refer to any (original) message fields, by using the format syntax: %(<field>)s, as shown in the example below.

resultsexporter:

    fields:
        - priority_msg:
            check_state:
                0: "%(servicecheckname)s HAS PRIORITY: LOW (OK)"
                2: "%(servicecheckname)s HAS PRIORITY: HIGH (CRITICAL)"
            default: "%(servicecheckname)s HAS PRIORITY: MEDIUM (%(check_state)s)"
        - check_state
        - stdout

This example results in behaviour as below:

This allows message fields to be 'renamed' easily if required by providing a one to one mapping with the original message field. For example to 'rename' the hostname field to name:

resultsexporter:
    fields:
        - name:
            default: "%(hostname)s"

This example results in behaviour as below:

🚧

Note: if you change you mapping values then you should review all your filters to ensure they will still work as expected

Reusing fields

To avoid duplication, the fields can be defined once using the & (yaml anchor) operator and reused multiple times using the * (anchor reference) operator.

resultsexporter:
    outputs:
        fields: &default_fields
            - hostname
            - servicecheckname
            - current_state
            - problem_has_been_acknowledged
            - is_hard_state
            - check_attempt
            - last_check
            - execution_time
            - stdout
        syslog:
            local_syslog_server:
                fields: *default_fields
        file:
            message_backup:
                fields: *default_fields

Alternatively, anchors can be declared as a list, and can have optional names for clarity, as in this example:

resultsexporter:
    fields:
        - basic_fields: &basic_fields
            - host_state
            - servicecheckname
            - last_check
            - stdout
        - &service_check_name_and_stdout:
            - servicecheckname
            - stdout
    outputs:
        syslog:
            local_syslog_server:
                fields: *basic_fields
        file:
            message_backup:
                fields: *service_check_name_and_stdout
        http:
            remote-api:
                fields:
                    - host_state
                    - is_hard_state

Performance Data Fields

The Results Exporter component exposes the individual metrics within the performance data of each message. To include the raw performance data string in your exported message, include the perf_data_raw field within your list of fields e.g for a service check with performance data of:

resultsexporter:

    outputs:
        syslog:
            local_syslog_server:
                fields:
                    - hostname
                    - stdout
                    - perf_data_raw

To include the entire performance data as a nested structure within your message, include the perf_data field:

resultsexporter:
    outputs:
        syslog:
            local_syslog_server:
                fields:
                    - hostname
                    - stdout
                    - perf_data

To include some of the nested fields, but not all, you can specify specific named metrics as below:

resultsexporter:
    outputs:
        syslog:
            local_syslog_server:
                fields:
                    - hostname
                    - stdout
                    - perf_data.rta
                    - perf_data.rtmin
                    - perf_data.rtmax

Supported fields

Filtering

A powerful feature of the Results Exporter is the ability to select exactly which messages you want to be exported via each output, using a filter string. Any message that meets the condition specified in the filter will be processed and exported by that output e.g:

resultsexporter:
    outputs:
        syslog:
            filter: '(hostname == "opsview") && (servicecheckname == "Connectivity - LAN")'

Specifying Filters

Using a filter allows you to focus on exporting the results that are important and ignore anything that isn't relevant to your chosen reporting/searching tool.

Inbuilt Filters

A filter string can also consist of a comparison between a key within the message and a value (the value for the key within each message being filtered).

More complex filters can be written by combining filter strings using the && (logical and) and || (logical or) operators.

Supported comparisons

Supported Fields

Within your filter, you can refer to any field listed as supported in Field Mapping, with the exception of perf_data, perf_data_raw and any extracted performance data fields. These are not supported by the filter.

Simple Filter Examples

# allow every message
filter: ''
  
# allow every message
filter: '*'
  
# allow no messages
filter: '!*'
  
# only allow messages where the hostname contains "opsview."
filter: '(hostname ~ "opsview.")'
  
# only allow messages related to the LAN Connectivity service check
filter: '(servicecheckname == "Connectivity - LAN")'
  
# only allow messages where the service state is anything except OK - where current_state values have not been remapped
filter: '(current_state !=0 )'

# only allow messages where the service state is anything except OK - where current_state values have been remapped
filter: '(current_state !~ "OK")'
  
# only allow messages where the hostname is in IPv4 address form
filter: '(hostname @ "^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$")'
  
# only allow messages where the host ID is 500 or above
filter: '(host_id >= 500)'
  
# only allow messages where state type is hard
filter: '(is_hard_state == True)'
  
# only allow messages where the service check name is null
filter: '(servicecheckname == null)'

Complex Filter Examples

# only allow messages where the hostname contains "opsview." and relates to the LAN Connectivity service check
filter: '(hostname ~ "opsview.") && (servicecheckname == "Connectivity - LAN")'
  
# only allow messages where the hostname contains "opsview." and relates to the LAN Connectivity service check,
# and the state type is HARD and the service state is anything except OK
filter: '(hostname ~ "opsview.") && (servicecheckname == "Connectivity - LAN") && (check_state_type == "HARD") && (current_state !~ "OK")'
  
# only allow messages where the hostname is in IPv4 address form
filter: '(hostname @ "^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$")'
  
# only allow messages from the 'Disk: /' and 'Disk: /tmp' service checks, and only when the host ID is 500 or above
filter: '((servicecheckname == "Disk: /") || (servicecheckname == "Disk: /tmp")) && (host_id >= 500)'

NOTE: It is advised to surround your filters with single quotes:

filter: '(hostname @ "^\d{2}-\d{4}$")'

If your filter is surrounded in double quotes, you will need to escape backlashes in your regular expressions:

filter: "(hostname @ '^\\d{2}-\\d{4}$')" 

Reusing Filters

To avoid duplication, the filters can be defined once using the & (yaml anchor) operator and reused multiple times using the * (anchor reference) operator.

resultsexporter:
    outputs:
        filter: &default_filter '(hostname == "Test Host")'
        syslog:
            local_syslog_server:
                filter: *default_filter

Alternatively, anchors can be declared as a list, and can have optional names for clarity, as in this example:

resultsexporter:
    filter:
        - opsview_host: &opsview_host
            '(hostname ~ "opsview")'
        - &not_ok:
            '(current_state !~ "OK")'
    outputs:
        syslog:
            local_syslog_server:
                filter: *opsview_host
        file:
            message_backup:
                filter: *not_ok

Multi-line Filters

Multi-line filters are possible as long as the entire filter is quoted - this can add clarity for complex filters as seen below:

resultsexporter:
    outputs:
        http:
            splunk:
                filter:
                    '(servicecheckname == "CPU Statistics")
                        ||
                     (servicecheckname == "Connectivity - LAN")'

Formatting Messages

The Results Exporter allows you to declare the format type of the exported messages for file and HTTP outputs, as well as adding any additional information to each message using a format string:

resultsexporter:
    outputs:
        file:
            format_type: json
            message_format: '{"my_message": %(message)s}'

Format Types

The list of currently supported format types is as follows:

• XML
• JSON
• KVP (Key Value Pairs)
• YAML

Example Job Result Message:

{
    "host_state": 0,
    "hostname": "My Host",
    "servicecheckname": null
    "perf_data": "rta=1ms;50;100;; rtmin=0.5ms;;;; rtmax=2ms;;;; pl=20%;60;90;;",
    "stdout": "OK - default msg"
}

Formatted into kvp:

[opsview_resultsexporter 2019-01-01 17:53:02] host_state=0, hostname="My Host",
servicecheckname=None, perf_data_raw="rta=1ms;50;100;; rtmin=0.5ms;;;; rtmax=2ms;;;; pl=20%;60;90;;",
perf_data.rta="1ms", perf_data.rtmin="0.5ms", perf_data.rtmax="2ms", perf_data.pl="20%", stdout="OK - default msg"

Formatted into json:

{
    "info": "opsview_resultsexporter 2019-01-01 17:53:02",
    "message": {
        "host_state":0,
        "hostname": "My Host",
        "servicecheckname": null,
        "perf_data_raw": "rta=1ms;50;100;; rtmin=0.5ms;;;; rtmax=2ms;;;; pl=20%;60;90;;",
        "perf_data": {
                "rta": "1ms",
                "rtmin": "0.5ms",
                "rtmax": "2ms",
                "pl": "20%"
        },
        "stdout":"OK - default msg"
    }
}

Formatted into xml:

<result>
    <info>opsview_resultsexporter 2019-01-01 17:53:02</info>
    <message>
        <host_state>0</host_state>
        <hostname>My Host</hostname>
        <servicecheckname />
        <perf_data_raw>rta=1ms;50;100;; rtmin=0.5ms;;;; rtmax=2ms;;;; pl=20%;60;90;;</perf_data_raw>
        <perf_data>
            <rta>1ms</rta>
            <rtmin>0.5ms</rtmin>
            <rtmax>2ms</rtmin>
            <pl>20%</pl>
        </perf_data_raw>
        <stdout>OK - default msg</stdout>
    </message>
</result>

Formatted into yaml:

- info: opsview_resultsexporter 2019-01-01 17:53:02
  message:
    host_state: 0
    hostname: My Host
    servicecheckname: null
    perf_data_raw: rta=1ms;50;100;; rtmin=0.5ms;;;; rtmax=2ms;;;; pl=20%;60;90;;
    perf_data:
        rta: 1ms
        rtmin: 0.5ms
        rtmax: 2ms
        pl: 20%
    stdout: OK - default msg