Skip to main content
Skip table of contents

Fluentd plugin service for API modules

Overview

The Delphix Fluentd plugin service assisted in a feature that provided options for configuring a Delphix engine that forwards events and metrics to a Splunk host. Delphix has developed methods to extend this capability for use with other monitoring packages (ELK Stack, Datadog, etc.). Fluentd plugins provide a mechanism to customize the Delphix engine’s forwarding capabilities for output to other data consumers. 

The Fluentd plugin is a tar file with a configuration template, including any gem files that are required to execute the directives in the configuration file. Once the plugin is uploaded, the configuration occurs by collecting user input for the variables required by the template. The interfaces will have changed since the set of variables needed is now mutable from plugin to plugin. Splunk configuration will be available out of the box with no additional plugins. This feature uses one UI for all Fluentd configurations.

The current implementation is limited to one data consumer at a time. Only one plugin, in addition to the default SplunkHec plugin, can be uploaded. This is an intended simplification for the initial Fluentd release version.

Technical details

Architectural diagram

GUI, API or CLI changes (if any)

APIs with a corresponding CLI for uploading plugins and Fluentd configuration have been added. The service/insight/plugins API displays the plugins that are available.

JSON
service insight plugins> ls

Objects   REFERENCE         PLUGIN       GEMS   ATTRIBUTEDEFINITIONS  SCHEMADEFINITION

FLUENTD_PLUGIN-1           splunkHec   …          …                                       …

Operations

requestUploadToken

The splunkHec plugin appears by default and cannot be deleted. Additionally, the requestUploadToken operation provides a token needed to upload a plugin via the data/upload API. If a second plugin is uploaded, it will appear in the list. Before a third plugin could be uploaded, the second must be selected and deleted. 

The following curl commands illustrate the use of the API to upload a plugin:

JSON
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/session -c ~/cookies.txt -H "Content-Type: application/json" <<EOF

{

   "type": "APISession",

   "version": {

       "type": "APIVersion",

       "major": 1,

       "minor": 4,

       "micro": 3

   }

}

EOF

curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/login -b ~/cookies.txt -c ~/cookies2.txt -H "Content-Type: application/json" <<EOF

{

"type": "LoginRequest",

"username": "sysadmin",

"password": "sysadmin",

"target": "SYSTEM"

}

EOF

curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/service/insight/plugins/requestUploadToken -b ~/cookies2.txt -H "Content-Type: application/json" <<EOF

Returns the token e.g:  {"type":"OKResult","status":"OK","result":{"type":"FileUploadResult","url":"/resources/json/delphix/data/upload","token":"59346df5-3ecd-4ced-afbf-97acefa156dc"},"job":null,"action":null}

curl  -b ~/cookies2.txt -X POST -F "file=@/Users/blewis/ws/far-dev/splunk_host_port.far" -F "token=<token>" http:/<engine-name>/resources/json/delphix/data/upload

The service/insight/plugins API displays the tokens that are available.

An example for creating a configuration with the default splunkHec plugin:

JSON
service insight configuration> create

service insight configuration create *> set plugin=splunkHec; set enabled=true; edit attributes

service insight configuration create attributes *> add; set name=hec_host; set value=http://vmname-splunkhost.delphix.com; back

service insight configuration create attributes *> add; set name=hec_port; set value=8088; back;

service insight configuration create attributes *> add; set type=FluentdSecretAttribute;

service insight configuration create attributes 2 *> set name=hec_token; set secretValue=bb75c032-bdea-4c19-b152-d158cf13e019; back

service insight configuration create attributes *> add; set name=eventsIndex; set value=delphix_events; back;

service insight configuration create attributes *> add; set name=protocol; set value=https; back

service insight configuration create attributes *> add; set name=metricsPushFrequency; set value=60; back

service insight configuration create attributes *> add; set name=metricsIndex; set value=delphix_metrics; back

service insight configuration create attributes *> add; set name=eventsPushFrequency; set value=60; back;

service insight configuration create attributes *> ls

Properties

    0:

        type: FluentdRegularAttribute (*)

        name: hec_host (*)

        value: http://vmname-splunkhost.delphix.com (*)

    1:

        type: FluentdRegularAttribute (*)

        name: hec_port (*)

        value: 8088 (*)

    2:

        type: FluentdSecretAttribute (*)

        name: hec_token (*)

        secretValue: ******** (*)

    3:

        type: FluentdRegularAttribute (*)

        name: eventsIndex (*)

        value: delphix_events (*)

    4:

        type: FluentdRegularAttribute (*)

        name: protocol (*)

        value: https (*)

    5:

        type: FluentdRegularAttribute (*)

        name: metricsPushFrequency (*)

        value: 60 (*)

    6:

        type: FluentdRegularAttribute (*)

        name: metricsIndex (*)

        value: delphix_metrics (*)

    7:

        type: FluentdRegularAttribute (*)

        name: eventsPushFrequency (*)

        value: 60 (*)

## Use the "add" command to add an element to this array.

service insight configuration create attributes *> back

service insight configuration create *> ls

Properties

    type: FluentdConfig

    name: (unset)

    attributes: [ ... ] (*)

    enabled: true (*)

    plugin: splunkHec (*)

service insight configuration create *> commit

Creating default splunkHec configuration from GUI

  1. Login as a sysadmin user and select Fluentd Configuration under the Preferences menu.
     

  2. The splunkHec plugin appears in the dropdown by default.
     

  3. Additional plugins can be added by using the + button on the right.

  4. The new plugin uploaded can be viewed in the dropdown menu and selected to add a configuration.

  5. The delete button on the right can be used to delete a selected plugin. 

  6. The name of fields for configuration would be the same as those provided in the uploaded plugin file.

  7. The configuration can be saved by clicking the save button on the bottom. 

  8. When loading an existing configuration on UI, the secret fields are masked.

Implementation

Secret attributes have been introduced in order to protect private data such as passwords and tokens. In the CLI and API, they are identified by the FluentdSecretAttribute type. In plugins, attribute names in the config file template beginning with _secret indicate secret data. 

Known issues

Fluentd runs in a systemd container to limit negative side-effects from a bad plugin. More security can be added to this container by running as a non-root user, in addition to limiting the size of the log and buffer files it can write.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.