Fluentd plugin service for API modules
Overview
The Delphix Fluentd plugin service assisted in a feature that provided options for configuring a Delphix engine that forwards events and metrics to a Splunk host. Delphix has developed methods to extend this capability for use with other monitoring packages (ELK Stack, Datadog, etc.). Fluentd plugins provide a mechanism to customize the Delphix engine’s forwarding capabilities for output to other data consumers.
The Fluentd plugin is a tar file with a configuration template, including any gem files that are required to execute the directives in the configuration file. Once the plugin is uploaded, the configuration occurs by collecting user input for the variables required by the template. The interfaces will have changed since the set of variables needed is now mutable from plugin to plugin. Splunk configuration will be available out of the box with no additional plugins. This feature uses one UI for all Fluentd configurations.
The current implementation is limited to one data consumer at a time. Only one plugin, in addition to the default SplunkHec plugin, can be uploaded. This is an intended simplification for the initial Fluentd release version.
Technical details
Architectural diagram
GUI, API or CLI changes (if any)
APIs with a corresponding CLI for uploading plugins and Fluentd configuration have been added. The service/insight/plugins API displays the plugins that are available.
service insight plugins> ls
Objects REFERENCE PLUGIN GEMS ATTRIBUTEDEFINITIONS SCHEMADEFINITION
FLUENTD_PLUGIN-1 splunkHec … … …
Operations
requestUploadTokenThe splunkHec plugin appears by default and cannot be deleted. Additionally, the requestUploadToken operation provides a token needed to upload a plugin via the data/upload API. If a second plugin is uploaded, it will appear in the list. Before a third plugin could be uploaded, the second must be selected and deleted.
The following curl commands illustrate the use of the API to upload a plugin:
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/session -c ~/cookies.txt -H "Content-Type: application/json" <<EOF
{
"type": "APISession",
"version": {
"type": "APIVersion",
"major": 1,
"minor": 4,
"micro": 3
}
}
EOF
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/login -b ~/cookies.txt -c ~/cookies2.txt -H "Content-Type: application/json" <<EOF
{
"type": "LoginRequest",
"username": "sysadmin",
"password": "sysadmin",
"target": "SYSTEM"
}
EOF
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/service/insight/plugins/requestUploadToken -b ~/cookies2.txt -H "Content-Type: application/json" <<EOF
Returns the token e.g: {"type":"OKResult","status":"OK","result":{"type":"FileUploadResult","url":"/resources/json/delphix/data/upload","token":"59346df5-3ecd-4ced-afbf-97acefa156dc"},"job":null,"action":null}
curl -b ~/cookies2.txt -X POST -F "file=@/Users/blewis/ws/far-dev/splunk_host_port.far" -F "token=<token>" http:/<engine-name>/resources/json/delphix/data/uploadThe service/insight/plugins API displays the tokens that are available.
An example for creating a configuration with the default splunkHec plugin:
service insight configuration> create
service insight configuration create *> set plugin=splunkHec; set enabled=true; edit attributes
service insight configuration create attributes *> add; set name=hec_host; set value=http://vmname-splunkhost.delphix.com; back
service insight configuration create attributes *> add; set name=hec_port; set value=8088; back;
service insight configuration create attributes *> add; set type=FluentdSecretAttribute;
service insight configuration create attributes 2 *> set name=hec_token; set secretValue=bb75c032-bdea-4c19-b152-d158cf13e019; back
service insight configuration create attributes *> add; set name=eventsIndex; set value=delphix_events; back;
service insight configuration create attributes *> add; set name=protocol; set value=https; back
service insight configuration create attributes *> add; set name=metricsPushFrequency; set value=60; back
service insight configuration create attributes *> add; set name=metricsIndex; set value=delphix_metrics; back
service insight configuration create attributes *> add; set name=eventsPushFrequency; set value=60; back;
service insight configuration create attributes *> ls
Properties
0:
type: FluentdRegularAttribute (*)
name: hec_host (*)
value: http://vmname-splunkhost.delphix.com (*)
1:
type: FluentdRegularAttribute (*)
name: hec_port (*)
value: 8088 (*)
2:
type: FluentdSecretAttribute (*)
name: hec_token (*)
secretValue: ******** (*)
3:
type: FluentdRegularAttribute (*)
name: eventsIndex (*)
value: delphix_events (*)
4:
type: FluentdRegularAttribute (*)
name: protocol (*)
value: https (*)
5:
type: FluentdRegularAttribute (*)
name: metricsPushFrequency (*)
value: 60 (*)
6:
type: FluentdRegularAttribute (*)
name: metricsIndex (*)
value: delphix_metrics (*)
7:
type: FluentdRegularAttribute (*)
name: eventsPushFrequency (*)
value: 60 (*)
## Use the "add" command to add an element to this array.
service insight configuration create attributes *> back
service insight configuration create *> ls
Properties
type: FluentdConfig
name: (unset)
attributes: [ ... ] (*)
enabled: true (*)
plugin: splunkHec (*)
service insight configuration create *> commitCreating default splunkHec configuration from GUI
Login as a sysadmin user and select Fluentd Configuration under the Preferences menu.
The splunkHec plugin appears in the dropdown by default.
Additional plugins can be added by using the + button on the right.
The new plugin uploaded can be viewed in the dropdown menu and selected to add a configuration.
The delete button on the right can be used to delete a selected plugin.
The name of fields for configuration would be the same as those provided in the uploaded plugin file.
The configuration can be saved by clicking the save button on the bottom.
When loading an existing configuration on UI, the secret fields are masked.
Troubleshooting
Occasionally, the fluentd service might fail to start or encounter problems when sending logs to the remote data consumer. Typical explanations might include syntax errors in the plugin’s config file or network configuration problems on the data consumer server. Such issues are generally reflected by error output in the fluent.log file, which can be accessed for download via the /service/fluentd/plugins/downloadFluentdLog API endpoint. Here is an example of downloading the log via curl:
curl -v -O -J "$delphix_engine/resources/json/delphix/service/fluentd/plugins/downloadFluentdLog" -b ~/cookies.txt
Fluentd rotates its logs once they have reached a certain maximum size to avoid memory issues should they grow too large. While unlikely, it is possible that the logs might be rotated in the midst of an ongoing issue you are trying to diagnose. If this occurs, it may be necessary to wait some time and retry the log download so that useful information is included.
Implementation
Secret attributes have been introduced in order to protect private data such as passwords and tokens. In the CLI and API, they are identified by the FluentdSecretAttribute type. In plugins, attribute names in the config file template beginning with _secret indicate secret data.
Known issues
Fluentd runs in a systemd container to limit negative side-effects from a bad plugin. More security can be added to this container by running as a non-root user, in addition to limiting the size of the log and buffer files it can write.