offlinemongodump

Perform the following steps to create a dSource with the offlinemongodump ingestion type.

Prerequisites

• The source and staging instances must meet the host requirements as described in Requirements for MongoDB staging/target hosts.

• For creating a dSource type of offlinemongodump, Backup Metadata File Name parameter is mandatory. The content of this file should be a single line datetime value, specified in format of MMDDYYYY_HH24MISS. The content of this file allows the Delphix Continuous Data Engine to validate if there is any new backup uploaded on the staging host that is different from the previous snapshot.

  • Delphix Continuous Data Engine checks this file based on a SnapSync policy, to check whether the new backup is available for ingestion or not. It performs no other checks on backups. If date is updated, Delphix Continuous Data Engine assumes that there are new backup files available in the backup location.

  • Delphix Continuous Data Engine will reingest backup and take a new snapshot. This file must be updated manually or as a part of the backup/copy script. The following command can be added to the backup script that will be executed as soon as the backup is successful or backup copy is successful.

    CODE

    echo "`date '+%m%d%Y_%H%M%S'`" > /<PATH>/mongo_bkp_metadata.txt  

The f​ile name and path must adhere to the organization standard naming convention.

•  It supports backups taken using the mongodump --oplog --gzip -o < bkp_dir > or mongodump --gzip -o < bkp_dir > command. Database dumps for one or more databases or one or more collections are supported.

• For MongoDB Atlas, a backup can be taken using the mongodump command line mentioned above.

Creating a dSource

1. Login to Delphix Management application.

2. Click Manage -> Environments.

3. Click on Add dSource.

4. In the Add dSource dialog window, select the MongoDB source config created on the staging host.

5. Select the dSource type from the drop-down list in the dSource dialog window. 

6. Enter the Mongo-specific parameters for your dSource configuration.

   1. dSource Type: This is the Type of dSource.

   2. dSource (Sharded/Replica Set) details:

      1. Staging Replicaset Name: This is the staging Database replica set name. The default value is dlpx_rs0.

      2. Staging MongoDB Host: This is the staging host for creating a dSource.

      3. Mount Path: This is the mount Path for dSource MongoDB instance data files.

      4. bindIP String: The default value is 0.0.0.0.

      5. Staging Port: This is the staging host port for MongoDB dSource instance.

   3. dSource tuning details:

      1. Oplog Size: This is the size of oplogs. 

      2. Journal Commit Interval: This is the journal commit interval, measured in ms. 

      3. Storage Engine: The default value is wiredTiger. 

      4. Enable SetParams: Enable the specified setParameters. 

      5. Set Parameter: These are the parameters that must be added in the mongod.conf. 

      6. Enable Auditlog: Enable the specified audit log parameters. 

      7. Auditlog Parameters: These are the parameters that must be added in the mongod.conf file.

   4. Mongodump Backup/Tuning details:

      1. Backup Metadata File Name: This is the file that contains backup datetime in the MMDDYYYY_HH24MISS format.

      2. Backup directory for Mongodump: This is the backup directory for Mongodump.

      3. Mongorestore Tuning Parameters: These are the pairs of parameter name and value for mongorestore command line.

   5. Enable Authentication: Enable member and client authentication for the database. 

   6. dSource authentication details:

      1. Mongo Admin User: This is the admin user of the staging database.This user must be created on the fly.

      2. MongoDB Password: This is the admin user password of the staging database.

      3. Cluster Auth Mode: The accepted values are; None, keyFile, x509, sendKeyFile, sendX509. 

      4. KeyAuth keyFile name: This is the absolute path of keyfile. 

   7. Enable SSL/TLS: Enable SSL/TLS member authentication. 

      1. SSL/TLS Parameters: Specify at least the following parameters; tlsMode, tlsCAFile, tlsCertificateKeyFile.

   8. Enable data encryption: Enable KMIP or keyfile based encryption.

   9. Data encryption details:

      1. Encryption Method: KeyFile or KMIP.

      2. Local Encryption Keyfile: This is the keyfile path for local encryption. 

      3. KMIP Parameters: This is the KMIP parameter details. For example, kmipClientCertificateFile, kmipPort, kmipServerCAFile, kmipServerName.

   10. Client Certificate for Mongos/Mongo session on Primary Node: This is the absolute path of client certificate to connect to staging database. 

   11. Client CA Certificate for Mongos/Mongo session on Primary Node: This is the absolute path of the root CA certificate that signed the client Certificate.  

7. Click Submit to create a dSource.

You must specify the inputs KeyAuth keyFile name and Cluster Auth Mode when Enable Authentication is selected.