shardedsource (using MongoDB Ops Manager backup files)

Perform the following steps to create a dSource with the shardedsource ingestion type.

Prerequisites

• The source and staging instances must meet the host requirements as described in Requirements for MongoDB staging/target hosts.

• For creating a dSource type of shardedsource, the Backup Metadata File Name parameter is mandatory. The content of this file should be a single line datetime value, specified in format of MMDDYYYY_HH24MISS. The content of this file allows the Delphix Continuous Data Engine to validate if there is any new backup uploaded on the staging host that is different from the previous snapshot.

  • Delphix Continuous Data Engine checks this file based on a SnapSync policy to check whether the new backup is available for ingestion or not. It performs no other checks on backups. If date is updated, Delphix Continuous Data Engine assumes that there are new backup files available in the backup location.

  • Delphix Continuous Data Engine will reingest backup and take a new snapshot. This file must be updated manually or as a part of the backup/copy script. The following command can be added to the backup script that will be executed as soon as the backup is successful or backup copy is successful.

    CODE

    echo "`date '+%m%d%Y_%H%M%S'`" > /<PATH>/mongo_bkp_metadata.txt

The file name and path must adhere to the organization standard naming convention.

• For shardedsource, it is expected to have one configserver backup file and one backup file per shard. It is important to keep the name of backup files same for every new backup. If the file names are changed, it needs to be updated in GUI parameters every time a new backup is uploaded. It is recommended to use some standard naming convention for sharded backups. 

• The expected backup file format is as follows: tar.gz or tar

The below list shows the sample names:

• Configsvr_bkp.tar.gz

• myShard_0_bkp.tar.gz

• myShard_1_bkp.tar.gz 

• myShard_2_bkp.tar.gz

Creating dSource

1. Login to Delphix Management application.

2. Click Manage > Environments.

3. Click on Add dSource.

4. In the Add dSource dialog window, select the MongoDB source config created on the staging host.

5. Select the dSource type from the drop-down list in the dSource dialog window. 

6. Enter the Mongo-specific parameters for your dSource configuration.

   1. dSource Type: This is the Type of dSource.

   2. dSource (Sharded/Replica set) details:

      1. Staging Replicaset Name: This is the staging Database replica set name. The default value is dlpx_rs0.

      2. Staging MongoDB Host: This is the staging host for creating a dSource.

      3. Mount Path: This is the mount Path for dSource MongoDB instance data files.

      4. bindIP String: The default value is 0.0.0.0.

      5. Staging Port: This is the staging host port for MongoDB dSource instance.

   3. dSource tuning details:

      1. Oplog Size: This is the size of oplogs. 

      2. Journal Commit Interval: This is the journal commit interval in ms. 

      3. Storage Engine: The default value is wiredTiger. 

      4. Enable SetParams: Enable the specified setParameters. 

      5. Set Parameter: These are the parameters that must be added in the mongod.conf. 

      6. Enable Auditlog: Enable the specified audit log parameters. 

      7. Auditlog Parameters: These are the parameters that must be added in the mongod.conf file.

   4. Sharded dSource details:

      1. Mongos Port Number: This is the port that must be used for MongoDB’s router instance.

   5. OpsManager backup details:

      1. Backup Metadata File Name: This is the file that contains backup datetime. The content of this file is in MMDDYYYY_HH24MISS format.

      2. Config Backup / Replicaset Backup: This is the absolute path of config backup file name (Sharded).

      3. Shard Backup Files: This is the absolute path of Shard Backup File. Add file for each Shard. One file per shard.

   6. Enable Authentication: Enable member and client authentication for the database. 

   7. dSource authentication details:

      1. Mongo Admin User: This is the admin user of the staging database. This user should be created on the fly.

      2. MongoDB Password: This is the admin user password of the staging database.

      3. Cluster Auth Mode: The accepted values are: None, keyfile, x509, sendKeyFile, sendX509. 

      4. KeyAuth keyfile name: This is the absolute path of keyfile.

      5. Enable SSL/TLS: Enable SSL/TLS member authentication. 

         1. SSL/TLS Parameters: Specify at least the following parameters; tlsMode, tlsCAFile, tlsCertificateKeyFile.

      6. Enable data encryption: Enable KMIP or keyfile based encryption.

      7. Data encryption details:

         1. Encryption Method: keyfile or KMIP.

         2. Local Encryption keyfile: This is the keyfile path for local encryption. 

         3. KMIP Parameters: Provide the KMIP parameter details. For example, kmipClientCertificateFile, kmipPort, kmipServerCAFile, kmipServerName.

      8. Client Certificate for Mongos/Mongo session on Primary Node: This is the absolute path of client certificate to connect to staging database. 

      9. Client CA Certificate for Mongos/Mongo session on Primary Node: This is the absolute path of root CA certificate that signed the client certificate.

   8. Click Submit to create a dSource.

• If MongoDB Ops Manager backup is taken when the source database is encrypted, you must select the Source is Encrypted checkbox and provide corresponding values at Source Encryption Method, KMIP Parameters or Local Encryption keyfile.

• You must specify the inputs KeyAuth keyfile name and Cluster Auth Mode when Enable Authentication is selected.