offlinemongodump

Perform the following steps to create a dSource with the offlinemongodump ingestion type.

Prerequisites

The source and staging instances must meet the host requirements as described in Requirements for MongoDB staging/target hosts. for creating a dSource type of offlinemongodump.
Backup Metadata File Name parameter is mandatory. The content of this file should be a single line datetime value, specified in format of MMDDYYYY_HH24MISS. The content of this file allows the Delphix Continuous Data Engine to validate if there is any new backup uploaded on the staging host that is different from the previous snapshot.
- Delphix Continuous Data Engine checks this file based on a SnapSync policy, to check whether the new backup is available for ingestion or not. It performs no other checks on backups. If date is updated, Delphix Continuous Data Engine assumes that there are new backup files available in the backup location.
- Delphix Continuous Data Engine will reingest backup and take a new snapshot. This file must be updated manually or as a part of the backup/copy script. The following command can be added to the backup script that will be executed as soon as the backup is successful or backup copy is successful.
  CODE
```
echo "`date '+%m%d%Y_%H%M%S'`" > /<PATH>/mongo_bkp_metadata.txt  
```
Backup directory for Mongodump is the backup directory of backups taken using mongodump tool.
- Make sure that the backup is taken using the below command line
  CODE
```
mongodump <mongo_uri> --gzip -o <bkp_dir>
```
  (append --oplog, if required)
- The mongorestore tool requires that the subdirectory structure of the backup must remain unchanged when restoring it. It should be exactly the same on the restore host as it was on the original host where the backup was created or else mongorestore will not ingest the backups and you will see the below error in the logs.
  CODE
```
don't know what to do with file <backup filen name>, skipping ...
```
- Database dumps for one or more databases or one or more collections are supported.
- Backups taken using --archive=<file-path> option are not yet supported.

Creating a dSource

Login to Delphix Management application.
Click Manage -> Environments.
Click on Add dSource.
In the Add dSource dialog window, select the MongoDB source config created on the staging host.
Select the dSource type from the drop-down list in the dSource dialog window.

Enter the Mongo-specific parameters for your dSource configuration.

dSource Type: This is the Type of dSource.
dSource (Sharded/Replica Set) details:
1. Staging Replicaset Name: This is the staging Database replica set name. The default value is dlpx_rs0.
2. Staging MongoDB Host: This is the staging host for creating a dSource.
3. Mount Path: This is the mount Path for dSource MongoDB instance data files.
4. bindIP String: The default value is 0.0.0.0.
5. Staging Port: This is the staging host port for MongoDB dSource instance.
dSource tuning details:
1. Oplog Size: This is the size of oplogs.
2. Journal Commit Interval: This is the journal commit interval, measured in ms.
3. Storage Engine: The default value is wiredTiger.
4. Enable SetParams: Enable the specified setParameters.
5. Set Parameter: These are the parameters that must be added in the mongod.conf.
6. Enable Auditlog: Enable the specified audit log parameters.
7. Auditlog Parameters: These are the parameters that must be added in the mongod.conf file.

Mongodump Backup/Tuning details:

Backup Metadata File Name: This is the file that contains backup datetime in the MMDDYYYY_HH24MISS format.
Backup directory for Mongodump: This is the backup directory for Mongodump.

Mongorestore Tuning Parameters: These are the pairs of parameter name and value for mongorestore command line.

CODE

input options:
      --objcheck                                            validate all objects before inserting
      --oplogReplay                                         replay oplog for point-in-time restore
      --oplogLimit=<seconds>[:ordinal]                      only include oplog entries before the provided Timestamp
      --oplogFile=<filename>                                oplog file to use for replay of oplog
      --archive=<filename>                                  restore dump from the specified archive file.  If flag is specified without a value, archive
                                                            is read from stdin
      --restoreDbUsersAndRoles                              restore user and role definitions for the given database
      --dir=<directory-name>                                input directory, use '-' for stdin
      --gzip                                                decompress gzipped input

restore options:
      --drop                                                drop each collection before import
      --dryRun                                              view summary without importing anything. recommended with verbosity
      --writeConcern=<write-concern>                        write concern options e.g. --writeConcern majority, --writeConcern '{w: 3, wtimeout: 500,
                                                            fsync: true, j: true}'
      --noIndexRestore                                      don't restore indexes
      --convertLegacyIndexes                                Removes invalid index options and rewrites legacy option values (e.g. true becomes 1).
      --noOptionsRestore                                    don't restore collection options
      --keepIndexVersion                                    don't update index version
      --maintainInsertionOrder                              restore the documents in the order of their appearance in the input source. By default the
                                                            insertions will be performed in an arbitrary order. Setting this flag also enables the
                                                            behavior of --stopOnError and restricts NumInsertionWorkersPerCollection to 1.
  -j, --numParallelCollections=                             number of collections to restore in parallel
      --numInsertionWorkersPerCollection=                   number of insert operations to run concurrently per collection
      --stopOnError                                         halt after encountering any error during insertion. By default, mongorestore will attempt to
                                                            continue through document validation and DuplicateKey errors, but with this option enabled,
                                                            the tool will stop instead. A small number of documents may be inserted after encountering
                                                            an error even with this option enabled; use --maintainInsertionOrder to halt immediately
                                                            after an error
      --bypassDocumentValidation                            bypass document validation
      --preserveUUID                                        preserve original collection UUIDs (off by default, requires drop)
      --fixDottedHashIndex                                  when enabled, all the hashed indexes on dotted fields will be created as single field
                                                            ascending indexes on the destination.

Enable Authentication: Enable member and client authentication for the database.
dSource authentication details:
1. Mongo Admin User: This is the admin user of the staging database.This user must be created on the fly.
2. MongoDB Password: This is the admin user password of the staging database.
3. Cluster Auth Mode: The accepted values are; None, keyFile, x509, sendKeyFile, sendX509.
4. KeyAuth keyFile name: This is the absolute path of keyfile.
Enable SSL/TLS: Enable SSL/TLS member authentication.
1. SSL/TLS Parameters: Specify at least the following parameters; tlsMode, tlsCAFile, tlsCertificateKeyFile.
Enable data encryption: Enable KMIP or keyfile based encryption.
Data encryption details:
1. Encryption Method: KeyFile or KMIP.
2. Local Encryption Keyfile: This is the keyfile path for local encryption.
3. KMIP Parameters: This is the KMIP parameter details. For example, kmipClientCertificateFile, kmipPort, kmipServerCAFile, kmipServerName.
Client Certificate for Mongos/Mongo session on Primary Node: This is the absolute path of client certificate to connect to staging database.
Client CA Certificate for Mongos/Mongo session on Primary Node: This is the absolute path of the root CA certificate that signed the client Certificate.

Click Submit to create a dSource.

You must specify the inputs KeyAuth keyFile name and Cluster Auth Mode when Enable Authentication is selected.