A closer look at TDE provisioning
The following is intended to provide a deeper understanding of the Delphix TDE implementation, and is not required for a general understanding of the general provisioning workflows.
Delphix Provisioning workflow with and without TDE
Provisioning a Virtual Pluggable Database (vPDB) first involves using the GUI or CLI to specify the vPDB parameters (such as the vPDB name and target container) along with the snapshot to provision from. Once the provision job is started with these parameters, the Delphix Engine does the following:
Mounts the snapshot files on the target host.
Creates and opens (in mount mode) the auxiliary container database on the target host, using the snapshot files. The auxiliary container database will have both the CDB and PDB data files from the dSource.
Completes recovery to bring the auxiliary container database into a consistent state.
Finalizes the state of the auxiliary database and unplugs the vPDB datafiles.
Plugs the vPDB into the target database, and opens it in read-write mode for general use.
If the dSource is TDE-enabled, then Delphix will need to perform additional operations to complete the provision of a TDE-enabled vPDB to a TDE-enabled target container (indicated in red below):
Mounts the snapshot files on the target host.
Creates and opens (in mount mode) the auxiliary container database on the target host, using the snapshot files. The auxiliary container database will have both the CDB and PDB data files from the dSource.
Creates a keystore for the auxiliary container database with the necessary keys to apply encrypted archived log files.
Completes recovery to bring the auxiliary container database into a consistent state.
Rotates the vPDB and auxiliary CDB master encryption keys by generating new keys that are unique to the vPDB / auxiliary CDB and not associated with the source PDB or CDB.
Exports only the newly generated keys to an exported keyfile to enable unplug.
Finalizes the state of the auxiliary database and unplugs the vPDB datafiles.
Imports the keys from the exported keyfile into the target keystore.
When provisioning to a new vCDB target, converts the auxiliary CDB into the final vCDB and creates the vCDB keystore from the auxiliary CDB keystore.
Plugs the vPDB into the target database, and opens it in read-write mode for general use.
If a plugin violation is detected after plugging the vPDB into the target database, the vPDB provision (and enable) operation will throw a critical error during the SnapSync operation, but it will succeed nonetheless. The vPDB will remain in a running state. You must manually resolve the plugin violations outside of Delphix, close and reopen the vPDB to make the changes effective, and then disable/enable the vPDB within Delphix.
The following diagram illustrates the provisioning steps.
At each stage of the provisioning process, the keys and exported keyfiles are always on user storage. The exported keyfile is located in the artifact directory, while the auxiliary and target keystores are in the auxiliary keystores directory. Both the artifact directory and auxiliary keystores directory are subdirectories of the TDE keystores root directory, which is either user specified, or if not specified defaults to the toolkit root directory. Similar to non-TDE-enabled vPDBs, the final vPDB (and vCDB, if applicable) is on Delphix storage while the target Linked CDB and its archive logs remain on user storage.
Validating the keystore password
To validate that the password for a given keystore is correct, Oracle provides the mkstore command-line utility. Navigate to the keystore folder on the target host where ewallet.p12 exists and run the command mkstore -wrl . list as the Oracle user. If the password is incorrect, you will see output similar to the following:
$ mkstore -wrl . -list
Oracle Secret Store Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.
Enter wallet password: xxxxxxxx
oracle.security.crypto.core.CipherException: Invalid padding string (or incorrect password)If the password is correct, you will see output similar to the following:
$ mkstore -wrl . -list
Oracle Secret Store Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.
Enter wallet password: xxxxxxxx
Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.AQtDPGje509PvxYeQuG7fmYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.AVXDkpbWnE9rv8fzTPiHTXcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.AWS1QwdQHk/jv0i5eJ3sb10AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.AZcpc0/QrU/Nv/Q54WRWRSMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.C204B25A34B93EB7E055000000000001
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.C204B30EA4A03FABE055000000000001
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.C204B407D46A3FFDE055000000000001
ORACLE.SECURITY.ID.ENCRYPTION.
ORACLE.SECURITY.KB.ENCRYPTION.
ORACLE.SECURITY.KM.ENCRYPTION.AQtDPGje509PvxYeQuG7fmYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.AVXDkpbWnE9rv8fzTPiHTXcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.AWS1QwdQHk/jv0i5eJ3sb10AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.AZcpc0/QrU/Nv/Q54WRWRSMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAExample Illustrating Generation of New Encryption Keys for vPDBs
This example illustrates the movement of encryption keys that occur as a result of provisioning a TDE-enabled vPDB. Consider a vPDB tde_vpdb that is provisioned from a dSource CDOMSHSR52CAPDB2 on the VM tde-source19, which is an Oracle database running version 19.11.0. Connecting to this database, we can query v$encryption_keys to determine the current keys in use by each PDB:
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
------ ---------------------- ---------- ----------
2 PDB$SEED READ ONLY NO
3 CDOMSHSR52CAPDB1 READ WRITE NO
4 CDOMSHSR52CAPDB2 READ WRITE NO
5 CDOMSHSR52CAPDB3 READ WRITE NO
SQL> select con_id, key_id from v$encryption_keys order by con_id;
CON_ID KEY_ID
------ ----------------------------------------------------
1 Ac9MY5kQwU8GvwlYMXImXmMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
3 AedrXL3aUk9zv+9t7J8ZsVYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
4 AdDdKibLKU9mv6PDAIvVvH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
5 AWdc3ZRaP09Pvw4+2FmLwHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAThe v$encryption_keys output for this environment shows that there are 3 PDBs within this CDB, all of which are TDE-enabled. In particular, the PDB used for the dSource has a con_id of 4, and an encryption key_id starting with AdDdKibL.
The vPDB tde_vpdb is provisioned to the CDB CDOMSHTG93CF on the VM tde-target19. Connecting to this database, we can again query v$encryption_keys to determine the keys in use by each PDB:
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
------ ---------------- ---------- ----------
2 PDB$SEED READ ONLY NO
3 CDOMSHTG93CFPDB1 READ WRITE NO
4 CDOMSHTG93CFPDB2 READ WRITE NO
5 CDOMSHTG93CFPDB3 READ WRITE NO
6 TDE_VPDB READ WRITE NO
SQL> select con_id, key_id from v$encryption_keys order by con_id;
CON_ID KEY_ID
------ ----------------------------------------------------
1 AZTc9eKqlk98v8GkQ8/AmaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
6 AZ3DQws5pE9LvxYDP14hDHwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAThe key which was originally present in the wallet on the dSource - AdDdKibL - is not present in the target keystore. Instead, Delphix has generated a new key - AZ3DQws5 for con_id of 6, which corresponds to the con_id of the vPDB. This key is unique to each vPDB and is not associated with the source PDB. This happens as a result of rotating the vPDB encryption keys as mentioned in the Delphix Provisioning workflow with and without TDE section.
There are several things to note about the behavior of Oracle and the v$encryption_keys table:
Keys are never deleted from existing keystores by Oracle, only new keys are added. Therefore, if we were to disable the vPDB, which will unmount and unplug it from the CDB,
v$encryption_keyswill still show the key as present, with its originalcon_id, even though it has been unplugged:CODESQL> show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ------ ---------------- ---------- ---------- 2 PDB$SEED READ ONLY NO 3 CDOMSHTG93CFPDB1 READ WRITE NO 4 CDOMSHTG93CFPDB2 READ WRITE NO 5 CDOMSHTG93CFPDB3 READ WRITE NO SQL> select con_id, key_id from v$encryption_keys order by con_id; CON_ID KEY_ID ------ ---------------------------------------------------- 1 AZTc9eKqlk98v8GkQ8/AmaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 6 AZ3DQws5pE9LvxYDP14hDHwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIf the wallet is closed for a particular PDB,
v$encryption_keyswill not show any entries for that PDB. The wallet status can be determined by queryingv$encryption_wallet.Querying
v$encryption_walletwhile the session is attached toCDB$ROOTwill return information about the entire CDB, otherwise, only the keys for the current PDB are returned.