Migrating a TDE-enabled vPDB

Migrating a vPDB from one container database to another container database on the same target host or a different target host involves the following steps:

Disable the vPDB.
Migrate.
For a TDE-enabled vPDB, there are additional steps that are required before the enable. The steps are necessary because the keystore and exported keyfiles are not present on Delphix storage. These steps are:
1. Copy the parent TDE keystore from the original target host to the new target host. If the vPDB is not being migrated to a new target host, then this step is not needed. Similarly, if the new host has the parent keystore in a different location, then the parent keystore path for the vPDB needs to be updated.
2. Copy the artifact directory for the vPDB from the original target host to the new target host. If the vPDB is not being migrated to a new host, then this step is not needed.
3. Ensure that the new target container has the TDE keystore password set.
4. Merge the original target keystore into the new target keystore. This is required to support a rewind operation to a snapshot taken before the migrate.
Enable.

The diagram below illustrates the scenario of a migrated TDE-enabled vPDB.

Example

Consider the following TDE-enabled vPDB:

This vPDB is currently provisioned to the container database CDOMLOTG9620 on the host tde-target18. Connecting to that database, we can see the keys for the target container and the vPDB:

CODE

SQL> show pdbs
CON_ID CON_NAME            OPEN MODE  RESTRICTED
------ ---------------------- ---------- ----------
     2 PDB$SEED            READ ONLY  NO
     3 CDOMLOTG9620PDB1        READ WRITE NO
     4 CDOMLOTG9620PDB2        READ WRITE NO
     5 CDOMLOTG9620PDB3        READ WRITE NO
     6 TDE_VPDB            READ WRITE YES
SQL> select con_id, key_id from v$encryption_keys;
CON_ID KEY_ID
------ ----------------------------------------------------
     1 ASFwmcfaMk8vv1LvzV0H8BEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     6 AdDdKibLKU9mv6PDAIvVvH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA

The key for the target container starts with ASFwmcfa and the vPDB key starts with AdDdKibL. We also have an artifact directory for this vPDB found in <toolkit_directory>/oracle_tde_keystores:

CODE

$ ls /toolkit/oracle_tde_keystores/
tde_vpdb_Encrypted_e9d2befb-9849-43c8-85f5-5ee8b760e334

We are going to migrate this vPDB to the container database CDOMLOTGCAE8 located on the host tde-target18b. Currently, that container has the following configuration:

CODE

SQL> show pdbs
CON_ID CON_NAME            OPEN MODE  RESTRICTED
------ ---------------------- ---------- ----------
     2 PDB$SEED            READ ONLY  NO
     3 CDOMLOTGCAE8PDB1        READ WRITE NO
     4 CDOMLOTGCAE8PDB2        READ WRITE NO
     5 CDOMLOTGCAE8PDB3        READ WRITE NO
SQL> select con_id, key_id from v$encryption_keys order by con_id;
CON_ID KEY_ID
------ ----------------------------------------------------
     1 Ad344VSUDk/jv1VDb1QNBHMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     3 AetaL+IKpE+ov5avXouApwUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     4 AXZTgaBbxk+6v6x2yHQ4ArgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     5 AayTgbjkJE/3v82/hBBBAWEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

The new target container has a CDB key starting with Ad344VSU. The migration steps are as follows:

Disable the vPDB in the original container on the host tde-target18a.
Migrate the vPDB to point to the new container on the new target host tde-target18b. The UI will report the following warnings during the migrate operation, which indicate the manual steps which need to be done:
Set the TDE keystore password for the new target container CDOMLOTGCAE8 on the new target host tde-target18b.
Copy the parent keystore to the new target host tde-target18b. In this case, we will keep the existing location /u01/app/oracle/product/18.11.0.0/dbhome_1/admin/CDOMSHSR52CA/wallet on both hosts, so the configuration does not need to be updated.
Copy the TDE artifact directory /toolkit/oracle_tde_keystores/tde_vpdb_Encrypted_e9d2befb-9849-43c8-85f5-5ee8b760e334 to the new target host tde-target18b. If there are no TDE-enabled vPDBs on the new target host, the oracle_tde_keystores directory may need to be created first.
1. On the new target host, check if /toolkit/oracle_tde_keystores exists. Create it, if not. While creating the directory, you should make sure the group ownership is set to oinstall and the directory's permissions are set to 0770. For example:
  CODE
```
mkdir -p /toolkit/oracle_tde_keystores 
chgrp oinstall /toolkit/oracle_tde_keystores
chmod 0770 /toolkit/oracle_tde_keystores
```
2. Copy the TDE artifact directory from the old target host to the new target host, preserving file permissions. For example, if using scp on the new target host:
  CODE
```
scp -rp <old_target_hostname>:/toolkit/oracle_tde_keystores/tde_vpdb_Encrypted_e9d2befb-9849-43c8-85f5-5ee8b760e334 /toolkit/oracle_tde_keystores/
```
3. If the copied files are now owned by a user that is not the Oracle home owner, add group-write permissions to key export files to prevent key imports from failing with ORA-46646: file from which keys are to be imported is invalid. For example:
  CODE
```
find /toolkit/oracle_tde_keystores/tde_vpdb_Encrypted_e9d2befb-9849-43c8-85f5-5ee8b760e334 -name "*.keys" -exec chmod g+w {} \;
```
Merge the original target keystore into the new target keystore. Oracle provides the ADMINISTER KEY MANAGEMENT MERGE KEYSTORE command to facilitate this. Note that you cannot successfully merge into a keystore that is currently in use by the database, so the recommended process is to first copy the original target keystore to the new host, merge the two keystores into a new keystore, replace the existing new target keystore and finally bounce the CDB to start using the merged keystore:
1. Copy the original keystore to /tmp/tde-target18a on tde-target18b.
2. Create a temporary directory /tmp/merged on tde-target18b.
3. While connected to the new container on tde-target18b, issue the merge keystore command:
  SQL
```
SQL> administer key management merge keystore '/tmp/tde-target18a' identified by
  *** and keystore '/u01/app/oracle/product/18.11.0.0/dbhome_1/admin/CDOMLOTGCAE8/wallet'
  identified by *** into new keystore '/tmp/merged' identified by ***;
```
4. Copy the new merged keystore /tmp/merged/ewallet.p12 into the existing keystore location on tde-target18b (backing up the existing keystore first).
5. If there is an autologin wallet configured for the container, it must be recreated:
  SQL
```
SQL> administer key management create auto_login keystore from keystore
 '/tmp/merged' identified by ***;
```
  Copy the new merged autologin keystore /tmp/merged/cwallet.ssointo the existing keystore location on tde-target18b (backing up the existing autologin keystore first).
6. Shutdown and restart the new target container database on the host tde-target18b.

Confirm that the keystore now contains both the original key and the new key:

CODE

SQL> select key_id from v$encryption_keys where con_id = 1;
KEY_ID
----------------------------------------------------
ASFwmcfaMk8vv1LvzV0H8BEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Ad344VSUDk/jv1VDb1QNBHMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Note that we have the key starting with ASFwmcfa from the original container, and the key starting with Ad344VSU from the new container.

Enable the vPDB to complete the migration operation. We can now see that the vPDB is successfully started in the new container, with its key starting with AdDdKibL:

CODE

SQL> show pdbs
CON_ID CON_NAME   		   OPEN MODE  RESTRICTED
------ ------------------------------ ---------- ----------
     2 PDB$SEED   		   READ ONLY  NO
     3 CDOMLOTGCAE8PDB1   	   READ WRITE NO
     4 CDOMLOTGCAE8PDB2   	   READ WRITE NO
     5 CDOMLOTGCAE8PDB3   	   READ WRITE NO
     6 TDE_VPDB   		   READ WRITE NO
SQL> select con_id, key_id from v$encryption_keys order by con_id;
CON_ID KEY_ID
------ ---------------------------------------------
     1 Ad344VSUDk/jv1VDb1QNBHMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     1 ASFwmcfaMk8vv1LvzV0H8BEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     3 AetaL+IKpE+ov5avXouApwUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     4 AXZTgaBbxk+6v6x2yHQ4ArgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     5 AayTgbjkJE//v82/hBBBAWEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
     6 AdDdKibLKU9mv6PDAIvVvH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
6 rows selected.