Azure object storage setup
From the Storage setup page, select Azure Blob Storage from the Storage Type for Data dropdown menu. For specific block storage requirements, see the Storage configuration section of Deployment for Microsoft Azure.
Delphix is compatible solely with the Online access tier (hot tier of Standard GPv2) for Azure-Blob.
Access method
You can choose between Managed Identities, including System-assigned and User-assigned types, or the Access Key method. Make sure you do not have versioning on your blob/bucket, otherwise deleted space will not be reclaimed since it will be held by the versioning logic of the cloud vendor.
Managed Identities are recommended to minimize security risks and maintenance related to access keys.
Storage capacity
Enter the desired data storage capacity. Azure Blob’s billing is based on actual space used, not provisioned. Therefore, if you allocate 10TB but only use 1TB, charges will apply only to the used space. The specified number also serves as a quota to limit the blob storage from expanding beyond an unforeseen point. This quota can be modified later through the sysadmin login or Setup UI, but not reduced below the current usage by the Delphix Engine.
Container setup
Input your storage container's name and test the connection to confirm VM accessibility to the container.
Block storage for cache
Block devices are utilized as cache, reducing latencies for frequently read data and acting as temporary storage for synchronous writes before the writes are persisted to object storage. Size the cache based on your knowledge of the frequently accessed data, adding an extra 10% for overhead, or start with a cache size at 50% of all dSources to be added to the engine.
For detailed block storage requirements, see the General storage section of Deployment for Microsoft Azure.
To support the engine's throughput, set up the disks accordingly. For example, Ultra disks are recommended for their customizable high IOPS and throughput, even at smaller sizes.
Ultra disks can be configured to have high IOPS and throughput at relatively small sizes. At 300 IOPS per GiB ratio, a 256 GB volume can have 76,800 IOPS and 4000 MB/s throughput.
Premium SSD disks can have high performance, though they need to be much larger. For example, a P80 32TiB volume has 20K IOPS and 900 MB/s. The new Premium SSD v2 disks have better IOPS per GiB ratio though they are not available in all regions. SSD v2 is currently untested due to this unavailability.
Storage account and permission setup
Storage container
Create a dedicated Storage Account for your Delphix Engine by navigating to Storage Accounts and clicking Create. Remember that Azure imposes ingress and egress limitations per storage account, which are detailed in their documentation: https://docs.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account
Next, create a Storage Container within the Storage Account to be used for your Delphix Engine by navigating to the chosen Storage Account, selecting Containers under Data Storage, and clicking + Container. Note the names of the Storage Container, Storage Account, and Resource Group.
Managed role
Forge a custom role to grant access to the storage account, which will be assigned to your virtual machine post-creation. Navigate to the Resource Group containing your Storage Container and select Access Control (IAM).
Then Create a custom role with a minimum scope of the storage account that your Delphix Engine will be using:
For the JSON view, navigate to the role's JSON view, click Edit, and replace the bolded sections with your specific values. Save the changes afterwards.
Use the example below, but replace the following placeholders with your values: Your_Role_Name, Your_Subscription_ID, Your_Resource_Group_Name, Your_Storage_Account_Name.
{
"properties": {
"roleName": "<Your_Role_Name>",
"description": "Delphix object storage Azure role permissions",
"assignableScopes": [
"/subscriptions/<Your_Subscription_ID>/resourceGroups/<Your_Resource_Group_Name>/providers/Microsoft.Storage/storageAccounts/<Your_Storage_Account_Name>"
],
"permissions": [
{
"actions": [
"Microsoft.Storage/storageAccounts/blobServices/containers/read"
],
"notActions": [],
"dataActions": [
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete"
],
"notDataActions": []
}
]
}
}Virtual machine
Ensure the system-assigned Managed Identity is enabled when creating your virtual machine.
Navigate to the virtual machine and click on Identity and then Azure role assignments. After creation, assign the newly created role to the virtual machine for the necessary permissions.
Click Add role assignments. For the scope, select Storage. Select your subscription. For Resource, select the Storage Account you will be using. For the Role, select the new role you created above.
Testing permissions
To ensure that the role permissions are functioning properly, assign the role to a generic Ubuntu VM with the Azure CLI installed. Follow the Azure CLI installation guide.
Then, execute the following commands to test access to your storage container:
delphix@demo-vm:~$ az login --identity
[
{
"environmentName": "AzureCloud",
"homeTenantId": "<Tenant>",
"id": "<ID>",
"isDefault": true,
"managedByTenants": [],
"name": "Pay-As-You-Go",
"state": "Enabled",
"tenantId": "<Tenant>",
"user": {
"assignedIdentityInfo": "MSI",
"name": "systemAssignedIdentity",
"type": "servicePrincipal"
}
}
]
# List the storage containers
delphix@demo-vm:~$ az storage container list --account-name <Your_Storage_Account_Name> --output table --auth-mode login
Name Lease Status Last Modified
----------------------------------------------------------- -------------- -------------------------
<Your_Storage_Container_Name> 2022-04-28T18:57:01+00:00
# List the contents of the new empty container
delphix@demo-vm:~$ - <Your_Storage_Account_Name> --container-name <Your_Storage_Container_Name> --output table --auth-mode login
# List the contents of the storage container again
delphix@demo-vm:~$ az storage blob list --account-name <Your_Storage_Account_Name> --container-name <Your_Storage_Container_Name> --output table --auth-mode login
Name Blob Type Blob Tier Length Content Type Last Modified Snapshot
-------- ----------- ----------- -------- -------------- ------------------------- ----------
test.txt BlockBlob 15 text/plain 2022-04-28T19:36:09+00:00
# Download the storage blob
delphix@demo-vm:~$ az storage blob download --account-name <Your_Storage_Account_Name> --container-name <Your_Storage_Container_Name> --output table --auth-mode login --name test.txt --file downloaded-test.txt
Finished[#############################################################] 100.0000%
Name Blob Type Blob Tier Length Content Type Last Modified Snapshot
-------- ----------- ----------- -------- -------------- ------------------------- ----------
test.txt BlockBlob 15 text/plain 2022-04-28T19:36:09+00:00
delphix@demo-vm:~$ cat downloaded-test.txt
This is a test
# Delete the storage blob
delphix@demo-vm:~$ az storage blob delete --account-name <Your_Storage_Account_Name> --container-name <Your_Storage_Container_Name> --output table --auth-mode login --name test.txt
delphix@demo-vm:~$azstoragebloblist--account-name<Your_Storage_Account_Name>--container-name<Your_Storage_Container_Name>--outputtable--auth-modelogiKey based access
If Managed Identities are not used, static Storage Account access keys can be an alternative. Azure's comprehensive guidelines on managing Storage Account access keys, including best practices for key security and rotation, can be found in the Storage account keys manage article.