GCP object storage setup
From the Storage setup page, select GCP Object Storage from the Storage Type for Data dropdown menu.
Access method
In GCP, the ideal way of accessing object storage is from an instance using a service account. A service account in GCP is similar to an IAM role in AWS, promoting secure and low-maintenance access to the object storage.
Storage capacity
When specifying storage capacity, note that GCP charges are based on the actual storage used. Thus, if you provision for 10TB but only utilize 1TB, billing will reflect the 1TB of actual usage. Adjusting this capacity is manageable through the sysadmin login or Setup UI, but cannot be decreased below what the Delphix Continuous Data Engine is currently utilizing. The specified number also serves as a quota to limit the object storage from expanding beyond an unforeseen point.
Bucket
Enter the name of your bucket in the Bucket field and test the connection to verify the VM's access to the bucket.
Block storage for cache
Block devices such as cache are used to reduce latencies for frequently read data and as temporary storage for synchronous writes before the writes are sent to object storage. If you already know the size of the frequently accessed data, then size the cache equal to (size of frequently accessed data + Extra 10% for bookkeeping purposes), If not, start with sizing the cache to 50% of the size of all dSources that will be added to the engine.
For the detailed block storage requirements, refer to the General storage configuration section in the Deployment for GCP documentation.
Set up the disks such that they can support the throughput of the engine. SSD Persistent Disks are recommended.
Elastic data engines on GCP: Permissions
Setup
Create a service account and custom role and assign that role to a virtual machine.
Enable private Google Access on the subnet used by the VM if it is not in the public subnet
Create a virtual machine
Create a bucket
Create a compute service account
Permissions
Create a service account and provide access scoped to storage APIs or all cloud APIs. The account creating the virtual machine should have serviceAccountUser permissions on the service account so that while creating the instance this account could be assigned.
scopes = ["cloud-platform"]
The service account should have permissions to the following roles:
legacyBucketWriter
legacyObjectOwner
Here is an example of applying permissions from this service account on a bucket.
You can also create and use a custom role for providing more restrictive permissions.
Bucket Protection and Lifecycle
Any protection policy (e.g. Object versioning, retention, event-based hold, etc…) or added Lifecycle rules are not supported. If present, the connectivity test to the bucket will fail or will have an adverse effect on the engine’s data storage stability.
Testing Bucket Permissions
You can test the connection from the Delphix engine to the bucket via the Setup UI, but there is no way in the Setup UI to test the connection without starting from the beginning of setup for the engine. To test only the engine connection to a bucket or multiple buckets, use the testConnection API in the Delphix Continuous Data Engine CLI and look for the result status after executing the commit operation. If the connection test is successful, the result status will be true. Here is an example of how to use the Delphix Continuous Data Engine CLI to test the engine connection to the object storage bucket, which results in a successful connection. The engine hostname in the example is test-gcp:
$ ssh sysadmin@test-gcp
test-gcp> storage objectStorage testConnection
test-gcp objectStorage testConnection *> set type=GcpObjectStoreTest
test-gcp storage objectStorage testConnection *>
test-gcp storage objectStorage testConnection *> ls
Properties
type: GcpObjectStoreTest (*)
bucket: (required)
test-gcp storage objectStorage testConnection *> set bucket=test-gcp-1
test-gcp storage objectStorage testConnection *> commit
type: ObjectStoreTestResult
errorMessage: (unset)
result: true