External database support

Overview

DCT uses a PostgreSQL database to store all the persistent data powering the application (engines, VDBS, compliance jobs, accounts, permissions, etc.). By default, a PostgreSQL container image is packaged along with the DCT application and deployed along other pods onto the Kubernetes cluster, storing its persistent data into the gwdatabase-data persistent volume.

Alternatively, DCT can be configured to use an external PostgreSQL database instead, to which DCT connects over TCP and can then run anywhere (typically outside of the Kubernetes cluster).

External database support is only available for Kubernetes and OpenShift deployments. Docker Compose installations are not eligible.

Requirements

Database type: PostgreSQL
Min supported version: 12.16
Max supported version: 14.10

Setup

PostgreSQL database setup

The following databases must be created prior to connecting DCT: app, jobs, data-library, bookmarks, masking, and virtualization.

masking is a newly added database for the masking service and is applicable only for release 12.0.0 and above.

virtualization is a newly added database for virtualization-app service and is applicable only for release 15.0.0 and above

The max_connections setting for database server must be greater than or equal to 200.

Quotes must be used to create or connect to the data-library databases, since its name contains a hyphen (-). Creating the database with a different name (such as data_library) will NOT work.

A PostgreSQL user must be created for DCT, with either the admin privilege or ALL privilege on the above databases. The following SQL script exemplifies the creation of required databases, granting privileges to a pre-existing dct_user (role) user.

NONE

CREATE DATABASE "bookmarks";
GRANT ALL PRIVILEGES ON DATABASE "bookmarks" TO dct_user;

CREATE DATABASE "data-library";
GRANT ALL PRIVILEGES ON DATABASE "data-library" TO dct_user;

CREATE DATABASE "jobs";
GRANT ALL PRIVILEGES ON DATABASE "jobs" TO dct_user;

CREATE DATABASE "app";
GRANT ALL PRIVILEGES ON DATABASE "app" TO dct_user;

CREATE DATABASE "masking";
GRANT ALL PRIVILEGES ON DATABASE "masking" TO dct_user;

CREATE DATABASE "virtualization";
GRANT ALL PRIVILEGES ON DATABASE "virtualization" TO dct_user;

DCT setup

Edit the values.yaml file to set useExternalDatabase to true and also provide the dbHost, dbPort, dbUser, and dbPassword, then run the following.

NONE

helm upgrade -f values.yaml dct-services delphix-dct

Previous versions of DCT required the aforementioned properties to be base64 encoded, but the values must be plain text values, as shown in the following excerpt:

CODE

useExternalDatabase: true
dbHost: "database-host.company.co"
dbPort: 5432
dbUser: "dct_user"
dbPassword: "dct_user_password"

If the password is stored as exemplified above in the values.yaml file, make sure to store this file in a secure location. Alternatively, set the value using the --set option flag in the HELM upgrade command, as shown in the following excerpt:

CODE

helm upgrade -f values.yaml --set dbPassword=dct_user_password dct-services delphix-dct

After upgrading the HELM chart, restart the pods to pick the changes:

NONE

kubectl rollout restart deployment data-library -n dct-services
kubectl rollout restart deployment data-bookmarks -n dct-services
kubectl rollout restart deployment jobs -n dct-services
kubectl rollout restart deployment gateway -n dct-services
kubectl rollout restart deployment masking -n dct-services
kubectl rollout restart deployment virtualization-app -n dct-services

Backup and recovery

When using an external database, the gwdatabase-data persistent volume (created at default by DCT) to store database data is not used. Backing up and restoring the external database is not managed by DCT. Frequent or continuous backups are required, otherwise, DCT data will be lost.

The gateway-data, masking-data, and virtualization-app-data persistent volume must still be backed up, because it contains an encryption key, which is used to encrypt sensitive data at the application-level before being sent to the database.

A backup of the external database cannot be restored successfully without a corresponding backup of gateway-data, masking-data, and virtualization-app-data, as DCT would not be able to decrypt some of the data in the database.

The encryption key in gateway-data, masking-data, and virtualization-app-data does not change after having been initially created, so backups of it do not need to be scheduled at the same time as database backups.

External database migration or upgrade

The external database can be migrated to a different host and/or upgraded to a different version at any time, as long as the version requirements above are met. If the database is unavailable for a period of time, the DCT application will temporarily fail (internal server errors on all API calls), but will recover automatically without the need for a restart. However, in case of planned maintenance, upgrade, or migration, the following procedure should be followed:

Stop (shutdown) the DCT application.
Upgrade or migrate the database.
If necessary, set the updated database properties in values.yaml, and run helm upgrade.
Start the DCT application.

DCT upgrade

Before upgrading to a new DCT version, review the documentation to identify if the external database version is compatible. If the external database version is unknown, call the metadata-database API endpoint to get the information.

NONE

curl -k --location --request GET 'https://<dct-server>/v3/management/metadata-database' \
--header 'Accept: application/json' \
--header 'Authorization: apk <api-key>'

{
  "external": false,
  "version": "14.9",
  "database_product_name": "PostgreSQL",
  "major_version": 14,
  "minor_version": 9,
  "min_supported_major_version": 12,
  "min_supported_minor_version": 16,
  "max_supported_major_version": 14,
  "max_supported_minor_version": 9,
  "compatible": true
}

If the version of the external database is not compatible with the requirements of the DCT version being upgraded to, follow the instructions in the External database migration or upgrade section above before upgrading.

The selected PostgreSQL version to upgrade to must be compatible with both the currently running DCT version and the upgrade version.

After the upgrade, verify if the external database is compatible with DCT by inspecting the compatible property of the metadata-database API endpoint (as shown above).