External database support
Overview
This feature is not available for DCT Appliance (OVA) deployments.
DCT uses a PostgreSQL database to store all the persistent data powering the application (engines, VDBS, compliance jobs, accounts, permissions, etc.). By default, a PostgreSQL container image is packaged along with the DCT application and deployed along other pods onto the Kubernetes cluster, storing its persistent data into the gwdatabase-data
persistent volume.
Alternatively, DCT can be configured to use an external PostgreSQL database instead, to which DCT connects over TCP and can then run anywhere (typically outside of the Kubernetes cluster).
External database support is only available for Kubernetes and OpenShift deployments. Docker Compose installations are not eligible.
Requirements
Database type: PostgreSQL
Min supported version: 12.16
Max supported version: 14.10
Setup
PostgreSQL database setup
The following databases must be created prior to connecting DCT: app, jobs, data-library, bookmarks, masking, and virtualization.
masking is a newly added database for the masking service and is applicable only for release 12.0.0 and above.
virtualization is a newly added database for virtualization-app service and is applicable only for release 15.0.0 and above
The max_connections
setting for database server must be greater than or equal to 200.
Quotes must be used to create or connect to the data-library databases, since its name contains a hyphen (-). Creating the database with a different name (such as data_library) will NOT work.
A PostgreSQL user must be created for DCT, with either the admin privilege or ALL privilege on the above databases. The following SQL script exemplifies the creation of required databases, granting privileges to a pre-existing dct_user
(role) user.
CREATE DATABASE "bookmarks";
GRANT ALL PRIVILEGES ON DATABASE "bookmarks" TO dct_user;
CREATE DATABASE "data-library";
GRANT ALL PRIVILEGES ON DATABASE "data-library" TO dct_user;
CREATE DATABASE "jobs";
GRANT ALL PRIVILEGES ON DATABASE "jobs" TO dct_user;
CREATE DATABASE "app";
GRANT ALL PRIVILEGES ON DATABASE "app" TO dct_user;
CREATE DATABASE "masking";
GRANT ALL PRIVILEGES ON DATABASE "masking" TO dct_user;
CREATE DATABASE "virtualization";
GRANT ALL PRIVILEGES ON DATABASE "virtualization" TO dct_user;
DCT setup
Edit the values.yaml file to set useExternalDatabase
to true and also provide the dbHost
, dbPort
, dbUser
, and dbPassword
, then run the following.
helm upgrade -f values.yaml dct-services delphix-dct
Previous versions of DCT required the aforementioned properties to be base64 encoded, but the values must be plain text values, as shown in the following excerpt:
useExternalDatabase: true
dbHost: "database-host.company.co"
dbPort: 5432
dbUser: "dct_user"
dbPassword: "dct_user_password"
If the password is stored as exemplified above in the values.yaml file, make sure to store this file in a secure location. Alternatively, set the value using the --set
option flag in the HELM upgrade command, as shown in the following excerpt:
helm upgrade -f values.yaml --set dbPassword=dct_user_password dct-services delphix-dct
After upgrading the HELM chart, restart the pods to pick the changes:
NONEkubectl rollout restart deployment data-library -n dct-services kubectl rollout restart deployment data-bookmarks -n dct-services kubectl rollout restart deployment jobs -n dct-services kubectl rollout restart deployment gateway -n dct-services kubectl rollout restart deployment masking -n dct-services kubectl rollout restart deployment virtualization-app -n dct-services
Backup and recovery
When using an external database, the gwdatabase-data
persistent volume (created at default by DCT) to store database data is not used. Backing up and restoring the external database is not managed by DCT. Frequent or continuous backups are required, otherwise, DCT data will be lost.
The gateway-data
, masking-data
, and virtualization-app-data
persistent volume must still be backed up, because it contains an encryption key, which is used to encrypt sensitive data at the application-level before being sent to the database.
A backup of the external database cannot be restored successfully without a corresponding backup of gateway-data
, masking-data
, and virtualization-app-data
, as DCT would not be able to decrypt some of the data in the database.
The encryption key in gateway-data
, masking-data
, and virtualization-app-data
does not change after having been initially created, so backups of it do not need to be scheduled at the same time as database backups.
External database migration or upgrade
The external database can be migrated to a different host and/or upgraded to a different version at any time, as long as the version requirements above are met. If the database is unavailable for a period of time, the DCT application will temporarily fail (internal server errors on all API calls), but will recover automatically without the need for a restart. However, in case of planned maintenance, upgrade, or migration, the following procedure should be followed:
Stop (shutdown) the DCT application.
Upgrade or migrate the database.
If necessary, set the updated database properties in values.yaml, and run
helm upgrade
.Start the DCT application.
DCT upgrade
Before upgrading to a new DCT version, review the documentation to identify if the external database version is compatible. If the external database version is unknown, call the metadata-database
API endpoint to get the information.
curl -k --location --request GET 'https://<dct-server>/v3/management/metadata-database' \
--header 'Accept: application/json' \
--header 'Authorization: apk <api-key>'
{
"external": false,
"version": "14.9",
"database_product_name": "PostgreSQL",
"major_version": 14,
"minor_version": 9,
"min_supported_major_version": 12,
"min_supported_minor_version": 16,
"max_supported_major_version": 14,
"max_supported_minor_version": 9,
"compatible": true
}
If the version of the external database is not compatible with the requirements of the DCT version being upgraded to, follow the instructions in the External database migration or upgrade section above before upgrading.
The selected PostgreSQL version to upgrade to must be compatible with both the currently running DCT version and the upgrade version.
After the upgrade, verify if the external database is compatible with DCT by inspecting the compatible
property of the metadata-database
API endpoint (as shown above).