DCT Concepts

Introduction

Data Control Tower provides new and novel approaches to general Delphix workflows to deliver a streamlined developer experience. This section will introduce new concepts to Delphix that Data Control Tower provides.

Concepts

VDB Groups

VDB Groups are a new concept to Delphix that enables the association of one or more VDBs as a single VDB Group. This allows for bulk operations to be performed on the grouped VDBs such as bookmark, provision, refresh, rewind, etc. This is particularly useful for complex application testing scenarios (e.g. integration and functional testing) that require multiple data sources to properly complete testing.

With VDB Groups, developers can now maintain data synchronicity between all grouped vdbs, which is particularly useful for complex timeflow operations. For example, updating VDBs to reflect a series of schema changes across data sources or to reflect an interesting event in all grouped datasets. In order to maintain synchronicity among grouped datasets, timeflow operations (refresh, rewind, etc.) must use a bookmark reference.

Bookmarks and vdb-groups are related loosely related: a vdb-group can exist in the absence of any bookmarks, and a bookmark can exist without any vdb-group. It is important to note that the bookmark represents data, while the vdb-group represents the the databases to make this data available.

Comparing Self Service Containers to VDB Groups

As mentioned above, VDB Groups are a crucial DCT concept that enables Self Service functionality outside of the Self Service application. One can consider VDB Groups to act similarly to Self Service containers in that it provides grouping and synchronization among vdbs, but VDB Groups can provide a more flexible approach for users. For example:

• The same vdb can be included in multiple vdb-groups
• Including a vdb in a vdb-group does not prevent operations on the vdb individually
• vdbs can be added to or removed from vdb-groups
• vdb-groups do not have their own timeline

DCT Bookmarks

DCT Bookmarks are a new concept that represents a human-readable snapshot reference that is maintained within DCT. This is not to be confused with Self Service bookmarks that are maintained separately within the Self Service application. With DCT Bookmarks, Developers can now reference meaningful data (e.g. capturing a schema version reference to pair with an associated code version, capturing test failure data so that developers can reproduce the error in a developer environment, etc.) and use those references for any number of use cases (e.g. versioning data as code, quickly provisioning a break/fix environment with relevant data, etc.). DCT Bookmarks are compatible with both VDBs and VDB Groups and can be used as a reference for common timeflow operations such as:

• Provisioning a VDB or VDB Group from a bookmark
• Refreshing a VDB or VDB Group to a bookmark
• Rewinding a VDB or VDB Group to a bookmark
• Share (Refresh a VDB or VDB Group from a compatible sibling VDB or VDB Groups’ bookmark) bookmarks with a compatible testing environment

DCT Jobs

Jobs in DCT are the primary means of providing operation feedback (PENDING, STARTED, TIMEDOUT, RUNNING, CANCELED, FAILED, SUSPENDED, WAITING, COMPLETED, ABANDONED) for top level operations run on DCT. Top level operations represent the parent operation that may have one or more child-based jobs (e.g. refreshing a VDB Group is the parent job to all of the individual refresh jobs for the grouped vdbs under the VDB Group reference).

Tags

DCT Tags enable a new business metadata layer for users and consumers to filter, sort, and identify common Delphix objects, to power any number of business-driven workflows. A tag is comprised of a (Key:Value) pair that associates business-level data (e.g. location, application, owner, etc.) with supported objects. The DCT 2.0 release supports the following Tags:

• Continuous Data Engines
• Environments
• dSources
• VDBs

Developers and Administrators add and remove tags using tag-specific object endpoints (e.g. /vdbs/{vdbId}/tags) and can leverage tags as search criteria when using the object-specific search endpoints (i.e. using filtering language to narrow results).

A few sample tag-based use cases include:

• Refreshing all the VDBs owned by a specific App Team using an “Application: Payment Processing” tag. This would be accomplished by querying “what VDBs have the (Application: Payment Processing) tag" and feeding those VDB IDs into the refresh endpoint.
• Driving Accountability for VDB ownership by tagging primary and secondary owners for each VDB (e.g. (primary_owner: John Smith), (secondary_owner: Jane Brown)). That way, if a VDB is overdue for a refresh, tracking down an owner is a simple tag query.

Tag-Based Filtering

All taggable objects support tag-based filtering for API queries that adhere to the search standards documented in API References. A few examples of how tag-based filtering can be used are as follows:

List all VDBs of type Oracle, of which ip address contains the “10.1.100” string and which have been tagged with the “app-dev-1” team tag:

database_type EQ 'Oracle' AND ip_address CONTAINS '10.1.100' and tags CONTAINS { key EQ 'team' AND value EQ 'app-dev-1'}

Nuances

Stateful APIs

All applicable DCT APIs are stateful so that running complex queries against a large Delphix deployment can be done rapidly and efficiently. DCT accomplishes this by periodically gathering and hosting telemetry-based Delphix metadata from each engine.

Local Data Availability

DCT currently relies on existing Continuous Data and Compliance constructs around data-environment-engine relationships. This means that DCT operations require VDBs to live on the engine where the parent dSource lives and so on.

Engine-to-DCT API mapping

Wherever possible, DCT has looked to provide an easier-to-consume developer experience. This means that in some cases, an API on DCT could have an identical API on an engine. However, there are many instances of providing a higher level abstraction for ease of consumption - one example is the data inventory APIs on DCT (sources, dSources, vdbs), which are a simplified representation of data represented by the source, sourceconfig, and reporsitory endpoints on the local engine (source, dsource, and vdb detail are all combined under those three endpoints).

Local references to Global UUIDs

In order to avoid collision of identically-named and referenced objects, DCT generates Universally Unique IDentifiers (UUID) for all objects. For existing objects on engines like dSources and VDBs, DCT will concatenate the local engine reference with the engine UUID (e.g. “Oracle-1” on engine “3cec810a-ee0f-11ec-8ea0-0242ac120002” will be represented as “Oracle-1-3cec810a-ee0f-11ec-8ea0-0242ac120002” on DCT).

Environment Representations

Environments within Delphix serve as a reference for the combination of a host and instance. This is coupled with the fact that environments can be leveraged by multiple engines at the same time and that engines often have a specific context to some of the elements that comprise an environment. For example, an environment could have both an Oracle and ASE instance installed and that Engine A leverages an Oracle-based workflow and Engine B leverages an ASE workflow. DCT will create two identifiers to represent the specific host and instance combinations. Thus, in DCT, Engine A will be connected to a different uniquely identified Environment than Engine B.

As mentioned earlier with Engine-to-DCT API mapping, DCT aims to simplify the user experience with Delphix APIs by combining different continuous data endpoints into a simplified DCT API. The Environment API does this by combining environment, repository, and host endpoints so that writing queries against Delphix data is a much simpler process. One example would be identifying all environments that have a compatible Oracle home for provisioning:

repositories CONTAINS { database_type EQ 'Oracle' and allow_provisioning EQ true AND version CONTAINS '19.2.3'}

Supported Data Sources/Configurations

DCT is currently compatible with the most common data sources and configurations supported by Delphix, which includes:

• Oracle Single Instance
• Oracle Clusters
• SQL Server Single Instance
• SQL Server Cluster (Availability Groups)
• ASE Single Instance
• ASE Clusters

Process Feedback

Whenever a DCT request completes, it will return a JOB ID as its response. This Job ID can be used in conjunction with the jobs endpoint to query the operation status.

API Metering Instructions

DCT employs a per API consumption model, which requires API metering and periodic reporting to Delphix Customer Success. To support reporting of API consumption, DCT offers an API consumption reporting endpoint: “api-usage-report”. This report will provide a list of all unique API endpoints and how often they were used over the specified time period sorted by API and method.

Required Inputs:

• File Type: CSV or JSON (CSV file types are compatible with most spreadsheet-style software like excel or google sheets)
• Start/End Date (default start date is “when DCT was installed” and default end date is the “time when the report was generated”)

Example cURL call:

curl --location --request GET 'https://[Inser_DCT_Server]/v2/reporting/api-usage-report/?end_date=2022-06-14T09:00-04:00&start_date=2022-06-01T00:00Z' \
--header 'Content-Type: application/json' \
--header 'Accept: text/csv' \
--header 'Authorization: apk  1.xxxxxxxx'

Example Output:

api_endpoint,api_method,api_count
"/v2/management/api-clients",GET,2
/v2/management/engines,GET,1
"/v2/management/engines/search",POST,1
"/v2/reporting/api-usage-report",GET,2