Config file driven multiple AWS Databricks workspaces deployment [Issue 187] #188

hwang-db · 2025-04-24T11:47:22Z

Revamping this example of modular AWS DB workspace deployment - we can now define multiple environments with yaml files to deploy m workspaces into n VPCs and create multiple catalogs with isolated underlying infra. All the workspaces are using backend PL and CMK feature; the VPCs are configured with NAT for egress.

nkvuong

half way through the review - some minor changes suggested

nkvuong · 2025-05-07T04:55:50Z

examples/aws-databricks-modular-privatelink/.terraform.lock.hcl

should keep this file, but make sure we update the providers to the latest versions - https://developer.hashicorp.com/terraform/language/files/dependency-lock

nkvuong · 2025-05-07T04:58:58Z

examples/aws-databricks-modular-privatelink/README.md

-  }
-}
-```
+You can separate out these 2 pipelines into different projects, instead of keeping everything in the same repo folder.


Suggested change

You can separate out these 2 pipelines into different projects, instead of keeping everything in the same repo folder.

You should separate out these 2 pipelines into different projects, instead of keeping everything in the same repo folder.

nkvuong · 2025-05-07T04:59:16Z

examples/aws-databricks-modular-privatelink/README.md

+You can separate out these 2 pipelines into different projects, instead of keeping everything in the same repo folder.

-Since we are using CMK (customer managed key) for encryption on root S3 bucket and Databricks managed resources, you also need to provide an AWS IAM ARN for `cmk_admin`. The format will be: `arn:aws:iam::123456:user/xxx`. You need to create this user and assign KMS admin role to it.
+> Step 1.1: Manually create a service principal with account admin role on Account Console, generate client secret; and note down client_id and client_secret values.


Suggested change

> Step 1.1: Manually create a service principal with account admin role on Account Console, generate client secret; and note down client_id and client_secret values.

> Step 1.1: Manually create a service principal with account admin role on Account Console, generate client secret; and note down `client_id` and `client_secret` values.

nkvuong · 2025-05-07T05:00:02Z

examples/aws-databricks-modular-privatelink/README.md

-    workspace_3 = var.workspace_3_config
-}
+export TF_VAR_aws_account_id=xxxx
+export AWS_ACCESS_KEY_ID=your_aws_role_access_key_id


let's avoid this, tf should just use awscli profile to auth

nkvuong · 2025-05-07T05:00:52Z

examples/aws-databricks-modular-privatelink/README.md

-In the default setting, this template creates one VPC (with one public subnet and one private subnet for hosting VPCEs). Each incoming workspace will add 2 private subnets into this VPC. If you need to create multiple VPCs, you should copy paste the VPC configs and change accordingly, or you can wrap VPC configs into a module, we leave this to you. 
-
-At this step, your workspaces deployment and VPC networking infra should have been successfully deployed and you will have `n` config json files for `n` workspaces deployed, under `/artifacts` folder, to be used in another Terraform project to deploy workspace objects including IP Access List. 
+Then in root level `variables.tf`, change the region default value to your region.


why don't you add this to the yml config as well?

nkvuong · 2025-05-07T10:15:24Z

examples/aws-databricks-modular-privatelink/configs/config-1.yaml

@@ -0,0 +1,43 @@
+# VPC Configuration


should start with three dashes

Suggested change

# VPC Configuration

---

# VPC Configuration

nkvuong · 2025-05-07T10:17:07Z

...ular-privatelink/databricks_account_tf_pipeline/modules/account_log_delivery/log_delivery.tf

@@ -0,0 +1,107 @@
+terraform {


do we still suggest customer to configure log delivery? system tables should replace this now as parsing the audit logs is quite time-consuming

nkvuong · 2025-05-07T10:18:43Z

examples/aws-databricks-modular-privatelink/configs/config-2.yaml

+deploy_metastore: "false"
+existing_metastore_id: "xxxxx-xxxx-xxxx-xxxx-xxxxxx"


we should combine these into a single flag existing_metastore_id

nkvuong · 2025-05-07T10:19:20Z

...ws-databricks-modular-privatelink/databricks_account_tf_pipeline/modules/aws_network/main.tf

+}
+
+module "vpc" {
+  source = "terraform-aws-modules/vpc/aws"


can we pin the module version?

nkvuong · 2025-05-07T10:20:25Z

...icks-modular-privatelink/databricks_account_tf_pipeline/modules/workspace_credential/main.tf

+}
+
+
+resource "aws_iam_role_policy" "cross_account" {


I thought we already have data.databricks_aws_crossaccount_policy

Copilot

Pull Request Overview

This PR revamps the AWS Databricks modular privatelink example to implement a config file-driven deployment approach. The changes enable deploying multiple Databricks workspaces across multiple VPCs using YAML configuration files, with support for Unity Catalog catalogs and isolated infrastructure.

Complete restructuring to use YAML configuration files instead of hardcoded Terraform variables
New modular architecture with account-level and workspace-level pipelines
Enhanced Unity Catalog support with isolated S3 buckets and external locations

Reviewed Changes

Copilot reviewed 59 out of 69 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
main.tf	Replaced hardcoded workspace configs with YAML-driven configuration parsing
variables.tf	Simplified to core variables, removed workspace-specific config variables
providers.tf	Updated provider versions and added time provider
outputs.tf	Modified to output workspace URLs from new module structure
identity_management.tf	Added comprehensive user and group management using YAML config
databricks_account_tf_pipeline/	New module for account-level resource deployment including workspaces and metastore
databricks_workspace_tf_pipeline/	New module for workspace-level resources including UC catalogs
configs/	New YAML configuration files for environment and user management
README.md	Updated documentation reflecting the new config-driven approach

Files not reviewed (1)

examples/aws-databricks-modular-privatelink/.terraform.lock.hcl: Language not supported

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-09T09:10:12Z

examples/aws-databricks-modular-privatelink/databricks_account_tf_pipeline/cmk.tf

    principals {
      type        = "AWS"
-      identifiers = [var.cmk_admin]
+      identifiers = [data.aws_caller_identity.current.account_id]


The account ID should be formatted as an ARN. This should be arn:aws:iam::${data.aws_caller_identity.current.account_id}:root to properly identify the root user of the account.

Suggested change

identifiers = [data.aws_caller_identity.current.account_id]

identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]

Copilot · 2025-10-09T09:10:12Z

examples/aws-databricks-modular-privatelink/databricks_account_tf_pipeline/cmk.tf

    principals {
      type        = "AWS"
-      identifiers = [var.cmk_admin]
+      identifiers = [data.aws_caller_identity.current.account_id]


The account ID should be formatted as an ARN. This should be arn:aws:iam::${data.aws_caller_identity.current.account_id}:root to properly identify the root user of the account.

Suggested change

identifiers = [data.aws_caller_identity.current.account_id]

identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]

hwang-db added 3 commits April 24, 2025 19:37

overhaul changes to multiple aws databricks workspaces

53ffba0

fix doc

3717da1

fix doc

96f514b

hwang-db requested review from a team as code owners April 24, 2025 11:47

hwang-db added 2 commits April 24, 2025 19:51

fix doc

2debcf4

fix doc

350591d

nkvuong suggested changes May 7, 2025

View reviewed changes

alexott requested review from a team and Copilot October 9, 2025 09:09

Copilot AI reviewed Oct 9, 2025

View reviewed changes

	You can separate out these 2 pipelines into different projects, instead of keeping everything in the same repo folder.
	You should separate out these 2 pipelines into different projects, instead of keeping everything in the same repo folder.

	> Step 1.1: Manually create a service principal with account admin role on Account Console, generate client secret; and note down client_id and client_secret values.
	> Step 1.1: Manually create a service principal with account admin role on Account Console, generate client secret; and note down `client_id` and `client_secret` values.

		deploy_metastore: "false"
		existing_metastore_id: "xxxxx-xxxx-xxxx-xxxx-xxxxxx"

	identifiers = [data.aws_caller_identity.current.account_id]
	identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]

Config file driven multiple AWS Databricks workspaces deployment [Issue 187] #188

Are you sure you want to change the base?

Config file driven multiple AWS Databricks workspaces deployment [Issue 187] #188

Uh oh!

Conversation

hwang-db commented Apr 24, 2025

Uh oh!

nkvuong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants