Synchronize cloud native secrets

11min
|
Enterprise
Vault
Video
Interactive

Deployment

Vault 1.16 includes a new feature for HashiCorp Vault Enterprise - Secrets sync. This allows you to automatically sync secrets from Vault Enterprise to a variety of third party platforms including AWS, Azure, GCP, GitHub, and Vercel.

This tutorial instructs the learner to create a static K/V secret in Vault Enterprise. Then you set up a destination and association to sync that secret between HashiCorp Vault Enterprise and AWS Secrets Manager.

Challenge

Vault Enterprise Secrets Sync addresses two common challenge to Vault adoption.

First, some services cannot communicate with Vault. Cloud providers naturally offer strong support with their own secrets management solutions. For example, AWS RDS Proxy only communicates with AWS Secrets Manager. This contributes to secrets sprawl, and being able to manage secrets in Vault Enterprise but allow access through platform specific tools is invaluable.

Second, organizations interested in leveraging the power of secrets management with Vault but access them in a platform native way. Though there are resources available, adoption of Vault requires developers to learn new technologies. Secrets Sync allows developers to focus on their preexisting platform knowledge and reduce effort an organization to adopt Vault. By focusing on cloud provider tools, they build on preexisting skills and reduce the adoption friction of new technologies.

Solution

secrets sync example

Vault Enterprise Secrets Sync automatically sync secrets between Vault and AWS Secrets Manager. Developers do not need to learn how to consume secrets from Vault. Operators continue utilizing the power of Vault Enterprise for secrets lifecycle management, but developers can consume the secrets through platform native tools and technologies.

associations and destinations

Sync destinations represent a location where to sync secrets with. There are five types of destinations: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, GitHub repository secret and Vercel Project environment variables. Multiple destinations may share a single secret. For example, a Jira token stored as a Vault secret used in CI workflows can sync to many GitHub repositories, each repository represented by a distinct Vault sync destination.

Secret associations represent a link between an existing Vault KV secret and a sync destination. Associations are granular and each secrets has a association. Many associations can share a destination.

Personas

A Vault administrator using Vault Enterprise for secrets lifecycle management but their organization requires that the secrets to be available through AWS Secrets Manager.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

To complete this tutorial you will need the following:

Vault Enterprise version 1.16 or later with valid license. Learn how to request a trial license.
Vault CLI
Terraform CLI
Amazon Web Services test account. This tutorial requires the creation of new cloud resources and but should not incur costs associated with the deployment and management of these resources. Your IAM user will need permission to create IAM User, and to create access keys for the user.
- Basic understanding of the AWS user interface, and the use of AWS Secrets Manager.
- Creating an IAM user with Terraform requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Lab set up

Clone an GitHub repository, create an IAM user with the proper policy and start an local instance of Enterprise Vault or HCP Vault to complete this lab.

Create AWS IAM user

To set up secrets sync Vault needs programmatic access to create, read, delete, update secrets in AWS Secrets Manager. In this section you will create an IAM User with least privileged access. You can create the IAM user manually or you can run the Terraform configuration included in the GitHub repository.

Create a IAM policy named vault_secrets_sync_policy with the following privileges.

{
  "Version": "2012-10-17",
  "Statement": [
     {
           "Sid": "SecretsManagerAccess",
           "Effect": "Allow",
           "Action": [
              "secretsmanager:DescribeSecret",
              "secretsmanager:GetSecretValue",
              "secretsmanager:CreateSecret",
              "secretsmanager:PutSecretValue",
              "secretsmanager:UpdateSecret",
              "secretsmanager:UpdateSecretVersionStage",
              "secretsmanager:DeleteSecret",
              "secretsmanager:RestoreSecret",
              "secretsmanager:TagResource",
              "secretsmanager:UntagResource"
           ],
           "Resource": "*"
     }
  ]
}

Create a IAM user named vault_secrets_sync_user and attach the vault_secrets_sync_policy.
Create AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the IAM user. See Managing access keys (console) for instructions.
Open a new terminal, set the VAULT_USER_ACCESS_KEY_ID and VAULT_USER_SECRET_ACCESS_KEY environment variables to the values from the IAM User vault_secrets_sync_user. If you are not running on us-east-1 please update the AWS_REGION.
```
$ export VAULT_USER_ACCESS_KEY_ID=<<IAM_USER_ACCESS_KEY>> && export VAULT_USER_SECRET_ACCESS_KEY=<<IAM_USER_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Clone the repository at learn-vault-enterprise-secrets-sync.

$ git clone https://github.com/hashicorp-education/learn-vault-enterprise-secrets-sync.git

Move into the terraform subdirectory of folder just created.
```
$ cd learn-vault-enterprise-secrets-sync/terraform
```
Working directory
This tutorial assumes that the learner executes the remainder of commands in the Terraform section in this directory.
Export your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables. This key pair should have permission to create the AWS IAM resources in the provided Terraform configuration.
If you are not running on us-east-1 please update the AWS_REGION.
```
$ export AWS_ACCESS_KEY_ID=<<YOUR_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<YOUR_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Initialize the configuration.

$ terraform init
...
Terraform has been successfully initialized!
...

Look at the plan.

$ terraform plan
...
Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:

access_key = (known after apply)
secret_key = (sensitive value) ...

Apply the configuration.

$ terraform apply -auto-approve
...

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
Outputs:
access_key = "AKIA3SCJYFFCO3FDO4P3"
secret_key = <sensitive>

In the terraform.tfstate file to locate the secret access key.

$ cat terraform.tfstate | head -n 15
{
"version": 4,
"terraform_version": "1.5.4",
"serial": 4,
"lineage": "76f54aef-b09b-647a-ab25-803ecb8f5a1f",
"outputs": {
   "access_key": {
      "value": "AKIA4V3NG3RZO3FDO4P3",
      "type": "string"
   },
   "secret_key": {
      "value": "tXMencyCLQpz1sRowsaMFGEZbsibIDJli9/Qy/E",
      "type": "string",
      "sensitive": true
   }
},

Near the top of the block just examined will find outputs block with a access_key and secret_key fields. Copy the value for each.
In the same terminal, set the VAULT_USER_ACCESS_KEY_ID and VAULT_USER_SECRET_ACCESS_KEY to the values from the IAM User vault_secrets_sync_user. If you are not running on us-east-1 please update the AWS_REGION.
```
$ export VAULT_USER_ACCESS_KEY_ID=<<IAM_USER_ACCESS_KEY>> && export VAULT_USER_SECRET_ACCESS_KEY=<<IAM_USER_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Set up Vault Enterprise

Open a new terminal.
Set your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.
```
$ export VAULT_LICENSE=02MV4UU43BK5....
```
Start a Vault development server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault development server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation
Do not run a Vault development server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Return to the other terminal session.
Export an environment variable for the Vault CLI to the address of the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the Vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

Check the status of Vault.

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.16.0+ent
Build Date      2023-09-08T17:10:08Z
Storage Type    inmem
Cluster Name    vault-cluster-42e92bf5
Cluster ID      fXXaMe47-nb77-2be6-9426-a7986s376t25
HA Enabled      false

The Vault server is ready.

Enterprise secrets sync

Create a KV secret

Create the KV secrets engine.

$ vault secrets enable -path=kv1 kv-v2
Success! Enabled the kv-v2 secrets engine at: kv1/

Create the secret.

$ vault kv put kv1/database/dev api_key="abc" key_id="123"
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T17:56:26.700919Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Configure destination and association

Configure the destination.

$ vault write sys/sync/destinations/aws-sm/aws-sm1 \
 access_key_id="${VAULT_USER_ACCESS_KEY_ID}" \
 secret_access_key="${VAULT_USER_SECRET_ACCESS_KEY}" \
 region="${AWS_REGION}" \
 secret_name_template="{{ .MountAccessor }}_{{ .SecretBaseName }}"

The output should resemble the following:

Key                   Value
---                   -----
connection_details    map[access_key_id:***** region:us-east-1 secret_access_key:*****]
name                  aws-sm1
options               map[custom_tags:map[] granularity_level:secret-path secret_name_template:{{ .MountAccessor }}_{{ .SecretBaseName }}]
type                  aws-sm

By default, the name of synced secrets follows this format: vault/<accessor>/<secret-path>, but each destination allows you to customize the name through the secret_name_template field. Refer to the Name Template documentation for details.

Set up the association.

$ vault write sys/sync/destinations/aws-sm/aws-sm1/associations/set \
   mount="kv1" \
   secret_name="database/dev"

The output should resemble the following:

Key                   Value
---                   -----
associated_secrets    map[kv_c6c5fb8c/database/dev:map[accessor:kv_c6c5fb8c secret_name:database/dev]]
store_name            aws-sm1
store_type            aws-sm

Log into your AWS Account and take a navigate to the AWS Secrets Manager.
Choose the link ending with /database/dev.
In AWS Secrets Manager choose Retrieve secret value in the Secret value section.
These are the same values as the kv secret in kv1/database/dev.

Modifications to the secret

Changes to the secret in Vault Enterprise are also reflected in AWS Secrets Manager.

Update the secret.

$ vault kv put kv1/database/dev api_key="abc" key_id="updated"
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-12T18:36:42.241359Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

Open up your AWS account and reload the secret in AWS Secrets Manager.

Use a patch command to add an last_key field.

$ vault kv patch -mount=kv1 database/dev last_key=new_one
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T18:00:49.720538Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            3

Remove associations (optional)

If you no longer want the secret to sync with AWS Secrets Manager, you can remove the association.

Remove the association between the secret and AWS Secrets Manager.

$ vault write sys/sync/destinations/aws-sm/aws-sm1/associations/remove \
   mount="kv1" \
   secret_name="database/dev"

The output should resemble the following:

Key                   Value
---                   -----
associated_secrets    map[]
store_name            aws-sm1
store_type            aws-sm

Return to your browser windows with AWS Secrets Manager and refresh the page. It should resemble the following:

Clean up

If you created the IAM user manually, delete the IAM user.
If you created the IAM User with Terraform, switch back to the original AWS credentials AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY used with Terraform to create the IAM User.

Destroy the resources Terraform created.

$ terraform destroy -auto-approve
...

Destroy complete! Resources: 3 destroyed.

Unset the environment variables used in this lab.

$ unset VAULT_USER_ACCESS_KEY \
VAULT_USER_SECRET_ACCESS_KEY \
VAULT_ADDR \
VAULT_TOKEN \

Stop the Vault server process.
```
$ pgrep -f vault | xargs kill
```
The Terraform version of this tutorial requires you to go back to the parent directory of learn-vault-enterprise-secrets-sync.
```
$ cd ../..
```

Delete the learn-vault-pki-engine directory.

$ rm -rf learn-vault-enterprise-secrets-sync

Summary

You learned how to set up Vault Enterprise secrets sync, which consists of destinations and associations. Destinations are platforms that Vault will sync a secret to. Associations create a link between a secret and sync destination. Changing a secret in Vault it will change the corresponding secret on the target platform service.

This tutorial guided you to set up a link between a secret in Vault Enterprise and AWS Secrets Manager, and demonstrated when changing secrets in Vault, the corresponding secret in AWS Secrets Manager is also changed.

References

Federated workload identities

Next Collection

Monitor & troubleshoot