Follow

Follow
How get started with Hashicorp Vault๐Ÿ›ก๏ธ

How get started with Hashicorp Vault๐Ÿ›ก๏ธ

Learn the whole HashiCorp Vault in a single blog

Dipankar Das's photo
Dipankar Das
ยทApr 5, 2023ยท

8 min read

Play this article

Table of contents

Introduction

It is a tool for managing secrets and sensitive data in modern computing environments. Especially in a dynamic environment. Which can be accessed using API, CLI as well as web app also if you don't want to manage the infrastructure on your own then you can use the SaaS offering by HashiCorp

Why?

In earlier app development we have to manually update the Secret / API Tokens which means human intervention for Key Rotation and updating the App Source Code's configurations. But, using the vault the app reaches out to the vault for fetching the token by providing valid authentication, and then the token will be used for authorization purposes, getting Required Secret Keys, and more. Tokens are frequency rotated based on max TTL(Time To Live) (i.e. Dynamic secrets ๐Ÿ˜‰ ) hence taking out the logic for managing the key rotation out of apps hence better security management and lowering the development overhead for adding security functionality

How to Install

There are various methods:

  • binary

  • package manager

If you have brew

brew tap hashicorp/tap
brew install hashicorp/tap/vault

๐Ÿ‘‰ Download Link

How to start the vault in dev mode โ›“๏ธ

Running development/testing mode

vault server -dev

then you need to set the required export for vault_addr and login as a root user using ROOT token. and check the status of the vault cluster which says the following:

  • the Cluster is unsealed (seal -> cluster is running it requires the keys to decrypt the master key which in itself decrypts the backend storage)

  • backend storage type is inmem which is in-memory

  • seal type is shamir

Note: ROOT TOKEN must be saved in highly secured environment

Now let's do it in prod environment ๐Ÿญ

vault server -config=config.hcl
# here the config.hcl will contain the required config for vault to run in production mode

Vault config (Example)๐Ÿ“ƒ

# for vault config -> /etc/vault.d/vault.hcl
storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
  token   = "XXXXXX-yyyy-zzz-aaaa-BBBBBB"
}

listener "tcp" {
 address = "0.0.0.0:8200"
 cluster_address = "0.0.0.0:8201"
 tls_disable = 0
 tls_cert_file = "/etc/vault.d/client.pem"
 tls_key_file = "/etc/vault.d/cert.key"
 tls_disable_client_certs = "true"
}

seal "awskms" {
  region = "<Region of KMS>"
  kms_key_id = "<AWS KMS key ID>",
  endpoint = "example.kms.<Region of KMS>.vpce.amazonaws.com"
}

api_addr = "https://vault-us-east-1.example.com:8200"
cluster_addr = " https://node-a-us-east-1.example.com:8201"
cluster_name = "vault-prod-us-east-1"
ui = true
log_level = "INFO"
license_path = "/opt/vault/vault.hcl"
disable_mlock=true

Key points

  • Backend storage - consul which is another storage which is a product of Hashicorp, and various field can be checked out ๐Ÿ‘‰ Docs link

  • Listener method - tcp means tcp connection as Transport Layer protocol ๐Ÿ‘‰ Docs link

  • Seal - we are using awskms for automatic unseal of the vault cluster ๐Ÿ‘‰ Docs link

  • UI - to set whether we want UI interface for the cluster

  • and more option ๐Ÿ‘‰ Docs link

What is AutoSeal, Why do we need and how to config? ๐Ÿ“

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand.

Unseal with Shamir's Secret Sharing

As, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

Instead of using shared keys are replaced by the vault transit key in a ault secrets engineor instead of creating another vault cluster for maintaining this process we can make some trusted cloud environment handle it for us like awskms

Unseal with Transit

for instance, when awskms is configured, the vault cluster during init stage creates and transfers data between kms and the cluster to store the vault keys. After any further shutdown or restart of cluster, it will reach out the awskms for unseal of cluster. It is an automatic process

seal "awskms" {
  #......
}
# Add this block to /etc/vault.d/vault.hcl

For more info on this topic ๐Ÿ‘‰ Refer Docs link

What is Backend, Why do we need and how to config? ๐Ÿ“

In vault language storage backend is untrusted storage where only encrypted data is stored and all the logic of encrypt and decrypt are inside a confined layer inside Vault cluster, whenever the data goes out of cluster it's always encrypted. Logging is given more priority than Performace as it's more important to log the event for a later security audit than to log after the data is processed. If the event is unable to be logged, vault does not process the event and throws an error

So in a nutshell, the storage backend is the location for the durable storage of Vault's information. When selecting which backend to use, remember each backend has pros, cons, advantages, and trade-offs.

For more info on this topic ๐Ÿ‘‰ Refer Docs link

Let's play around and learn ๐Ÿ

Let's get started with configuring the vault server

Prerequisites

  • Vault binary

  • root access to the server

  • AWS account

  • server is authenticated to AWS cloud

Step 1: Setup AWS Credentials ๐Ÿ›ถ

Install the credentials using AWS CLI or directly create a file

cat <<EOF > ~/.aws/credentials
[default]
aws_access_key_id = AABBCCXyz
aws_secret_access_key = Abdcdcdiweif43323

Now the AWS credential part is ready!

Step 2: Make the installed vault package to start automatically by systemd ๐Ÿšค

let's move forward and add systemd file to make automatic startup of vault binary on startup

sudo su
cat << EOF > /usr/lib/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl  # important
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
User=root # important
Group=root # important
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl # important
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity # important

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload

As now the startup script is ready, we can now config the vault server settings

Step 3: Create AWS S3 bucket for storage of the vault ๐Ÿ›ฅ๏ธ

We are using cloud-based storage as it is highly available. No data loss

  • with the name "<Bucket_Name>" or make it something unique as it will be our storage with default settings

  • Make sure you allow all public access

    NOTE: only for demonstration purpose

  • Link for how to create ๐Ÿ‘‰ S3 bucket

  • Once created the bucket it will be empty

  • copy the bucket name

Step 4: Create a key in AWS KMS for AutoSeal โ›ด๏ธ

It will help a lot when you vault server/cluster restarts, or you start it after shutdown it will automatically unseal it

  • Go to AWS KMS dashboard

  • Click on the customer-managed key

  • Select the checkbox to enable a specific IAM user to have permission to delete the KMS key

  • Save the KMS ID and region where its created

Step 5: Create an Endpoint in VPC (Regional based service) to access the key(s) ๐Ÿšข

As the endpoint is created we can now be able to use the KMS service using that VPC

Note: Make sure KMS and endpoint are in the same region

Step 6: vault configuration ๐Ÿ›ณ๏ธ

cat <<EOF > /etc/vault.d/vault.hcl
# Full configuration options can be found at https://www.vaultproject.io/docs/configuration

ui = true

#mlock = true
#disable_mlock = true

disable_mlock = true
#storage "file" {
#  path = "/home/dipankar/vault/data"
#}

storage "s3" {
  bucket = "<Bucket_Name>"
}

# HTTP listener
#listener "tcp" {
#  address = "127.0.0.1:8200"
#  tls_disable = 1
#}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_disable = 1  # no TLS i.e. no HTTPS
  tls_cert_file = "/etc/vault.d/vault.crt"
  tls_key_file  = "/etc/vault.d/vault.key"
  tls_disable_client_certs = "true"
}

seal "awskms" {
  region = "<KMS_KEY_REGION>"
  kms_key_id = "<KMS_KEY_ID>"
  endpoint = "kms.<KMS_KEY_REGION>.amazonaws.com"
}
EOF

Let's bootup the vault ๐Ÿคž

sudo systemctl start vault
sudo systemctl enable vault
sudo systemctl stop vault # to stop the vault server

# vault is uninitialized and sealed
vault status

# lets initialize the vault server
vault operator init # it will output the root login token(KEEP IT SAVED)

# so now the vault must be initialized and unsealed
vault status

# login
vault login <ROOT_TOKEN>

# In order to use login method other than token and it is
# userpass then we first need to enable the auth method
vault auth enable userpass

# now we need to specify the nwe username and password
vault write auth/userpass/users/<username> password=<password>

# Lets add a new secret
vault secrets enable kv

# In next section we will enable specific auth method and policies

login into specific userpass user

Created the secret key and how to access them

Turn on Specific auth method and policies

# for restrictive access for the user to kv store only
cat <<EOF > user-policy.hcl
path "kv/*" {
  capabilities = ["list", "read", "update", "delete"]
}

path "kv" {
  capabilities = ["list", "delete"]
}
EOF

vault policy write user-dipankar user-policy.hcl
vault write auth/userpass/users/dipankar policy=user-dipankar password=1234
vault secrets enable kv

the above policy user-dipankar will allow user who has this policy attached to read, update, list, and delete permissions in the path kv/......

but you may ask them to then allow kv path its because to query the secret the user must have permission to access kv path i.e. able to see kv option in UI and then can navigate to other subdirectories

For more info on CLI commands, do refer ๐Ÿ‘‰ CLI Docs link

End remarks ๐Ÿช‚

I hope you liked this blog. If any mistake, do comment down below ๐Ÿš€.

Do like and share this blogโ™ฅ๏ธ

Here are my socials:

Twitter https://twitter.com/DipankarDas011

Linkedin https://www.linkedin.com/in/dipankar-das-1324b6206/

Follow Kubesimplify on Hashnode, Twitter and LinkedIn. Join our Discord server to learn with us.

Did you find this article valuable?

Support Kubesimplify by becoming a sponsor. Any amount is appreciated!

See recent sponsors |ย Learn more about Hashnode Sponsors
ย 
Share this