Nomad constraint "${attr.vault.version} version >= 0.6.1" to access vault - token

Question

I'm trying to deploy a Nomad job which has a template that fetches some secrets from a Vault.
My problem is that it keeps on giving this Placement Failure because of a constraint which I can't understand why:
Constraint ${attr.vault.version} version >= 0.6.1 filtered 1 node
Nomad config
datacenter = "dc1"
data_dir = "/var/lib/nomad"
advertise {
# Defaults to the first private IP address.
http = "10.134.43.195"
rpc = "10.134.43.195"
serf = "10.134.43.195"
}
server {
enabled = true
bootstrap_expect = 1
server_join {
retry_join = ["provider=digitalocean api_token=[SECRET] tag_name=nomad_auto_join"],
}
}
client {
enabled = true
}
# Consul is installed locally and clustered
consul {
address = "http://127.0.0.1:8500"
server_auto_join = true
client_auto_join = true
auto_advertise = true
}
vault {
enabled = true
address = "http://vault.service.consul:8200"
token = "[NOMAD_VAULT_TOKEN]"
create_from_role = "nomad-cluster"
task_token_ttl = "1h"
}
autopilot {
cleanup_dead_servers = true
last_contact_threshold = "200ms"
max_trailing_logs = 250
server_stabilization_time = "10s"
enable_redundancy_zones = false
disable_upgrade_migration = false
enable_custom_upgrades = false
}
telemetry {
publish_allocation_metrics = true
publish_node_metrics = true
prometheus_metrics = true
}
### Nomad Token to access Vault
NOMAD_VAULT_TOKEN is generated with this command:
vault token create -policy nomad-server -period 72h -orphan
Vault policy nomad-server
nomad-server vault policy is as such:
# Allow creating tokens under "nomad-cluster" token role.Z
path "auth/token/create/nomad-cluster" {
capabilities = ["update"]
}
# Allow looking up "nomad-cluster" token role.
path "auth/token/roles/nomad-cluster" {
capabilities = ["read"]
}
# Allow looking up the token passed to Nomad to validate # the token has the
# proper capabilities. This is provided by the "default" policy.
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow looking up incoming tokens to validate they have permissions to access
# the tokens they are requesting. This is only required if
# `allow_unauthenticated` is set to false.
path "auth/token/lookup" {
capabilities = ["update"]
}
# Allow revoking tokens that should no longer exist. This allows revoking
# tokens for dead tasks.
path "auth/token/revoke-accessor" {
capabilities = ["update"]
}
# Allow checking the capabilities of our own token. This is used to validate the
# token upon startup.
path "sys/capabilities-self" {
capabilities = ["update"]
}
# Allow our own token to be renewed.
path "auth/token/renew-self" {
capabilities = ["update"]
}
# This is where needed secretes are fetched from
path "kv/*" {
capabilities = ["update", "read", "create"]
}
Nomad job definition
My Nomad job definition is:
job "api" {
datacenters = ["dc1"]
type = "service"
group "api" {
count = 1
update {
max_parallel = 1
min_healthy_time = "30s"
healthy_deadline = "10m"
progress_deadline = "11m"
auto_revert = true
}
task "api" {
driver = "docker"
config {
image = "registry.gitlab.com/[GROUP]/[PROJECT]/${ENVIRONMENT}:${BUILD_NUMBER}"
port_map {
nginx = 80
}
auth {
username = "${REGISTRY_USER}"
password = "${REGISTRY_PASS}"
}
force_pull = true
hostname = "api"
}
vault {
policies = ["kv"]
change_mode = "signal"
change_signal = "SIGINT"
// env = "false"
}
template {
data = <<EOT
APP_NAME={{ key "services/api/app/${ENVIRONMENT}/APP_NAME" }}
APP_ENV={{ key "services/api/app/${ENVIRONMENT}/APP_ENV" }}
APP_KEY={{with secret "kv/services/api/app/${ENVIRONMENT}"}}{{.Data.APP_KEY.value}}{{end}}
APP_DEBUG={{ key "services/api/app/${ENVIRONMENT}/APP_DEBUG" }}
APP_URL={{ key "services/api/app/${ENVIRONMENT}/APP_URL" }}
LOG_CHANNEL={{ key "services/api/log/${ENVIRONMENT}/LOG_CHANNEL" }}
DB_CONNECTION={{ key "services/api/db/${ENVIRONMENT}/DB_CONNECTION" }}
DB_HOST={{ key "services/api/db/${ENVIRONMENT}/DB_HOST" }}
DB_PORT={{ key "services/api/db/${ENVIRONMENT}/DB_PORT" }}
DB_DATABASE={{with secret "kv/services/api/db/${ENVIRONMENT}/DB_DATABASE"}}{{.Data.value}}{{end}}
DB_USERNAME={{with secret "kv/services/api/db/${ENVIRONMENT}/DB_USERNAME"}}{{.Data.value}}{{end}}
DB_PASSWORD={{with secret "kv/services/api/db/${ENVIRONMENT}/DB_PASSWORD"}}{{.Data.value}}{{end}}
BROADCAST_DRIVER={{ key "services/api/broadcast/${ENVIRONMENT}/BROADCAST_DRIVER" }}
CACHE_DRIVER={{ key "services/api/cache/${ENVIRONMENT}/CACHE_DRIVER" }}
SESSION_DRIVER={{ key "services/api/session/${ENVIRONMENT}/SESSION_DRIVER" }}
SESSION_LIFETIME={{ key "services/api/session/${ENVIRONMENT}/SESSION_LIFETIME" }}
QUEUE_DRIVER={{ key "services/api/queue/${ENVIRONMENT}/QUEUE_DRIVER" }}
REDIS_HOST={{ key "services/api/redis/${ENVIRONMENT}/REDIS_HOST" }}
REDIS_PORT={{ key "services/api/redis/${ENVIRONMENT}/REDIS_PORT" }}
REDIS_PASSWORD={{with secret "kv/services/api/redis/${ENVIRONMENT}/REDIS_PASSWORD"}}{{.Data.value}}{{end}}
MAIL_DRIVER={{ key "services/api/mail/${ENVIRONMENT}/MAIL_DRIVER" }}
MAIL_HOST={{ key "services/api/mail/${ENVIRONMENT}/MAIL_HOST" }}
MAIL_PORT={{ key "services/api/mail/${ENVIRONMENT}/MAIL_PORT" }}
MAIL_USERNAME={{ key "services/api/mail/${ENVIRONMENT}/MAIL_USERNAME" }}
MAIL_PASSWORD={{ key "services/api/mail/${ENVIRONMENT}/MAIL_PASSWORD" }}
MAIL_ENCRYPTION={{ key "services/api/mail/${ENVIRONMENT}/MAIL_ENCRYPTION" }}
MAIL_FROM_ADDRESS={{ key "services/api/mail/${ENVIRONMENT}/MAIL_FROM_ADDRESS" }}
MAIL_FROM_NAME={{ key "services/api/mail/${ENVIRONMENT}/MAIL_FROM_NAME" }}
MAILGUN_DOMAIN={{ key "services/api/mailgun/${ENVIRONMENT}/MAILGUN_DOMAIN" }}
MAILGUN_SECRET={{with secret "kv/services/api/mailgun/${ENVIRONMENT}/MAILGUN_SECRET"}}{{.Data.value}}{{end}}
DO_SPACES_KEY={{with secret "kv/services/api/spaces/${ENVIRONMENT}/DO_SPACES_KEY"}}{{.Data.value}}{{end}}
DO_SPACES_SECRET={{with secret "kv/services/api/spaces/${ENVIRONMENT}/DO_SPACES_SECRET"}}{{.Data.value}}{{end}}
DO_SPACES_ENDPOINT={{ key "services/api/spaces/${ENVIRONMENT}/DO_SPACES_ENDPOINT" }}
DO_SPACES_REGION={{ key "services/api/spaces/${ENVIRONMENT}/DO_SPACES_REGION" }}
DO_SPACES_BUCKET={{ key "services/api/spaces/${ENVIRONMENT}/DO_SPACES_BUCKET" }}
JWT_SECRET={{with secret "kv/services/api/jwt/${ENVIRONMENT}"}}{{.Data.JWT_SECRET}}{{end}}
EOT
destination = "custom/.env"
// change_mode = "signal"
// change_signal = "SIGINT"
env = true
}
service {
name = "api"
tags = [
"urlprefix-${ENVIRONMENT_URL}/"
]
port = "nginx"
check {
type = "tcp"
port = "nginx"
interval = "10s"
timeout = "2s"
}
}
resources {
cpu = 500
memory = 256
network {
mbits = 100
port "nginx" {}
}
}
}
}
}
Nomad logs
In Nomad logs, I can check that it correctly gets a token from Vault:
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.333Z [INFO ] client.fingerprint_mgr.vault: Vault is available
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.334Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.354Z [DEBUG] nomad.vault: starting renewal loop: creation_ttl=72h0m0s
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.375Z [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs="[arch cgroup consul cpu host network nomad signal storage vault]"
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.380Z [DEBUG] nomad.vault: successfully renewed server token
Oct 16 13:29:19 nomad-server-0 nomad: 2019-10-16T13:29:19.380Z [INFO ] nomad.vault: successfully renewed token: next_renewal=35h59m59.999975432s
I'm stuck here, if anyone can provide any insight, I would really appreciate!
[EDIT]
I'm using Vault v1.2.0-rc1.

Answer 1

sometimes this error means that the vault is still sealed or has become sealed after a restart of vault
this can be tested by doing a dig on the vaults dns address for example vault.service.consul

Answer 2

According to the docs, Note: Vault integration requires Vault version 0.6.2 or higher.
Your error message backs that up -- Constraint ${attr.vault.version} version >= 0.6.1 filtered 1 node. Normally, constraints are things that you specify in the nomad spec, but in this case, it looks like it's coming from nomad.
I think you'll need to upgrade vault to 0.6.2

Answer 3

I've upgraded Vault to version v1.2.3 and it started working.