[ansible-project] Inventory and group_vars layout advice

[ddffgpmfn]

Seeking some best-practice advice about where to store variables within the
following structure:

* multiple inventory folders ./inventory/x, ./inventory/y, ./inventory/z
* inside each inventory folder, combinations of dynamic and static
inventory sources
* inside each inventory folder, environment specific group vars

Currently I am copy-pasting variables common to all of the inventories
(environments) between the all.yml group_vars files.


My question: is there a way to have a 'global' group variables file I can
use for this type of variable?


I know I could source a file with include_vars: at the start of each of the
multiple plays inside the playbooks, but hoped there might be something
more elegant and/or automatic (like group_vars) available. Maybe even a
simple restructure I can't think of.

.
├── inventory
│ ├── aws-dev
│ │ ├── aws-dev
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ ├── security_group_app.yml
│ │ └── security_group_util
│ │ ├── secrets.yml
│ │ └── security_group_util.yml
│ ├── aws-prod
│ │ ├── aws-prod
│ │ └── group_vars
│ │ └── all
│ │ ├── all.yml
│ │ └── secrets.yml
│ └── vmware-dev
│ ├── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ └── tag_role_app.yml
│ ├── vmware-dev
│ ├── vmware.ini
│ └── vmware.py
├── roles
├── aws-configure.yml
├── aws-provision.yml
├── vmware-configure.yml
└── vmware-provision.yml

[Mark McCoy]

Have you thought about using role default variables instead? If all
environments have the same setting, this would be simpler. Role default
variables can then be overridden for specific groups in the group vars file.

Post by ddffgpmfn
Seeking some best-practice advice about where to store variables within
* multiple inventory folders ./inventory/x, ./inventory/y, ./inventory/z
* inside each inventory folder, combinations of dynamic and static
inventory sources
* inside each inventory folder, environment specific group vars
Currently I am copy-pasting variables common to all of the inventories
(environments) between the all.yml group_vars files.
My question: is there a way to have a 'global' group variables file I can
use for this type of variable?
I know I could source a file with include_vars: at the start of each of
the multiple plays inside the playbooks, but hoped there might be something
more elegant and/or automatic (like group_vars) available. Maybe even a
simple restructure I can't think of.
.
├── inventory
│ ├── aws-dev
│ │ ├── aws-dev
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ ├── security_group_app.yml
│ │ └── security_group_util
│ │ ├── secrets.yml
│ │ └── security_group_util.yml
│ ├── aws-prod
│ │ ├── aws-prod
│ │ └── group_vars
│ │ └── all
│ │ ├── all.yml
│ │ └── secrets.yml
│ └── vmware-dev
│ ├── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ └── tag_role_app.yml
│ ├── vmware-dev
│ ├── vmware.ini
│ └── vmware.py
├── roles
├── aws-configure.yml
├── aws-provision.yml
├── vmware-configure.yml
└── vmware-provision.yml
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/306e15e9-3a22-4c24-a346-791029131072%40googlegroups.com
<https://groups.google.com/d/msgid/ansible-project/306e15e9-3a22-4c24-a346-791029131072%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[ddffgpmfn]

Already doing that where possible, but for things like microservices that
have templated configs that use the same variable, having it in defaults
still requires copy-paste, but at that (potentially larger) level instead.

Might it be possible to add an ansible configuration param that forces a
particular file to always be loaded, in addition to normal?

Abuse a vars_plugin to always return the vars in a file...?

[Hagai Kariti]

I'd like to hear advices for this as well. I do have one idea that I don't
like however. You should be able to use group_vars relative to your
playbooks as well. So put a group_vars dir outside of your inventories,
next to your playbooks, and that will be evaluated for all envs.

Post by ddffgpmfn
Already doing that where possible, but for things like microservices that
have templated configs that use the same variable, having it in defaults
still requires copy-paste, but at that (potentially larger) level instead.
Might it be possible to add an ansible configuration param that forces a
particular file to always be loaded, in addition to normal?
Abuse a vars_plugin to always return the vars in a file...?

[ddffgpmfn]

Thanks Hafai,
Confirming that works, unbelievably. A group_vars/all.yml next to
playbooks, as you say, is evaluated as well as in the inner group_vars.

I'm warming to the idea of writing a vars plugin that is essentially just a
pointer to a file somewhere. Advantage is that the target could be anywhere
you like.

[Hagai Kariti]

Problem with a vars plugin is that you'll have to specify it in each
playbook. New idea: you could put another inventory script in your dir that
evaluates your common file.

Post by ddffgpmfn
Thanks Hafai,
Confirming that works, unbelievably. A group_vars/all.yml next to
playbooks, as you say, is evaluated as well as in the inner group_vars.
I'm warming to the idea of writing a vars plugin that is essentially just
a pointer to a file somewhere. Advantage is that the target could be
anywhere you like.
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/T9RWNQbLRWs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/15a4f5b0-b92b-4005-9955-1eea03d8b39c%40googlegroups.com
<https://groups.google.com/d/msgid/ansible-project/15a4f5b0-b92b-4005-9955-1eea03d8b39c%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[ddffgpmfn]

So vars plugins have to be specified per play, like 'include_vars:' do? If
so, no savings there.

An inventory script would be pretty good as it would be a write once, then
stays the same for every extra inventory.

Can you clarify how it might work?

This command turns will output JSON from the Ansible .yml file:
python -c 'import sys, yaml, json; json.dump(yaml.load(sys.stdin),
sys.stdout, indent=4)' < all.yml > all.json

The other inventory scripts return groups. Would I have to wrap that output
in a 'all' group for it to work? Is that enough?

Post by Hagai Kariti
Problem with a vars plugin is that you'll have to specify it in each
playbook. New idea: you could put another inventory script in your dir that
evaluates your common file.

[ddffgpmfn]

Some of the variables in the all.yml are also templated from other
variables - would they be parsed properly if coming in from an inventory
script?

[Hagai Kariti]

You need to wrap it in an 'all' group, yes. The output should look like
this:

{
"all": {
"vars": {
..your vars here..
}
}
}

And templates are evaluated when used, so the vars can be templated.

Post by ddffgpmfn
Some of the variables in the all.yml are also templated from other
variables - would they be parsed properly if coming in from an inventory
script?
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/T9RWNQbLRWs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/3fbb98b7-de69-4ae3-9f3d-16b740bd7f4e%40googlegroups.com
<https://groups.google.com/d/msgid/ansible-project/3fbb98b7-de69-4ae3-9f3d-16b740bd7f4e%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[ddffgpmfn]

Thanks to Hagai I have a good solution worth sharing.

* add common_vars.sh bash script to each of the inventories I wish to share
variables. Make it executable. Point it where your common Ansible YAML
variables are (in my case inventory/common_vars.yml).
* add common_vars.yml and populate it

Run your playbooks with the specified inventory path as normal
(inventory/aws-dev, inventory/aws-prod etc)

Benefits
* inventory files are all kept in the inventory folder
* normal Ansible group_vars are left alone
* only need one extra file per inventory, and no copy paste
* if you need a few top level common_vars, you can just point your
common_vars to onprem_common.yml and cloudstack_common.yml, or as you like.
* feels like a proper Ansible way to do things without hacks

New dir structure below (compared to original). Bash script code after.

.
├── inventory
│ ├── common_vars.yml
│ ├── aws-dev
│ │ ├── aws-dev
│ │ ├── common_vars.sh
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ ├── security_group_app.yml
│ │ └── security_group_util
│ │ ├── secrets.yml
│ │ └── security_group_util.yml
│ ├── aws-prod
│ │ ├── aws-prod
│ │ ├── common_vars.sh
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ │ └── all
│ │ ├── all.yml
│ │ └── secrets.yml
│ └── vmware-dev
│ ├── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ └── tag_role_app.yml
│ ├── common_vars.sh
│ ├── vmware-dev
│ ├── vmware.ini
│ └── vmware.py
├── roles
├── aws-configure.yml
├── aws-provision.yml
├── vmware-configure.yml
└── vmware-provision.yml

common_vars.sh
#!/usr/bin/env bash

#
http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
# echo $DIR
#
http://stackoverflow.com/questions/27382552/converting-yaml-to-json-with-python-block-end-found
python -c 'import sys, yaml, json; json.dump({"all": {"vars":
yaml.load(sys.stdin) } }, sys.stdout, indent=4)' < $DIR/../common_vars.yml
# > all.json

This accomplishes everything required without complexity.

Thanks Hagai!

Post by Hagai Kariti
You need to wrap it in an 'all' group, yes. The output should look like
{
"all": {
"vars": {
..your vars here..
}
}
}
And templates are evaluated when used, so the vars can be templated.

[Hagai Kariti]

Thanks for sharing! Glad I could help. :)

Post by ddffgpmfn
Thanks to Hagai I have a good solution worth sharing.
* add common_vars.sh bash script to each of the inventories I wish to
share variables. Make it executable. Point it where your common Ansible
YAML variables are (in my case inventory/common_vars.yml).
* add common_vars.yml and populate it
Run your playbooks with the specified inventory path as normal
(inventory/aws-dev, inventory/aws-prod etc)
Benefits
* inventory files are all kept in the inventory folder
* normal Ansible group_vars are left alone
* only need one extra file per inventory, and no copy paste
* if you need a few top level common_vars, you can just point your
common_vars to onprem_common.yml and cloudstack_common.yml, or as you like.
* feels like a proper Ansible way to do things without hacks
New dir structure below (compared to original). Bash script code after.
.
├── inventory
│ ├── common_vars.yml
│ ├── aws-dev
│ │ ├── aws-dev
│ │ ├── common_vars.sh
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ ├── security_group_app.yml
│ │ └── security_group_util
│ │ ├── secrets.yml
│ │ └── security_group_util.yml
│ ├── aws-prod
│ │ ├── aws-prod
│ │ ├── common_vars.sh
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ │ └── all
│ │ ├── all.yml
│ │ └── secrets.yml
│ └── vmware-dev
│ ├── group_vars
│ │ ├── all
│ │ │ ├── all.yml
│ │ │ └── secrets.yml
│ │ └── tag_role_app.yml
│ ├── common_vars.sh
│ ├── vmware-dev
│ ├── vmware.ini
│ └── vmware.py
├── roles
├── aws-configure.yml
├── aws-provision.yml
├── vmware-configure.yml
└── vmware-provision.yml
common_vars.sh
#!/usr/bin/env bash
#
http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
# echo $DIR
#
http://stackoverflow.com/questions/27382552/converting-yaml-to-json-with-python-block-end-found
yaml.load(sys.stdin) } }, sys.stdout, indent=4)' < $DIR/../common_vars.yml
# > all.json
This accomplishes everything required without complexity.
Thanks Hagai!

Post by Hagai Kariti
You need to wrap it in an 'all' group, yes. The output should look like
{
"all": {
"vars": {
..your vars here..
}
}
}
And templates are evaluated when used, so the vars can be templated.

--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/T9RWNQbLRWs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/c21d0da1-f461-4ff6-8a5a-dd49f87b17de%40googlegroups.com
<https://groups.google.com/d/msgid/ansible-project/c21d0da1-f461-4ff6-8a5a-dd49f87b17de%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Tom Paine]

This has worked incredibly well in practice. We've deduped all our
variables and only have a handful of inventory specific vars. Perfect.

Any idea how this might be extended to use ansible-vault encrypted files?

Is the vault-password put into somewhere accessible, so that a modified
script could do the same JSON-ification as it does with the clear YAML vars?

Post by Hagai Kariti
Thanks for sharing! Glad I could help. :)

[Hagai Kariti]

Hm, can't think of something. I think we must have ansible pass it as an
environment variable for the script to use.

Post by Tom Paine
This has worked incredibly well in practice. We've deduped all our
variables and only have a handful of inventory specific vars. Perfect.
Any idea how this might be extended to use ansible-vault encrypted files?
Is the vault-password put into somewhere accessible, so that a modified
script could do the same JSON-ification as it does with the clear YAML vars?

Post by Hagai Kariti
Thanks for sharing! Glad I could help. :)

--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/T9RWNQbLRWs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/7491dcf2-4674-4f1b-be13-399ce4d9ca30%40googlegroups.com
<https://groups.google.com/d/msgid/ansible-project/7491dcf2-4674-4f1b-be13-399ce4d9ca30%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Quentin Stafford-Fraser]

Nice concept.

If you prefer not to mix python and bash, you could create a common_vars.py
instead. Something like:

#! /usr/bin/env python

import os, sys, yaml, json
srcfile = os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), '..',
'common_vars.yml'))
with open(srcfile) as f:
json.dump({"all": {"vars": yaml.load(f) } }, sys.stdout, indent=4)


Remember to make it executable with chmod +x as with the bash script.

[Barry Kaplan]

I have, and and still are, struggling with this as well. Our inventory is
very similar to yours. We have three AWS environments, and dozens of other
on-prem environments. For the AWS we need to share a bunch of variables.
Putting those in playbook group_vars does not work because the are only for
AWS, not the other environments.

Hence I have resorted to soft-linking an aws-common.yml into each of the
aws group_vars. Since git handles the links its really no pain, just a bit
of confusion when searching.

I do a similar thing in the playbooks for other reasons. But now reading
below the under playbooks ansible will look up the tree for group_vars I
may not need those.