[ansible-project] delegate_to does not work for array of hosts passed in to role

[Barry Kaplan]

I posted this as a bug (https://github.com/ansible/ansible/issues/11102).
But it was closed saying it was a discussion. Not sure why, seems like a
clear bug to me. But I'm open to be dissuaded.

----

Ansible 1.9.1 running from OSX.
in playbook

- role: mongodb
mongodb_replica_set_name: "{{mongodb_rs_name}}"
mongodb_replica_set_hosts: "{{groups.mongod_rs_member}}"

mongodb role

- set_fact:
mongodb_candidate_master: "{{mongodb_replica_set_hosts[0]}}"
run_once: true

- shell: echo "HELLO groups.mongod_rs_member +++++++++++++++++"
delegate_to: "{{groups.mongod_rs_member[0]}}"
run_once: true

- shell: echo "HELLO mongodb_candidate_master +++++++++++++++++"
delegate_to: "{{mongodb_candidate_master}}"
run_once: true

- shell: echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"
delegate_to: "{{mongodb_replica_set_hosts[0]}}"
run_once: true

output

TASK: [mongodb | shell echo "HELLO groups.mongod_rs_member +++++++++++++++++"] ***
<10.0.139.56> ESTABLISH CONNECTION FOR USER: ubuntu
<10.0.139.56> REMOTE_MODULE command echo "HELLO groups.mongod_rs_member +++++++++++++++++" #USE_SHELL
<10.0.139.56> EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o ControlPath="/Users/bkaplan/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 10.0.139.56 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=enrekuxbmqjtjdgsxvlukuvqlhnudrkw] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-enrekuxbmqjtjdgsxvlukuvqlhnudrkw; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'"'"''
changed: [10.0.139.56 -> 10.0.139.56] => {"changed": true, "cmd": "echo \"HELLO groups.mongod_rs_member +++++++++++++++++\"", "delta": "0:00:00.005216", "end": "2015-05-31 04:13:32.378158", "rc": 0, "start": "2015-05-31 04:13:32.372942", "stderr": "", "stdout": "HELLO groups.mongod_rs_member +++++++++++++++++", "warnings": []}

TASK: [mongodb | shell echo "HELLO mongodb_candidate_master +++++++++++++++++"] ***
<10.0.139.56> ESTABLISH CONNECTION FOR USER: ubuntu
<10.0.139.56> REMOTE_MODULE command echo "HELLO mongodb_candidate_master +++++++++++++++++" #USE_SHELL
<10.0.139.56> EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o ControlPath="/Users/bkaplan/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 10.0.139.56 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=xnkqbskjckeetqglyftpndfusnsyhoyq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-xnkqbskjckeetqglyftpndfusnsyhoyq; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'"'"''
changed: [10.0.139.56 -> 10.0.139.56] => {"changed": true, "cmd": "echo \"HELLO mongodb_candidate_master +++++++++++++++++\"", "delta": "0:00:00.005059", "end": "2015-05-31 04:13:33.277391", "rc": 0, "start": "2015-05-31 04:13:33.272332", "stderr": "", "stdout": "HELLO mongodb_candidate_master +++++++++++++++++", "warnings": []}

TASK: [mongodb | shell echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"] ***
<{> ESTABLISH CONNECTION FOR USER: ubuntu
<{> REMOTE_MODULE command echo "HELLO mongodb_replica_set_hosts +++++++++++++++++" #USE_SHELL
<{> EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o ControlPath="/Users/bkaplan/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 { /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=mqgojxehsqcpahfvfcdimgoedvygtvvb] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-mqgojxehsqcpahfvfcdimgoedvygtvvb; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'"'"''
fatal: [10.0.139.56 -> {] => SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh

[Brian Coca]

From the output of the 3rd task it seems that
mongodb_replica_set_hosts[0] is being interpreted as '{' which, of
course, does not resolve. This does not seem to be an array issue but
of too many nested templating attempts.

[Barry Kaplan]

What do you think would be causing the nested templating? Something deep in
variable resolution?

The third echo is the direct version of first setting the value to a fact,
which works.

Post by Brian Coca
From the output of the 3rd task it seems that
mongodb_replica_set_hosts[0] is being interpreted as '{' which, of
course, does not resolve. This does not seem to be an array issue but
of too many nested templating attempts.
--
Brian Coca

[Brian Coca]

not sure what 'works' means in this context

Post by Barry Kaplan
What do you think would be causing the nested templating? Something deep in
variable resolution?
The third echo is the direct version of first setting the value to a fact,
which works.

Post by Brian Coca
From the output of the 3rd task it seems that
mongodb_replica_set_hosts[0] is being interpreted as '{' which, of
course, does not resolve. This does not seem to be an array issue but
of too many nested templating attempts.
--
Brian Coca

--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/a747482f-5ed6-4ac2-8dc3-752a0f9adf9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry Kaplan]

Post by Brian Coca
not sure what 'works' means in this context

"works" is that the third task delegates to the same host as the second
task.

Don't you expect the exact same output for the last two tasks? Are you
seeing something that I am missing?

[Brian Coca]

the task fails with e 'fatal', what looks the same is the verbose
output of what will be executed

Post by Barry Kaplan

Post by Brian Coca
not sure what 'works' means in this context

"works" is that the third task delegates to the same host as the second
task.
Don't you expect the exact same output for the last two tasks? Are you
seeing something that I am missing?
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/79164104-61fa-466c-8ced-fae92d8ba19b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry Kaplan]

2) ssh ... -o ConnectTimeout=10 10.0.139.56 /bin/sh -c ...
3) ssh ... -o ConnectTimeout=10 { /bin/sh -c ...

So you don't think that the ssh ip in the 3 task should not be the same as
2?

I'm sorry Brian, but I am not all following what the miscommunication is
here.

[Brian Coca]

the ssh ip in task 3 is '{' which indicates that the [0] is applied
to the 'template string' and not to the resulting array you expect.
This makes the 3rd task fail as it cannot ssh to '{'.

Post by Barry Kaplan
2) ssh ... -o ConnectTimeout=10 10.0.139.56 /bin/sh -c ...
3) ssh ... -o ConnectTimeout=10 { /bin/sh -c ...
So you don't think that the ssh ip in the 3 task should not be the same as
2?
I'm sorry Brian, but I am not all following what the miscommunication is
here.
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/39fb66d6-bcb3-4f05-be8e-0f3c1267f589%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry Kaplan]

Post by Brian Coca
the ssh ip in task 3 is '{' which indicates that the [0] is applied
to the 'template string' and not to the resulting array you expect.
This makes the 3rd task fail as it cannot ssh to '{'.

Exactly. Is that a mistake I am making? Or a bug? I filed the bug because I
don't see a mistake on my part.

[Brian Coca]

can you show that variable in a debug during execution? it does not
look like it is being set properly.

Post by Barry Kaplan

Post by Brian Coca
the ssh ip in task 3 is '{' which indicates that the [0] is applied
to the 'template string' and not to the resulting array you expect.
This makes the 3rd task fail as it cannot ssh to '{'.

Exactly. Is that a mistake I am making? Or a bug? I filed the bug because I
don't see a mistake on my part.
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/8c07227d-d1e3-46c8-80e4-a5e78dc01e15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry Kaplan]

Here's the new test:

- set_fact:
mongodb_candidate_master: "{{mongodb_replica_set_hosts[0]}}"
run_once: true


- name: var groups.mongod_rs_member[0]
debug: var=groups.mongod_rs_member[0]
run_once: true


- shell: echo "HELLO groups.mongod_rs_member +++++++++++++++++"
delegate_to: "{{groups.mongod_rs_member[0]}}"
run_once: true


- name: var mongodb_candidate_master
debug: var=mongodb_candidate_master
run_once: true


- shell: echo "HELLO mongodb_candidate_master +++++++++++++++++"
delegate_to: "{{mongodb_candidate_master}}"
run_once: true


- name: var mongodb_replica_set_hosts[0]
debug: var=mongodb_replica_set_hosts[0]
run_once: true


- name: msg mongodb_replica_set_hosts[0]
debug: msg="{{mongodb_replica_set_hosts[0]}}"
run_once: true


- debug: msg="{{mongodb_replica_set_hosts[0]}}"
run_once: true


- shell: echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"
delegate_to: "{{mongodb_replica_set_hosts[0]}}"
run_once: true


And the output:

ansible❯ ansible-playbook -i inventory/aws/staging playbooks/configure/
mongod.yml --skip-tags base
ops/git/master !+


PLAY [Base ansible configuration (packages)]
**********************************


GATHERING FACTS
***************************************************************
ok: [10.0.137.189]
ok: [10.0.139.51]
ok: [10.0.136.46]


PLAY [Base ansible configuration (configure)]
*********************************


GATHERING FACTS
***************************************************************
ok: [10.0.137.189]
ok: [10.0.139.51]
ok: [10.0.136.46]


PLAY [Base ansible configuration (users)]
*************************************


GATHERING FACTS
***************************************************************
ok: [10.0.137.189]
ok: [10.0.139.51]
ok: [10.0.136.46]


PLAY [Configure mongodb replicaset cluster]
***********************************


GATHERING FACTS
***************************************************************
ok: [10.0.136.46]
ok: [10.0.137.189]
ok: [10.0.139.51]


TASK: [debug var=groups.mongod_rs_member]
*************************************
ok: [10.0.136.46] => {
"var": {
"groups.mongod_rs_member": [
"10.0.136.46",
"10.0.139.51",
"10.0.137.189"
]
}
}
ok: [10.0.139.51] => {
"var": {
"groups.mongod_rs_member": [
"10.0.136.46",
"10.0.139.51",
"10.0.137.189"
]
}
}
ok: [10.0.137.189] => {
"var": {
"groups.mongod_rs_member": [
"10.0.136.46",
"10.0.139.51",
"10.0.137.189"
]
}
}


TASK: [mongodb | set_fact ]
***************************************************
ok: [10.0.136.46]


TASK: [mongodb | var groups.mongod_rs_member[0]]
******************************
ok: [10.0.136.46] => {
"var": {
"groups.mongod_rs_member[0]": "10.0.136.46"
}
}


TASK: [mongodb | shell echo "HELLO groups.mongod_rs_member
+++++++++++++++++"] ***
changed: [10.0.136.46 -> 10.0.136.46]


TASK: [mongodb | var mongodb_candidate_master]
********************************
ok: [10.0.136.46] => {
"var": {
"mongodb_candidate_master": "10.0.136.46"
}
}


TASK: [mongodb | shell echo "HELLO mongodb_candidate_master
+++++++++++++++++"] ***
changed: [10.0.136.46 -> 10.0.136.46]


TASK: [mongodb | var mongodb_replica_set_hosts[0]]
****************************
ok: [10.0.136.46] => {
"var": {
"mongodb_replica_set_hosts[0]": "10.0.136.46"
}
}


TASK: [mongodb | msg mongodb_replica_set_hosts[0]]
****************************
ok: [10.0.136.46] => {
"msg": "10.0.136.46"
}


TASK: [mongodb | debug msg="{"]
***********************************************
ok: [10.0.136.46] => {
"msg": "10.0.136.46"
}


TASK: [mongodb | shell echo "HELLO mongodb_replica_set_hosts
+++++++++++++++++"] ***
fatal: [10.0.136.46 -> {] => SSH Error: data could not be sent to the
remote host. Make sure this host can be reached over ssh


FATAL: all hosts have already failed -- aborting


The last debug is interesting. I does not have a -name and it seems to
render the value as '{' just like the delegate_to.

[Barry Kaplan]

Is it still unclear Brian?

I also seem to be seeing a related problem. Consider:

- name: Provision Ops RDS instance
hosts: localhost
connection: local
gather_facts: no
tags: [control]


tasks:
...
- shell: |
echo {{groups.consul_server[0]}} && hostname -i && curl localhost:
8500/v1/catalog/services
register: __catalog
delegate_to: groups.consul_server[0]

Here the shell commands runs on localhost, not the host specified. The
output is

TASK: [shell echo {{groups.consul_server[0]}} && hostname -i && curl
localhost:8500/v1/catalog/services
] ***
<groups.consul_server[0]> REMOTE_MODULE command echo 10.0.196.116 &&
hostname -i && curl localhost:8500/v1/catalog/services #USE_SHELL
<groups.consul_server[0]> EXEC ['/bin/sh', '-c', 'mkdir -p
$HOME/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713 && chmod a+rx
$HOME/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713 && echo
$HOME/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713']
<groups.consul_server[0]> PUT /tmp/tmpTGBlM3 TO /home/bkaplan/.ansible/tmp/
ansible-tmp-1437044596.27-54518627543713/command
<groups.consul_server[0]> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8 /usr/bin/python
/home/bkaplan/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713/command;
rm -rf /home/bkaplan/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713/

/dev/null 2>&1']

failed: [localhost -> groups.consul_server[0]] => {"changed": true, "cmd": "echo
10.0.196.116 && hostname -i && curl localhost:8500/v1/catalog/services",
"delta": "0:00:00.012358", "end": "2015-07-16 16:33:16.332372", "rc": 7,
"start": "2015-07-16 16:33:16.320014", "warnings": []}
stderr: % Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0curl: (7) Failed to connect to localhost port 8500: Connection refused
stdout: 10.0.196.116
127.0.1.1


Where '10.0.196.116' is groups.consul_server[0]. So the variable resolves
correctly, but delegate to does not seem to use it. This is not specific to
shell, it has the same behavior for other modules (eg, uri).

[Brian Coca]

you need moustaches:

delegate_to: "{{groups.consul_server[0]}}"

only conditionals (when: ) do not need templating, with_ has a
haphazard support for it which we plan to deprecate.

[Barry Kaplan]

Hmm, thought I tried that, but will check. Thanks Brian.

I wonder whether it would be better for it to be error if delegate_to does
not resolve rather than silently fall back to localhost... Depending on the
task fallback to localhost could do some serious damage.

[Barry Kaplan]

I see the same behavior with the moustaches:

- name: Test delegate to
shell: echo "{{ groups.consul_server[0] }}" && hostname -i
register: __hostname

delegate_to: "{{ groups.consul_server[0] }}"
- debug: var=__hostname


TASK: [Test delegate to] *********************************************
<10.0.196.116> REMOTE_MODULE command echo "10.0.196.116" && hostname -i
#USE_SHELL
<10.0.196.116> EXEC ['/bin/sh', '-c', 'mkdir -p
$HOME/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917 && chmod a+rx
$HOME/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917 && echo
$HOME/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917']
<10.0.196.116> PUT /tmp/tmpDXWJFH TO /home/bkaplan/.ansible/tmp/ansible-tmp-
1437108695.64-26819269533917/command
<10.0.196.116> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8 /usr/bin/python
/home/bkaplan/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917/command;
rm -rf /home/bkaplan/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917/

/dev/null 2>&1']

changed: [localhost -> 10.0.196.116] => {"changed": true, "cmd": "echo
\"10.0.196.116\" && hostname -i", "delta": "0:00:00.004458", "end": "2015-07-17
10:21:35.693136", "rc": 0, "start": "2015-07-17 10:21:35.688678", "stderr":
"", "stdout": "10.0.196.116\n127.0.1.1", "warnings": []}


TASK: [debug var=__hostname]
**************************************************
ok: [localhost] => {
"var": {
"__hostname": {
"changed": true,
"cmd": "echo \"10.0.196.116\" && hostname -i",
"delta": "0:00:00.004458",
"end": "2015-07-17 10:21:35.693136",
"invocation": {
"module_args": "echo \"10.0.196.116\" && hostname -i",
"module_name": "shell"
},
"rc": 0,
"start": "2015-07-17 10:21:35.688678",
"stderr": "",
"stdout": "10.0.196.116\n127.0.1.1",
"stdout_lines": [
"10.0.196.116",
"127.0.1.1"
],
"warnings": []
}
}
}

[Brian Coca]

well, the first difference is that you are not delegating to a host
named 'groups.consul_server[0]' but to 10.0.196.116

Sorry for not noticing before, but the delegate_to on debug is
irrelevant, debug always executes on 'the master' or machine you
execute ansible on, it does not need to connect to any machine, i'm
not sure what you are expecting there.

delegate_to on shell or any module that executes remotely goes to the
delegated server, but things that operate on 'master' like debug,
prompt, add_host, group_by make no sense to delegate.

[Barry Kaplan]

Brian, I'm not following you at all. It was the shell command that uses the
delegate_to. The debug is only showing the register from the shell.

- name: Test delegate to
shell: echo "{{ groups.consul_server[0] }}" && hostname -i
register: __hostname
delegate_to: "{{ groups.consul_server[0] }}"

- debug: var=__hostname


"stdout_lines": [
"10.0.196.116", <--- echo "{{ groups.consul_server[0] }}"
"127.0.1.1" <--- hostname -i
],

[Brian Coca]

ah, formatting of previous email threw me off, the only reason i can
think of for this happening is that you have connection=local for the
task (from play or command line) and the host does not have it
specified through ansible_connection.

[Barry Kaplan]

All my hosts come from the ec2 inventory plugin. I can access the hosts in
all other ways without having to explicitly specify an ansible_connection.

So then is it better to specify the target host in the play and use
local_action? I kinda thought these
two were inverses of each other.

-barry

[Barry Kaplan]

Perfect! I have removed the connection=local from the play and now all
works as expected. I guess I don't fully understand why connection=local
should be used. When I was first starting with ansible plays that I studied
that did ec2 stuff used it, so I did too.

[Brian Coca]

The connection: local is used when you want a plugin to execute on
master that normally would execute on the target. You normally want to
do this with cloud modules or modules that use some web api. Some
modules always execute on the 'master' as they are meant to just
change the data in the play like, debug, set_fact, add_host, group_by
so these don't need any connection settings.