[ansible-project] Ansible copy module hangs on CentOS 7

[Martin Ilchev]

Hi Guys,

I have a playbook to copy DNS zone files from local directory to a remote
one and restart the named service. The playbook is as follows:
---
# This playbook upgrades zone data files from the control machine to all
DNS servers 1 machine at a time

- name: Upgrade zone data files. Reload named-chroot if required
hosts: dnsservers
gather_facts: no
sudo: yes
serial: 1
tasks:

- name: copy dns zone files from control machine to DNS servers
copy: src=zones/ dest=/var/named owner=root group=named mode=0440
register: namedzones

- name: restart dns service if required
service: name=named-chroot state=restarted
when: namedzones.changed

I test run the playbook as follows:

ansible-playbook -vv dns-copy-zones.yml --ask-become-pass --check

The output is:
SUDO password:

PLAY [Upgrade zone data files. Reload named-chroot if required] ***

TASK: [copy dns zone files from control machine to DNS servers] ***
fatal: [192.168.16.195] => ssh connection error while waiting for sudo
password prompt

FATAL: all hosts have already failed -- aborting



PLAY RECAP
********************************************************************
to retry, use: --limit @/home/martin/dns-copy-zones.retry

192.168.16.195 : ok=0 changed=0 unreachable=1 failed=0

When I increase the verbosity I can see 5-6 files are copied then copy
hangs locally on transferring the next file and eventually times out.

On the remote machine if I tail /var/log/secure I can see all the files up
to the last one going through.

The machine I run the playbook from and the destination machine are both
CentOS 7. The version is ansible 1.9.2.

Below is the more verbose output I have highlighted what I think are the
important messages. I am only including the last file to go through
successfully and the next one to hang (let me know if you require the full
output):
ON CONTROL MACHINE:
<192.168.16.195> EXEC ssh -C -tt -v -o ControlMaster=auto -o
ControlPersist=60s -o
ControlPath="/home/martin/.ansible/cp/ansible-ssh-%h-%p-%r" -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10 192.168.16.195 /bin/sh -c
'sudo -k && sudo -H -S -p "[sudo via ansible,
key=zasgrvyvetzjqebxkfgjmaeohbealypu] password: " -u root /bin/sh -c
'"'"'echo BECOME-SUCCESS-zasgrvyvetzjqebxkfgjmaeohbealypu; rc=flag; [ -r
/var/named/named.empty ] || rc=2; [ -f /var/named/named.empty ] || rc=1; [
-d /var/named/named.empty ] && rc=3; python -V 2>/dev/null || rc=4; [
x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.empty && exit 0;
(python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536; hasher =
hashlib.sha1(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.empty'"'"'"'"'"'"'"'"'", "rb") buf
= afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c
'"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.empty'"'"'"'"'"'"'"'"'", "rb") buf
= afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo
'"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/var/named/named.empty)'"'"''
<192.168.16.195> EXEC ssh -C -tt -v -o ControlMaster=auto -o
ControlPersist=60s -o
ControlPath="/home/martin/.ansible/cp/ansible-ssh-%h-%p-%r" -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10 192.168.16.195 /bin/sh -c
'sudo -k && sudo -H -S -p "[sudo via ansible,
key=bnbotucjqsepetntcsddztjuluwdvzam] password: " -u root /bin/sh -c
'"'"'echo BECOME-SUCCESS-bnbotucjqsepetntcsddztjuluwdvzam; rc=flag; [ -r
/var/named/named.localhost ] || rc=2; [ -f /var/named/named.localhost ] ||
rc=1; [ -d /var/named/named.localhost ] && rc=3; python -V 2>/dev/null ||
rc=4; [ x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.localhost &&
exit 0; (python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536;
hasher = hashlib.sha1(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.localhost'"'"'"'"'"'"'"'"'", "rb")
buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c
'"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.localhost'"'"'"'"'"'"'"'"'", "rb")
buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo
'"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/var/named/named.localhost)'"'"''

ON THE DESTINATION MACHINE (output in /var/log/secure):
Jul 16 12:48:00 dominus sudo: martin : TTY=pts/2 ; PWD=/home/martin ;
USER=root ; COMMAND=/bin/sh -c echo
BECOME-SUCCESS-zasgrvyvetzjqebxkfgjmaeohbealypu; rc=flag; [ -r
/var/named/named.empty ] || rc=2; [ -f /var/named/named.empty ] || rc=1; [
-d /var/named/named.empty ] && rc=3; python -V 2>/dev/null || rc=4; [
x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.empty && exit 0;
(python -c 'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1();
afile = open("'/var/named/named.empty'", "rb")
buf = afile.read(BLOCKSIZE)
while len(buf) > 0:
hasher.update(buf)
buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (python -c 'import sha;
BLOCKSIZE = 65536; hasher = sha.sha();
afile = open("'/var/named/named.empty'", "rb")
buf = afile.read(BLOCKSIZE)
while len(buf) > 0:
hasher.update(buf)
buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (echo '0
'/var/named/named.empty)
*Jul 16 12:51:05 dominus sudo: pam_unix(sudo:auth): conversation failed*
*Jul 16 12:51:05 dominus sudo: pam_unix(sudo:auth): auth could not identify
password for [martin]*
Jul 16 12:51:05 dominus sudo: martin : 1 incorrect password attempt ;
TTY=pts/0 ; PWD=/home/martin ; USER=root ; COMMAND=/bin/sh -c echo
BECOME-SUCCESS-akpnodnhiuvxfpmhtrqwonulsepeazkv; rc=flag; [ -r
/var/named/db.sub1.testdns1.com ] || rc=2; [ -f
/var/named/db.sub1.testdns1.com ] || rc=1; [ -d
/var/named/db.sub1.testdns1.com ] && rc=3; python -V 2>/dev/null || rc=4; [
x"$rc" != "xflag" ] && echo "${rc} "/var/named/db.sub1.testdns1.com &&
exit 0; (python -c 'import hashlib; BLOCKSIZE = 65536; hasher =
hashlib.sha1();
afile = open("'/var/named/db.sub1.testdns1.com'", "rb")
buf = afile.read(BLOCKSIZE)
while len(buf) > 0:
hasher.update(buf)
buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (python -c 'import sha;
BLOCKSIZE = 65536; hasher = sha.sha();
afile = open("'/var/named/db.sub1.testdns1.com'", "rb")
buf = afile.read(BLOCKSIZE)
while len(buf) > 0:
hasher.update(buf)
buf =
Jul 16 12:51:05 dominus sudo: martin : (command continued)
afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (echo '0
'/var/named/db.sub1.testdns1.com)
Jul 16 12:53:25 dominus sudo: pam_unix(sudo:auth): conversation failed
Jul 16 12:53:25 dominus sudo: pam_unix(sudo:auth): auth could not identify
password for [martin]
Jul 16 12:54:25 dominus sshd[21162]: Received disconnect from
192.168.16.55: 11: disconnected by user
Jul 16 12:54:25 dominus sshd[21159]: pam_unix(sshd:session): session closed
for user martin

Apologies for the massive post however let me know if you require more
information.

Kind Regards,
Martin

[Martin Ilchev]

After a lot of digging I was able to find the source of the problem. There
was an sshd_config.j2 template that was taken from Debian based system. In
it the default settings for Subsystem is different than CentOS
# Debian
Subsystem sftp /usr/lib/openssh/sftp-server
#Centos
Subsystem sftp /usr/libexec/openssh/sftp-server

so lib vs libexec after restoring the config file back to normal the
playbook completes without errors.

Hopefully this will save someone else a couple of days.
M

Post by Martin Ilchev
Hi Guys,
I have a playbook to copy DNS zone files from local directory to a remote
---
# This playbook upgrades zone data files from the control machine to all
DNS servers 1 machine at a time
- name: Upgrade zone data files. Reload named-chroot if required
hosts: dnsservers
gather_facts: no
sudo: yes
serial: 1
- name: copy dns zone files from control machine to DNS servers
copy: src=zones/ dest=/var/named owner=root group=named mode=0440
register: namedzones
- name: restart dns service if required
service: name=named-chroot state=restarted
when: namedzones.changed
ansible-playbook -vv dns-copy-zones.yml --ask-become-pass --check
PLAY [Upgrade zone data files. Reload named-chroot if required] ***
TASK: [copy dns zone files from control machine to DNS servers] ***
fatal: [192.168.16.195] => ssh connection error while waiting for sudo
password prompt
FATAL: all hosts have already failed -- aborting
PLAY RECAP
********************************************************************
192.168.16.195 : ok=0 changed=0 unreachable=1 failed=0
When I increase the verbosity I can see 5-6 files are copied then copy
hangs locally on transferring the next file and eventually times out.
On the remote machine if I tail /var/log/secure I can see all the files up
to the last one going through.
The machine I run the playbook from and the destination machine are both
CentOS 7. The version is ansible 1.9.2.
Below is the more verbose output I have highlighted what I think are the
important messages. I am only including the last file to go through
successfully and the next one to hang (let me know if you require the full
<192.168.16.195> EXEC ssh -C -tt -v -o ControlMaster=auto -o
ControlPersist=60s -o
ControlPath="/home/martin/.ansible/cp/ansible-ssh-%h-%p-%r" -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10 192.168.16.195 /bin/sh -c
'sudo -k && sudo -H -S -p "[sudo via ansible,
key=zasgrvyvetzjqebxkfgjmaeohbealypu] password: " -u root /bin/sh -c
'"'"'echo BECOME-SUCCESS-zasgrvyvetzjqebxkfgjmaeohbealypu; rc=flag; [ -r
/var/named/named.empty ] || rc=2; [ -f /var/named/named.empty ] || rc=1; [
-d /var/named/named.empty ] && rc=3; python -V 2>/dev/null || rc=4; [
x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.empty && exit 0;
(python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536; hasher =
hashlib.sha1(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.empty'"'"'"'"'"'"'"'"'", "rb") buf
= afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c
'"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.empty'"'"'"'"'"'"'"'"'", "rb") buf
= afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo
'"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/var/named/named.empty)'"'"''
<192.168.16.195> EXEC ssh -C -tt -v -o ControlMaster=auto -o
ControlPersist=60s -o
ControlPath="/home/martin/.ansible/cp/ansible-ssh-%h-%p-%r" -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10 192.168.16.195 /bin/sh -c
'sudo -k && sudo -H -S -p "[sudo via ansible,
key=bnbotucjqsepetntcsddztjuluwdvzam] password: " -u root /bin/sh -c
'"'"'echo BECOME-SUCCESS-bnbotucjqsepetntcsddztjuluwdvzam; rc=flag; [ -r
/var/named/named.localhost ] || rc=2; [ -f /var/named/named.localhost ] ||
rc=1; [ -d /var/named/named.localhost ] && rc=3; python -V 2>/dev/null ||
rc=4; [ x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.localhost &&
exit 0; (python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536;
hasher = hashlib.sha1(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.localhost'"'"'"'"'"'"'"'"'", "rb")
buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c
'"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile =
open("'"'"'"'"'"'"'"'"'/var/named/named.localhost'"'"'"'"'"'"'"'"'", "rb")
buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf =
afile.read(BLOCKSIZE) afile.close()
print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo
'"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/var/named/named.localhost)'"'"''
Jul 16 12:48:00 dominus sudo: martin : TTY=pts/2 ; PWD=/home/martin ;
USER=root ; COMMAND=/bin/sh -c echo
BECOME-SUCCESS-zasgrvyvetzjqebxkfgjmaeohbealypu; rc=flag; [ -r
/var/named/named.empty ] || rc=2; [ -f /var/named/named.empty ] || rc=1; [
-d /var/named/named.empty ] && rc=3; python -V 2>/dev/null || rc=4; [
x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.empty && exit 0;
(python -c 'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1();
afile = open("'/var/named/named.empty'", "rb")
buf = afile.read(BLOCKSIZE)
hasher.update(buf)
buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (python -c 'import sha;
BLOCKSIZE = 65536; hasher = sha.sha();
afile = open("'/var/named/named.empty'", "rb")
buf = afile.read(BLOCKSIZE)
hasher.update(buf)
buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (echo '0
'/var/named/named.empty)
*Jul 16 12:51:05 dominus sudo: pam_unix(sudo:auth): conversation failed*
*Jul 16 12:51:05 dominus sudo: pam_unix(sudo:auth): auth could not
identify password for [martin]*
Jul 16 12:51:05 dominus sudo: martin : 1 incorrect password attempt ;
TTY=pts/0 ; PWD=/home/martin ; USER=root ; COMMAND=/bin/sh -c echo
BECOME-SUCCESS-akpnodnhiuvxfpmhtrqwonulsepeazkv; rc=flag; [ -r /var/named/
db.sub1.testdns1.com ] || rc=2; [ -f /var/named/db.sub1.testdns1.com ] ||
rc=1; [ -d /var/named/db.sub1.testdns1.com ] && rc=3; python -V
2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc} "/var/named/
db.sub1.testdns1.com && exit 0; (python -c 'import hashlib; BLOCKSIZE =
65536; hasher = hashlib.sha1();
afile = open("'/var/named/db.sub1.testdns1.com'", "rb")
buf = afile.read(BLOCKSIZE)
hasher.update(buf)
buf = afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (python -c 'import sha;
BLOCKSIZE = 65536; hasher = sha.sha();
afile = open("'/var/named/db.sub1.testdns1.com'", "rb")
buf = afile.read(BLOCKSIZE)
hasher.update(buf)
buf =
Jul 16 12:51:05 dominus sudo: martin : (command continued)
afile.read(BLOCKSIZE)
afile.close()
print(hasher.hexdigest())' 2>/dev/null) || (echo '0 '/var/named/
db.sub1.testdns1.com)
Jul 16 12:53:25 dominus sudo: pam_unix(sudo:auth): conversation failed
Jul 16 12:53:25 dominus sudo: pam_unix(sudo:auth): auth could not identify
password for [martin]
Jul 16 12:54:25 dominus sshd[21162]: Received disconnect from
192.168.16.55: 11: disconnected by user
Jul 16 12:54:25 dominus sshd[21159]: pam_unix(sshd:session): session
closed for user martin
Apologies for the massive post however let me know if you require more
information.
Kind Regards,
Martin