Discussion:
[ansible-project] Send playbook output to Logstash/Elasticsearch
David Reagan
2015-07-06 23:50:55 UTC
Permalink
I'm starting to run playbooks automatically via a push server. Basically
emulating what you can get with Puppet. As part of that, I'd like to send
the output of my cronjob playbooks into my ELK stack. Which is hard to do
when the default output is so unreadable.

Here's effectively what I want to get: http://pastebin.com/eVVcLfKn

Basically, a log entry per task per host.

From what I've seen googling around, the way to do this is via a callback
plugin. So I looked
at https://github.com/ansible/ansible/blob/devel/plugins/callbacks/log_plays.py
and got lost. My Python skills are very low, and apparently my general dev
skills are currently not up to the task.

I tried modifying log_plays to just dump the data variable raw, but I keep
running into errors....

Any tips?

Has anyone already solved this problem?
--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+***@googlegroups.com.
To post to this group, send email to ansible-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/d5e3c925-1df9-43b0-ad80-dbb1cb5f6a31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Brian Coca
2015-07-07 00:25:58 UTC
Permalink
log_plays was designed to drop the json to syslog-ng which would then
push it to elastic search w/o need for logstash, probably easier to
just setup syslog-ng to do the same.
--
Brian Coca
--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+***@googlegroups.com.
To post to this group, send email to ansible-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAJ5XC8n722sXnV547ZCjrTxBeaDQOqxzxQ1QEs0fYeq5AuQ6Sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
David Reagan
2015-07-07 00:37:29 UTC
Permalink
The default log_plays doesn't actually output all the information I need.

I have lots of stuff going to syslog and then into ELK already, but in this
case, I figured I'd just let logstash-forwarder watch the ansible log file.
Then format the output so that logstash doesn't have to filter it at all.

Currently, I figured out how to get valid json out per line. But I'm stuck
figuring out how to get the task name, the role name, and the command line
command information.

Are there global vars I can reference from the plugin? Where could I find a
list of them?


--David Reagan
Post by Brian Coca
log_plays was designed to drop the json to syslog-ng which would then
push it to elastic search w/o need for logstash, probably easier to
just setup syslog-ng to do the same.
--
Brian Coca
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/zOjsYxLN578/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAJ5XC8n722sXnV547ZCjrTxBeaDQOqxzxQ1QEs0fYeq5AuQ6Sg%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+***@googlegroups.com.
To post to this group, send email to ansible-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANo%2B_AcRMnkzn-atKOPtN7kLwQdFTxDyAi%2BJYn2g5TmFRAikqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
David Reagan
2015-07-07 18:26:06 UTC
Permalink
With some work I have almost exactly what I want. See:
http://pastebin.com/G819PEZY

Questions: The status of that restart apache 2 task should be "CHANGED",
since it actually did change during the play run. But it seems like the
value for changed tasks is always "OK". Is there a way to change that?



--David Reagan
Post by David Reagan
The default log_plays doesn't actually output all the information I need.
I have lots of stuff going to syslog and then into ELK already, but in
this case, I figured I'd just let logstash-forwarder watch the ansible log
file. Then format the output so that logstash doesn't have to filter it at
all.
Currently, I figured out how to get valid json out per line. But I'm stuck
figuring out how to get the task name, the role name, and the command line
command information.
Are there global vars I can reference from the plugin? Where could I find
a list of them?
--David Reagan
Post by Brian Coca
log_plays was designed to drop the json to syslog-ng which would then
push it to elastic search w/o need for logstash, probably easier to
just setup syslog-ng to do the same.
--
Brian Coca
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/zOjsYxLN578/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAJ5XC8n722sXnV547ZCjrTxBeaDQOqxzxQ1QEs0fYeq5AuQ6Sg%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+***@googlegroups.com.
To post to this group, send email to ansible-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANo%2B_AfMFvypiqYdFWbOopLNNa3yx_Y-vxQBmY%2BHiY85fYCSYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Brian Coca
2015-07-07 20:19:58 UTC
Permalink
status will be OK in both cases, you need to look at changed : true/false
Post by David Reagan
http://pastebin.com/G819PEZY
Questions: The status of that restart apache 2 task should be "CHANGED",
since it actually did change during the play run. But it seems like the
value for changed tasks is always "OK". Is there a way to change that?
--David Reagan
Post by David Reagan
The default log_plays doesn't actually output all the information I need.
I have lots of stuff going to syslog and then into ELK already, but in
this case, I figured I'd just let logstash-forwarder watch the ansible log
file. Then format the output so that logstash doesn't have to filter it at
all.
Currently, I figured out how to get valid json out per line. But I'm stuck
figuring out how to get the task name, the role name, and the command line
command information.
Are there global vars I can reference from the plugin? Where could I find
a list of them?
--David Reagan
Post by Brian Coca
log_plays was designed to drop the json to syslog-ng which would then
push it to elastic search w/o need for logstash, probably easier to
just setup syslog-ng to do the same.
--
Brian Coca
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/zOjsYxLN578/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAJ5XC8n722sXnV547ZCjrTxBeaDQOqxzxQ1QEs0fYeq5AuQ6Sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CANo%2B_AfMFvypiqYdFWbOopLNNa3yx_Y-vxQBmY%2BHiY85fYCSYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
Brian Coca
--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+***@googlegroups.com.
To post to this group, send email to ansible-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAJ5XC8nvqO_3HKZmfBD_NhxW6nY7mfoZHPCuZTfz%2B2QiRM1QJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
David Reagan
2015-07-07 22:32:21 UTC
Permalink
Ah, that makes sense. Thanks!

--David Reagan
Post by Brian Coca
status will be OK in both cases, you need to look at changed : true/false
Post by David Reagan
http://pastebin.com/G819PEZY
Questions: The status of that restart apache 2 task should be "CHANGED",
since it actually did change during the play run. But it seems like the
value for changed tasks is always "OK". Is there a way to change that?
--David Reagan
Post by David Reagan
The default log_plays doesn't actually output all the information I
need.
Post by David Reagan
Post by David Reagan
I have lots of stuff going to syslog and then into ELK already, but in
this case, I figured I'd just let logstash-forwarder watch the ansible
log
Post by David Reagan
Post by David Reagan
file. Then format the output so that logstash doesn't have to filter it
at
Post by David Reagan
Post by David Reagan
all.
Currently, I figured out how to get valid json out per line. But I'm
stuck
Post by David Reagan
Post by David Reagan
figuring out how to get the task name, the role name, and the command
line
Post by David Reagan
Post by David Reagan
command information.
Are there global vars I can reference from the plugin? Where could I
find
Post by David Reagan
Post by David Reagan
a list of them?
--David Reagan
Post by Brian Coca
log_plays was designed to drop the json to syslog-ng which would then
push it to elastic search w/o need for logstash, probably easier to
just setup syslog-ng to do the same.
--
Brian Coca
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/zOjsYxLN578/unsubscribe.
Post by David Reagan
Post by David Reagan
Post by Brian Coca
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAJ5XC8n722sXnV547ZCjrTxBeaDQOqxzxQ1QEs0fYeq5AuQ6Sg%40mail.gmail.com
.
Post by David Reagan
Post by David Reagan
Post by Brian Coca
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CANo%2B_AfMFvypiqYdFWbOopLNNa3yx_Y-vxQBmY%2BHiY85fYCSYg%40mail.gmail.com
.
Post by David Reagan
For more options, visit https://groups.google.com/d/optout.
--
Brian Coca
--
You received this message because you are subscribed to a topic in the
Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/ansible-project/zOjsYxLN578/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAJ5XC8nvqO_3HKZmfBD_NhxW6nY7mfoZHPCuZTfz%2B2QiRM1QJQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+***@googlegroups.com.
To post to this group, send email to ansible-***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANo%2B_Ae%2B5uS3PPHo7S984HvP%2B6oGLMt4zcHUfCcPPfN-noQxJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...