tools to gather variant information from thousands of servers and update into database

Question

Recently, I use ansible to gather information periodically from thousands of servers(which already installed collectd to gather CPU, memory, disks, netowrking info) in our company.

For example: I need to get those information daily

• kernel version
• installed packages versions(JDK, docker, etc)
• running process info
• some commands output (lldp, docker ps, etc)
• some files content (cat /etc/some/config | grep word)

I use ansible shell module to execute those variant commands and parse the output into our inventory NOSQL.

By using ansible which use SSH/WINRM to access thousands servers seems inefficient.(which fine tune with more forks, enabled ControlMaster pipeline, etc).

I are trying to find a better way to replace ansible for those information gathering and parsing.

Below are some solutions which I am surveying.

facebook/osquery is flexible and powerful, but maybe not easy(comparing to bash script) to run a customized commands and parsing the output by writing an extension.

collectd with exec plugin to run customized scripts seems straightforward. in this way I need to deploy new scripts to all servers from day to day.

Are there some tools which is more suitable for such variant commands output gathering? and easy to deploy, reporting, or may with some dashboard and API for operation.

Answer 1

Doing something with collectd feeding into Prometheus or datadog is definitely a good idea for gathering metrics, but it doesn't help you with things like the kernel version. Handling process monitoring might be better handled with nagios or datadog as well. But for the bulk of what you're doing I'd say that it makes the most sense to stick with ansible.

What you're doing sounds very similar to the ansible-cmdb project:

Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page (and other things) containing system configuration information.

It supports multiple types of output (html, csv, sql, etc) and extending information gathered by Ansible with custom data. For each host it also shows the groups, host variables, custom variables and machine-local facts.

I've been using this for the last couple of years and it continues to evolve and improve. I hope you'll find it useful as well.