Server Fault
Server Fault

Questions tagged [fluentd]

Fluentd is an open source log aggregator.

Fluentd is a fully free and fully open-source log collector that instantly enables you to have a "Log Everything" architecture with 125+ types of systems.

Fluentd treats logs as JSON, a popular machine-readable format. It is written primarily in C with a thin-Ruby wrapper that gives users flexibility.

Fluentd's performance has been proven in the field: its largest user currently collects logs from 5000+ servers, 5 TB of daily data, handling 50,000 msgs/sec at peak time.

50 questions
5
votes
0 answers

Consuming multiline json in fluentd

I am new to fluentd. I have applications that run in Docker containers. They are Java apps that log in JSON format. The JSON messages are usually split over multiple lines. I would like to use the Docker fluentd log driver to send these messages to…
wimnat
  • 309
  • 2
  • 13
5
votes
1 answer

How to maximize utilization of fluentd server?

I have a fluentd server that is processing an average of 1000 req/s. This server is composed by 32 cpus. In this server are running 32 fluentd docker containers with the same configuration. The entry point of the server is an nginx as a load…
MiquelB
  • 53
4
votes
1 answer

How to specify the log tag from docker compose on google logging?

In my docker compose file, I can specify the logging driver as a "fluentd" and by default, it pushes the log to the listener of google-fluentd on port 24224 I can see the log appear on the logging page along with the tag as the container Id.…
Thai Tran
  • 213
4
votes
1 answer

logstash-forward equivalent for fluentd?

Is there something equivalent to logstash-forwarder that can ship logfiles to fluentd? I am trying to send log files from an application to a remote fluentd but have not seen whether this is possible via an agent somehow. I understand that you can…
adamo
  • 7,045
3
votes
3 answers

Disable fluentd on on Container-Optimized OS (GCE)

I would like to disable fluentd on a VM instance running Container-Optimized OS on Google compute engine. The reason being that it is eating up a lot of RAM for functionalities I do not require. $ ps aux --sort -rss USER PID %CPU %MEM VSZ…
Nils André
  • 131
3
votes
2 answers

fluent configuration not working

I have setup fluent with elasticsearch and Kibana with nginx. I followed the instruction and when I go to http:/192.168.10.25:8888 I get the following message 400 Bad Request 'json' or 'msgpack' parameter is required Below is my fluent.conf ##…
rahrahruby
  • 587
2
votes
0 answers

Monitoring Cassandra Server Logs?

I've done some searching and have not found any examples of people using something like FluentD to forward Cassandra log file data (system.log) to a log indexer. Is this something any is doing? If not, why? If yes, are there example fluentD configs…
Drew
  • 123
2
votes
0 answers

fluentd modular configuration

Is there any way to setup fluentd/td-agent in a way that it's configuration will be modular? I know there is @include directive but this works only if every time I add something new I modify main td-agent.conf file adding new rule to replace tag…
termil0r
  • 133
2
votes
2 answers

How to collect logs for statistics in site?

I want to collect logs from nginx ( several servers, 1 000 000 lines in logs for minute for all servers ) to central stat server for statistics processing. I see 2 variants: write logs to local log files for each servers rename logs for template…
Bdfy
  • 181
2
votes
1 answer

Forward linux logs to fluentd on linux

On one VM I have this in /etc/rsyslog.d/50-default.conf *.* @192.168.29.1:42185 # Default rules for rsyslog. On the vm with fluentd I have this: I have this in /etc/td-agent/td-agent.conf type syslog port 42185 tag …
James Black
  • 133
2
votes
2 answers

Fluent Bit as Syslog Server Alternative

We have an application deployed on AWS EKS cluster, which has a requirement of sending the application error messages to a syslog server. Our default logs shipper is Fluent Bit, deployed as a Daemonset in the AWS EKS cluster. Going over the official…
edwio
  • 121
2
votes
2 answers

How to get JSON-tokenized logs into cloudwatch with fluent bit in EKS 1.22 (containerd)

I would like to get the message of my log entry into AWS with correct json tokenization from CRI application logs when running in AWS EKS (version 1.22) My application outputs valid json, but the log line is prepended with YYYY-MM-DDTHH:MM:SS.MILLIZ…
Anonymouslemming
  • 951
2
votes
2 answers

Rsyslog - "Warning: CA certificate is not set" but TLS forwarding still works

I am configuring rsyslog to forward over TCP/TLS. I was curious why, unlike HTTPS, I had to add the certificate to my client, so I went against the README and commented out the cert config, leaving me with this: #…
threewordphrase
  • 33
  • 1
  • 4
2
votes
1 answer

GCE stackdriver logging agent (fluentd) memory leak with COS

I have a VM on GCE where I run a custom Docker image. I installed it on COS (cos-stable-74-11895-125-0) to a g1-small (1 vCPU, 1.7 GB memory) instance. It is an Elasticsearch with locked memory settings. It consumes exactly 1 GB of RAM. The setup…
Krisztián Velez
  • 121
1
vote
1 answer

Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search

I have very complex log messages, that I want to reduce to the most important fields in order to save quota. The log messages are multiline and there is a lot of redundant information in them. A solution is to script something to reformat these logs…
gspoosi
  • 141
1
2 3 4