Setup
I initially had TIG-stack (Telegraf, InfluxDB and Grafana) deployed in a Linux server where it worked as expected. I've recently deployed it on OKD (OpenShift upstream). I have several dashboards each of which work when the time range is small enough and data can be loaded 'instantaneously'.
Problem
But when I specify a time range of over an hour, I start to see POST requests getting blocked by browser from the network tab of webpage inspect. This behavior is consistent across several browsers (Firefox, Edge and Chrome).
The POST requests are blocked almost right as they are sent and don't fail after a timeout. In Firefox, I see the error NS_BINDING_ABORTED. Grafana dashboard keeps sending the request with no luck (as seen from inspect screenshot below).
After a minute or two, these POST requests intermittently start to succeed out of the blue. I have not been able to figure out what's causing this issue.
I copied the request body and headers of these requests into postman and tried sending from there which worked flawlessly, leading me to think this may not be a grafana issue or a load balancer issue. To further isolate the load-balancer, I exposed grafana on a nodeport service to access it outside of a load-balancer and got the same behavior.
To see if this is an issue with influxdb, I ran the queries specified in the request body directly within influxdb pod and got an appropriate output.
Some online posts suggest firewalls can cause these issues but I don't have any firewall installed. The client request goes to the load-balancer which routes it to one of the OKD servers.
Another post suggested HTTP traffic to HTTPS connection can be a cause of this. I have self-signed certificates for the HTTPS connections, so could this be an issue? So far I haven't found any evidence backing this in my problem. To negate this issue, I created an HTTP/insecure route, and got the same behavior leading me to think this may also be not the issue.
It appears to me some sort of configuration issue that would resolve it for the browser. Please let me know how I can further troubleshoot this as I've spent significant time on this issue without any luck.
