I'm scraping some metrics (openstack cinder volume sizes) every 15 minutes, and the results produce a discontinuous graph, like this:
(That's the result of the simple query cinder_volume_size_gb).
The metrics "exist" for about five minutes, but then disappear until the next scrape interval. What configuration settings would influence this behavior?
To the title question - yes, it is documented at https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness:
If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. This effectively means that time series "disappear" from graphs at times where their latest collected sample is older than 5 minutes or after they are marked stale.
To the other question - it is poorly documented, but there is a command-line option to change the default:
--query.lookback-delta=5m The maximum lookback duration for retrieving metrics during expression evaluations.
