Apologies in advance if I'm mixing some terminology. I'm relatively new to Splunk.
I'm building a Splunk app to monitor our product, [Mattermost](https://mattermost.com/). We expose Prometheus style metrics and I'm using [this Prometheus data input type](https://github.com/lukemonahan/splunk_modinput_prometheus) by @luke.monahan@rivium.com.au to get the metrics in (thanks Luke!).
We have a metric `mattermost_db_master_connections_total` that is displayed in a Single Value chart at the top of a dashboard, as well as in amongst some time series charts below. The time series chart seems to match with what I see in our equivalent Grafana dashboard, but the Single Value stat seems to bounce between `0` and the value I would expect, depending on the timing of when I refresh the dashboard.
Is there something I should be doing in my query to smooth out those drops to `0` on the single value panel? What is happening here? Missing values I don't see on the time series?
Single Value Query:
`
| mstats max(_value) prestats=true WHERE metric_name="mattermost_db_master_connections_total" AND sourcetype=prometheus:metric span=15s
| timechart max(_value) span=15s
`
Time Series Query:
`| mstats max(_value) prestats=true WHERE metric_name="mattermost_db_master_connections_total" AND sourcetype=prometheus:metric span=15s BY host
| timechart max(_value) span=15s agg=max useother=false BY host
| addtotals
`
Screenshots:
Single Value (after refreshing... and seemingly majority of the time it looks like this... every few refreshes I'll get `3` as expected):
![alt text][1]
Time Series Plot:
![alt text][2]
[1]: /storage/temp/288572-screen-shot-2020-04-17-at-71127-am.png
[2]: /storage/temp/288573-screen-shot-2020-04-17-at-71318-am.png
↧