I've set up a two-node Graphite cluster connected using carbon-relay:
I sent 500 counters for my metric cluster-test-500 over ~1min period, at approximately 60/10 seconds.
Based on my aggregation rules:
Meta data:
aggregation method: sum
max retention: 31536000
xFilesFactor: 0
Archive 0 info:
offset: 52
seconds per point: 10
points: 2160
retention: 21600
size: 25920
Archive 1 info:
offset: 25972
seconds per point: 60
points: 10080
retention: 604800
size: 120960
Archive 2 info:
offset: 146932
seconds per point: 600
points: 52560
retention: 31536000
size: 630720
The data appeared in both nodes' Whisper DBs:
data:
12
63
61
62
63
60
61
57
0
data:
261
239
0
data:
500
0
Then, when I queried Graphite using the summarize function, over the past day:
http://graphite.mysite.com/render?target=summarize(stats.counters.cluster-test-500.*.*.*.count,"1day","sum","False")&format=json&tz=UTC&from=18:00_20130923&until=18:00_20130924
It returns inconsistent results by simply re-running the same query:
[{"target": "summarize(stats.counters.cluster-test-500.count, \"1day\", \"sum\", true)", "datapoints": [[302.0, 1379959260]]}]
[{"target": "summarize(stats.counters.cluster-test-500.count, \"1day\", \"sum\", true)", "datapoints": [[500.0, 1379959260]]}]
[{"target": "summarize(stats.counters.cluster-test-500.count, \"1day\", \"sum\", true)", "datapoints": [[261.0, 1379959260]]}]
[{"target": "summarize(stats.counters.cluster-test-500.count, \"1day\", \"sum\", true)", "datapoints": [[63.0, 1379959260]]}]
Which appears to result from seemingly random summations of time segments in the WhisperDB file. What on earth could be causing this?
