4

I have logs coming from repositories into grafana (loki). I am trying to get the number of repositories (that have the msg="Repository finished") for a given duration of, say 6h"

I used these queries but they don't deliver any results

sum by (repository) (sum_over_time ({job=~"$cronjob", job_name=~"$job"} | json  | durationMs != "" | unwrap durationMs | __error__="" [6h] ))

the same for this query:

sum by (repository) (sum_over_time ({job=~"$cronjob", job_name=~"$job"} | json  | durationMs != "" | unwrap time| __error__="" [6h] ))

My logs look like this after performing {job=~"$cronjob", job_name=~"$job"} | json | msg="Repository finished" on them

{"name":"reno","hostname":"reno-01234","pid":9,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3216,"msg":"Repository finished","time":"2022-08-09T12:00:25.580Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":9,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3558,"msg":"Repository finished","time":"2022-08-09T11:00:12.767Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":9,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3036,"msg":"Repository finished","time":"2022-08-09T10:01:30.224Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":8,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3275,"msg":"Repository finished","time":"2022-08-09T09:00:31.077Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":8,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3233,"msg":"Repository finished","time":"2022-08-09T08:00:18.020Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":9,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3581,"msg":"Repository finished","time":"2022-08-09T07:00:28.657Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":7,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3592,"msg":"Repository finished","time":"2022-08-09T06:00:19.073Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":10,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":8509,"msg":"Repository finished","time":"2022-08-09T05:00:34.047Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":8,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3730,"msg":"Repository finished","time":"2022-08-09T04:00:22.514Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":9,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3876,"msg":"Repository finished","time":"2022-08-09T03:00:42.023Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":10,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3754,"msg":"Repository finished","time":"2022-08-09T02:00:29.661Z","v":0}
{"name":"reno","hostname":"reno-01234","pid":10,"level":30,"logContext":"abc","repository":"abc/bmw/gmc","durationMs":3360,"msg":"Repository finished","time":"2022-08-09T01:00:17.274Z","v":0}
QBits
  • 121
  • 1
  • 11

1 Answers1

2

While I don't have direct answer to Your question, I do have something similar that might help anyone who is searching for an answer.

I've decided to provide answer because it turns out that in 99% of search cases it's just hitting a wall - there are no examples or answers to quesion such as presented in here.

My goal was to count number of entries with given string, grouped by days or months and I've managed to get what I wanted.

See:

enter image description here

First of all logs example:

[2022-03-12T13:42:56.782023+00:00] NotifierProxyLogger.INFO: Now calling api:  {"calledMethod":"insertDiscordMessage","baseUrl":"http://...:8902","requestUri":"/api/external/discord/insert-message","dataBag":{"webhookName":"NewsFeed","messageContent":"\n**Title: ** Katarzyna Pełczyńska-Nałęcz gościnią Jacka Gądka w Porannej Rozmowie Gazeta.pl [NA ŻYWO],\n**Description: ** W poniedziałek w Porannej Rozmowie Gazeta.pl gościmy Katarzynę Płeczyńską-Nałęcz, byłą ambasadorkę Polski w Moskwie i członkinię Instytutu Strategie 2050. Rozmowę prowadzi Jacek Gądek.\n**Link: ** https://wiadomosci.gazeta.pl/wiadomosci/7,114884,28135035,katarzyna-pelczynska-nalecz-goscinia-jacka-gadka-w-porannej.html\n        ","messageTitle":"**[NEWS]** Keywords: (wezwanie armia) OR (pobór wojskowy) OR (atak na polskę)","source":"NewsFeed"}} []

[2022-03-19T12:00:03.372023+00:00] NotifierProxyLogger.INFO: Now calling api:  {"calledMethod":"insertDiscordMessage","baseUrl":"http://...","requestUri":"/api/external/discord/insert-message","dataBag":{"webhookName":"NewsFeed","messageContent":"\n**Title: ** Polska mocarstwem światowym,\n**Description: ** Wojna toczy się już 20 dzień, choć nie wiadomo dlaczego.\n**Link: ** https://wolnemedia.net/polska-mocarstwem-swiatowym/\n        ","messageTitle":"**[NEWS]** Keywords: (wezwanie armia) OR (pobór wojskowy) OR (atak na polskę) OR (atak nato) OR (wojna polska)","source":"NewsFeed"}} []

Per day

  • Query: count_over_time({filename="/var/log/external/NAME/news-feed/notifier-proxy-logger-bridge.log"} |= `insertDiscordMessage` | regexp `(?P<timestamp>[0-9]{4}-[0-9]{1,2}-[0-9]{1,2})` [$__interval])
  • Query options:
    • Legend: {{timestamp}}
  • Visualisation: Bar gauge
  • Visualisation config:
    • Orientation: Horizontal
    • Calculation: Total

Per day for year 2023

  • Query: count_over_time({filename="/var/log/external/NAME/news-feed/notifier-proxy-logger-bridge.log"} |= `insertDiscordMessage` |= `2023` | regexp `(\[)(?P<timestamp>2023-[0-9]{1,2}-[0-9]{2})(T)` | timestamp != `` [$__interval])
  • Query options:
    • Legend: {{timestamp}}
  • Visualisation: Bar gauge
  • Visualisation config:
    • Orientation: Vertical
    • Calculation: Total

Per day for year 2023 + total sum

  • Query 1: count_over_time({filename="/var/log/external/NAME/news-feed/notifier-proxy-logger-bridge.log"} |= `insertDiscordMessage` |= `2023` | regexp `(\[)(?P<timestamp>2023-[0-9]{1,2}-[0-9]{2})(T)` | timestamp != `` [$__interval])
  • Query options:
    • Legend: {{timestamp}}
  • Query 2: sum(count_over_time({filename="/var/log/external/NAME/news-feed/notifier-proxy-logger-bridge.log"} |= `insertDiscordMessage` |= `2023` | regexp `(\[)(?P<timestamp>2023-[0-9]{1,2}-[0-9]{2})(T)` | timestamp != `` [$__interval]))
  • Query options:
    • Legend: Total
  • Visualisation: Bar gauge
  • Visualisation config:
    • Orientation: Vertical
    • Calculation: Total
Volmarg Reiso
  • 437
  • 4
  • 11