how to ignore printing Max depth limit reached go colly

Question

i have a go colly crawler that i am trying to crawl many sites . on my terminal it prints a lot of :

2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached

it makes it hard for me to read some prints that i have placed . i wanted to know if there is any way to ignore printing this in termial . thanks

Are you sure `gocolly` logs it? – zerkms May 30 '23 at 00:07 — zerkms, May 30 '23 at 00:07

score 1 · Accepted Answer · answered May 30 '23 at 01:49

Max depth limit reached is colly.ErrMaxDepth. There must be codes like this in your project:

c := colly.NewCollector(colly.MaxDepth(5))

// ...

if err := c.Visit("http://go-colly.org/"); err != nil {
    log.Println(err)
}

If you don't want to log this error, add a simple check to exclude it:

c := colly.NewCollector(colly.MaxDepth(5))

// ...

if err := c.Visit("http://go-colly.org/"); err != nil {
    // log the error only when the error is not ErrMaxDepth.
    if err != colly.ErrMaxDepth {
        log.Println(err)
    }
}

Another option is to redirect the output to a file:

go run . 2>&1 >log.txt

or copy the output to a file and also to standard output with tee:

go run . 2>&1 | tee log.txt

that did the job i didnt know the maxDepth has special error type thanks <3 — Farshad, May 30 '23 at 18:10

how to ignore printing Max depth limit reached go colly

1 Answers1