Golang Concurrency Code Review of Codewalk

Question

I'm trying to understand best practices for Golang concurrency. I read O'Reilly's book on Go's concurrency and then came back to the Golang Codewalks, specifically this example:

https://golang.org/doc/codewalk/sharemem/

This is the code I was hoping to review with you in order to learn a little bit more about Go. My first impression is that this code is breaking some best practices. This is of course my (very) unexperienced opinion and I wanted to discuss and gain some insight on the process. This isn't about who's right or wrong, please be nice, I just want to share my views and get some feedback on them. Maybe this discussion will help other people see why I'm wrong and teach them something.

I'm fully aware that the purpose of this code is to teach beginners, not to be perfect code.

Issue 1 - No Goroutine cleanup logic

func main() {
    // Create our input and output channels.
    pending, complete := make(chan *Resource), make(chan *Resource)

    // Launch the StateMonitor.
    status := StateMonitor(statusInterval)

    // Launch some Poller goroutines.
    for i := 0; i < numPollers; i++ {
        go Poller(pending, complete, status)
    }

    // Send some Resources to the pending queue.
    go func() {
        for _, url := range urls {
            pending <- &Resource{url: url}
        }
    }()

    for r := range complete {
        go r.Sleep(pending)
    }
}

The main method has no way to cleanup the Goroutines, which means if this was part of a library, they would be leaked.

Issue 2 - Writers aren't spawning the channels

I read that as a best practice, the logic to create, write and cleanup a channel should be controlled by a single entity (or group of entities). The reason behind this is that writers will panic when writing to a closed channel. So, it is best for the writer(s) to create the channel, write to it and control when it should be closed. If there are multiple writers, they can be synced with a WaitGroup.

func StateMonitor(updateInterval time.Duration) chan<- State {
    updates := make(chan State)
    urlStatus := make(map[string]string)
    ticker := time.NewTicker(updateInterval)
    go func() {
        for {
            select {
            case <-ticker.C:
                logState(urlStatus)
            case s := <-updates:
                urlStatus[s.url] = s.status
            }
        }
    }()
    return updates
}

This function shouldn't be in charge of creating the updates channel because it is the reader of the channel, not the writer. The writer of this channel should create it and pass it to this function. Basically saying to the function "I will pass updates to you via this channel". But instead, this function is creating a channel and it isn't clear who is responsible of cleaning it up.

Issue 3 - Writing to a channel asynchronously

This function:

func (r *Resource) Sleep(done chan<- *Resource) {
    time.Sleep(pollInterval + errTimeout*time.Duration(r.errCount))
    done <- r
}

Is being referenced here:

for r := range complete {
    go r.Sleep(pending)
}

And it seems like an awful idea. When this channel is closed, we'll have a goroutine sleeping somewhere out of our reach waiting to write to that channel. Let's say this goroutine sleeps for 1h, when it wakes up, it will try to write to a channel that was closed in the cleanup process. This is another example of why the writters of the channels should be in charge of the cleanup process. Here we have a writer who's completely free and unaware of when the channel was closed.

Please

If I missed any issues from that code (related to concurrency), please list them. It doesn't have to be an objective issue, if you'd have designed the code in a different way for any reason, I'm also interested in learning about it.

Biggest lesson from this code

For me the biggest lesson I take from reviewing this code is that the cleanup of channels and the writing to them has to be synchronized. They have to be in the same for{} or at least communicate somehow (maybe via other channels or primitives) to avoid writing to a closed channel.

Answer 1

1. It is the main method, so there is no need to cleanup. When main returns, the program exits. If this wasn't the main, then you would be correct.

2. There is no best practice that fits all use cases. The code you show here is a very common pattern. The function creates a goroutine, and returns a channel so that others can communicate with that goroutine. There is no rule that governs how channels must be created. There is no way to terminate that goroutine though. One use case this pattern fits well is reading a large resultset from a database. The channel allows streaming data as it is read from the database. In that case usually there are other means of terminating the goroutine though, like passing a context.

3. Again, there are no hard rules on how channels should be created/closed. A channel can be left open, and it will be garbage collected when it is no longer used. If the use case demands so, the channel can be left open indefinitely, and the scenario you worry about will never happen.

Answer 2

1. As you are asking about if this code was part of a library, yes it would be poor practice to spawn goroutines with no cleanup inside a library function. If those goroutines carry out documented behaviour of the library, it's problematic that the caller doesn't know when that behaviour is going to happen. If you have any behaviour that is typically "fire and forget", it should be the caller who chooses when to forget about it. For example:

func doAfter5Minutes(f func()) {
   go func() {
       time.Sleep(5 * time.Minute)
       f()
       log.Println("done!")
   }()
}

Makes sense, right? When you call the function, it does something 5 minutes later. The problem is that it's easy to misuse this function like this:

// do the important task every 5 minutes
for {
    doAfter5Minutes(importantTaskFunction)
}

At first glance, this might seem fine. We're doing the important task every 5 minutes, right? In reality, we're spawning many goroutines very quickly, probably consuming all available memory before they start dropping off.

We could implement some kind of callback or channel to signal when the task is done, but really, the function should be simplified like so:

func doAfter5Minutes(f func()) {
   time.Sleep(5 * time.Minute)
   f()
   log.Println("done!")
}

Now the caller has the choice of how to use it:

// call synchronously
doAfter5Minutes(importantTaskFunction)
// fire and forget
go doAfter5Minutes(importantTaskFunction)

2. This function arguably should also be changed. As you say, the writer should effectively own the channel, as they should be the one closing it. The fact that this channel-reading function insists on creating the channel it reads from actually coerces itself into this poor "fire and forget" pattern mentioned above. Notice how the function needs to read from the channel, but it also needs to return the channel before reading. It therefore had to put the reading behaviour in a new, un-managed goroutine to allow itself to return the channel right away.

func StateMonitor(updates chan State, updateInterval time.Duration) {
    urlStatus := make(map[string]string)
    ticker := time.NewTicker(updateInterval)
    defer ticker.Stop() // not stopping the ticker is also a resource leak

    for {
        select {
        case <-ticker.C:
            logState(urlStatus)
        case s := <-updates:
            urlStatus[s.url] = s.status
        }
    }

}

Notice that the function is now simpler, more flexible and synchronous. The only thing that the previous version really accomplishes, is that it (mostly) guarantees that each instance of StateMonitor will have a channel all to itself, and you won't have a situation where multiple monitors are competing for reads on the same channel. While this may help you avoid a certain class of bugs, it also makes the function a lot less flexible and more likely to have resource leaks.

3. I'm not sure I really understand this example, but the golden rule for channel closing is that the writer should always be responsible for closing the channel. Keep this rule in mind, and notice a few points about this code:

• The Sleep method writes to r
• The Sleep method is executed concurrently, with no method of tracking how many instances are running, what state they are in, etc.

Based on these points alone, we can say that there probably isn't anywhere in the program where it would be safe to close r, because there's seemingly no way of knowing if it will be used again.