I needed to call a web service a specific number of times, and add the results to a file (order being unimportant). It might be possible to simply spawn a goroutine for every request, but a more elegant solution is to use the producer-consumer pattern:
func main() {
jobs := make(chan bool)
results := make(chan string)
done := make(chan bool)
numberOfWorkers:= 5
numberOfJobs:= 1000
for i := 0; i < numberOfWorkers; i++ {
go worker(jobs, results, done)
}
go func() {
for i := 0; i < numberOfJobs; i++ {
jobs <- true
}
}()
go func() {
count := 0
for {
result := <-results
println(result)
count++
if count >= numberOfJobs {
done <- true
return
}
}
}()
<-done
}
func worker(jobs chan bool, result chan string, done chan bool) {
for {
select {
case <-jobs:
res, err := getResult()
if err != nil {
panic(err)
}
results <- res
case <-done:
return
}
}
}
func getResult() (string, error) {
resp, err := http.Get("http://localhost/foo")
if err != nil {
return 0, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return 0, err
}
return body, nil
}
This solution uses 3 channels: one to specify the work still remaining (a bool in this case, but this channel could include data to vary the request), one to return results from the workers; and one to signify that work was complete.
This turned out to be a more interesting problem than it looked. My first attempt was using a WaitGroup, but I couldn’t get that to work. And debugging revealed an interesting gap in my mental model of how channels work: I hadn’t fully internalised the fact that pushing a message onto a go channel will block unless someone is ready to consume it. Hence the need to push work onto the jobs channel in a separate goroutine. I had it in my head that they were more like an Erlang/F# mailbox, despite the fact I knew that buffered channels were a thing.
The error handling could also be improved, currently if any request fails it just panics.
