Look on my works, ye Mighty, and despair!

I needed to call a web service a specific number of times, and add the results to a file (order being unimportant). It might be possible to simply spawn a goroutine for every request, but a more elegant solution is to use the producer-consumer pattern:

func main() {
	jobs := make(chan bool)
	results := make(chan string)
	done := make(chan bool)
	numberOfWorkers:= 5
	numberOfJobs:= 1000
	for i := 0; i < numberOfWorkers; i++ {
		go worker(jobs, results, done)
	}
	go func() {
		for i := 0; i < numberOfJobs; i++ {
			jobs <- true
		}
	}()
	go func() {
		count := 0
		for {
			result := <-results
			println(result)
			count++
			if count >= numberOfJobs {
				done <- true
				return
			}
		}
	}()
	<-done
}

func worker(jobs chan bool, result chan string, done chan bool) {
	for {
		select {
		case <-jobs:
			res, err := getResult()
			if err != nil {
				panic(err)
			}
			results <- res
		case <-done:
			return
		}
	}
}

func getResult() (string, error) {
	resp, err := http.Get("http://localhost/foo")
	if err != nil {
		return 0, err
	}
	defer resp.Body.Close()
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return 0, err
	}
	return body, nil
}

This solution uses 3 channels: one to specify the work still remaining (a bool in this case, but this channel could include data to vary the request), one to return results from the workers; and one to signify that work was complete.

This turned out to be a more interesting problem than it looked. My first attempt was using a WaitGroup, but I couldn’t get that to work. And debugging revealed an interesting gap in my mental model of how channels work: I hadn’t fully internalised the fact that pushing a message onto a go channel will block unless someone is ready to consume it. Hence the need to push work onto the jobs channel in a separate goroutine. I had it in my head that they were more like an Erlang/F# mailbox, despite the fact I knew that buffered channels were a thing.

The error handling could also be improved, currently if any request fails it just panics.