github.com/beauhoyt/goconcurrencypatterns
该文档源自 Rob Pike 在google IO中的演讲，版权归原作者所有。如果你在天朝，可以点击这里观看演讲视频。所有的代码皆为本人根据演讲稿内容编写，并调试通过。如发现bug欢迎提交更新。
People seemed fascinated by the concurrency features of Go when the language was firest announced.
Questions:
Look around you, What do you see?
Do you see a single-stepping world doing one thing at a time?
Or do you see a complex world of interacting, independently behaving pieces?
That's why. Sequential processing on its own does not model the world's behavior.
Concurrency is the composition of independently executing computations.
Concurrency is a way to structure software, particaularly as a way to write clean code that interacts well with the real world.
It is not parallelism.
Concurrency is not paralleism, although it enables parallelism.
If you have only one processor, your program can still be concurrent but it cannot be parallel.
On the other hand, a well-written concurrent program might run efficiently in parallel on a multiprocessor. That property could be important...
See tinyurl.com/goconcnotpar for more on that distinction. Too much to discuss here.
Easy to understand.
Easy to use.
Easy to reason about.
You don't need to be an expert!
(Much nicer than dealing with the minutiae of parallelism (threads, semaphores, locks, barries, etc.))
To many, the concurrency features of Go seemed new.
But they are rooted in a long history, reaching back to Hoare's CSP in 1978 and even Dijkstra's guarded commands(1975).
Languages with similar features:
Go is the latest on the Newsqueak-Alef-Limbo branch, distinguished by first-class channels.
Erlang is closer to the original CSP, where you communicate to a process by name rather than over a channel.
The models are equivalent but express things differently
Rough analogy: writing to a file by name(process, Erlang) vs. writing to a file descriptor (channel, Go).
We need an example to show the interesting properties of the concurrency primitives.
To avoid distraction, we make it a boring example.
func boring(msg string) {
for i := 0; ; i++ {
fmt.Println(msg, i)
time.Sleep(time.Second)
}
}
Make the intervals between messages unpredictable (still under a second).
func boring(msg string) {
for i := 0; ; i++ {
fmt.Println(msg, i)
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}
The boring function runs on forever, like a boring party guest.
func main() {
boring("boring!")
}
func boring(msg string) {
for i := 0; ; i++ {
fmt.Println(msg, i)
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}
The go statement runs the function as usual, but doesn't make the caller wait.
It launches a goroutine.
The functionality is analogous to the & on the end of a shell command.
package main
import (
"fmt"
"time"
"math/rand"
)
func main() {
go boring("boring!")
}
When main returns, the program exits and takes the boring function down with it.
We can hang around a little, and on the way show that both main and the launched goroutine are running.
func main() {
go boring("boring!")
fmt.Println("I'm listening")
time.Sleep(2 * time.Second)
fmt.Println("You're boring; I'm leaving.")
}
What is a goroutine? It's an independently executing function, launched by a go statement.
It has its own call stack, which grows and shrinks as required.
It's very cheap. It's practical to have thousands, even hundreds of thousands of goroutines.
It's not a thread.
There might be only one thread in a program with thousands of goroutines.
Instead, goroutines are multiplexed dynamically onto threads are needed to keep all the goroutines running.
But if you think of it as a very cheap thread, you won't be far off.
Our boring examples cheated: the main function couldn't see the output from the other goroutine.
It was just printed to the screen, where we pretended we saw a conversation.
Real conversations require communication.
A channel in Go provides a connection betwwen two goroutines, allowing them to communicate.
// Declaring and initializing.
var c chan int
c = make(chan int)
// or
c := make(chan int)
// Sending on a channel.
c <- 1
// Receiving from a channel.
// The "arrow" indicates the direction of data flow.
value = <- c
A channel connects the main and boring goroutines so they can communicate.
func main() {
c := make(chan string)
go boring("boring!", c)
for i := 0; i < 5; i++ {
fmt.Printf("You say: %q\n", <-c) // Receive expression is just a value.
}
fmt.Println("You're boring; I'm leaving.")
}
func boring(msg string, c chan string) {
for i := 0; ; i++ {
c <- fmt.Sprintf("%s %d", msg, i) // Expression to be sent can be any suitable value.
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}
When the main function executes <-c, it will wait for a value to be sent.
Similarly, when the boring function executes c <- value, it waits for a receiver to be ready.
A sender and receiver must both be ready to play their part in the communication. Otherwise we wait until they are.
Thus channels both communicate and synchronize.
Note for experts: Go channels can also be created with a buffer.
Buffering removes synchronization.
Buffering makes them more like Erlang's mailboxes.
Buffered channels can be important for some problems but they are more subtle to reason about.
We won't need them today.
Don't communicate by sharing memory, share memory by communicating.
Channels are first-class values, just like strings or integers.
func main() {
c := boring("boring!") // Function returning a channel.
for i := 0; i < 5; i++ {
fmt.Printf("You say: %q\n", <-c)
}
fmt.Println("You're boring; I'm leaving.")
}
func boring(msg string) <-chan string{ // Returns receive-only channel of strings.
c := make(chan string)
go func() {
for i := 0; ; i++ {
c <- fmt.Sprintf("%s %d", msg, i)
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}()
return c // Return the channel to the caller.
}
Our boring function returns a channel that lets us communicate with the boring service it provides.
We can have more instances of the service.
func main(){
joe := boring("Joe")
ann := boring("Ann")
for i := 0; i < 5; i++ {
fmt.Println(<-joe)
fmt.Println(<-ann)
}
fmt.Println("You're both boring; I'm leaving.")
}
These programs make Joe and Ann count in lockstep. We can instead use a fan-in function to let whosoever is ready talk.
func fanIn(input1, input2 <-chan string) <-chan string {
c := make(chan string)
go func() { for { c <- <-input1 } }()
go func() { for { c <- <-input2 } }()
return c
}
func main(){
c := fanIn(boring("Joe"), boring("Ann"))
for i := 0; i < 10; i++ {
fmt.Println(<-c)
}
fmt.Println("You're both boring; I'm leaving.")
}
Send a channel on a channel, making goroutine wait its turn.
Receive all messages, then enable them again by sending on a private channel.
First we define a message type that contains a channel for the reply.
type Message struct {
str string
wait chan bool
}
Each speaker must wait for a go-ahead.
for i := 0; i < 5; i++ {
msg1 := <-c; fmt.Println(msg1.str)
msg2 := <-c; fmt.Println(msg2.str)
msg1.wait <- true
msg2.wait <- true
}
waitForIt := make(chan bool) // Shared between all messages.
c <- Message( fmt.Sprintf("%s: %d", msg, i), waitForIt )
time.Sleep(time.Duration(rand.Intn(2e3)) * time.Millisecond)
<- waitForIt
A control structure unique to concurrency.
The reason channels and goroutines are built into the language.
The select statement provides another way to handle multiple channels. It's like a switch, but each case is a communication:
select {
case v1 := <-c1:
fmt.Printf("received %v from c1\n", v1)
case v2 := <-c2:
fmt.Printf("received %v from c2\n", v2)
case c3 <- 23:
fmt.Printf("sent %v to c3\n", 23)
default:
fmt.Printf("no one was ready to communiction\n")
}
Rewrite our original fanin function. Only one goroutine is needed. Old:
func fanIn(input1, input2 <-chan string) <-chan string {
c := make(chan string)
go func() { for { c <- <-input1 } }()
go func() { for { c <- <-input2 } }()
return c
}
Rewrite our original fanin function. Only one goroutine is needed. New:
func fanIn(input1, input2 <-chan string) <-chan string {
c := make(chan string)
go func() {
for {
select {
case s := <-input1: c <- s
case s := <-input2: c <- s
}
}
}()
return c
}
The time.After function returns a channel that blocks for the specified duration. After the interval, the channel delivers the current time, once.
func main(){
c := boring("Joe")
for {
select {
case s:= <-c:
fmt.Println(s)
case <-time.After(1 * time.Second):
fmt.Println("You're too slow.")
return
}
}
fmt.Println("You're both boring; I'm leaving.")
}
Create the timer once, outside the loop, to time out the entire conversation. (In the previous program, we had a timeout for each message.)
func main(){
c := boring("Joe")
timeout := time.After(5 * time.Second)
for {
select {
case s:= <-c:
fmt.Println(s)
case <-timeout:
fmt.Println("You're talk too much.")
return
}
}
fmt.Println("You're both boring; I'm leaving.")
}
We can turn this around and tell Joe to stop when we're tired of listening to him.
quit := make(chan bool)
c := boring("Joe", quit)
for i := rand.Intn(20); i >= 0; i-- { fmt.Println(<-c) }
quit <- true
select {
case c <- fmt.Sprintf("%s %d", msg, i):
// do nothing
case <-quit:
return
}
How do we know it's finished? Wait for it to tell us it's done: receive on the quit channel
quit := make(chan string)
c := boring("Joe", quit)
for i := rand.Intn(20); i >= 0; i-- { fmt.Println(<-c) }
quit <- "Bye!"
fmt.Printf("Joe says: %q\n", <-quit)
select {
case c <- fmt.Sprintf("%s %d", msg, i):
// do nothing
case <-quit:
cleanup()
quit <- "See you!"
return
}
func f(left, right chan int) {
left <- 1 + <-right
}
func main() {
const n = 10000
leftmost := make(chan int)
right := leftmost
left := leftmost
for i := 0; i < n; i++ {
right = make(chan int)
go f(left, right)
left = right
}
go func(c chan int) { c <- 1}(right)
fmt.Println(<-leftmost)
}
Go was designed for writing systems software. Let's see how the concurrency features come into play.
Q: What does Google search do?
A: Given a query, return a page of search results (and some ads).
Q: How do we get the search results?
A: Send the query to Web search, Image search, YouTube, Maps, News, etc., then mix the results.
How do we implement this?
We can simulate the search function, much as we simulated conversation before.
The Google functio takes a query and returns a slice of Results (which are just strings).
Google invokes Web, Image, and Video searches serially, appending them to the results slice.
func Google(query string) []Result{
results := make([]Result, 3, 10)
results = append(results, Web(query))
results = append(results, Image(query))
results = append(results, Video(query))
return results
}
Run the Web, Image, and Video searchs concurently, and wait for all results.
No locks, No condition variables. No callbacks.
func Google(query string) (results []Result) {
c := make(chan Result)
go func() { c <- Web(query) } ()
go func() { c <- Image(query) } ()
go func() { c <- Video(query) } ()
for i :=0; i < 3; i++ {
result := <-c
results = append(results, result)
}
return
}
Don't wait for slow servers. No locks. No condition variables. No callbacks.
func Google(query string) (results []Result) {
c := make(chan Result)
go func() { c <- Web(query) } ()
go func() { c <- Image(query) } ()
go func() { c <- Video(query) } ()
timeout := time.After(80 * time.Millisecond)
for i :=0; i < 3; i++ {
select {
case result := <-c:
results = append(results, result)
case <-timeout:
fmt.Println("timed out")
return
}
}
return
}
Q: How do we avoid discarding srsults from slow servers?
A: Replicate the servers. Send request to multiple replicas, and use the first response.
func First(query string, replicas ...Search) Result {
c := make(chan Result)
searchReplica := func(i int) { c <- replicas[i](query) }
for i := range replicas {
go searchReplica(i)
}
return <-c
}
func main() {
rand.Seed(time.Now().UnixNano())
start := time.Now()
result := First("golang", fakeSearch("replica 1"),
fakeSearch("replica 2"))
elapsed := time.Since(start)
fmt.Println(results)
fmt.Println(elapsed)
}
Reduce tail latency using replicated search servers.
func Google(query string) (results []Result) {
c := make(chan Result)
go func() { c <- First(query, Web1, Web2) } ()
go func() { c <- First(query, Image1, Image2) } ()
go func() { c <- First(query, Video1, Video2) } ()
timeout := time.After(80 * time.Millisecond)
for i :=0; i < 3; i++ {
select {
case result := <-c:
results = append(results, result)
case <-timeout:
fmt.Println("timed out")
return
}
}
return
}
No locks. No condition variables. No callbacks.
In just a few simple transformations we used Go's concurrency primitives to convert a
program into one that is
There are endless ways to use these tools, many presented elsewhere.
Chatroulette toy:
Load balancer:
Concurrent prime sieve.
Concurrent power series (by Mcllroy):
The're fun to play with, but don't overuse these ideas
Goroutines and channels are big ideas. They're tools for program construnction.
But sometimes all you need is a reference counter.
Go has "sync" and "sync/atomic" packages that provide mutexes, condition variables, etc. They provide tools for smaller problems.
Ofter, these things will work together to solve a bigger problem.
Always use the right tool for the job.
Goroutines and channels make it easy to express complex operations dealing with
And they're fun to use.
Go Home Page:
Go Tour (learn Go in your browser)
Package documentation:
Articles galore:
Concurrency is not parallelism:
FAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.