54 KiB
TODO: to other notes.
Initialization
Although it doesn't look superficially very different from initialization in C or C++, initialization in Go is more powerful. Complex structures can be built during initialization and the ordering issues among initialized objects, even among different packages, are handled correctly. Constants
Constants in Go are just that—constant. They are created at compile time, even when defined as locals in functions, and can only be numbers, characters (runes), strings or booleans. Because of the compile-time restriction, the expressions that define them must be constant expressions, evaluatable by the compiler. For instance, 1<<3 is a constant expression, while math.Sin(math.Pi/4) is not because the function call to math.Sin needs to happen at run time.
In Go, enumerated constants are created using the iota enumerator. Since iota can be part of an expression and expressions can be implicitly repeated, it is easy to build intricate sets of values.
type ByteSize float64
const ( _ = iota // ignore first value by assigning to blank identifier KB ByteSize = 1 << (10 * iota) MB GB TB PB EB ZB YB )
The ability to attach a method such as String to any user-defined type makes it possible for arbitrary values to format themselves automatically for printing. Although you'll see it most often applied to structs, this technique is also useful for scalar types such as floating-point types like ByteSize.
func (b ByteSize) String() string { switch { case b >= YB: return fmt.Sprintf("%.2fYB", b/YB) case b >= ZB: return fmt.Sprintf("%.2fZB", b/ZB) case b >= EB: return fmt.Sprintf("%.2fEB", b/EB) case b >= PB: return fmt.Sprintf("%.2fPB", b/PB) case b >= TB: return fmt.Sprintf("%.2fTB", b/TB) case b >= GB: return fmt.Sprintf("%.2fGB", b/GB) case b >= MB: return fmt.Sprintf("%.2fMB", b/MB) case b >= KB: return fmt.Sprintf("%.2fKB", b/KB) } return fmt.Sprintf("%.2fB", b) }
The expression YB prints as 1.00YB, while ByteSize(1e13) prints as 9.09TB.
The use here of Sprintf to implement ByteSize's String method is safe (avoids recurring indefinitely) not because of a conversion but because it calls Sprintf with %f, which is not a string format: Sprintf will only call the String method when it wants a string, and %f wants a floating-point value. Variables
Variables can be initialized just like constants but the initializer can be a general expression computed at run time.
var ( home = os.Getenv("HOME") user = os.Getenv("USER") gopath = os.Getenv("GOPATH") )
The init function
Finally, each source file can define its own niladic init function to set up whatever state is required. (Actually each file can have multiple init functions.) And finally means finally: init is called after all the variable declarations in the package have evaluated their initializers, and those are evaluated only after all the imported packages have been initialized.
Besides initializations that cannot be expressed as declarations, a common use of init functions is to verify or repair correctness of the program state before real execution begins.
func init() { if user == "" { log.Fatal("$USER not set") } if home == "" { home = "/home/" + user } if gopath == "" { gopath = home + "/go" } // gopath may be overridden by --gopath flag on command line. flag.StringVar(&gopath, "gopath", gopath, "override default GOPATH") }
Methods Pointers vs. Values
As we saw with ByteSize, methods can be defined for any named type (except a pointer or an interface); the receiver does not have to be a struct.
In the discussion of slices above, we wrote an Append function. We can define it as a method on slices instead. To do this, we first declare a named type to which we can bind the method, and then make the receiver for the method a value of that type.
type ByteSlice []byte
func (slice ByteSlice) Append(data []byte) []byte { // Body exactly the same as the Append function defined above. }
This still requires the method to return the updated slice. We can eliminate that clumsiness by redefining the method to take a pointer to a ByteSlice as its receiver, so the method can overwrite the caller's slice.
func (p *ByteSlice) Append(data []byte) { slice := *p // Body as above, without the return. *p = slice }
In fact, we can do even better. If we modify our function so it looks like a standard Write method, like this,
func (p *ByteSlice) Write(data []byte) (n int, err error) { slice := *p // Again as above. *p = slice return len(data), nil }
then the type *ByteSlice satisfies the standard interface io.Writer, which is handy. For instance, we can print into one.
var b ByteSlice
fmt.Fprintf(&b, "This hour has %d days\n", 7)
We pass the address of a ByteSlice because only *ByteSlice satisfies io.Writer. The rule about pointers vs. values for receivers is that value methods can be invoked on pointers and values, but pointer methods can only be invoked on pointers.
This rule arises because pointer methods can modify the receiver; invoking them on a value would cause the method to receive a copy of the value, so any modifications would be discarded. The language therefore disallows this mistake. There is a handy exception, though. When the value is addressable, the language takes care of the common case of invoking a pointer method on a value by inserting the address operator automatically. In our example, the variable b is addressable, so we can call its Write method with just b.Write. The compiler will rewrite that to (&b).Write for us.
By the way, the idea of using Write on a slice of bytes is central to the implementation of bytes.Buffer. Interfaces and other types Interfaces
Interfaces in Go provide a way to specify the behavior of an object: if something can do this, then it can be used here. We've seen a couple of simple examples already; custom printers can be implemented by a String method while Fprintf can generate output to anything with a Write method. Interfaces with only one or two methods are common in Go code, and are usually given a name derived from the method, such as io.Writer for something that implements Write.
A type can implement multiple interfaces. For instance, a collection can be sorted by the routines in package sort if it implements sort.Interface, which contains Len(), Less(i, j int) bool, and Swap(i, j int), and it could also have a custom formatter. In this contrived example Sequence satisfies both.
type Sequence []int
// Methods required by sort.Interface. func (s Sequence) Len() int { return len(s) } func (s Sequence) Less(i, j int) bool { return s[i] < s[j] } func (s Sequence) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
// Copy returns a copy of the Sequence. func (s Sequence) Copy() Sequence { copy := make(Sequence, 0, len(s)) return append(copy, s...) }
// Method for printing - sorts the elements before printing. func (s Sequence) String() string { s = s.Copy() // Make a copy; don't overwrite argument. sort.Sort(s) str := "[" for i, elem := range s { // Loop is O(N²); will fix that in next example. if i > 0 { str += " " } str += fmt.Sprint(elem) } return str + "]" }
Conversions
The String method of Sequence is recreating the work that Sprint already does for slices. (It also has complexity O(N²), which is poor.) We can share the effort (and also speed it up) if we convert the Sequence to a plain []int before calling Sprint.
func (s Sequence) String() string { s = s.Copy() sort.Sort(s) return fmt.Sprint([]int(s)) }
This method is another example of the conversion technique for calling Sprintf safely from a String method. Because the two types (Sequence and []int) are the same if we ignore the type name, it's legal to convert between them. The conversion doesn't create a new value, it just temporarily acts as though the existing value has a new type. (There are other legal conversions, such as from integer to floating point, that do create a new value.)
It's an idiom in Go programs to convert the type of an expression to access a different set of methods. As an example, we could use the existing type sort.IntSlice to reduce the entire example to this:
type Sequence []int
// Method for printing - sorts the elements before printing func (s Sequence) String() string { s = s.Copy() sort.IntSlice(s).Sort() return fmt.Sprint([]int(s)) }
Now, instead of having Sequence implement multiple interfaces (sorting and printing), we're using the ability of a data item to be converted to multiple types (Sequence, sort.IntSlice and []int), each of which does some part of the job. That's more unusual in practice but can be effective. Interface conversions and type assertions
Type switches are a form of conversion: they take an interface and, for each case in the switch, in a sense convert it to the type of that case. Here's a simplified version of how the code under fmt.Printf turns a value into a string using a type switch. If it's already a string, we want the actual string value held by the interface, while if it has a String method we want the result of calling the method.
type Stringer interface { String() string }
var value interface{} // Value provided by caller. switch str := value.(type) { case string: return str case Stringer: return str.String() }
The first case finds a concrete value; the second converts the interface into another interface. It's perfectly fine to mix types this way.
What if there's only one type we care about? If we know the value holds a string and we just want to extract it? A one-case type switch would do, but so would a type assertion. A type assertion takes an interface value and extracts from it a value of the specified explicit type. The syntax borrows from the clause opening a type switch, but with an explicit type rather than the type keyword:
value.(typeName)
and the result is a new value with the static type typeName. That type must either be the concrete type held by the interface, or a second interface type that the value can be converted to. To extract the string we know is in the value, we could write:
str := value.(string)
But if it turns out that the value does not contain a string, the program will crash with a run-time error. To guard against that, use the "comma, ok" idiom to test, safely, whether the value is a string:
str, ok := value.(string) if ok { fmt.Printf("string value is: %q\n", str) } else { fmt.Printf("value is not a string\n") }
If the type assertion fails, str will still exist and be of type string, but it will have the zero value, an empty string.
As an illustration of the capability, here's an if-else statement that's equivalent to the type switch that opened this section.
if str, ok := value.(string); ok { return str } else if str, ok := value.(Stringer); ok { return str.String() }
Generality
If a type exists only to implement an interface and will never have exported methods beyond that interface, there is no need to export the type itself. Exporting just the interface makes it clear the value has no interesting behavior beyond what is described in the interface. It also avoids the need to repeat the documentation on every instance of a common method.
In such cases, the constructor should return an interface value rather than the implementing type. As an example, in the hash libraries both crc32.NewIEEE and adler32.New return the interface type hash.Hash32. Substituting the CRC-32 algorithm for Adler-32 in a Go program requires only changing the constructor call; the rest of the code is unaffected by the change of algorithm.
A similar approach allows the streaming cipher algorithms in the various crypto packages to be separated from the block ciphers they chain together. The Block interface in the crypto/cipher package specifies the behavior of a block cipher, which provides encryption of a single block of data. Then, by analogy with the bufio package, cipher packages that implement this interface can be used to construct streaming ciphers, represented by the Stream interface, without knowing the details of the block encryption.
The crypto/cipher interfaces look like this:
type Block interface { BlockSize() int Encrypt(dst, src []byte) Decrypt(dst, src []byte) }
type Stream interface { XORKeyStream(dst, src []byte) }
Here's the definition of the counter mode (CTR) stream, which turns a block cipher into a streaming cipher; notice that the block cipher's details are abstracted away:
// NewCTR returns a Stream that encrypts/decrypts using the given Block in // counter mode. The length of iv must be the same as the Block's block size. func NewCTR(block Block, iv []byte) Stream
NewCTR applies not just to one specific encryption algorithm and data source but to any implementation of the Block interface and any Stream. Because they return interface values, replacing CTR encryption with other encryption modes is a localized change. The constructor calls must be edited, but because the surrounding code must treat the result only as a Stream, it won't notice the difference. Interfaces and methods
Since almost anything can have methods attached, almost anything can satisfy an interface. One illustrative example is in the http package, which defines the Handler interface. Any object that implements Handler can serve HTTP requests.
type Handler interface { ServeHTTP(ResponseWriter, *Request) }
ResponseWriter is itself an interface that provides access to the methods needed to return the response to the client. Those methods include the standard Write method, so an http.ResponseWriter can be used wherever an io.Writer can be used. Request is a struct containing a parsed representation of the request from the client.
For brevity, let's ignore POSTs and assume HTTP requests are always GETs; that simplification does not affect the way the handlers are set up. Here's a trivial implementation of a handler to count the number of times the page is visited.
// Simple counter server. type Counter struct { n int }
func (ctr *Counter) ServeHTTP(w http.ResponseWriter, req *http.Request) { ctr.n++ fmt.Fprintf(w, "counter = %d\n", ctr.n) }
(Keeping with our theme, note how Fprintf can print to an http.ResponseWriter.) In a real server, access to ctr.n would need protection from concurrent access. See the sync and atomic packages for suggestions.
For reference, here's how to attach such a server to a node on the URL tree.
import "net/http" ... ctr := new(Counter) http.Handle("/counter", ctr)
But why make Counter a struct? An integer is all that's needed. (The receiver needs to be a pointer so the increment is visible to the caller.)
// Simpler counter server. type Counter int
func (ctr *Counter) ServeHTTP(w http.ResponseWriter, req *http.Request) { *ctr++ fmt.Fprintf(w, "counter = %d\n", *ctr) }
What if your program has some internal state that needs to be notified that a page has been visited? Tie a channel to the web page.
// A channel that sends a notification on each visit. // (Probably want the channel to be buffered.) type Chan chan *http.Request
func (ch Chan) ServeHTTP(w http.ResponseWriter, req *http.Request) { ch <- req fmt.Fprint(w, "notification sent") }
Finally, let's say we wanted to present on /args the arguments used when invoking the server binary. It's easy to write a function to print the arguments.
func ArgServer() { fmt.Println(os.Args) }
How do we turn that into an HTTP server? We could make ArgServer a method of some type whose value we ignore, but there's a cleaner way. Since we can define a method for any type except pointers and interfaces, we can write a method for a function. The http package contains this code:
// The HandlerFunc type is an adapter to allow the use of // ordinary functions as HTTP handlers. If f is a function // with the appropriate signature, HandlerFunc(f) is a // Handler object that calls f. type HandlerFunc func(ResponseWriter, *Request)
// ServeHTTP calls f(w, req). func (f HandlerFunc) ServeHTTP(w ResponseWriter, req *Request) { f(w, req) }
HandlerFunc is a type with a method, ServeHTTP, so values of that type can serve HTTP requests. Look at the implementation of the method: the receiver is a function, f, and the method calls f. That may seem odd but it's not that different from, say, the receiver being a channel and the method sending on the channel.
To make ArgServer into an HTTP server, we first modify it to have the right signature.
// Argument server. func ArgServer(w http.ResponseWriter, req *http.Request) { fmt.Fprintln(w, os.Args) }
ArgServer now has same signature as HandlerFunc, so it can be converted to that type to access its methods, just as we converted Sequence to IntSlice to access IntSlice.Sort. The code to set it up is concise:
http.Handle("/args", http.HandlerFunc(ArgServer))
When someone visits the page /args, the handler installed at that page has value ArgServer and type HandlerFunc. The HTTP server will invoke the method ServeHTTP of that type, with ArgServer as the receiver, which will in turn call ArgServer (via the invocation f(w, req) inside HandlerFunc.ServeHTTP). The arguments will then be displayed.
In this section we have made an HTTP server from a struct, an integer, a channel, and a function, all because interfaces are just sets of methods, which can be defined for (almost) any type. The blank identifier
We've mentioned the blank identifier a couple of times now, in the context of for range loops and maps. The blank identifier can be assigned or declared with any value of any type, with the value discarded harmlessly. It's a bit like writing to the Unix /dev/null file: it represents a write-only value to be used as a place-holder where a variable is needed but the actual value is irrelevant. It has uses beyond those we've seen already. The blank identifier in multiple assignment
The use of a blank identifier in a for range loop is a special case of a general situation: multiple assignment.
If an assignment requires multiple values on the left side, but one of the values will not be used by the program, a blank identifier on the left-hand-side of the assignment avoids the need to create a dummy variable and makes it clear that the value is to be discarded. For instance, when calling a function that returns a value and an error, but only the error is important, use the blank identifier to discard the irrelevant value.
if _, err := os.Stat(path); os.IsNotExist(err) { fmt.Printf("%s does not exist\n", path) }
Occasionally you'll see code that discards the error value in order to ignore the error; this is terrible practice. Always check error returns; they're provided for a reason.
// Bad! This code will crash if path does not exist. fi, _ := os.Stat(path) if fi.IsDir() { fmt.Printf("%s is a directory\n", path) }
Unused imports and variables
It is an error to import a package or to declare a variable without using it. Unused imports bloat the program and slow compilation, while a variable that is initialized but not used is at least a wasted computation and perhaps indicative of a larger bug. When a program is under active development, however, unused imports and variables often arise and it can be annoying to delete them just to have the compilation proceed, only to have them be needed again later. The blank identifier provides a workaround.
This half-written program has two unused imports (fmt and io) and an unused variable (fd), so it will not compile, but it would be nice to see if the code so far is correct.
package main
import ( "fmt" "io" "log" "os" )
func main() { fd, err := os.Open("test.go") if err != nil { log.Fatal(err) } // TODO: use fd. }
To silence complaints about the unused imports, use a blank identifier to refer to a symbol from the imported package. Similarly, assigning the unused variable fd to the blank identifier will silence the unused variable error. This version of the program does compile.
package main
import ( "fmt" "io" "log" "os" )
var _ = fmt.Printf // For debugging; delete when done. var _ io.Reader // For debugging; delete when done.
func main() { fd, err := os.Open("test.go") if err != nil { log.Fatal(err) } // TODO: use fd. _ = fd }
By convention, the global declarations to silence import errors should come right after the imports and be commented, both to make them easy to find and as a reminder to clean things up later. Import for side effect
An unused import like fmt or io in the previous example should eventually be used or removed: blank assignments identify code as a work in progress. But sometimes it is useful to import a package only for its side effects, without any explicit use. For example, during its init function, the net/http/pprof package registers HTTP handlers that provide debugging information. It has an exported API, but most clients need only the handler registration and access the data through a web page. To import the package only for its side effects, rename the package to the blank identifier:
import _ "net/http/pprof"
This form of import makes clear that the package is being imported for its side effects, because there is no other possible use of the package: in this file, it doesn't have a name. (If it did, and we didn't use that name, the compiler would reject the program.) Interface checks
As we saw in the discussion of interfaces above, a type need not declare explicitly that it implements an interface. Instead, a type implements the interface just by implementing the interface's methods. In practice, most interface conversions are static and therefore checked at compile time. For example, passing an *os.File to a function expecting an io.Reader will not compile unless *os.File implements the io.Reader interface.
Some interface checks do happen at run-time, though. One instance is in the encoding/json package, which defines a Marshaler interface. When the JSON encoder receives a value that implements that interface, the encoder invokes the value's marshaling method to convert it to JSON instead of doing the standard conversion. The encoder checks this property at run time with a type assertion like:
m, ok := val.(json.Marshaler)
If it's necessary only to ask whether a type implements an interface, without actually using the interface itself, perhaps as part of an error check, use the blank identifier to ignore the type-asserted value:
if _, ok := val.(json.Marshaler); ok { fmt.Printf("value %v of type %T implements json.Marshaler\n", val, val) }
One place this situation arises is when it is necessary to guarantee within the package implementing the type that it actually satisfies the interface. If a type—for example, json.RawMessage—needs a custom JSON representation, it should implement json.Marshaler, but there are no static conversions that would cause the compiler to verify this automatically. If the type inadvertently fails to satisfy the interface, the JSON encoder will still work, but will not use the custom implementation. To guarantee that the implementation is correct, a global declaration using the blank identifier can be used in the package:
var _ json.Marshaler = (*RawMessage)(nil)
In this declaration, the assignment involving a conversion of a *RawMessage to a Marshaler requires that *RawMessage implements Marshaler, and that property will be checked at compile time. Should the json.Marshaler interface change, this package will no longer compile and we will be on notice that it needs to be updated.
The appearance of the blank identifier in this construct indicates that the declaration exists only for the type checking, not to create a variable. Don't do this for every type that satisfies an interface, though. By convention, such declarations are only used when there are no static conversions already present in the code, which is a rare event. Embedding
Go does not provide the typical, type-driven notion of subclassing, but it does have the ability to “borrow” pieces of an implementation by embedding types within a struct or interface.
Interface embedding is very simple. We've mentioned the io.Reader and io.Writer interfaces before; here are their definitions.
type Reader interface { Read(p []byte) (n int, err error) }
type Writer interface { Write(p []byte) (n int, err error) }
The io package also exports several other interfaces that specify objects that can implement several such methods. For instance, there is io.ReadWriter, an interface containing both Read and Write. We could specify io.ReadWriter by listing the two methods explicitly, but it's easier and more evocative to embed the two interfaces to form the new one, like this:
// ReadWriter is the interface that combines the Reader and Writer interfaces. type ReadWriter interface { Reader Writer }
This says just what it looks like: A ReadWriter can do what a Reader does and what a Writer does; it is a union of the embedded interfaces. Only interfaces can be embedded within interfaces.
The same basic idea applies to structs, but with more far-reaching implications. The bufio package has two struct types, bufio.Reader and bufio.Writer, each of which of course implements the analogous interfaces from package io. And bufio also implements a buffered reader/writer, which it does by combining a reader and a writer into one struct using embedding: it lists the types within the struct but does not give them field names.
// ReadWriter stores pointers to a Reader and a Writer. // It implements io.ReadWriter. type ReadWriter struct { *Reader // *bufio.Reader *Writer // *bufio.Writer }
The embedded elements are pointers to structs and of course must be initialized to point to valid structs before they can be used. The ReadWriter struct could be written as
type ReadWriter struct { reader *Reader writer *Writer }
but then to promote the methods of the fields and to satisfy the io interfaces, we would also need to provide forwarding methods, like this:
func (rw *ReadWriter) Read(p []byte) (n int, err error) { return rw.reader.Read(p) }
By embedding the structs directly, we avoid this bookkeeping. The methods of embedded types come along for free, which means that bufio.ReadWriter not only has the methods of bufio.Reader and bufio.Writer, it also satisfies all three interfaces: io.Reader, io.Writer, and io.ReadWriter.
There's an important way in which embedding differs from subclassing. When we embed a type, the methods of that type become methods of the outer type, but when they are invoked the receiver of the method is the inner type, not the outer one. In our example, when the Read method of a bufio.ReadWriter is invoked, it has exactly the same effect as the forwarding method written out above; the receiver is the reader field of the ReadWriter, not the ReadWriter itself.
Embedding can also be a simple convenience. This example shows an embedded field alongside a regular, named field.
type Job struct { Command string *log.Logger }
The Job type now has the Print, Printf, Println and other methods of *log.Logger. We could have given the Logger a field name, of course, but it's not necessary to do so. And now, once initialized, we can log to the Job:
job.Println("starting now...")
The Logger is a regular field of the Job struct, so we can initialize it in the usual way inside the constructor for Job, like this,
func NewJob(command string, logger *log.Logger) *Job { return &Job{command, logger} }
or with a composite literal,
job := &Job{command, log.New(os.Stderr, "Job: ", log.Ldate)}
If we need to refer to an embedded field directly, the type name of the field, ignoring the package qualifier, serves as a field name, as it did in the Read method of our ReadWriter struct. Here, if we needed to access the *log.Logger of a Job variable job, we would write job.Logger, which would be useful if we wanted to refine the methods of Logger.
func (job *Job) Printf(format string, args ...interface{}) { job.Logger.Printf("%q: %s", job.Command, fmt.Sprintf(format, args...)) }
Embedding types introduces the problem of name conflicts but the rules to resolve them are simple. First, a field or method X hides any other item X in a more deeply nested part of the type. If log.Logger contained a field or method called Command, the Command field of Job would dominate it.
Second, if the same name appears at the same nesting level, it is usually an error; it would be erroneous to embed log.Logger if the Job struct contained another field or method called Logger. However, if the duplicate name is never mentioned in the program outside the type definition, it is OK. This qualification provides some protection against changes made to types embedded from outside; there is no problem if a field is added that conflicts with another field in another subtype if neither field is ever used. Concurrency Share by communicating
Concurrent programming is a large topic and there is space only for some Go-specific highlights here.
Concurrent programming in many environments is made difficult by the subtleties required to implement correct access to shared variables. Go encourages a different approach in which shared values are passed around on channels and, in fact, never actively shared by separate threads of execution. Only one goroutine has access to the value at any given time. Data races cannot occur, by design. To encourage this way of thinking we have reduced it to a slogan:
Do not communicate by sharing memory; instead, share memory by communicating.
This approach can be taken too far. Reference counts may be best done by putting a mutex around an integer variable, for instance. But as a high-level approach, using channels to control access makes it easier to write clear, correct programs.
One way to think about this model is to consider a typical single-threaded program running on one CPU. It has no need for synchronization primitives. Now run another such instance; it too needs no synchronization. Now let those two communicate; if the communication is the synchronizer, there's still no need for other synchronization. Unix pipelines, for example, fit this model perfectly. Although Go's approach to concurrency originates in Hoare's Communicating Sequential Processes (CSP), it can also be seen as a type-safe generalization of Unix pipes. Goroutines
They're called goroutines because the existing terms—threads, coroutines, processes, and so on—convey inaccurate connotations. A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management.
Prefix a function or method call with the go keyword to run the call in a new goroutine. When the call completes, the goroutine exits, silently. (The effect is similar to the Unix shell's & notation for running a command in the background.)
go list.Sort() // run list.Sort concurrently; don't wait for it.
A function literal can be handy in a goroutine invocation.
func Announce(message string, delay time.Duration) { go func() { time.Sleep(delay) fmt.Println(message) }() // Note the parentheses - must call the function. }
In Go, function literals are closures: the implementation makes sure the variables referred to by the function survive as long as they are active.
These examples aren't too practical because the functions have no way of signaling completion. For that, we need channels. Channels
Like maps, channels are allocated with make, and the resulting value acts as a reference to an underlying data structure. If an optional integer parameter is provided, it sets the buffer size for the channel. The default is zero, for an unbuffered or synchronous channel.
ci := make(chan int) // unbuffered channel of integers cj := make(chan int, 0) // unbuffered channel of integers cs := make(chan *os.File, 100) // buffered channel of pointers to Files
Unbuffered channels combine communication—the exchange of a value—with synchronization—guaranteeing that two calculations (goroutines) are in a known state.
There are lots of nice idioms using channels. Here's one to get us started. In the previous section we launched a sort in the background. A channel can allow the launching goroutine to wait for the sort to complete.
c := make(chan int) // Allocate a channel. // Start the sort in a goroutine; when it completes, signal on the channel. go func() { list.Sort() c <- 1 // Send a signal; value does not matter. }() doSomethingForAWhile() <-c // Wait for sort to finish; discard sent value.
Receivers always block until there is data to receive. If the channel is unbuffered, the sender blocks until the receiver has received the value. If the channel has a buffer, the sender blocks only until the value has been copied to the buffer; if the buffer is full, this means waiting until some receiver has retrieved a value.
A buffered channel can be used like a semaphore, for instance to limit throughput. In this example, incoming requests are passed to handle, which sends a value into the channel, processes the request, and then receives a value from the channel to ready the “semaphore” for the next consumer. The capacity of the channel buffer limits the number of simultaneous calls to process.
var sem = make(chan int, MaxOutstanding)
func handle(r *Request) { sem <- 1 // Wait for active queue to drain. process(r) // May take a long time. <-sem // Done; enable next request to run. }
func Serve(queue chan *Request) { for { req := <-queue go handle(req) // Don't wait for handle to finish. } }
Once MaxOutstanding handlers are executing process, any more will block trying to send into the filled channel buffer, until one of the existing handlers finishes and receives from the buffer.
This design has a problem, though: Serve creates a new goroutine for every incoming request, even though only MaxOutstanding of them can run at any moment. As a result, the program can consume unlimited resources if the requests come in too fast. We can address that deficiency by changing Serve to gate the creation of the goroutines. Here's an obvious solution, but beware it has a bug we'll fix subsequently:
func Serve(queue chan *Request) { for req := range queue { sem <- 1 go func() { process(req) // Buggy; see explanation below. <-sem }() } }
The bug is that in a Go for loop, the loop variable is reused for each iteration, so the req variable is shared across all goroutines. That's not what we want. We need to make sure that req is unique for each goroutine. Here's one way to do that, passing the value of req as an argument to the closure in the goroutine:
func Serve(queue chan *Request) { for req := range queue { sem <- 1 go func(req *Request) { process(req) <-sem }(req) } }
Compare this version with the previous to see the difference in how the closure is declared and run. Another solution is just to create a new variable with the same name, as in this example:
func Serve(queue chan *Request) { for req := range queue { req := req // Create new instance of req for the goroutine. sem <- 1 go func() { process(req) <-sem }() } }
It may seem odd to write
req := req
but it's legal and idiomatic in Go to do this. You get a fresh version of the variable with the same name, deliberately shadowing the loop variable locally but unique to each goroutine.
Going back to the general problem of writing the server, another approach that manages resources well is to start a fixed number of handle goroutines all reading from the request channel. The number of goroutines limits the number of simultaneous calls to process. This Serve function also accepts a channel on which it will be told to exit; after launching the goroutines it blocks receiving from that channel.
func handle(queue chan *Request) { for r := range queue { process(r) } }
func Serve(clientRequests chan *Request, quit chan bool) { // Start handlers for i := 0; i < MaxOutstanding; i++ { go handle(clientRequests) } <-quit // Wait to be told to exit. }
Channels of channels
One of the most important properties of Go is that a channel is a first-class value that can be allocated and passed around like any other. A common use of this property is to implement safe, parallel demultiplexing.
In the example in the previous section, handle was an idealized handler for a request but we didn't define the type it was handling. If that type includes a channel on which to reply, each client can provide its own path for the answer. Here's a schematic definition of type Request.
type Request struct { args []int f func([]int) int resultChan chan int }
The client provides a function and its arguments, as well as a channel inside the request object on which to receive the answer.
func sum(a []int) (s int) { for _, v := range a { s += v } return }
request := &Request{[]int{3, 4, 5}, sum, make(chan int)} // Send request clientRequests <- request // Wait for response. fmt.Printf("answer: %d\n", <-request.resultChan)
On the server side, the handler function is the only thing that changes.
func handle(queue chan *Request) { for req := range queue { req.resultChan <- req.f(req.args) } }
There's clearly a lot more to do to make it realistic, but this code is a framework for a rate-limited, parallel, non-blocking RPC system, and there's not a mutex in sight. Parallelization
Another application of these ideas is to parallelize a calculation across multiple CPU cores. If the calculation can be broken into separate pieces that can execute independently, it can be parallelized, with a channel to signal when each piece completes.
Let's say we have an expensive operation to perform on a vector of items, and that the value of the operation on each item is independent, as in this idealized example.
type Vector []float64
// Apply the operation to v[i], v[i+1] ... up to v[n-1]. func (v Vector) DoSome(i, n int, u Vector, c chan int) { for ; i < n; i++ { v[i] += u.Op(v[i]) } c <- 1 // signal that this piece is done }
We launch the pieces independently in a loop, one per CPU. They can complete in any order but it doesn't matter; we just count the completion signals by draining the channel after launching all the goroutines.
const numCPU = 4 // number of CPU cores
func (v Vector) DoAll(u Vector) { c := make(chan int, numCPU) // Buffering optional but sensible. for i := 0; i < numCPU; i++ { go v.DoSome(i*len(v)/numCPU, (i+1)*len(v)/numCPU, u, c) } // Drain the channel. for i := 0; i < numCPU; i++ { <-c // wait for one task to complete } // All done. }
Rather than create a constant value for numCPU, we can ask the runtime what value is appropriate. The function runtime.NumCPU returns the number of hardware CPU cores in the machine, so we could write
var numCPU = runtime.NumCPU()
There is also a function runtime.GOMAXPROCS, which reports (or sets) the user-specified number of cores that a Go program can have running simultaneously. It defaults to the value of runtime.NumCPU but can be overridden by setting the similarly named shell environment variable or by calling the function with a positive number. Calling it with zero just queries the value. Therefore if we want to honor the user's resource request, we should write
var numCPU = runtime.GOMAXPROCS(0)
Be sure not to confuse the ideas of concurrency—structuring a program as independently executing components—and parallelism—executing calculations in parallel for efficiency on multiple CPUs. Although the concurrency features of Go can make some problems easy to structure as parallel computations, Go is a concurrent language, not a parallel one, and not all parallelization problems fit Go's model. For a discussion of the distinction, see the talk cited in this blog post. A leaky buffer
The tools of concurrent programming can even make non-concurrent ideas easier to express. Here's an example abstracted from an RPC package. The client goroutine loops receiving data from some source, perhaps a network. To avoid allocating and freeing buffers, it keeps a free list, and uses a buffered channel to represent it. If the channel is empty, a new buffer gets allocated. Once the message buffer is ready, it's sent to the server on serverChan.
var freeList = make(chan *Buffer, 100) var serverChan = make(chan *Buffer)
func client() { for { var b *Buffer // Grab a buffer if available; allocate if not. select { case b = <-freeList: // Got one; nothing more to do. default: // None free, so allocate a new one. b = new(Buffer) } load(b) // Read next message from the net. serverChan <- b // Send to server. } }
The server loop receives each message from the client, processes it, and returns the buffer to the free list.
func server() { for { b := <-serverChan // Wait for work. process(b) // Reuse buffer if there's room. select { case freeList <- b: // Buffer on free list; nothing more to do. default: // Free list full, just carry on. } } }
The client attempts to retrieve a buffer from freeList; if none is available, it allocates a fresh one. The server's send to freeList puts b back on the free list unless the list is full, in which case the buffer is dropped on the floor to be reclaimed by the garbage collector. (The default clauses in the select statements execute when no other case is ready, meaning that the selects never block.) This implementation builds a leaky bucket free list in just a few lines, relying on the buffered channel and the garbage collector for bookkeeping. Errors
Library routines must often return some sort of error indication to the caller. As mentioned earlier, Go's multivalue return makes it easy to return a detailed error description alongside the normal return value. It is good style to use this feature to provide detailed error information. For example, as we'll see, os.Open doesn't just return a nil pointer on failure, it also returns an error value that describes what went wrong.
By convention, errors have type error, a simple built-in interface.
type error interface { Error() string }
A library writer is free to implement this interface with a richer model under the covers, making it possible not only to see the error but also to provide some context. As mentioned, alongside the usual *os.File return value, os.Open also returns an error value. If the file is opened successfully, the error will be nil, but when there is a problem, it will hold an os.PathError:
// PathError records an error and the operation and // file path that caused it. type PathError struct { Op string // "open", "unlink", etc. Path string // The associated file. Err error // Returned by the system call. }
func (e *PathError) Error() string { return e.Op + " " + e.Path + ": " + e.Err.Error() }
PathError's Error generates a string like this:
open /etc/passwx: no such file or directory
Such an error, which includes the problematic file name, the operation, and the operating system error it triggered, is useful even if printed far from the call that caused it; it is much more informative than the plain "no such file or directory".
When feasible, error strings should identify their origin, such as by having a prefix naming the operation or package that generated the error. For example, in package image, the string representation for a decoding error due to an unknown format is "image: unknown format".
Callers that care about the precise error details can use a type switch or a type assertion to look for specific errors and extract details. For PathErrors this might include examining the internal Err field for recoverable failures.
for try := 0; try < 2; try++ { file, err = os.Create(filename) if err == nil { return } if e, ok := err.(*os.PathError); ok && e.Err == syscall.ENOSPC { deleteTempFiles() // Recover some space. continue } return }
The second if statement here is another type assertion. If it fails, ok will be false, and e will be nil. If it succeeds, ok will be true, which means the error was of type *os.PathError, and then so is e, which we can examine for more information about the error. Panic
The usual way to report an error to a caller is to return an error as an extra return value. The canonical Read method is a well-known instance; it returns a byte count and an error. But what if the error is unrecoverable? Sometimes the program simply cannot continue.
For this purpose, there is a built-in function panic that in effect creates a run-time error that will stop the program (but see the next section). The function takes a single argument of arbitrary type—often a string—to be printed as the program dies. It's also a way to indicate that something impossible has happened, such as exiting an infinite loop.
// A toy implementation of cube root using Newton's method. func CubeRoot(x float64) float64 { z := x/3 // Arbitrary initial value for i := 0; i < 1e6; i++ { prevz := z z -= (zzz-x) / (3zz) if veryClose(z, prevz) { return z } } // A million iterations has not converged; something is wrong. panic(fmt.Sprintf("CubeRoot(%g) did not converge", x)) }
This is only an example but real library functions should avoid panic. If the problem can be masked or worked around, it's always better to let things continue to run rather than taking down the whole program. One possible counterexample is during initialization: if the library truly cannot set itself up, it might be reasonable to panic, so to speak.
var user = os.Getenv("USER")
func init() { if user == "" { panic("no value for $USER") } }
Recover
When panic is called, including implicitly for run-time errors such as indexing a slice out of bounds or failing a type assertion, it immediately stops execution of the current function and begins unwinding the stack of the goroutine, running any deferred functions along the way. If that unwinding reaches the top of the goroutine's stack, the program dies. However, it is possible to use the built-in function recover to regain control of the goroutine and resume normal execution.
A call to recover stops the unwinding and returns the argument passed to panic. Because the only code that runs while unwinding is inside deferred functions, recover is only useful inside deferred functions.
One application of recover is to shut down a failing goroutine inside a server without killing the other executing goroutines.
func server(workChan <-chan *Work) { for work := range workChan { go safelyDo(work) } }
func safelyDo(work *Work) { defer func() { if err := recover(); err != nil { log.Println("work failed:", err) } }() do(work) }
In this example, if do(work) panics, the result will be logged and the goroutine will exit cleanly without disturbing the others. There's no need to do anything else in the deferred closure; calling recover handles the condition completely.
Because recover always returns nil unless called directly from a deferred function, deferred code can call library routines that themselves use panic and recover without failing. As an example, the deferred function in safelyDo might call a logging function before calling recover, and that logging code would run unaffected by the panicking state.
With our recovery pattern in place, the do function (and anything it calls) can get out of any bad situation cleanly by calling panic. We can use that idea to simplify error handling in complex software. Let's look at an idealized version of a regexp package, which reports parsing errors by calling panic with a local error type. Here's the definition of Error, an error method, and the Compile function.
// Error is the type of a parse error; it satisfies the error interface. type Error string func (e Error) Error() string { return string(e) }
// error is a method of *Regexp that reports parsing errors by // panicking with an Error. func (regexp *Regexp) error(err string) { panic(Error(err)) }
// Compile returns a parsed representation of the regular expression. func Compile(str string) (regexp *Regexp, err error) { regexp = new(Regexp) // doParse will panic if there is a parse error. defer func() { if e := recover(); e != nil { regexp = nil // Clear return value. err = e.(Error) // Will re-panic if not a parse error. } }() return regexp.doParse(str), nil }
If doParse panics, the recovery block will set the return value to nil—deferred functions can modify named return values. It will then check, in the assignment to err, that the problem was a parse error by asserting that it has the local type Error. If it does not, the type assertion will fail, causing a run-time error that continues the stack unwinding as though nothing had interrupted it. This check means that if something unexpected happens, such as an index out of bounds, the code will fail even though we are using panic and recover to handle parse errors.
With error handling in place, the error method (because it's a method bound to a type, it's fine, even natural, for it to have the same name as the builtin error type) makes it easy to report parse errors without worrying about unwinding the parse stack by hand:
if pos == 0 { re.error("'*' illegal at start of expression") }
Useful though this pattern is, it should be used only within a package. Parse turns its internal panic calls into error values; it does not expose panics to its client. That is a good rule to follow.
By the way, this re-panic idiom changes the panic value if an actual error occurs. However, both the original and new failures will be presented in the crash report, so the root cause of the problem will still be visible. Thus this simple re-panic approach is usually sufficient—it's a crash after all—but if you want to display only the original value, you can write a little more code to filter unexpected problems and re-panic with the original error. That's left as an exercise for the reader. A web server
Let's finish with a complete Go program, a web server. This one is actually a kind of web re-server. Google provides a service at chart.apis.google.com that does automatic formatting of data into charts and graphs. It's hard to use interactively, though, because you need to put the data into the URL as a query. The program here provides a nicer interface to one form of data: given a short piece of text, it calls on the chart server to produce a QR code, a matrix of boxes that encode the text. That image can be grabbed with your cell phone's camera and interpreted as, for instance, a URL, saving you typing the URL into the phone's tiny keyboard.
Here's the complete program. An explanation follows.
package main
import ( "flag" "html/template" "log" "net/http" )
var addr = flag.String("addr", ":1718", "http service address") // Q=17, R=18
var templ = template.Must(template.New("qr").Parse(templateStr))
func main() { flag.Parse() http.Handle("/", http.Hand
lerFunc(QR)) err := http.ListenAndServe(*addr, nil) if err != nil { log.Fatal("ListenAndServe:", err) } }
func QR(w http.ResponseWriter, req *http.Request) { templ.Execute(w, req.FormValue("s")) }
const templateStr = `
<html>
<head>
<title>QR Link Generator</title>
</head>
<body>
{{if .}}
<img src="http://chart.apis.google.com/chart?chs=300x300&cht=qr&choe=UTF-8&chl={{.}}" />
<br>
{{.}}
<br>
<br>
{{end}}
<form action="/" name=f method="GET">
<input maxLength=1024 size=70 name=s value="" title="Text to QR Encode">
<input type=submit value="Show QR" name=qr>
</form>
</body>
</html>
`
The pieces up to main should be easy to follow. The one flag sets a default HTTP port for our server. The template variable templ is where the fun happens. It builds an HTML template that will be executed by the server to display the page; more about that in a moment.
The main function parses the flags and, using the mechanism we talked about above, binds the function QR to the root path for the server. Then http.ListenAndServe is called to start the server; it blocks while the server runs.
QR just receives the request, which contains form data, and executes the template on the data in the form value named s.
The template package html/template is powerful; this program just touches on its capabilities. In essence, it rewrites a piece of HTML text on the fly by substituting elements derived from data items passed to templ.Execute, in this case the form value. Within the template text (templateStr), double-brace-delimited pieces denote template actions. The piece from {{if .}} to {{end}} executes only if the value of the current data item, called . (dot), is non-empty. That is, when the string is empty, this piece of the template is suppressed.
The two snippets {{.}} say to show the data presented to the template—the query string—on the web page. The HTML template package automatically provides appropriate escaping so the text is safe to display.
The rest of the template string is just the HTML to show when the page loads. If this is too quick an explanation, see the documentation for the template package for a more thorough discussion.
And there you have it: a useful web server in a few lines of code plus some data-driven HTML text. Go is powerful enough to make a lot happen in a few lines.