About this post

The Go programming language is easy to learn. Go has only about 25 reserved words as compared to Java 67 (not including frameworks) or C# 79. The Go toolchain to compile Go codes, manage dependencies, generate documentation and perform the analysis is also very easy to use.

I was introduced to Go after years of working initially with Java, QT/C++, Objective C then Swift, and my employer told that Go was to be the working lanugage. Despite having no prior experience using Go, I became very comfortable using the language very quickly. I felt ready to work on a production-grade project using Go.

However, when I embarked on my first production grade project, I found architecting Go projects that had clean, clear, concise and maintainable layout challenging. I found little consensus on how to layout Go projects. So I applied architectural paradigms that I had learnt from my previous working experience, which was a mix of object-oriented and, sort of, functional programming. I had assumed that “good” architectural paradiums transcend all programming languages. However, I found, belatedly that I could not apply architectural paradigm intended, for example, Java to Go. When I applied the Java way of organising code I ended up with a structure that was too complex and hard to maintain – i.e. deeply nested layers of abstractions.

To avoid repeating the mistakes for future projects, I pivoted from what I knew about organising projects and delved into the philosophy behind the Go’s creation. I turned to a presentation by Rob Pike, one of the creators of the language, where he presented the Go Proverbs. The proverb gave me the necessary philosophical foundation to break from my past and exploit the goodness of Go which I will share my thoughts in a series of blogs. In this post, I will address the aspect of the proverb titled “design the architecture, name the components, document the details”.

Go packages

A Go package is the basic unit of the language. It is a folder containing one or more source files. A Go source file is one with the extension *.go. The Go source files are organised as either part of a main package or a named package.

A main package contains a source file or files with the signature package main and one main function. The folder containing such files can have names different from its package signature. There is no requirement for these folders to be specifically named main like so:

  /something
    main.go <-- containts a statement "package main" at the top of the file

NOTE: main.go is a convention but you can name the file anything.

A named package contains a source file or files with a signature that matches its containing folder. For example, source files with a signature in the file package something must be found under this folder named something, like so:

  /something
    source.go <-- contain a statement "package something" at the top of the file

With some exceptions that I will not discuss here, Go is not opinionated about naming packages or naming source files in each package, thus I found myself confronted with these questions:

• How should I group my source files into a Go package?
• How should I name my Go packages and their source files?
• If we have more than one package, how should I aggregate them?
• How do I export or avoid exporting Go packages for used by other projects?
• How do I use Go packages and their contents to model the solution my project is intended to implement concisely whilst minimising complexity?

Deciding what to include in and naming a Go package

If you applied an object-oriented programming mindset, your instinct is organise your code around “classes”. A class is construct you typically find in object-oriented languages such as Java or C#.

The basic element of a class is a data structure with member methods. There are other more complex construct associated with classes in object-oriented langagues such as class (or static) methods, access protections (e.g. public, protected and private), and inheritence or polymorphism. I’ll discuss how these mechanisms impact architectural decisions in future blogs.

In the case of Go, the equivalent, very loosely, of a class is a struct representing the structure of a piece of data with a “member” function declared like this:

type Person struct{
  Name string
}

func (p Person) GreetsWith(message string){
  fmt.Printf("%s greets with message: %s", p.Name, message)
}

However, it is worth noting that declaring a Go “class” as illustrated above is simply a syntatic sugar for this:

type Person struct{
  Name string
}

func SendGreetings(p Person, message string){
  fmt.Printf("%s greets with message: %s", p.Name, message)
}

This second approach is a more functional approach where the variable p represents a receiver type.

Working on my first Go project, I started organising my code around object oriented style classes. I would then group these classes into Go packages, treating the packages as “namespaces” commonly found in object-oriented languages. There were times where I packaged all kind of classes that did not fit into any grouping – i.e. lacking internal cohension. I also realise that with the intended in the design of Go, a package name and its content have a coupled relationship. This means whenever you consumed a stucts or function you have to prefix it with a package name. Without much thought I found myself creating packages that were too nested and difficult to maintain or extend or exported for third parties use.

In order to find source of good practices, I examined Go’s standard library package time. In that package there is a function call Now() which coupled with the package time.Now(). If you follow the official Go package naming convention described in the Go blog, it recommends:

• package names are lowercase with no underscore or mixed caps (but the Go compiler will accept these names);
• package names are short and clear often simple nouns;
• avoid ambiguous names, such as util or common, be specific;
• avoid repeating the content and package name, for example, the server – e.g. function name – provided by the package http should be called Server not HttpServer so when the function is called, it will be http.Server instead of http.HttpServer which would suggest that under the http package, there might be other http server.

The recommendations seemed to reinforce the idea that in the context of Go, it probably make sense to use package in lieu of classes. Consider the example of the class Person, we could model the it this way:

package person

type Detail struct{
  Firstname string
  Surname   string
}

func SendGreetings(from Detail, to Detail, msg string) {
  fmt.Printf("%s greets %s with message: %s", from.Firstname, to.Firstmame, msg)
}

I could easily consume the package this way person.Detail{} or person.SendGreetings(from, to, message). By treating packages in lieu of classes, treating functions in packages as forms of member methods, I found it easier to organise packages in a coherent fashion.

What about the goodness of object-oriented programming? Inheritence and polymorphism?

My view is the loss of object-orientation is more than compensated by Go features that the designer of the language intended it to be. Namely to keep things small and most importantly compatible to Go concurrency paradigm by being more functional.

Organising your project structure

Having decided how I would create Go packages, the next challenge I found was how best to organise them in a project structure.

When working on a project with some degree of complexity, I often found that I need to layout my Go packages to exist alongside non-Go-related artefacts in a way that was coherent not only to myself but my collaborators. Examples of non-Go artefacts are deployment scripts, build scripts, etc that support the creation of Go binaries. The other descisions I also had to make was whether to export my Go packages for consumption by other projects or not export my Go packages.

Having then worked on several Go projects, I learned that there are basically two types of projects; a library-oriented project or a service-oriented project. A library-oriented project is intended purely for sharing Go artefacts with other projects. A Service-oriented project is intended to build one or more executable binaries. I also learned that there is no standard way of laying out a service-oriented or library project. After much experimentation I found the following layouts that enable me to maintain a clean and extensible architecture.

Library-oriented

A Go library, also known as a Go module, is a collection of Go source codes or Go packages intended for consumption by other projects. There is no need to precompile in order to share packaged. Consuming projects simply import library from source.

There are three main types:

• a single package library;
• a multi-package library;
• a generated library.

In the case of a single package library, the root of a source code repository and its name is the equivalent of a Go package. All I needed to do was to maintain the source files in a flat structure. You can see an example in Corbra library.

A multi-package library has a collection of Go packages. It could lay out the Go packages like this:

<repository>/
  fibgen/  <-- Go package containing functions to generate fibonacci sequence
  cryptoutil/ <-- Go package containing functions to generate cryptographic keys
  README.md
  go.mod

However, there is also the option of a layout around the folder pkg and it would look like this:

<repository name>/
  docs/ <-- companion folders
  examples/ <-- companion folders
  pkg/
    fibgen/
    cryptoutil/
  README.md
  go.mod

Using the pkg folder helps distinguish exportable Go codes from non-Go artefacts like docs containing markdowns or examples containing runnable Go examples or illustrations of how to consume pkg. This layout could also be extended where the Go packages are generated from a third-party specification such as protobuf as shown below:

<repository>/
  build/ <-- contain generator scripts
  docs/
  pkg/
    pb/  <-- generated Go package from protos
  protos/
    model.proto <-- protobuf spec
  /scripts <-- script as hooks for users or CI/CD to kick off generator
  README.md
  go.mod

Notes:

1. In a multipackage library, Ensure that all packages are independent of each other. Do not have packages dependent on each other in this layout. Whilst the Go tooling does not forbade dependencies, it can cause problems with it comes to versioning.

2. the use of pkg is a hotly debated topic in the Go community. Some feel that it is unnecessay as the folder has no significance from a Go tooling perspective. Others feel that it is a useful organisational signalling device. It helps people maintaining the code disambiguate from other aspects of the projects. I am in favour of using pkg and recommends it as an organisational device but it is a personal choice.

Service-oriented

A service oriented project primary focus is building one or more executable. Whilst you can elect to export Go artefacts, my recommendation is not to export Go packages for consumption by other projects. I feel you should think of such layout as the basis of a monorepo for creating multiple services.

A basic service oriented project with one main package could easily be organised this way.

<repository>/
   utility.go
   main.go <-- contain only main function
   go.mod

At the next level are projects involving multiple main packages, in particular, to build microservices organised around a monorepo. For this type project, a layout like this would be most appropriate:

<repository>/
  cmd/
    service_a/
      main.go
    service_b/
      main.go
  internal/ <-- shared internal library codes for service_a and service_b
    config/
      config.go

The folders cmd and internal are the two organising folders for your codes to build executables (service_a and service_b). Note: The folder internal has a special meaning in Go. Packages under internal cannot be exported to other Go packages beyond adjacent folders.

A more realistic project will involve not just Go codes but also non-Go artefacts like build or deployment scripts. For this type of project, a better layout is this:

<repository>/
  build/ <-- build scripts
  deployment/ <-- k8s yaml manifest
  cmd/ <-- folder containing multiple main packages
    service_a/
        internal/ <-- Go packages internal to service_a but not service_b
          restapi/
      main.go
    service_b/
        internal/
          restapi/
      main.go
  docs/ <-- folder for markdown documents
  internal/ <-- folder for packages shared with main packages under cmd
    fibgen/
    cryptoutil/
  scripts/ <-- script as hooks for users or CI/CD to kick off build process
  README.md
  go.mod

This rationale behind the layout is based on the hexagon architecture as shown in Figure 1. In this case, packages under cmd is dependent on internal packages specific to main or dependent on top level internal. Within internal packages should ideally not be dependent on each other to avoid circular dependencies but if you have to do so, dependencies should be one way only.

I also found that the layout also allow me to combine Go with other languages in a coherent manner, separating cleanly between Go and other languages, such as Javascript. Please check out my example here. It demonstrates a service-oriented project that combines Go with ReactJS source codes clearly demarcated.

Document the details

Go emphasises short package names. The consensus amongst the Go community is that it keeps code compact. Some have argued that naming Go packages with short names is not helpful as readers cannot glean its context. Fortunately, the Go toolchain makes it easy to document and add details to Go package names and their content.

I can add detail about a Go package via godoc style documentation in Go codes. Here is an example of documentation about a Go package:

// Package sort provides primitives for sorting slices and user-defined
// collections.
package sort

The same applies to the content of a Go package. I could document shown in this example:

// Fprint formats using the default formats for its operands and writes to w.
// Spaces are added between operands when neither is a string.
// It returns the number of bytes written and any write error encountered.
func Fprint(w io.Writer, a ...interface{}) (n int, err error) {

The godoc format is not the only way to present details about a Go package and its contents. Readers might prefer information in other forms, as Rob Pike noted in another Go proverb, documentation is for users. With a few keystrokes of the godoc command, part of the Go toolchain, documentation about packages can be displayed in a web browser (see Figure 1) for a more human-readable and searchable form.

The same information about a specific Go package can also be displayed from the godoc cli (see Figure 2).

I found using godoc formatting a great way of enabling others to learn about my Go packages and their content and is sufficient for library-oriented projects. However, for service-oriented projects where I had to build artefacts, perform tests and deploy, I would include a README.md and a docs folder following the Standard Go Project Layout template. Sometimes, I find it useful also to add examples to help others who are maintaining my code appreciate the content. My recommended layout is this:

<repository>/
  docs/
    deployment.md
    test.md
    build.md
  examples/
  README.md

Conclusion

To summarise, for any Go projects and putting Rob Pike’s proverb on architecture, naming things and documentation into action, I recommend these principles when you work with Go:

• Do not replicate paradigms from other languages blindly.
• Use short, clear, and concise names for your Go packages and their content.
• When you layout your project, decide if you are working on a library or a service oriented project.
• For library based project, keep it as flat (ideally a collection of Go files only and treat the repository as a single package) as possible.
• For library based project, organise it around pkg folder to diambiguate it from other non-go artefacts.
• For library based project with multiple Go packages, ensure that each package is independent of others (i.e. no dependencies between packages).
• For Service based project, keep all packages internal, do not export any Go packages (keep the packages under cmd or internal folder).
• For Service based project, follow the hexagonal architecture pattern ensuring that dependencies flows from cmd to internal packages and avoid circular dependencies.
• Use godoc and markdowns to document things.