State of Web Development in Go

Background

Go was initially created at Google to solve the problem of writing large software in a simple, performant manner. It was presented as a replacement for C++ when writing fast servers. One of the first uses of Go at Google was to implement the Google Downloads server (dl.google.com) that was initially written in C++.

Instead, over the years, it ended up replacing languages like Python, Java, PHP and developers started using Go to write high-level applications. It made perfect sense given the simplicity of Go, rock-solid standard library with native support for HTTP servers, and the ease of deployment - it is very hard to beat a single binary that can be copied over and simply run to start the server.

While Go provides a lot of what you need to build a web server, the community started filling in gaps to reduce the burden of reinventing the wheel. Packages like Gorilla provide support for complex routing and middlewares. Frameworks like gin and beego aim to provide a more feature-full set of packages for fast development.

This post discusses what you need to build a server app in 2021 and how Go developers usually solve these problems. Some problems are solved more elegantly than others while others are still an open problem for the community to solve.

First, let’s get started with topics that may apply to any type of application written in Go and then topics that are more specific to web development:

Project Layout

The first thing that you would do when getting started with a project is decide on the project layout. Go does not enforce any layout - you can have a main.go file and run it using go run main.go. Since that is obviously not going to scale, you need a slightly more sophisticated layout. The GoLang Standards project has a repository showcasing what the basic layout for Go projects should look like - golang-standards/project-layout

It’s a fairly comprehensive layout and you won’t need every directory for every project but here’s the minimum layout that you would need for most projects:

cmd/
├─ my-app/
│  ├─ main.go
pkg/
├─ component-a/
│  ├─ models.go
│  ├─ repo.go
│  ├─ svc.go
│  ├─ router.go
├─ component-b/
configs/
├─ dev.toml
├─ prod.toml
scripts/

The layout provides a structured way to have multiple entry points under the cmd/ directory, all of the application logic under pkg/ split by their domain, and other directories such as configs/ or scripts/ that are required in many projects.

Configuration

Storing API secrets, database credentials, and other configuration is a necessary part of any projects. Depending on the configuration, it may be feasible to share defaults and check-in to source control whereas in other cases like secrets, they need to be provided based on the environment (dev vs. prod) and not shared by any means.

The most popular HTTP frameworks in GoLang tend to not provide a solution for this. A simple Google search for golang config reader will land on the spf13/viper Github repository. It provides everything you need to read configuration values but the burden of integrating this necessary part of any application is left up to the developer.

In smaller applications, it may not even be necessary to integrate a dependency like spf13/viper. Simple, shareable config values can be stored in the code or in a JSON file that can be read trivially in-app. Secrets can be stored in system environment variables. In my opinion, this should be the go-to solution for most projects unless you explicitly feel the need for something more complex.

Errors

Errors in Go have been discussed a LOT. The only other topic that might have beaten errors is generics in the last year. With newer features such as Unwrap, Is, and As, the need for third-party packages has reduced significantly. Without these features, packages like pkg/errors were used in most projects. Now, they are a nice-to-have with some extra features.

Whichever solution you pick for your project, there is a pattern that has emerged in handling errors that is useful and should be the default for everyone. I see many projects handling errors like so:

val, err := doSomething()
if err != nil {
  return err // DON'T DO THIS!
}

This is not useful for any non-trivial application. An error that is simply bubbled up the chain with no additional context may have no value at the the top of the stack.

Let’s say that an HTTP route in your app that is responsible for creating a new user calls a function CreateUser and receives an error: “file does not exist”. Is this error useful? What file are we talking about? Unfortunately, this is exactly what you will get if you follow the pattern above.

Instead, try the pattern below:

val, err := doSomething()
if err != nil {
	return fmt.Errorf("failed to do something; %v", err)
}

This embeds the returned error from doSomething and adds additional context. If needed, we can even add parameters to our error message like an identifier or a file path.

Using the pattern above, our error message from CreateUser could be more helpful: “failed to create user; failed to read config file (prod.toml); file does not exist”. Now, we know exactly what happened - the user failed to create because our implementation tried to read a config file (prod.toml) that was not found.

Initializing Apps

If you have worked on a medium or a large-scale application written in Go, you know that the main.go file can be a long set of messy instructions that are responsible for initializing the entire app including reading configs, flags, creating database connections, app-specific objects, pass dependencies as needed, and more. If you try to make your main.go shorter, you may end up with series of function calls that are essentially doing the same thing - you have passed the problem down the chain but not eliminated it.

Every project has a slightly different way of solving it. Larger companies like Facebook, Google, and Uber have all proposed solutions to this problem. Now, of course, the applications that we are writing are not at their scale and we don’t need to use their solutions to our problems. But, some of the solutions can be applied to smaller applications to introduce modularity, easier testing, extensibility, and saner main.go files.

All of the solutions from the three companies mentioned assume Dependency Injection as the pattern to organize your app. There has been some debate on whether this is pattern belongs in GoLang or not. In my opinion, the pattern is useful and can be adapted to be used in an idiomatic way in Go.

// No dependency injection:
func NewRepository() *Repository {
	return &Repository{
		db: NewGormDB(),
  }
}

// With dependency injection:
func NewRepository(db *gorm.DB) *Repository {
	return &Repository{
		db: db,
  }
}

type Repository struct {
	db *gorm.DB
}

Given the example above, the NewRepository function with no dependency injection has to create the database connection in the initializer. This makes it harder to test the repository layer and hides its true dependencies inside the implementation. As the number of dependencies grow, the NewRepository function will get more complex as well.

In the example with dependency injection, the database connection is passed as a parameter and the repository can be initialized far more easily. We can even pass in a different database connection, easily share it across different parts of our app, and keep the database initialization logic in one place.

However, it is the example with the dependency injection that usually results in a large main.go file because all of the initialization logic has been moved all the way up.

func main() {
	config, err := NewConfig()
	if err != nil { ... }

	db, err := NewDB(config)
	if err != nil { ... }

	usersRepo, err := NewUsersRepo(db)
	if err != nil { ... }

	usersSvc, err := NewUsersSvc(usersRepo)
	if err != nil { ... }

	... repeat for every part of your app ..

	server.Start()
}

Facebook came up with one of the earliest solution to this with the inject library. However, it has been deprecated and archived so we won’t discuss it any further.

The two primary “types” of solution to this are: Runtime DI and Compile-time DI. The goal for both of these solutions is to handle the “wiring up” of your application i.e. you provide the dependency graph and the library takes care of initializing your app.

Uber’s solution, uber-go/fx uses runtime dependency injection. On app launch, it initializes and creates all of the dependencies, and passes them as needed. Failures such as any unmet dependency requirements will result in the application not being able to launch. Using this pattern is powerful but with the downside that there is some “magic” and no compile-time safety.

Google’s solution, google/wire takes a different approach to the same problem. Given a dependency graph, it generates the initialization function for you. This provides compile time safety and the generated code is readable and usually very close to what you would write by hand. The downside is an additional build step required to generate these files.

In my opinion, one of the biggest benefits of using DI in your application is modularity. It has the power to break down your application into reusable components in a way that is very hard to match when wiring components manually. In other words, it can act as the “glue” between different parts of your app.

If you are able to standardize the glue that connects different parts of your app, it can provide a layer for you to build reusable components that can work across completely different apps.


Now that we have covered a lot of the common concepts, let’s discuss areas that are specific to web development:

Routing & Middleware

When building a web server, routing is a core part of it. From my experience, this is where the bulk of frameworks and packages have invested their efforts. gorilla/mux is one of the most influential and powerful set of packages that helps solve this problem. If your project doesn’t rely on a framework, gorilla/mux is the goto solution for this problem.

Other frameworks like gin-gonic/gin provide similar functionality but since they are ‘frameworks’, they tend to provide a more end-to-end solution that include reading and writing request/responses to/from different formats, grouping routers, server startup/shutdown, and more.

Choosing between the two types of solution comes down to deciding whether you want to glue together various dependencies and keep control of your project or instead invest in a framework that can provide a core set of features out-of-the-box. While they are both reasonable choices, in my experience, the more idiomatic way has been to glue together small dependencies such as gorilla/mux. On the other hand, frameworks like gin-gonic/gin have no shortage of Github stars showing that the community also needs more full-featured frameworks.

Request, Response, and Validation

When handling a request in a HTTP request handler, there are a series of steps that you would usually follow:

  1. Read & validate the request body (if needed)
  2. Call your business logic
  3. Present the response

Assuming that you are building an API server that reads and responds using JSON, the Go standard library comes with the encoding/json package that can help marshal and unmarshal JSON. For example, writing JSON to the response body may look something like this:

result, err := doBusinessLogic()
if err != nil {
	...
}

w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)

err = json.NewEncoder(w).Encode(result)
if err != nil {
	...
}

Similarly, the Decoder in the json package provide utilities on reading request bodies. The standard library even provides utilities on reading form values and url parameters. Clearly, there is no need for a third-party package for reading and writing request/response bodies.

When you read a request body, it is considered a good practice to perform some validation on it before passing the data to your business logic layer. The asaskevich/govalidator package among others has a huge library of validators and can be extended to support custom use cases.

Overall, a complete route handler may have a structure similar to:

func (ro *Router) HandleCreateUser(w http.ResponseWriter, r *http.Request) {
	var body struct{
		Name string `json:"name" valid:"required"`
	}

	err := json.NewDecoder(r.Body).Decode(&body)
	if err != nil {
		// ...
	}

	ok, err := govalidator.ValidateStruct(body)
	if !ok {
		// ...
	}

	result, err := ro.svc.CreateUser(body.Name)
	if err != nil {
		// ...
	}

	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)

	err = json.NewEncoder(w).Encode(result)
	if err != nil {
		...
	}

In my opinion, the above pattern provides flexibility and is idiomatic but leaves a lot to be desired in terms of having a simple pattern that needs to be repeated many times in a single app.

Database & ORM

Go’s standard library comes with the database/sql package. So, do you need an ORM or a query builder to handle your queries? You don’t /need/ it but it can save some time. gorm.io/gorm is one of the most popular ORM libraries for Go.

Of course, even if you use a library like gorm, you need to learn SQL. What gorm provides is a query builder and easy mapping from SQL rows to your Go structs. Everything else like migrations is a nice-to-have and can be replaced by any other tool or library you may prefer.

Even if we use a library like gorm, it is critical to have a “repository” layer that can encapsulate all of your queries that returns concrete application-level structs. Here’s an example:

func (r *Repository) FindAssignmentsByCourseID(ctx context.Context, courseID string) ([]Assignment, error) {
	var assignments []Assignment

	err := r.db.
		Where(&Assignment{CourseID: courseID}).
		Find(&assignments).
		Error

	if err != nil {
		...
	}

	return assignments, nil
}

What’s Missing?

Clearly, there is a ton of options available if you want to build your web app using Go. The standard library provides a solid foundation and the community has built a lot on top to meet any requirements that you might have. So, what’s missing and what can we do to fill more gaps?

In my experience, the Go web development ecosystem provides a set of packages that solve a few things at once and can be used together to build complex apps. However, the complexity of integrating them together is left up to the developer and the cost of this is underestimated. To better explain this, let’s take the example of the Ruby on Rails ecosystem. They are widely known for having the ability to ‘install gems’ that can provide powerful functionality with a few lines of code. While the philosophy of Ruby on Rails may or may not be compatible with the Go ecosystem, there are a few lessons to be learned.

I hope that going forward, the Go community can work together to build tools and packages that work seamlessly together. This requires work at the foundational layer of app development (patterns around App Initialization, Errors, Logging, Configs, etc.), and all the way to the top (routing, middleware, database, auth, sessions etc.).

In an ideal world, I should be able to create a standardized Go app with all of the basics figured out, fill in my app-specific business logic, pull in powerful modules to complement my app’s functionality, and deploy easily to the cloud. I will be working on making progress towards this vision and writing more about my progress.

Follow me on Twitter to get future post updates