Lessons Learned Building Web Applications With Go

Chris Gansen · chris@publicgood.com · @cgansen

Hello

Day job

I'm CTO at Public Good Software, a startup in Chicago.

Prior to Public Good, I built technology for the Obama 2012 campaign, IBM, and Smart Chicago Collaborative.

I've built a few Go webapps:

HealthNear.me
tool for people to find public health resources
ChicagoWorksForYou.com
scrape and analyze Chicago's 311 data
PublicGood.com
today's topic.

PublicGood.com

"doing good done right"

What is it?

Public Good helps nonprofits raise more money and meet new donors by making online fundraising easy and accessible to all.

Technology

We've built a data platform using Go, a public website using AngularJS, and run it all on Amazon Web Services.

We've evolved from "hello, world" 2 years ago to powering online fundraising for hundreds of nonprofits.

How we use Go

Our entire backend – API, data platform, workers, and maintenance utilities – is written in 27K lines of Go.

              $ sloc parade/
                Language  Files   Code  Comment  Blank  Total
                   Total    545  27199     1516   5742  34287
                      Go    544  27175     1514   5731  34250
                    Make      1     24        2     11     37
            
code count via sloc

Why Go?

We came out of a high-pressure, high-stakes environment and weren't happy with Ruby and Python.

Our initial plan for Public Good was a high-performance data integration platform.

We wanted a language that was fast, easy to deploy and maintain, and could scale very quickly.

Go advantages

(circa 2013)

  1. Strong bonafides
  2. Statically typed
  3. Super fast
  4. Concurrency is built-in
  5. Build/test/deploy/run cycle is straight-forward.

Go disadvantages

(circa 2013)

  1. (relatively) new
  2. Limited 3rd party tools and libraries
  3. Not many examples of big production apps using Go

Lessons

  1. Structuring code is hard.
  2. Concurrency is awesome.
  3. Deploying and running Go apps is easy.
  4. The Go ecosystem is robust.

Lesson №1

Structuring code is hard.

Our core platform has gone through three phases:

  1. Single API endpoint and two background workers
  2. Half-dozen services and workers, lots of spaghetti code
  3. 19 services, two dozen data models, many background jobs, "clean" architecture

Small Go apps are simple and powerful

When we had a single endpoint, builds were really fast, code was clean, and everything worked as a single binary.

Growing code

Many languages have prescriptive frameworks (e.g. Rails and Django) that enforce application structure.

Go is a language, not a framework; therefore, code structure is left as an excercise for the reader.

Use packages wisely

For a while we had two core packages: services and data.

This was a pretty simple structure, but led to slower builds and bloated packages.

Enter the Clean Architecture

source: 8th Light

Our application stucture

                ./parade:
                    cmd
                        apiserver
                        ...
                    jobs
                        advisementmonitor
                        ...
                    lib
                        attrs
                        cache
                        config
                        ...
                    model
                        account
                        activity
                        advisement
                        ...
                    service
                        accounts
                        activities
                        ...
                    usecases
                        AddCardForPerson.go
                        AddContactForOrg.go
                        AddContactForPerson.go
                        ...
                    worker
                        ...
              

Splitting code into model, service, and usecases enforces a separation of interests, makes the code more modular, and keeps packages small and lean.

"Applying the Clean Architecture to Go Applications" by Manuel Kiessling is a must-read and informed much of our work.

Code generation

Our next frontier is automatically generating much our code.

go 1.4 and go generate are very good at this.

Enter mapper

Small tool that creates basic model code (DB CRUD, and some other bits)

package test

//go:generate mapper $GOFILE

type Item struct {
        ID           int64   `json:"id,string"`
        Name         string  `json:"name"`
        OmitMe       float32 `json:"magic" db:"-"`
        SpecialField string  `db:"different_and_special" json:"special_field"`
}

// generated by mapper; DO NOT EDIT

package test

// imports...

type Mapper struct {
    DB dbconn.Queryer
}

var TableName = ItemName

var DataCols = []string{ "id", "name", "different_and_special",  }

var ColumnNames = append(model.MetaCols, DataCols...)

func (mapper Mapper) Count(clause string, args ...interface{}) (int, error) {
    var c int
    query := squirrel.Select("COUNT(*)").From(TableName).RunWith(mapper.DB)
    if clause != "" {
        query = query.Where(clause, args...)
    }

    if err := query.QueryRow().Scan(&c); err != nil {
        return 0, err
    }

    return c, nil
}

Code generation

Huge wins for the team:

  1. We eliminated tedious boilerplate code.
  2. It is now simple to make sweeping changes to all generated code.
  3. Eliminated small discrepancies across the codebase.

Lesson №2

Concurrency is awesome.

healthcheck example

var checks = []*healthCheck{
  &healthCheck{
      Name: "db",
      Fn: healthcheck.Database{DB: c.DB}.Check,
  },
  &healthCheck{
      Name: "elasticsearch",
      Fn: healthcheck.Elasticsearch{}.Check,
  },
  &healthCheck{
      Name: "memcache",
      Fn: healthcheck.Memcache{MC: c.Config["PARADE_MEMCACHE_SERVERS"].(string)}.Check,
  },
}
              healthy := 1
done := make(chan bool, len(checks))
var results []*healthCheck

for _, chk := range checks {
	chk := chk
	go func() {
		chk.Do()
		healthy = healthy & chk.Healthy
		done <- true
	}()
}

for i := 0; i < len(checks); i++ {
	<-done
}
              
            

if !healthy {
	// oh crap!
	c.Log.Warning("healthcheck: system failed health check")
}

c.Response["checks"] = results
c.Response["overall_healthy"] = healthy
               

Fork work to the background

A common API service handler pattern is:

  1. read client input
  2. marshal input to a data structure
  3. do "work" with the input
  4. marshal result of work to an output
  5. send response to client
  1. read in a JSON respresentation of a donation
  2. process the donation
  3. send email notifications
  4. update metrics services
  5. publish activities to feeds
  6. queue for later processing
  7. respond to client with success message
  1. read in a JSON respresentation of a donation
  2. process the donation
  3. respond to client with success message
  4. send email notifications
  5. update metrics services
  6. publish activities to feeds
  7. queue for later processing

Anonymous funcs = :-)

go func(){
        go counters()
        go emails()
        go alerts()
}()

Caveat: you still need to check for an recover from errors in goroutines!

We've gone through our code with a fine-toothed comb to see what work absolutely has to happen synchronously, and what can happen in the background.

We've found that in most cases, a bulk of our work can happen asynchronously.

YMMV.

Lesson №3

Deploying and running Go apps is easy.

Wasn't always easy

We started with cross-compiling on local dev machines, then rsync-ing binaries to the production machines, then doing manual restarts.

It worked, but with more than a single developer, it quickly became a nightmare.

  1. dev runs ./build_release.sh 1.2.3, which pushs a new tag to Github, and starts release job on Jenkins box.
  2. release job runs a docker container with our build environment
  3. docker build container runs go build
  4. release job packages binaries into a .rpm and publishes to a private S3 bucket
  5. We run automated tests to sanity check the release
  6. We run yum update ... on app nodes and restart supervisord

time to production is ~ 5 minutes.

We have containerized all of our background tasks.

Once we have time and/or energy to migrate into VPC, we'll containerize the API and start using AWS ECS.

Lesson №4

The Go ecosystem is robust.

We heard this a lot when we started

We wanted to use Go for our _________, but we looked and _________ wasn't there, so we used _________ instead.

Since then: 4x growth in "golang" searches

OSS libraries

data layer
lib/pq, squirrel, go-memcache, elastigo
api/web
gocraft/web, goauth2, osin, jwt-go, grace
backend
statsd, iron_go, braintree-go, gochimp
dev
forego, goconvey, goimports, goreturns, godep, ngrok

Dependency management :-(

Don't ever rely on someone else's master to be stable.

godep is the best out there, but still a blunt and cranky tool.

The recent interest from the Go core team in making dep. management easier is promising.

Code generation is the future

We're really excited to see what tools the community release that leverage go generate.

Our initial experience with stringer and jsonenums is very positive.

We plan to open source our data layer generation code when it's ready.

Resources

aka: where I go for help

"Effective Go"

I read this every day for three months, and still go back to it regularly.

https://golang.org/doc/effective_go.html

Go standard library

The standard library code is full of clear, concise Go code, and lots of comments. It is an inspiration.

Golang Weekly

http://golangweekly.com/issues

Great way to stay abreast of new libraries, topics in the community, and Go success stories.

Thank you

This presentation is available online

https://pgs.io/golessons

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Keep in touch

email
chris@publicgood.com
twitter
@cgansen