Writing a URL Shortener in Go

Note: This article was published a few months after I originally wrote the go-short scratch project. The conclusion includes some comments on how I’d further improve this project now that I have additional Go experience.

I recently started learning the Go Programming Language and decided to build a small project to better explore the language. After a bit of deliberation, I settled on writing a URL Shortening Service that leverages Redis as it’s backing datastore (based on an exercise from 7 Databases in 7 Weeks). My hope was that by picking a simple project I could better focus on leveraging the Go Tools, Go Testing, and learn more about Go best practices. The basic URL Shortener only needs 3 routes:

• Page entering URLs to shorten
• Endpoint to create a new shortened URL
• Endpoint that redirects shortened URLs

This article is divided into sections describing each of my coding sessions. I entered each session with a few fixed goals and try to outline my different takeaways and lessons learned. I’m not a Go Expert, so if you see instances of unidiomatic code, please let me know below!

Lastly, you can find the project source code on Gitlab.

Session 1: Project Startup, CI and Docker

My main goals for this session was to get the project in a state where it compiles, passes a dummy test and integrates with Gitlab CI to verify the Build/Test stages.

The Go language comes with a number of excellent utilities out of the box that made Docker and CI integration pretty straightfoward for someone new to the language. My initial thought was to try leveraging the go build and go test commands in different stages of the .gitlab-ci.yml file and see if that would be sufficient. I decided that I’d go ahead and create the basic project structure and add a dummy entrypoint and test that the CI/CD could run.

I began by creating a new directory in my GOPATH for the project, initializing the project repository and creating go-short.go, seen below:

package main

import (
	"fmt"
	"os"
)

// Hello returns a formatted Welcome Message
func Hello(target string) string {
	return fmt.Sprintf("Hello, %s!", target)
}

func main() {
	target := "World"
	if len(os.Args) >= 2 {
		target = os.Args[1]
	}

	fmt.Println(Hello(target))
}

After verifying that the project could be built using go build, I went ahead and added a simple test in the file go-short_test.go:

package main

import (
	"strings"
	"testing"
)

func TestHello(t *testing.T) {
	var tests = []string{
		"Hello", "World", "John", "Jane", "abc",
	}

	for _, input := range tests {
		if !strings.Contains(Hello(input), input) {
			t.Errorf("Hello(%q) contains %v, failed", input, input)
		}
	}
}

For this dummy test case I implemented a few testing guidelines outlined in the Go Programming Language book. I used table-driven testing, in which multiple test cases are specified as a slice of structs or values and evaluated in a loop. Additionally, I leverged t.Errorf to report failures, rather than t.Fatal, as t.Error will mark the test case failure without terminating execution of the test. This allows us to review all possible test failures for a single table-driven test, rather than aborting at the first error. I also used an expressive error formatting message that explicitly states the expectation that failed and the corresponding parameters. Finally, in order to make the test more robust, I checked for the inclusion of a given substring, rather than evaluating the entire returned string. This way any minor tweaks to the Hello function wouldn’t invalidate our test cases completely.

Next I verified these tests ran and passed by running go test. As always, if you aren’t writing the test prior to implementing the function body you should force the test to fail to verify the test is making the correct assertions (ex: if strings.Contains(Hello(input), input)), or break the Hello function implementation).

With a basic program and test in place, we can finally add a .gitlab-ci.yml file to integrate CI/CD. This ended up being very simple:

image: golang:1.11
stages:
  - build
  - test

build:
  stage: build
  script:
    - go build

test:
  stage: test
  script:
    - go test

As a bonus, I also went ahead and added a Dockerfile based on the official image:

FROM golang:1.11

WORKDIR /go/src/gitlab.com/cfilby/go-short
COPY . .

RUN go install
EXPOSE 8080
CMD ["go-short"]

With these files in place, the last step was to commit and push the project to Gitlab, completing the development session.

Takeaways:

• The go-short.go code written for day one is ultimately going to be removed from the project, but my goal was to have anything that compiles and can integrate with CI. I always aim to have something running in projects as soon as possible, even if it’s dummy functionality.
• The strong emphasis on built in tooling in Go is phenomenal. It removes a lot of the guesswork and reduces the number of decisions that need to be made before being productive. It was nice to have consistent formatting out of the box, a solid test framework, easy builds, etc.
• CI/CD Integration for a basic Go project was pretty painless
• The Go Testing framework is definitely different. It seems like some aspects of testing require more boilerplate, but the resulting tests seem more expressive and easier to read and debug. It will be interesting to see how this feels on more complex functions.
• While GOPATH is pretty straightforward after some reading, I plan to look into the new Go Modules system and see how that feels. Not being constrained to a specific directory structure would be nice.

Session 2: Modules, Storage and Testing

I embarked on this session with the goal of integrating the new Go Module system, adding a few third party dependencies, integrating Redis and implementing the URL shortening functionality. I also wanted to add more tests for more complex functionality.

Go Modules

It looks like you can only initialize Go Modules outside of the GOPATH, so I moved the project to another directory and ran go mod init gitlab.com/bindersfullofcode/go-short. Running the command generates a go.mod and go.sum file for tracking project dependencies.

With the Go Module setup, all that’s left to do is add some dependencies and get coding! Go Module will save any dependencies you go get from the project directory to the go.mod and go.sum files. For this project I opted to use go-redis/redis for ‘persistence’ and teris-io/shortid to generate the IDs that map to full length URLs.

go get -u github.com/go-redis/redis
go get -u github.com/teris-io/shortid

Implementing Storage

Given that this URL shortener is a simple proof of concept, I decided to store the URL ShortID as a key in Redis and have the corresponding value be the URL. A more robust implementation would store statistics, include a timeout, and more.

To implement this in code we need to use the Redis client to execute the following commands:

// Save the ShortID and URL
SET abcdef http://google.com
OK

// Fetch a URL for a ShortID
GET abcdef
http://google.com

// Check if the ShortID already exists to avoid overwritting
EXISTS abcdef
1

My initial thought was to create a generic key/value Storage Interface for the rest of the application to leverage in order to abstract away the underlying use of Redis.

// Storage Exposes a basic Key/Value Storage Interface
type Storage interface {
	Get(key string) (string, error)
	Set(key string, value interface{}) error
	Exists(key string) (bool, error)
}

The RedisStorage struct that conforms to this interface was fairly simple to implement. For the sake of brevity, I’ll only include the Exists method below, but you can see the rest of the code on Gitlab:

// Exists checks if the specified key is in use
// Returns false when an error occurs.
func (r *RedisStorage) Exists(key string) (bool, error) {
	exists, err := r.client.Exists(key).Result()
	if err != nil {
		return false, err
	}

	return exists == 1, nil
}

With the Storage Interface and Redis implementation in place, I swapped my focus to the URL Shortening code. Using the Storage interface means we could easily swap out the underlying implementation for testing or for a different database altogether.

Again, I created an interface to wrap the URLShortening functionality (this should make testing components that use the Shortener easier):

// Shortener is the interface that wraps UrlShortening operations
type Shortener interface {
	Shorten(url string) (string, error)
	Get(key string) (string, error)
}

The implementation for the Shorten function is particularly interesting as it needs to generate an id, check if that id is already in use, and store the new shortened URL and report any errors:

// Shorten creates a shortened URL Key and persists the key to the underlying store
// Returns the shortened url key if successful
func (u *Store) Shorten(url string) (string, error) {
	id, err := generateID()
	if err != nil {
		return "", fmt.Errorf("generating id for %s: %v", url, err)
	}

	exists, err := u.storage.Exists(id)
	if err != nil {
		return "", fmt.Errorf("checking existance of key %s for url %s: %v", id, url, err)
	} else if exists {
		return "", fmt.Errorf("key %s for url %s already exists in storage", id, url)
	}

	if err := u.storage.Set(id, url); err != nil {
		return "", fmt.Errorf("storing key %s for url %s: %v", id, url, err)
	}

	return id, nil
}

The generateID function is a package level variable to allow for easy swapping of the implementation during testing (seen below).

The final step was to create some tests to verify the functionality and error handling of the Get and Shorten functions. Below you can see the TestShortenGenerateID function that stubs out the generateID function for testing purposes (and eventually restores it):

// This test is pretty clunky, but I wanted to try my hand at testing external dependencies
// as outlined in The Go Programming Language 11.2.3
// TODO: Complete Testing of Shorten
func TestShortenGenerateID(t *testing.T) {
	tests := []struct {
		key         string
		generateID  func() (string, error)
		shouldError bool
	}{
		{"should succeed", func() (string, error) { return "KEY", nil }, false},
		{"should error", func() (string, error) { return "", errors.New("Error") }, true},
	}

	idGenerator := generateID

	for _, test := range tests {
		store := createURLStorage(nil)
		generateID = test.generateID

		_, err := store.Shorten(test.key)
		if test.shouldError && err == nil {
			t.Errorf("store.Shorten(%s) expected error, found none", test.key)
		}

		if !test.shouldError && err != nil {
			t.Errorf("store.Shorten(%s) found error, expected none: %v", test.key, err)
		}
	}

	generateID = idGenerator
}

Note: The createURLStorage(map[string]string) function creates a fakeStorage instance backed by a map.

I felt that this coding session definitely exposed me to some areas I need to study further in Go. While I’m content with the Shortener and Storage abstractions, I think that I should have used finer grained interfaces to simplify testing and make the Storage more generalizable. Likewise, my current tests feel a bit clunky, especially when it comes to testing error paths.

Takeaways:

• I need to spend more time learning about Go Interfaces and their proper usage.
• Implicit interface satisfaction in Go is a super powerful and flexible language feature.
• I’m probably underutilizing the Dependency Inversion Protocol. Go implicit interface satisfaction would make it trivial to define the URLStorage interface at the URLShortener level and have the Redis Storage implicitly satisfy it, rather than having the URL Shortener (Domain Logic) depend on a low level Storage Interface (“boring detail”).
• It might be worth creating some custom error types. I opted to use fmt.Errorf to wrap and extend existing errors, but it might be worth integrating a more robust error handling library.
• I may be overusing methods and may want to look more into using functions that receive interfaces instead. This combined with fine grained interfaces would make passing stubs to testing trivial.
• Setting up a fakeStorage implementation for testing was very straightforward with the Storage interface in place, but my tests for URLShorten were fairly lengthy.

Next up: integrate net/http!

Session 3: net/http

The goal of this session was to create the HTTP request handlers and wire up the URL Shortening functionality. Our HTTP Handlers need to respond to three types of requests: showing the Home Page, creating a URL Redirect and responding to a URL Redirect request.

These types of requests correspond to the following three REST Methods:

• GET / - Get the Home Page
• POST / - Create a Shortened Link
• GET /:redirect_key - Redirect to the specified link

The go standard library includes the net/http and html/template packages, both of which are great starting points for a basic server rendered pages project. If our project were more complex, I would go ahead and integrate gorilla/mux for the cleaner routing abstraction and middleware support, but we can make do without it for now.

Entrypoint

We’ll start by rewriting the go-short.go entry point to start a web server.

func main() {
	mux := http.NewServeMux()
	web.RegisterHandlers(mux)

	if err := http.ListenAndServe(":8080", mux); err != nil {
		log.Fatalf("unable to start server: %v\n", err)
	}
}

We begin by creating a new ServeMux that we’ll register our request handlers on. net/http includes a number of methods to register request handlers (such as http.HandlerFunc(pattern string, handler func(ResponseWriter, *Request))) which register handlers onto the http.DefaultServeMux. The ListenAndServe function starts the webserver and accepts an http.Handler which is responsible for handling incoming requests. When the handler argument is nil, the DefaultServeMux is used. In general, creating a ServeMux is preferred as using the DefaultServeMux can allow third party packages to unexpectedly register handlers onto your webserver. Alex Edwards explains go routing and the DefaultMux in greater detail (here)[https://www.alexedwards.net/blog/a-recap-of-request-handling]. Lastly, we create a function in our shortner package that accepts a ServeMux and registers the request handlers.

Views

The go standard library includes both a (text/template)[https://golang.org/pkg/text/template/] and (html/template)[https://golang.org/pkg/html/template/] that implement the same interface interface. The Template interface allows you to load/parse individual templates, or load them all via a pattern or variadic argument list and reference them by name. Seeing how we have two templates, I’ve opted to load all templates matching the views/*.html glob.

The views/ folder contains two templates, home.html and created.html

var tmpl = template.Must(template.ParseGlob("views/*.html"))

When rendering a page, we must provide the an io.Writer to write the rendered template to, the name of the desired template, and the context to inject into the template. For simplicity I’ve created the following helper function in handlers.go.

func renderPage(w http.ResponseWriter, template string, content interface{}) {
	if err := tmpl.ExecuteTemplate(w, template, content); err != nil {
		http.Error(w, fmt.Sprintf("rendering template: %v", err), http.StatusInternalServerError)
	}
}

The template syntax itself has vague similarities to the Handlebars templating language: {{ }} is used to wrap actions. You can reference properties on the context/data struct using the cursor (a dot/period). Ex: {{ .URL }} would correspond to the URL field of the following data struct type Data struct { URL string }.

You can see the content of created.html below:

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>URL Shortener</title>
    </head>
    <body>
        <h2>Go Short!</h2>
        <p>New URL Created!</p>
        <a href="/{{.ShortID}}">{{.ShortID}}</a>
    </body>
</html>

This templating library is fairly expressive and allows for conditional statements, loops and the nesting/combining of various several templates into a single rendered view. For simplicity I repeated the whole document structure in both templates, but a more complete implementation would share a common site layout.

Additional Resources:

• Build Web Applications With Golang
• Template Syntax

Handlers

We’ll have a centralized router for our HTTP Requests that looks like the following:

func handleHomePage(w http.ResponseWriter, r *http.Request) {
	if r.URL.Path == "/" {
		if r.Method == http.MethodPost {
			createShortenedURL(w, r)
		} else if r.Method == http.MethodGet {
			renderPage(w, "home.html", nil)
		}
	} else if r.Method == http.MethodGet {
		handleRedirect(w, r)
	}
}

When GET - / is invoked, we’ll render the home page. When POST -/ is invoked we’ll create a new redirect (if possible) and render the created page. Lastly, if a non-explicit path is provided we’ll attempt to perform a redirect. Using gorilla/mux would provide a much more robust path matching solution, but this basic router should serve our purposes.

The handlers all leverage a package level urlShortener object var urlShortener = shortener.New(storage.New("localhost:6379")).

The createShortenedURL handler must parse the request body, validate the arguments, create a new URL and render the created page. It also renders appropriate status codes where possible:

func createShortenedURL(w http.ResponseWriter, r *http.Request) {
	if err := r.ParseForm(); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	if r.Form["url"] == nil || len(r.Form["url"]) == 0 {
		http.Error(w, "no url provided", http.StatusBadRequest)
		return
	}

	url := r.Form["url"][0]
	shortID, err := urlShortener.Shorten(url)
	if err != nil {
		http.Error(w, fmt.Sprintf("unable to shorten %s: %v", url, err), http.StatusInternalServerError)
		return
	}

	renderPage(w, "created.html", struct {
		ShortID string
	}{shortID})
}

Notice the correspondence between the anonymous struct and the created.html template shown in the View section.

The redirect handler slices the URL path to remove the leading /, finds the corresponding key in the Shortener and redirects to the stored URL if found.

func handleRedirect(w http.ResponseWriter, r *http.Request) {
	key := r.URL.Path[1:]
	url, err := urlShortener.Get(key)
	if err != nil {
		http.Error(w, fmt.Sprintf("unable to find %s: %v", key, err), http.StatusNotFound)
		return
	}

	http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}

Finally, this central router is registered using a public RegisterHandlers function.

func RegisterHandlers(mux *http.ServeMux) {
	mux.HandleFunc("/", handleHomePage)
}

Takeaways:

• The net/http library is pretty powerful, but RESTful APIs/Path Matching requires manual implementation or an external library
• Translating lower level errors into http status codes could be awkward without explicit error types
• The view templating library is fairly robust, but layout setup requires some initial work.
• The handlers/applciation could be tested using the http/httpest package but were skipped given the retroactive nature of this writeup.

Conclusions

These conclusions are written with the benefit of hindsight given that this article was finished several months I initially started writing it. Since then I’ve learned a bit more about Golang and have several ideas on how I’d further tweak this application.

• I’d have leveraged a (multistage build)[https://medium.com/travis-on-docker/multi-stage-docker-builds-for-creating-tiny-go-images-e0e1867efe5a] for the Docker image.
• I would have leveraged the (internal directory)[https://blog.learngoprogramming.com/special-packages-and-directories-in-go-1d6295690a6b]
• I would have leveraged gorilla/mux. I get that Golang has a very strong emphasis on building libraries yourself, but I’d happily accept the external dependency for more explicit router definitions.
• I’d have written tests for the handlers. httptest.ResponseRecorder is pretty painless to use in my work experience and writing tests here would have been pretty straightforward.
• I’d probably have hoisted the Shortener/Storage interfaces to a higher level. The handlers/Shortener should have depended on abstractions at their level, rather than a lower level abstraction.
• I’d explore the golang template library further and make a base layout file. Just for the sake of finishing the project I opted to duplicate the layout in both templates, but it probably wouldn’t have required that much effort to do it correctly.

For those that are new to Go, I’d definitely recommend writing a basic web applciation with minimal dependencies. The standard library has a lot of powerful abstractions out of the box, and understanding these abstractions can give you a better sense of how to write idiomatic Go code. You can always view the standard library source code in the godocs if you’re ever wondering how it’s implemented under the hood (Ex: (http.HandleFunc)[https://golang.org/src/net/http/server.go?s=73427:73498#L2396])

If you’ve made it this far, thanks for taking the time to read this article! If you have any comments, suggestions or corrections, please let me know below!