The easiest way to embed static files into a binary file in your Golang app (no external dependencies)

Introduction

Hello, everyone! 👋 I have a very interesting find for you. But first, I'm going to tell you a little background story, so...

Earlier, I tried go.rice and go-bindata, then I used packr and switched to pkger from the same author (Hi, Mark Bates, if you're suddenly reading this). Yep, I was satisfied with this packages and quite happy!

But when I saw this method somewhere on the Internet, I immediately herded it into all of my Golang apps (who needed it, of course).

...and what's problem with ready-made packages?

First of all, it's external packages with their dependencies. It's good when you need to quickly make a working prototype OR don't really care about the size of your code base.

Documentation may not be complete or cover actually what you need at this moment. I have been frustrated to have to wait a long time (sometimes, up to several weeks) to get an answer OR a bug fix.

I have nothing against the authors of these great Go packages! I personally support many of Open Source repositories on GitHub and understand they very well! 😉

Disclaimer

This article was written before Go 1.16 released, which added awesome features for adding files to the final binary.

Why I need this?

Good question, by the way, because many programmers do not always understand advantages of this approach to packaging a product (the final program, which you sell to your customer).

☝️ Especially, who came from scripting languages like JavaScript, Python, PHP or else.

I highlight the following:

• It's safe. Files, which added to a binary, are impossible (or extremely difficult) to extract back.

• It's convenient. There's no need to always remember to include a folder with static files, when you give production-ready product to your customer.

OK! So what am I suggesting? 🤔

My suggest will be very simple and elegant: why not write our own packer for static files using only built-in Go tools? Yes, gophers have already figured out what I'm talking about... Go generate function!

⭐️ Please, read first this awesome blog post by Rob Pike (22 Dec 2014) for more understand how it works! Unfortunately, I can't explain this principle better and shorter than Rob did...

No time for further reading? 🚀

I created repository with full code example on my GitHub especially for you:

Just git clone and read instructions from README.

Project structure

Let's go mod init ... and create some files 👌

$ tree .
.
├── Makefile
├── go.mod
├── cmd
│   └── app
│       └── main.go          <-- main file of your Go app
├── internal
│   └── box                  <-- template and functions for create blob
│       ├── box.go
│       └── generator.go
└── static                   <-- folder with static files
    └── index.html

Add most used commands to Makefile:

.PHONY: generate

# ⚠️
generate:
    @go generate ./...
    @echo "[OK] Files added to embed box!"

security:
    @gosec ./...
    @echo "[OK] Go security check was completed!"

build: generate security
    @go build -o ./build/server ./cmd/app/*.go
    @echo "[OK] App binary was created!"

run:
    @./build/server

⚠️ I strongly recommend to use Go security checker for your code! Maybe I'll write a separate article about this great package later.

Create methods for use on code (box.go)

//go:generate go run generator.go

package box

type embedBox struct {
    storage map[string][]byte
}

// Create new box for embed files
func newEmbedBox() *embedBox {
    return &embedBox{storage: make(map[string][]byte)}
}

// Add a file to box
func (e *embedBox) Add(file string, content []byte) {
    e.storage[file] = content
}

// Get file's content
func (e *embedBox) Get(file string) []byte {
    if f, ok := e.storage[file]; ok {
        return f
    }
    return nil
}

// Embed box expose
var box = newEmbedBox()

// Add a file content to box
func Add(file string, content []byte) {
    box.Add(file, content)
}

// Get a file from box
func Get(file string) []byte {
    return box.Get(file)
}

All magic included into first line: //go:generate .... This call Go compiler to use methods from this file for code generate by another.

Create template and functions for blob file (generator.go)

//+build ignore

package main

import (
    "bytes"
    "fmt"
    "go/format"
    "io/ioutil"
    "log"
    "os"
    "path/filepath"
    "strings"
    "text/template"
)

const (
    blobFileName string = "blob.go"
    embedFolder  string = "../../static"
)

// Define vars for build template
var conv = map[string]interface{}{"conv": fmtByteSlice}
var tmpl = template.Must(template.New("").Funcs(conv).Parse(`package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    {{- range $name, $file := . }}
        box.Add("{{ $name }}", []byte{ {{ conv $file }} })
    {{- end }}
}`),
)

func fmtByteSlice(s []byte) string {
    builder := strings.Builder{}

    for _, v := range s {
        builder.WriteString(fmt.Sprintf("%d,", int(v)))
    }

    return builder.String()
}

func main() {
    // Checking directory with files
    if _, err := os.Stat(embedFolder); os.IsNotExist(err) {
        log.Fatal("Configs directory does not exists!")
    }

    // Create map for filenames
    configs := make(map[string][]byte)

    // Walking through embed directory
    err := filepath.Walk(embedFolder, func(path string, info os.FileInfo, err error) error {
        relativePath := filepath.ToSlash(strings.TrimPrefix(path, embedFolder))

        if info.IsDir() {
            // Skip directories
            log.Println(path, "is a directory, skipping...")
            return nil
        } else {
            // If element is a simple file, embed
            log.Println(path, "is a file, packing in...")

            b, err := ioutil.ReadFile(path)
            if err != nil {
                // If file not reading
                log.Printf("Error reading %s: %s", path, err)
                return err
            }

            // Add file name to map
            configs[relativePath] = b
        }

        return nil
    })
    if err != nil {
        log.Fatal("Error walking through embed directory:", err)
    }

    // Create blob file
    f, err := os.Create(blobFileName)
    if err != nil {
        log.Fatal("Error creating blob file:", err)
    }
    defer f.Close()

    // Create buffer
    builder := &bytes.Buffer{}

    // Execute template
    if err = tmpl.Execute(builder, configs); err != nil {
        log.Fatal("Error executing template", err)
    }

    // Formatting generated code
    data, err := format.Source(builder.Bytes())
    if err != nil {
        log.Fatal("Error formatting generated code", err)
    }

    // Writing blob file
    if err = ioutil.WriteFile(blobFileName, data, os.ModePerm); err != nil {
        log.Fatal("Error writing blob file", err)
    }
}

At this point, we set //+build ignore on first line, because this code needed only for code generation process and don't be in main build.

We're almost done! 👍

Let's go through the block file generation process itself in more detail.

So, running command:

$ make generate

...it created for us blob.go file (on current project folder) with this code:

package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    box.Add("/index.html", []byte{35, 32, ...}
    // ...
}

Basically, we converted the files in the ./static folder to a slice of bytes and now we can easily add them (in this clean form) to our binary executable! 🎉

Time to use it all in your code

package main

import (
    "log"
    "net/http"
    "text/template"

    "github.com/koddr/example-embed-static-files-go/internal/box"
)

type PageData struct {
    Title       string
    Heading     string
    Description string
}

func main() {
    // Define embed file (for a short)
    index := string(box.Get("/index.html"))

    // Template
    tmpl := template.Must(template.New("").Parse(index))

    // Handle function
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        data := PageData{
            Title:       "The easiest way to embed static files into a binary file",
            Heading:     "This is easiest way",
            Description: "My life credo is 'If you can not use N, do not use N'.",
        }

        // Execute template with data
        if err := tmpl.Execute(w, data); err != nil {
            log.Fatal(err)
        }
    })

    // Start server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatal(err)
    }
}

☝️ Always use / for looking up! For example: /images/logo.jpg is actually ./static/images/logo.jpg.

Exercises 🤓

• Improve box.Get() function for getting all files from embed folder at once call (for example, by using wildcard). Like box.Get("/images/*") or similar.
• Exclude path to directory of static files to flags and when call it on go:generate.

Final words

I understand, that it would be much better to take a ready-made library and use its API... but then you would never get to know this wonderful tool and write your first code generator!

Invest in yourself and your horizons! 🎁

Photo by

[Title] Vitor Santos https://unsplash.com/photos/GOQ32dlahDk
[1] Brunno Tozzo https://unsplash.com/photos/t32lrFimPlU
[2] Christin Hume https://unsplash.com/photos/Hcfwew744z4

P.S.

If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻

And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!

My projects that need your help (and stars) 👇

• 🔥 gowebly: A next-generation CLI tool for easily build amazing web applications with Go on the backend, using htmx & hyperscript and the most popular atomic/utility-first CSS frameworks on the frontend.
• ✨ create-go-app: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.
• 🏃 yatr: Yet Another Task Runner allows you to organize and automate your routine operations that you normally do in Makefile (or else) for each project.
• 📚 gosl: The Go Snippet Library provides snippets collection for working with routine operations in your Go programs with a super user-friendly API and the most efficient performance.
• 🏄‍♂️ csv2api: The parser reads the CSV file with the raw data, filters the records, identifies fields to be changed, and sends a request to update the data to the specified endpoint of your REST API.
• 🚴 json2csv: The parser can read given folder with JSON files, filtering and qualifying input data with intent & stop words dictionaries and save results to CSV files by given chunk size.