A test suite for any package is explicitly testing different code paths and identifies how your code works using different inputs. But does it actually manage to cover all the code branches that exist in your code? How about all the different functions?
One way to answer these questions is to thoroughly look at all the tests written (by you and your colleagues) and understand if there are any gaps. The easiest way to do that though is to use a tool, and in the case of Go this tool is called cover.
There is an excellent blog post on the Go Blog, called “The cover story” that describes how Go uses the cover tool to generate reports that I definitely recommend reading. In this post we are going to analyze a few parts of how the cover tool works and also provide a way to use it for packages which include sub-packages.
Setting up
We are going to use a package called go_test_coverage that includes 2 sub-packages (we’ll see why in a minute but for now let’s just go with it 😉). You can find the code on my GitHub account and the directory would roughly look like this:
. ├── calculator │ ├── calculator.go │ └── calculator_test.go └── size ├── measurements.go └── measurements_test.go
🖥️ The full code is on my Github repo: https://github.com/efrag/blog-posts
There size package and code are from the Go blog post mentioned above and the calculator package defines a few functions that do basic arithmetic operations (sum, subtract, multiple and abs).
Test coverage 101
Running the tests for our package is simply done by running
$ go test --count=1 ./... ok github.com/efrag/blog-posts/go_test_coverage/calculator 0.002s ok github.com/efrag/blog-posts/go_test_coverage/size 0.001s
This output tells us that our tests for the 2 sub-packages run successfully and the time it took to run them. Now if we wanted to see how much of the code in our packages was covered by the tests that we have written we would run:
$ go test --count=1 --cover ./... ok github.com/efrag/blog-posts/go_test_coverage/calculator 0.002s coverage: 16.7% of statements ok github.com/efrag/blog-posts/go_test_coverage/size 0.001s coverage: 42.9% of statements
So by adding the --cover parameter to the command our output is enriched by the coverage, which says that we have covered 16.7% of the statements in the calculator package and 42.9% of the statements in the size package.
Counting Statements and Coverage
The default output already gives us a great idea of how much of our code is actually tested for each sub-package. Go actually uses a really neat way of determining what constitutes a statement and to see of it has been executed. When running the cover tool as part of our tests, Go does the following.
1️⃣ Identifies code branches
A code branch is the most nested opening / closing of curly braces that contains statements that can be executed. In the image below we can see what that translates to for part of the calculator.go file included in the repo.
Lines 15-17 define a code branch that actually includes the entirety of the multiply function. The abs function is a bit more complicated in this view as it actually defines 3 code branches:
- lines 19-25 which define the entire function
- lines 20-22 that define the execution of the if statement
- and finally line 24 that defined the code branch with the return of the positive number
a
2️⃣ Defines a new struct and counts statements
Now that the tool has the ability to identify these branches it also needs a way of keeping track of them and also of counting the number of statements each of them includes. This is done in 2 steps:
- Go defines and adds a new variable in our program called
GoCoverthat holds this information - Go also sets parts of the struct in the entry point of every code branch
You can see exactly what Go generates by running the following command for our file:
$ go tool cover -mode=set calculator/calculator.go
//line calculator/calculator.go:1
package calculator
type Calculator struct {
name string
}
func sum(a, b uint8) uint16 {GoCover.Count[0] = 1;
return uint16(a) + uint16(b)
}
func subtract(a, b int8) int16 {GoCover.Count[1] = 1;
return int16(a) - int16(b)
}
func multiply(a, b int8) int16 {GoCover.Count[2] = 1;
return int16(a) * int16(b)
}
func abs(a int8) int8 {GoCover.Count[3] = 1;
if a < 0 {GoCover.Count[5] = 1;
return -a
}
GoCover.Count[4] = 1;return a
}
var GoCover = struct {
Count [6]uint32
Pos [3 * 6]uint32
NumStmt [6]uint16
} {
Pos: [3 * 6]uint32{
7, 9, 0x2001d, // [0]
11, 13, 0x20020, // [1]
15, 17, 0x20020, // [2]
19, 20, 0xb0017, // [3]
24, 24, 0xa0002, // [4]
20, 22, 0x3000b, // [5]
},
NumStmt: [6]uint16{
1, // 0
1, // 1
1, // 2
1, // 3
1, // 4
1, // 5
},
}At the bottom of the output printed on our terminal we can see the new Struct with its 3 fields:
- Pos: defines the lines where the code branches start and finish
- NumStmt: defines the number of statements (go commands) that will be executed if we enter a code branch
- Count: defines an integer slice with as many elements as code branches that we have. If a code branch is executed during a test its value is going to be set to 1 in the slice.
We can also see the values of Count being set in the code, in lines 9, 13, 17, 21, 22 and 26 for the 6 branches we have in this code.
3️⃣ Calculates statements run / branch
And finally the last step here is to calculate the coverage which is just the total number of statements run for each of the branches that got executed during the test run, divided by the total number of statements in our package 🎉.
Coverage profile: detailed output of cover
So far we have only seen the final output of the cover tool which tells us the percentage of the statements that get executed when we run our suite. If you would like to see the specific numbers calculated for your packages you can check out the output of the coverage profile.
The Coverage profile is a file that gets generated by running the following command:
$ go test --count=1 -coverprofile=coverage.out ./... ok github.com/efrag/blog-posts/go_test_coverage/calculator 0.003s coverage: 16.7% of statements ok github.com/efrag/blog-posts/go_test_coverage/size 0.001s coverage: 42.9% of statements
The output of the command is exactly the same as before but it will have also generated a coverage.out file which in this case looks like this:
mode: set github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:3.25,4.9 1 1 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:16.2,16.19 1 0 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:5.13,6.20 1 1 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:7.14,8.16 1 0 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:9.14,10.17 1 1 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:11.15,12.15 1 0 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:13.16,14.16 1 0 github.com/efrag/blog-posts/go_test_coverage/calculator/calculator.go:7.29,9.2 1 1 github.com/efrag/blog-posts/go_test_coverage/calculator/calculator.go:11.32,13.2 1 0 github.com/efrag/blog-posts/go_test_coverage/calculator/calculator.go:15.32,17.2 1 0 github.com/efrag/blog-posts/go_test_coverage/calculator/calculator.go:19.23,20.11 1 0 github.com/efrag/blog-posts/go_test_coverage/calculator/calculator.go:24.2,24.10 1 0 github.com/efrag/blog-posts/go_test_coverage/calculator/calculator.go:20.11,22.3 1 0
The contents of the file are the detailed output that lists all the code branches in the packages with the number of statements that they include and how many of those got executed with our tests. More specifically these 2 lines:
github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:3.25,4.9 1 1 github.com/efrag/blog-posts/go_test_coverage/size/measurements.go:16.2,16.19 1 0
Say that there is a code branch in the measurements.go file (lines 3-4) with 1 statement (lines 3-4) which indeed executed by our tests and another code branch (line 16) with 1 statement that we did not execute.
Adding the numbers for the size package we see that we have 7 statements in total 3 of which were covered by tests, which means the coverage for the package is 3*100/7 = 42.86%. Performing the same calculation for the size package we have 6 statements with 1 covered by tests which gives us 1*100/6 = 16.67% coverage for the package.
Package level coverage
The last piece of the puzzle for today is how can we actually get the coverage percentage for our entire package, in this case that go_test_coverage, which as we have seen is made up of two sub-packages the size and calculator. Simply using the percentages that we can see in the output of cover is not the right approach as these tell us nothing for the total number of statements executed overall.
For that we are going to use the cover profile that we saw earlier and instead of isolating our calculations per sub-package we are going to use the full output and calculate the value for our entire package. This would give us a total of 13 statements 4 of which where executed by our tests which means a coverage of 4*100/13 = 30.77%.
Ideally we would like a way of doing this calculation automatically rather than manually inspecting the file. Fear not – shell and awk are here:
$ go test --count=1 -coverprofile=coverage.out ./... ; \
cat coverage.out | \
awk 'BEGIN {cov=0; stat=0;} \
$3!="" { cov+=($3==1?$2:0); stat+=$2; } \
END {printf("Total coverage: %.2f%% of statements\n", (cov/stat)*100);}'This will:
- create the coverage profile as before using the go test command
- then it will read it and print it on screen
- and finally it will pass it through an
awkprogram
The program defines two accumulators (one for the statements and one for the covered lines) and looping through each line of output it adds the values together. Finally it prints the Total coverage: % of statements for your package. Running the above will output this 💪🎉:
ok github.com/efrag/blog-posts/go_test_coverage/calculator 0.002s coverage: 16.7% of statements ok github.com/efrag/blog-posts/go_test_coverage/size 0.002s coverage: 42.9% of statements Total coverage: 30.77% of statements
Final thoughts
I found the way that Go does code coverage very interesting. Had not actually contemplated before how these tools work under the hood and seeing the output of the cover tool and the generated GoCover struct overlaid in my code was the cherry on top 🤩 of this entire process.