C++ Compilation: Lies, Damned Lies, and Statistics

no, seriously, tl;dr…

How the Test Works
Optimizing the Test
Reasonable Confidence
Bias
TL;DR
Results by Compiler
Results by C++ Standard

• Visual Studio 2015
• Visual Studio 2017
• clang 7.0.1

Heuristics

In the first post, C++ Headers Are Expensive!, I posted a bunch of compilation times for includes, gave some vague hand wavy explanation of how I captured the data and pretty much left it at that. In the second post, C++ Compliation: What’s Slowing Us Down? I examined some things interfering with the compilation speed which were outside of the test. In this post I’m updating the compilation numbers, adding some more details on how the testing works, and most importantly, I’m going to show reasonable confidence in the numbers and what this does for future posts.

How the Test Works

The test iterates over a list of compilers, C++ standard versions, and headers. For each, it invokes the compiler on a source file that includes the relevant header and times how long it takes. The source code for each test is a simple function which returns an integer and takes no parameters; it only requires core logic of the language and is intentionally neither special nor complicated. When running the tests, if the header requires a higher C++ standard than is being currently tested, it is marked as “unsupported” and skipped, otherwise the test is run some number of times, storing the elapsed time for each iteration for later processing.

I exclude the time it takes to start the compiler and parse a constant block of code by compiling the unmodified code and calculate how long it takes. This is done hundreds of times and averaged together. The higher the iteration count of the baseline test, the higher accuracy the baseline time is, assuming the compilation ends up taking generally the same amount of time. The baseline is then subtracted from test times, giving us the true cost of the change.

The baseline code looks like this:

int testFunction() { return 0; }

The test code adds the code we’re going to test. In this case, we’re adding an include so we can see how much time this costs:

#include <assert.h>
int testFunction() { return 0; }

Optimizing the Test

The full test takes 10 to 24 hours to run, so any speed gains allow me to running more iterations, or a have a higher frequency of tests.

In the second post, I fixed some of the “easy” stuff which was causing variability in testing. I wanted to avoid making changes that affect the compilation itself, instead only reducing outside interference, reducing the amount of time the tests take as well as increase their repeatability.

I tried running one compilation per core but that made the results unreliable due to the compilers stomping each other in various ways.

Outside of that, I haven’t done anything, even the oft suggested “dude, ssd!”

Reasonable Confidence

There can be no meaningful change without meaningful measurement. In order to present the results, I need to have some reasonable confidence that the numbers are correct.

For my purposes, “reasonable confidence” is saying “Header soandso took N seconds” and have a way to prove that the result is statistically likely. To do this, I run the test a large number of times. The time of each iteration is plotted to a graph and then a sliding window of time is adjusted until we cover a high percentage of results.

An important note here is that heuristics like this cannot actually predict future results, instead, it can only say “of the data given, X percentage of them occurred within Y range.” Given that the test is done the same way every time on the same machine under the same conditions, we would expect that the results will be similar. So, if you’re keeping score, yes, we expect it’ll be the same, but no we can’t actually predict it… it’s all just so confusing!

Once we’ve got some data, there are three things we can do to get try to make some meaningful sense out of them:

1. Grow the window (fits a higher percentage of tests within the window, but also a widens the range of result times)
2. Shrink the window (defines a narrower range of result times, but that lowers the percent of tests within the window)
3. Gather More Data! Assuming repeatability, this should allow us to shrink the window and still have a narrow range of result times.

So the goal is to figure out how many iterations need to be run in order to achieve a desired percentage of tests falling within a certain range of time.

That number is currently between 128 and 256 iterations, but I use 128 as the test runs in “only” 10 hours. Surprisingly, warmup tests (running the test to preload hard drive/memory caches) actually has negligible influence on this. Previously, I was running the number of test iterations twice and skipping the results of the first test, but it turns out, this was a huge waste of time. Oh well!

Bias

I try to avoid injecting my opinion, desires, or hopeful results into the tests. It’s easy to tweak statistics to give desired results versus reality.

None of that is good, so to avoid it, I am providing as much information as I have that a) is required to have reasonable confidence in my results, and b) I think people will reasonably care about. I’ll provide high level results and confidence(s) I have in that range of data, and I’ll sometimes (but not all of the time) provide raw results. To be honest, though, maybe 1% of the folks that’ll ever read this will want or actually do anything with that data. If I don’t provide it (or at least proof that it’s accurate) I may as well just type in my own numbers into the table.

That being said, if you _do_ want that full data for future posts, let me know (twitter: @virtuallyrandom) and I’ll make a reasonable attempt to get it to you.

TL;DR

These tests covered Microsoft Visual Studio 2015, Microsoft Visual Studio 2017, and clang 7.0.1 for C++ standards C++20, C++17, C++14, C++11, C++03, and C++98. Baseline time was calculated by compiling basic source code 128 times. Result time was calculated by compiling adjusted source code (include + basic source code) 128 times.

For the sake of brevity, compilers are annoted as VS15 for Visual Studio 2015’s cl.exe, VS17 for Visual Studio 2017’s cl.exe, or clang for, well, clang.

Due to the amount of content, I’m posting several batches of data. The first batch is a table which contains the results for each header in each compiler under only the C++14 standard. The second batch is a table for each compiler for each header under each C++ standard. The third batch is hueristic data that gives the confidence time range and occupancy for each compiler, each header, and each standard. Finally, there are zips of all of this data (in tabe delimited text files) so folks can peruse and validate the data themselves.

Results by Compiler

This is a table of compilation times, in seconds, for each compiler using the C++14 standard. That standard was chosen as it supported by all three compilers tested.

Header	VS15	VS17	clang
algorithm	0.169	0.191	0.316
any	unsupported	unsupported	unsupported
array	0.284	0.353	0.506
assert.h	0.009	0.008	0.014
atomic	0.326	0.405	0.659
bit	unsupported	unsupported	unsupported
bitset	0.329	0.415	0.527
cassert	0.01	0.009	0.017
cctype	0.012	0.011	0.023
cerrno	0.011	0.009	0.018
cfenv	0.012	0.011	0.022
cfloat	0.011	0.009	0.019
charconv	0.341	0.432	0.542
chrono	0.144	0.195	0.306
cinttypes	0.014	0.011	0.021
climits	0.01	0.009	0.018
clocale	0.011	0.009	0.019
cmath	0.036	0.045	0.084
codecvt	unsupported	unsupported	unsupported
compare	unsupported	unsupported	unsupported
complex	0.344	0.44	0.558
concepts	unsupported	unsupported	unsupported
condition_variable	0.286	0.372	0.51
contract	unsupported	unsupported	unsupported
csetjmp	0.011	0.01	0.018
csignal	0.01	0.009	0.017
cstdarg	0.01	0.01	0.018
cstddef	0.01	0.013	0.021
cstdint	0.011	0.009	0.018
cstdio	0.043	0.043	0.073
cstdlib	0.01	0.027	0.058
cstring	0.018	0.019	0.038
ctime	0.013	0.013	0.025
ctype.h	0.013	0.01	0.02
cuchar	0.011	0.01	0.018
cwchar	0.04	0.045	0.082
cwctype	0.014	0.01	0.02
deque	0.183	0.216	0.339
errno.h	0.009	0.008	0.015
exception	0.048	0.068	0.103
execution	unsupported	unsupported	unsupported
fenv.h	0.01	0.009	0.018
filesystem	unsupported	unsupported	unsupported
float.h	0.009	0.008	0.016
forward_list	0.183	0.214	0.338
fstream	0.271	0.331	0.476
functional	0.187	0.222	0.361
future	1.059	1.317	1.474
initializer_list	0.011	0.013	0.022
inttypes.h	0.01	0.009	0.016
iomanip	0.272	0.335	0.482
ios	0.259	0.318	0.456
iosfwd	0.08	0.094	0.172
iostream	0.264	0.325	0.464
istream	0.264	0.324	0.463
iterator	0.265	0.327	0.468
limits	0.079	0.089	0.163
limits.h	0.004	0.006	0.012
list	0.183	0.214	0.338
locale	0.337	0.427	0.545
locale.h	0.01	0.01	0.016
map	0.194	0.231	0.363
math.h	0.015	0.016	0.03
memory	0.173	0.2	0.324
memory_resource	unsupported	unsupported	unsupported
mutex	0.28	0.364	0.498
new	0.052	0.067	0.106
numeric	0.109	0.144	0.236
optional	unsupported	unsupported	unsupported
ostream	0.261	0.321	0.459
queue	0.198	0.235	0.374
random	0.305	0.394	0.553
ranges	unsupported	unsupported	unsupported
ratio	0.046	0.062	0.103
regex	0.392	0.505	0.634
scoped_allocator	0.171	0.198	0.323
set	0.185	0.217	0.341
setjmp.h	0.004	0.006	0.013
shared_mutex	0.289	0.379	0.508
signal.h	0.012	0.008	0.015
span	unsupported	unsupported	unsupported
sstream	0.329	0.416	0.528
stack	0.186	0.216	0.341
stdarg.h	0.004	0.007	0.006
stddef.h	0.009	0.008	0.007
stdexcept	0.177	0.208	0.33
stdint.h	0.005	0.006	0.013
stdio.h	0.037	0.042	0.069
stdlib.h	0.017	0.018	0.039
streambuf	0.254	0.311	0.444
string	0.327	0.413	0.523
string_view	unsupported	unsupported	unsupported
string.h	0.016	0.017	0.034
strstream	0.285	0.358	0.492
syncstream	unsupported	unsupported	unsupported
system_error	0.208	0.246	0.365
thread	0.227	0.289	0.448
time.h	0.011	0.012	0.022
tuple	0.123	0.163	0.263
type_traits	0.043	0.06	0.096
typeindex	0.052	0.069	0.109
typeinfo	0.051	0.068	0.107
uchar.h	0.009	0.009	0.015
unordered_map	0.204	0.245	0.384
unordered_set	0.194	0.231	0.365
utility	0.098	0.127	0.212
valarray	0.174	0.198	0.319
variant	unsupported	unsupported	unsupported
vector	0.185	0.217	0.344
version	unsupported	unsupported	unsupported
wchar.h	0.038	0.043	0.078
wctype.h	0.01	0.008	0.017
windows.h	0.423	0.517	1.038

Results by C++ Standard

Microsoft Visual Studio 2015

Header	c++14	c++11	c++03	c++98
algorithm	0.169	0.17	0.17	unsupported
any	unsupported	unsupported	unsupported	unsupported
array	0.284	0.284	unsupported	unsupported
assert.h	0.009	0.01	0.01	0.01
atomic	0.326	0.327	unsupported	unsupported
bit	unsupported	unsupported	unsupported	unsupported
bitset	0.329	0.329	0.329	unsupported
cassert	0.01	0.011	0.011	unsupported
cctype	0.012	0.013	0.013	unsupported
cerrno	0.011	0.011	0.011	unsupported
cfenv	0.012	0.012	unsupported	unsupported
cfloat	0.011	0.016	0.012	unsupported
charconv	unsupported	unsupported	unsupported	unsupported
chrono	0.144	0.145	unsupported	unsupported
cinttypes	0.014	0.012	unsupported	unsupported
climits	0.01	0.011	0.011	unsupported
clocale	0.011	0.011	0.011	unsupported
cmath	0.036	0.037	0.037	unsupported
codecvt	0.341	0.342	unsupported	unsupported
compare	unsupported	unsupported	unsupported	unsupported
complex	0.344	0.346	0.345	unsupported
concepts	unsupported	unsupported	unsupported	unsupported
condition_variable	0.286	0.285	unsupported	unsupported
contract	unsupported	unsupported	unsupported	unsupported
csetjmp	0.011	0.011	0.011	unsupported
csignal	0.01	0.011	0.011	unsupported
cstdarg	0.01	0.012	0.011	unsupported
cstddef	0.01	0.011	0.011	unsupported
cstdint	0.011	0.011	unsupported	unsupported
cstdio	0.043	0.04	0.039	unsupported
cstdlib	0.01	0.02	0.02	unsupported
cstring	0.018	0.018	0.019	unsupported
ctime	0.013	0.014	0.014	unsupported
ctype.h	0.013	0.011	0.011	0.011
cuchar	0.011	0.011	unsupported	unsupported
cwchar	0.04	0.041	0.041	unsupported
cwctype	0.014	0.012	0.014	unsupported
deque	0.183	0.184	0.184	unsupported
errno.h	0.009	0.012	0.01	0.01
exception	0.048	0.049	0.049	unsupported
execution	unsupported	unsupported	unsupported	unsupported
fenv.h	0.01	0.012	0.011	0.011
filesystem	unsupported	unsupported	unsupported	unsupported
float.h	0.009	0.01	0.01	0.01
forward_list	0.183	0.184	unsupported	unsupported
fstream	0.271	0.268	0.269	unsupported
functional	0.187	0.187	0.188	unsupported
future	1.059	1.058	unsupported	unsupported
initializer_list	0.011	0.014	unsupported	unsupported
inttypes.h	0.01	0.01	0.011	0.011
iomanip	0.272	0.273	0.272	unsupported
ios	0.259	0.26	0.259	unsupported
iosfwd	0.08	0.08	0.08	unsupported
iostream	0.264	0.264	0.265	unsupported
istream	0.264	0.264	0.264	unsupported
iterator	0.265	0.266	0.266	unsupported
limits	0.079	0.08	0.08	unsupported
limits.h	0.004	0.005	0.005	0.005
list	0.183	0.184	0.184	unsupported
locale	0.337	0.338	0.339	unsupported
locale.h	0.01	0.01	0.01	0.013
map	0.194	0.194	0.195	unsupported
math.h	0.015	0.015	0.017	0.016
memory	0.173	0.173	0.173	unsupported
memory_resource	unsupported	unsupported	unsupported	unsupported
mutex	0.28	0.28	unsupported	unsupported
new	0.052	0.05	0.051	unsupported
numeric	0.109	0.109	0.11	unsupported
optional	unsupported	unsupported	unsupported	unsupported
ostream	0.261	0.261	0.264	unsupported
queue	0.198	0.198	0.2	unsupported
random	0.305	0.306	unsupported	unsupported
ranges	unsupported	unsupported	unsupported	unsupported
ratio	0.046	0.046	unsupported	unsupported
regex	0.392	0.392	unsupported	unsupported
scoped_allocator	0.171	0.172	unsupported	unsupported
set	0.185	0.186	0.186	unsupported
setjmp.h	0.004	0.005	0.006	0.005
shared_mutex	0.289	0.291	0.289	0.29
signal.h	0.012	0.01	0.01	0.01
span	unsupported	unsupported	unsupported	unsupported
sstream	0.329	0.33	0.329	unsupported
stack	0.186	0.185	0.185	unsupported
stdarg.h	0.004	0.005	0.005	0.006
stddef.h	0.009	0.01	0.01	0.01
stdexcept	0.177	0.178	0.178	unsupported
stdint.h	0.005	0.005	0.005	0.006
stdio.h	0.037	0.039	0.038	0.038
stdlib.h	0.017	0.018	0.018	0.018
streambuf	0.254	0.256	0.255	unsupported
string	0.327	0.328	0.328	unsupported
string_view	unsupported	unsupported	unsupported	unsupported
string.h	0.016	0.017	0.018	0.018
strstream	0.285	0.286	0.285	unsupported
syncstream	unsupported	unsupported	unsupported	unsupported
system_error	0.208	0.208	unsupported	unsupported
thread	0.227	0.228	unsupported	unsupported
time.h	0.011	0.012	0.013	0.013
tuple	0.123	0.123	unsupported	unsupported
type_traits	0.043	0.044	unsupported	unsupported
typeindex	0.052	0.053	unsupported	unsupported
typeinfo	0.051	0.053	0.052	unsupported
uchar.h	0.009	0.01	0.01	0.013
unordered_map	0.204	0.205	unsupported	unsupported
unordered_set	0.194	0.194	unsupported	unsupported
utility	0.098	0.098	0.098	unsupported
valarray	0.174	0.173	0.173	unsupported
variant	unsupported	unsupported	unsupported	unsupported
vector	0.185	0.187	0.186	unsupported
version	unsupported	unsupported	unsupported	unsupported
wchar.h	0.038	0.039	0.04	0.04
wctype.h	0.01	0.01	0.011	0.011
windows.h	0.423	0.42	0.421	0.421

Microsoft Visual Studio 2017

Header	c++20	c++17	c++14	c++11	c++03	c++98
algorithm	0.191	0.195	0.191	0.191	0.189	unsupported
any	error	0.196	unsupported	unsupported	unsupported	unsupported
array	0.353	0.366	0.353	0.354	unsupported	unsupported
assert.h	0.007	0.007	0.008	0.008	0.006	0.007
atomic	0.407	0.409	0.405	0.406	unsupported	unsupported
bit	error	unsupported	unsupported	unsupported	unsupported	unsupported
bitset	0.415	0.442	0.415	0.415	0.414	unsupported
cassert	0.009	0.008	0.009	0.009	0.007	unsupported
cctype	0.011	0.013	0.011	0.012	0.01	unsupported
cerrno	0.01	0.008	0.009	0.01	0.008	unsupported
cfenv	0.011	0.01	0.011	0.011	unsupported	unsupported
cfloat	0.01	0.01	0.009	0.01	0.009	unsupported
charconv	error	error	unsupported	unsupported	unsupported	unsupported
chrono	0.194	0.196	0.195	0.195	unsupported	unsupported
cinttypes	0.018	0.01	0.011	0.011	unsupported	unsupported
climits	0.009	0.008	0.009	0.009	0.009	unsupported
clocale	0.009	0.011	0.009	0.012	0.008	unsupported
cmath	0.042	0.041	0.045	0.042	0.041	unsupported
codecvt	unsupported	0.457	0.432	0.431	unsupported	unsupported
compare	error	unsupported	unsupported	unsupported	unsupported	unsupported
complex	0.44	0.466	0.44	0.44	0.439	unsupported
concepts	error	unsupported	unsupported	unsupported	unsupported	unsupported
condition_variable	0.372	0.411	0.372	0.372	unsupported	unsupported
contract	error	unsupported	unsupported	unsupported	unsupported	unsupported
csetjmp	0.01	0.011	0.01	0.01	0.009	unsupported
csignal	0.009	0.009	0.009	0.009	0.008	unsupported
cstdarg	0.009	0.009	0.01	0.009	0.008	unsupported
cstddef	0.012	0.013	0.013	0.013	0.011	unsupported
cstdint	0.013	0.008	0.009	0.009	unsupported	unsupported
cstdio	0.045	0.043	0.043	0.044	0.042	unsupported
cstdlib	0.026	0.026	0.027	0.027	0.025	unsupported
cstring	0.018	0.018	0.019	0.018	0.017	unsupported
ctime	0.014	0.012	0.013	0.013	0.011	unsupported
ctype.h	0.009	0.009	0.01	0.01	0.008	0.012
cuchar	0.009	0.01	0.01	0.011	unsupported	unsupported
cwchar	0.045	0.044	0.045	0.045	0.044	unsupported
cwctype	0.01	0.01	0.01	0.011	0.009	unsupported
deque	0.214	0.222	0.216	0.215	0.213	unsupported
errno.h	0.008	0.009	0.008	0.008	0.006	0.008
exception	0.065	0.067	0.068	0.066	0.064	unsupported
execution	error	error	unsupported	unsupported	unsupported	unsupported
fenv.h	0.009	0.009	0.009	0.01	0.009	0.009
filesystem	0.773	0.772	unsupported	unsupported	unsupported	unsupported
float.h	0.008	0.008	0.008	0.008	0.007	0.008
forward_list	0.214	0.221	0.214	0.214	unsupported	unsupported
fstream	0.331	0.344	0.331	0.339	0.329	unsupported
functional	0.222	0.285	0.222	0.222	0.22	unsupported
future	1.317	1.353	1.317	1.319	unsupported	unsupported
initializer_list	0.013	0.013	0.013	0.013	unsupported	unsupported
inttypes.h	0.008	0.008	0.009	0.012	0.007	0.008
iomanip	0.336	0.35	0.335	0.336	0.334	unsupported
ios	0.319	0.332	0.318	0.319	0.317	unsupported
iosfwd	0.094	0.094	0.094	0.094	0.092	unsupported
iostream	0.325	0.338	0.325	0.324	0.324	unsupported
istream	0.324	0.338	0.324	0.324	0.322	unsupported
iterator	0.328	0.343	0.327	0.329	0.328	unsupported
limits	0.089	0.09	0.089	0.09	0.087	unsupported
limits.h	0.008	0.005	0.006	0.007	0.004	0.006
list	0.214	0.222	0.214	0.215	0.212	unsupported
locale	0.427	0.452	0.427	0.427	0.425	unsupported
locale.h	0.008	0.007	0.01	0.008	0.006	0.008
map	0.232	0.239	0.231	0.231	0.23	unsupported
math.h	0.015	0.014	0.016	0.015	0.013	0.015
memory	0.2	0.202	0.2	0.2	0.198	unsupported
memory_resource	error	error	unsupported	unsupported	unsupported	unsupported
mutex	0.365	0.404	0.364	0.365	unsupported	unsupported
new	0.066	0.068	0.067	0.067	0.065	unsupported
numeric	0.143	0.145	0.144	0.143	0.141	unsupported
optional	error	0.196	unsupported	unsupported	unsupported	unsupported
ostream	0.322	0.335	0.321	0.323	0	unsupported
queue	0.235	0.243	0.235	0.235	0.233	unsupported
random	0.395	0.407	0.394	0.394	unsupported	unsupported
ranges	error	unsupported	unsupported	unsupported	unsupported	unsupported
ratio	0.063	0.064	0.062	0.063	unsupported	unsupported
regex	0.506	0.53	0.505	0.505	unsupported	unsupported
scoped_allocator	0.198	0.2	0.198	0.198	unsupported	unsupported
set	0.218	0.226	0.217	0.217	0.215	unsupported
setjmp.h	0.007	0.005	0.006	0.007	0.005	0.006
shared_mutex	0.379	0.418	0.379	0.379	0.377	0.379
signal.h	0.008	0.007	0.008	0.008	0.006	0.008
span	error	unsupported	unsupported	unsupported	unsupported	unsupported
sstream	0.415	0.442	0.416	0.425	0.414	unsupported
stack	0.216	0.224	0.216	0.216	0.214	unsupported
stdarg.h	0.006	0.005	0.007	0.006	0.004	0.006
stddef.h	0.007	0.008	0.008	0.008	0.006	0.008
stdexcept	0.207	0.215	0.208	0.207	0.205	unsupported
stdint.h	0.006	0.006	0.006	0.006	0.005	0.006
stdio.h	0.042	0.041	0.042	0.042	0.04	0.042
stdlib.h	0.017	0.017	0.018	0.02	0.016	0.017
streambuf	0.311	0.325	0.311	0.313	0.309	unsupported
string	0.413	0.439	0.413	0.415	0.412	unsupported
string_view	error	0.343	unsupported	unsupported	unsupported	unsupported
string.h	0.016	0.016	0.017	0.017	0.015	0.017
strstream	0.36	0.375	0.358	0.36	0.357	unsupported
syncstream	error	unsupported	unsupported	unsupported	unsupported	unsupported
system_error	0.249	0.256	0.246	0.247	unsupported	unsupported
thread	0.29	0.337	0.289	0.291	unsupported	unsupported
time.h	0.011	0.01	0.012	0.012	0.01	0.011
tuple	0.162	0.165	0.163	0.163	unsupported	unsupported
type_traits	error	0.064	0.06	0.064	unsupported	unsupported
typeindex	0.006	0.073	0.069	0.07	unsupported	unsupported
typeinfo	0.068	0.07	0.068	0.068	0.067	unsupported
uchar.h	0.008	0.008	0.009	0.008	0.006	0.008
unordered_map	0.245	0.253	0.245	0.245	unsupported	unsupported
unordered_set	0.231	0.239	0.231	0.232	unsupported	unsupported
utility	0.125	0.128	0.127	0.126	0.124	unsupported
valarray	0.198	0.2	0.198	0.198	0.197	unsupported
variant	error	0.223	unsupported	unsupported	unsupported	unsupported
vector	0.218	0.226	0.217	0.217	0.216	unsupported
version	error	unsupported	unsupported	unsupported	unsupported	unsupported
wchar.h	0.045	0.042	0.043	0.043	0.041	0.043
wctype.h	0.009	0.008	0.008	0.009	0.007	0.01
windows.h	0.516	0.516	0.517	0.521	0.519	0.519

clang

Header	c++20	c++17	c++14	c++11	c++03	c++98
algorithm	0.321	0.322	0.316	error	error	unsupported
any	0.32	0.324	unsupported	unsupported	unsupported	unsupported
array	0.527	0.527	0.506	error	unsupported	unsupported
assert.h	0.011	0.013	0.014	0.016	0.016	0.017
atomic	0.664	0.668	0.659	error	unsupported	unsupported
bit	error	unsupported	unsupported	unsupported	unsupported	unsupported
bitset	0.549	0.55	0.527	error	error	unsupported
cassert	0.015	0.019	0.017	0.018	error	unsupported
cctype	0.02	0.021	0.023	0.024	error	unsupported
cerrno	0.015	0.016	0.018	0.019	error	unsupported
cfenv	0.019	0.021	0.022	0.022	unsupported	unsupported
cfloat	0.016	0.018	0.019	0.02	error	unsupported
charconv	error	error	unsupported	unsupported	unsupported	unsupported
chrono	0.304	0.306	0.306	error	unsupported	unsupported
cinttypes	0.018	0.019	0.021	0.021	unsupported	unsupported
climits	0.015	0.016	0.018	0.019	error	unsupported
clocale	0.016	0.017	0.019	0.019	error	unsupported
cmath	0	0.082	0.084	0.087	error	unsupported
codecvt	unsupported	0.569	0.542	error	unsupported	unsupported
compare	error	unsupported	unsupported	unsupported	unsupported	unsupported
complex	error	0.582	0.558	error	error	unsupported
concepts	error	unsupported	unsupported	unsupported	unsupported	unsupported
condition_variable	0.567	0.569	0.51	error	unsupported	unsupported
contract	error	unsupported	unsupported	unsupported	unsupported	unsupported
csetjmp	0.015	0.016	0.018	0.019	error	unsupported
csignal	0.015	0.016	0.017	0.019	error	unsupported
cstdarg	0.014	0.016	0.018	0.019	error	unsupported
cstddef	0.02	0.022	0.021	0.023	error	unsupported
cstdint	0.016	0.017	0.018	0.02	unsupported	unsupported
cstdio	0.071	0.071	0.073	0.074	error	unsupported
cstdlib	0.055	0.056	0.058	0.059	error	unsupported
cstring	0.035	0.036	0.038	0.039	error	unsupported
ctime	0.022	0.023	0.025	0.025	error	unsupported
ctype.h	0.017	0.018	0.02	0.021	0.022	0.022
cuchar	0.015	0.017	0.018	0.018	unsupported	unsupported
cwchar	0.079	0.08	0.082	0.083	error	unsupported
cwctype	0.018	0.019	0.02	0.021	error	unsupported
deque	0.357	0.359	0.339	error	error	unsupported
errno.h	0.012	0.014	0.015	0.016	0.016	0.018
exception	0.106	0.108	0.103	error	error	unsupported
execution	error	error	unsupported	unsupported	unsupported	unsupported
fenv.h	0.015	0.017	0.018	0.019	0.02	0.021
filesystem	0.873	0.875	unsupported	unsupported	unsupported	unsupported
float.h	0.013	0.014	0.016	0.017	0.018	0.018
forward_list	0.355	0.357	0.338	error	unsupported	unsupported
fstream	0.495	0.497	0.476	error	error	unsupported
functional	0.449	0.45	0.361	error	error	unsupported
future	1.508	1.509	1.474	error	unsupported	unsupported
initializer_list	0.021	0.023	0.022	0.023	unsupported	unsupported
inttypes.h	0.014	0.015	0.016	0.018	0.018	0.019
iomanip	0.503	0.504	0.482	error	error	unsupported
ios	0.475	0.477	0.456	error	error	unsupported
iosfwd	0.171	0.172	0.172	error	error	unsupported
iostream	0.484	0.485	0.464	error	error	unsupported
istream	0.484	0.486	0.463	error	error	unsupported
iterator	0.489	0.491	0.468	error	error	unsupported
limits	0.162	0.164	0.163	error	error	unsupported
limits.h	0.01	0.011	0.012	0.014	0.014	0.015
list	0.355	0.357	0.338	error	error	unsupported
locale	0.568	0.57	0.545	error	error	unsupported
locale.h	0.013	0.014	0.016	0.018	0.018	0.018
map	0.381	0.382	0.363	error	error	unsupported
math.h	0.028	0.03	0.03	0.032	0.033	0.033
memory	0.329	0.331	0.324	error	error	unsupported
memory_resource	error	error	unsupported	unsupported	unsupported	unsupported
mutex	0.554	0.555	0.498	error	unsupported	unsupported
new	0.107	0.109	0.106	error	error	unsupported
numeric	error	0.244	0.236	error	error	unsupported
optional	0.325	0.327	unsupported	unsupported	unsupported	unsupported
ostream	0.48	0.48	0.459	error	error	unsupported
queue	0.393	0.395	0.374	error	error	unsupported
random	0.571	0.575	0.553	error	unsupported	unsupported
ranges	error	unsupported	unsupported	unsupported	unsupported	unsupported
ratio	0.105	0.106	0.103	error	unsupported	unsupported
regex	0.656	0.659	0.634	error	unsupported	unsupported
scoped_allocator	0.328	0.33	0.323	error	unsupported	unsupported
set	0.36	0.362	0.341	error	error	unsupported
setjmp.h	0.01	0.011	0.013	0.014	error	error
shared_mutex	0.567	0.568	0.508	error	error	error
signal.h	0.012	0.013	0.015	0.016	0.017	0.017
span	error	unsupported	unsupported	unsupported	unsupported	unsupported
sstream	0.55	0.552	0.528	error	error	unsupported
stack	0.36	0.362	0.341	error	error	unsupported
stdarg.h	0.003	0.004	0.006	0.007	0.008	0.008
stddef.h	0.004	0.005	0.007	0.008	0.008	0.009
stdexcept	0.348	0.351	0.33	error	error	unsupported
stdint.h	0.01	0.011	0.013	0.014	0.015	0.015
stdio.h	0.067	0.068	0.069	0.071	0.071	0.072
stdlib.h	0.035	0.037	0.039	0.04	0.04	0.041
streambuf	0.466	0.466	0.444	error	error	unsupported
string	0.545	0.547	0.523	error	error	unsupported
string_view	0.482	0.484	unsupported	unsupported	unsupported	unsupported
string.h	0.032	0.033	0.034	0.036	0.037	0.037
strstream	0.513	0.515	0.492	error	error	unsupported
syncstream	error	unsupported	unsupported	unsupported	unsupported	unsupported
system_error	0.384	0.386	0.365	error	unsupported	unsupported
thread	0.509	0.511	0.448	error	unsupported	unsupported
time.h	0.019	0.021	0.022	0.023	0.024	0.025
tuple	0.268	0.27	0.263	error	unsupported	unsupported
type_traits	0.098	0.1	0.096	error	unsupported	unsupported
typeindex	0.111	0.113	0.109	error	unsupported	unsupported
typeinfo	0.109	0.11	0.107	error	error	unsupported
uchar.h	0.012	0.013	0.015	0.017	error	error
unordered_map	0.402	0.404	0.384	error	unsupported	unsupported
unordered_set	0.384	0.385	0.365	error	unsupported	unsupported
utility	0.214	0.219	0.212	error	error	unsupported
valarray	error	0.326	0.319	error	error	unsupported
variant	0.376	0.377	unsupported	unsupported	unsupported	unsupported
vector	0.362	0.363	0.344	error	error	unsupported
version	error	unsupported	unsupported	unsupported	unsupported	unsupported
wchar.h	0.075	0.077	0.078	0.08	0.08	0.081
wctype.h	0.014	0.016	0.017	0.019	0.019	0.02
windows.h	1.054	1.055	1.038	1.034	error	error

Heuristics

The data here is provided to show reasonable confidence in the above numbers. I won’t be posting this type of data for everything as it’s just a ton of numbers to walk through. If you need or want this data, contact me on twitter @virtuallyrandom.

Bell curve data is divided into standard deviations. The count of hits within each are then counted to give the “Confidence” column, which is simply the percent of things in that bucket (number_of_things_in_bucket / total_number_of_things.) As the range increases, so does the confidence.

Each confidence percentage is followed by a range. As the range is increase, the confidence rating does as well. With 128 tests and a range of 5 milliseconds, the results are mostly around 90%, except for clang which has a wider variation. With 256 tests, we get to 92-94%, but the tests take twice as long.

Below is the bell curve for assert.h, VC15, C++14. They all pretty much follow this same pattern where there is a very high occurrence count at a small time interval with a few deviations outside of that.

Header	Compiler	Standard	Average	Deviation	Percent 1	Range Start 1	Range Stop 1	Percent 3	Range Start 3	Range Stop 3	Percent 5	Range Start 5	Range Stop 5
assert.h	VS15	c++14	0.008906	0.001	46.90%	0.087	0.088	86.70%	0.086	0.089	96.10%	0.085	0.09
assert.h	VS15	c++11	0.009674	0.001	36.70%	0.088	0.089	82.00%	0.087	0.09	96.90%	0.086	0.091
assert.h	VS15	c++03	0.009569	0.001	35.90%	0.088	0.089	80.50%	0.087	0.09	96.90%	0.086	0.091
assert.h	VS15	c++98	0.009864	0.001	42.20%	0.088	0.089	82.80%	0.087	0.09	96.10%	0.086	0.091
assert.h	VS17	c++20	0.007073	0.001	32.80%	0.084	0.085	80.50%	0.083	0.086	96.10%	0.082	0.087
assert.h	VS17	c++17	0.006631	0.001	32.80%	0.084	0.085	71.90%	0.083	0.086	89.80%	0.082	0.087
assert.h	VS17	c++14	0.00775	0.001	29.70%	0.084	0.085	64.80%	0.083	0.086	89.10%	0.082	0.087
assert.h	VS17	c++11	0.007733	0.001	22.70%	0.085	0.086	60.20%	0.084	0.087	85.90%	0.083	0.088
assert.h	VS17	c++03	0.005695	0.001	25.80%	0.086	0.087	64.10%	0.085	0.088	96.90%	0.084	0.089
assert.h	VS17	c++98	0.00682	0.001	24.20%	0.085	0.086	65.60%	0.084	0.087	85.90%	0.083	0.088
assert.h	clang	c++20	0.011184	0.001	29.70%	0.103	0.104	62.50%	0.102	0.105	81.30%	0.101	0.106
assert.h	clang	c++17	0.012967	0.001	25.80%	0.105	0.106	62.50%	0.104	0.107	84.40%	0.103	0.108
assert.h	clang	c++14	0.014325	0.001	22.70%	0.107	0.108	51.60%	0.106	0.109	75.80%	0.105	0.11
assert.h	clang	c++11	0.015795	0.001	23.40%	0.108	0.109	53.10%	0.107	0.11	71.10%	0.106	0.111
assert.h	clang	c++03	0.015588	0.001	21.90%	0.107	0.108	50.80%	0.106	0.109	78.90%	0.105	0.11
assert.h	clang	c++98	0.016876	0.001	21.10%	0.109	0.11	55.50%	0.108	0.111	68.80%	0.107	0.112

If you got this far, good job! In the next post, I’m going to delve into things you can do to affect compile times, hopefully for the better!

Upon request, I’ve uploaded the tab delimited text results of the tests.

• output.txt – contains the speed of the processor that ran the tests followed by the name and compilation time for each header on each compiler for each standard.
• bell.txt – contains the header, compiler, standard, average deviation, percent and range followed by column of data that contain the count of times within that range.
• time.txt – contains the header, compiler, standard, base time (in clocks) for the empty test, followed by the time per iteration of the test in clock cycles.

Edit: I found a typo in the test data which caused codecvt and filesystem headers to be marked as unsupported. I have since fixed it and uploaded an addendum to the test results.