Compiling your specification from source

Richard D Jones

·

Follow

Published in

ITNEXT

·

8 min read

·

Jul 17, 2018

--

Listen

Share

Specifications, and especially Open Standards rather than one-offs, have to be highly specific and detailed. They must also be clearly laid out so that implementers have a chance of getting it right.

As a result of the detail required, specifications contain a lot of minutae which apply in some contexts and not others, and in which information is repeated for clarity and context throughout.

This in turn creates a maintenance headache for the owner of the spec. As ideas and requirements evolve (as they should, through constant community feedback), updating the documentation while keeping internal consistency becomes extraordinarily challenging. The larger the spec, the larger the problem, and some specifications are very very large and require a lot of supporting information.

I talk about this abstractly, but I want to ground what follows in a concrete use case: the SWORD 3.0 specification. SWORD 3.0 is a protocol enabling clients and servers to communicate around complex digital objects, especially with regard to supporting the deposit of these objects into a service like a digital repository. I have been technical lead for the 2.0 and 3.0 versions of this specification, so have witnessed first-hand the challenges of assembling the documentation.

In this post I wanted to share some experiences and approaches that we took to maximise the accuracy of the SWORD 3.0 spec at the same time as minimising the maintenance work.

Normalisation

The single most important thing you can do is normalise the information in your specification to the highest degree possible. This minimises repetition in your source materials, which maximises the chance for your specification to be accurate.

But hold on, didn’t we just say that in order to help implementers, repetition in context is required?

Absolutely, and once we’ve normalised the context we’re going to introduce mechanisms to produce the final documentation with all the repetition that it needs, and without the management overhead.

Put specialist knowledge in specialist formats

Specs are generally written in prose (albeit a highly technical style), and yet implementations are written in code. This creates a huge divide for implementers. We can cross this divide a little by writing our specification in code wherever possible.

In this case I mean you should use existing standards to document the specification wherever possible. In our specific case for SWORD 3.0, we used the following:

1. OpenAPI for the REST API that the specification defined.
2. JSON Schema for the structure of all the JSON documents defined by the specification

By doing this we can short-cut the implementation process where these documents are likely needed anyway, and made those artefacts the primary source of information for the spec.

Make re-usable examples

Instead of writing in-line usage examples, you should produce re-usable examples that can be shown to the reader in appropriate contexts. For example, if you have defined JSON formats, have full concrete examples of them stored as actual .json files in your specification source tree.

Tabulate content

A substantial portion of specification content is in the business of defining, explaining or otherwise expanding on particular terms or combinations of terms. All of these are suitable for tabulation, by which I mean put it in a spreadsheet. I’d recommend storing your sheets as CSVs in your source tree, and then they’ll be machine readable.

For example, if you have a set of URLs that you use throughout the documentation, define them in a table like this

| URL          | Definition                                      |
| ------------ | ----------------------------------------------- |
| Service-URL  | The root URL for the service                    |
| Resource-URL | The URL to be used to access a defined resource |

The advantage of doing this in a machine-readable way is that when we compile our specification as HTML (that’s where we’re heading) we can always link back to these definitions, or surface them in-context as we discuss another aspect of the specification, all without ever having to actually repeat the content itself.

You can also do highly sophisticated things with this tabularisation approach, and you can read my post about how we did this for SWORD 3.0 for some ideas.

Define your outputs

In addition to normalising the content, we also need to decide what outputs we want to produce. For a specification this might include the following

• The Specification: this is going to be your formal, no frills, pure spec definition. It won’t be fully normalised, there will undoubtedly be some repetition, and we’ll be looking to keep that to a minimum.
• Appendices and Supporting Information: This might include prior work such as requirements lists or use cases, or background reading, or deep-dives into the consequences of areas of the spec.
• Implementation Guide: Large specifications can be quite dry and leave implementers with little idea where to start. Additional documentation like implementation guides, tutorials or worked examples can improve the chances of getting quality implementations.
• Presentations and Marketing Collateral: Believe it or not, successful specifications (especially Open ones) require a substantial amount of marketing. Presentations, blog posts, information sheets, etc.

It’ll be immediately clear to you that the same kinds of content is going to appear over and over in different contexts throughout.

For example, if you have defined a set of rules for interacting with a set of API endpoints, and you have abstracted these rules in the main specification, it may not be immediately clear to an implementer which rules apply to a specific endpoint. In that case you can re-use the same content from the main specification in a separate document, to be absolutely explicit about how to implement each endpoint. You can see a detailed discussion of this approach in my other post, and here’s a quick example:

Suppose we have the following (simple, contrived) rules:

1. All endpoints MUST authenticate
2. Clients sending body content MUST provide the Content-Length header
3. Clients sending JSON MUST provide the header X-JSON-Type (a header we just invented for argument’s sake)
4. Clients sending Binary content MUST provide the Content-Type header

Then in my supporting information I can provide a section which reads:

Sending Binary content to the Resource-URL

1. MUST authenticate

2. Must provide the Content-Length header

3. Must provide theContent-Type header

It’ll be important that when I change a rule in the main specification that the supporting documentation gets updated too. This is a strong motivation for us to normalise our spec content, and put it into machine-readable formats, as otherwise we have to do that manually.

Define a compilation process

Now we know what documents we want, and we have all our base information in normalised data sources, it’s time to start compiling those into the final documentation.

Here I’ll describe our compiler for SWORD 3.0 in brief. Our compiler is custom built for this purpose, though it shares a lot of similarities with a static site generator and you may be able to use something like Jekyl or Hugo for your purposes. I’ll describe our compiler in detail in a later post.

Each document, such as the main spec, or the implementer guide, has its own text file which contains all the assembly information. Ours are written in Markdown, which gets rid of the need to write HTML by hand. They also contain custom tags that tell our compiler what to do. Each of these documents is fed into the compiler, which ultimately outputs HTML.

Here are the basic steps it takes:

1. Read in the master file for the output document. For the most part, our master files are just a list of sections to be included in a suitable order
2. Import the include files. The includes are also Markdown files which contain the text of the specification, and functions which tell the compiler what documentation to generate from the normalised source data.
3. Index the document. This means that we look for all headers and prefix them with a suitable section number (like 2.4.17), and store all the information for generating a Table of Contents later.
4. Run all the functions in the document. This will generate all the text from the normalised sources. This include operations like: describe the API from the OpenAPI specification, or include tables of definitions, or create links between sections.
5. Render the file to HTML. Convert the resulting file from Markdown to HTML, enabling some useful extensions like those covering tables and code blocks (which are not part of the base Markdown language).

To give you a more concrete picture, here’s a snippet from the main specification file:

Last modified: {% date now,%Y-%m-%d %}
{% include sections/INTRODUCTION.md %}
{% include sections/TERMINOLOGY.md %}
{% include sections/STRUCTURE.md %}
{% include sections/HTTP_HEADERS.md %}
{% include sections/PROTOCOL_OPERATIONS.md %}
{% include sections/PROTOCOL_REQUIREMENTS.md %}

It basically says: print a timestamp for the build, and then include the following sections in order.

If you go further and dig through those files you’ll find functions like the following:

{% json_extract
    source=examples/service-document.json,
    keys=byReferenceDeposit
%}

or:

{% http_exchange
    source=schemas/openapi.json,
    method=get,
    url=/Service-URL,
    response=200
%}

These tell the compiler to substitute in some information from our normalised source data.

For example the json_extract function loads the specified JSON file, and prints one or more keys from it in a quoted block. You might use this to explain how a specific feature is represented in a document, for example. The above usage of the function is used to describe how “byReferenceDeposits” are announced in the service description, and gives us:

{
    "byReferenceDeposit": true
}

The advantage here is that if we choose to change the value of that field in the example file, the value in the extract also gets updated.

Or consider the http_exchange — this provides us with a view on the OpenAPI specification that is of a form commonly seen in specifications; it details the on-the-wire interactions between a client and server over HTTP. The above function call would output something like this:

GET /Service-URL HTTP/1.1 
Authorization: ... 
On-Behalf-Of: ...   
HTTP/1.1 200 
Content-Type: application/json  [Service Document]

Again, this is taking advantage of the normalised data, and if we chose to add HTTP headers, the body content or any of the other aspects of this exchange to the OpenAPI definition, these would be reflected in our example.

Examples from SWORDv3

To see some concrete examples, you can take a look at our SWORD specification git repository (if you manage your spec like this, it’s also easy to version control):

• A directory of normalised source data using open standards
• A directory of tabulated source data
• A directory of examples of document types from the specification
• The main specification source and resulting document
• An implementers’ supporting guide detailing the requirements for every possible protocol operation: source and resulting document
• An overview presentation built from the same normalised source materials as the main specification: source and resulting presentation

Over the years I’ve been involved in a number of specification efforts. Not only SWORDs 1 through 3, also OAI-ORE, and ResourceSync (which passed through formal NISO procedures). Not to mention many client-specific API definitions. In all of them, a large amount of documentation needed to be managed, and in SWORD 3.0 we had an opportunity to explore a way to make this easier.

I hope that this approach can help you with that kind of work, and you’ll find it relevant not only if you work on Open Standards, but in all aspects of technical specification.

In time we may be able to formally release the software that assembles the specification. Right now it’s a prototype written specifically for SWORD 3.0, though you are welcome to take a look.

Richard is Founder and Senior Partner at Cottage Labs, a software development consultancy specialising in all aspects of the data lifecycle. He’s occasionally on Twitter at @richard_d_jones