RFC: JSONx Framework

Hi Jan, thanks for your message and your interest. I would be happy to answer your questions:

In 2015, I was working on a project that required parsing complex JSON documents. There was a lot of complexity in the documents, and a lot of variability. One approach was to use lightweight parsing techniques, and another was to use a binding framework.

Lightweight parsing techniques are not great for complex JSON documents with high variability. This is because all errors (of assumptions as to what a particular property or object or whatever) end up being pushed to the runtime. That is, if there is an error with the logic, I would only catch that error in the runtime. Due to the large number and size of the documents, I needed a better solution, because otherwise there would be a significant risk that the implementation of my application would begin to accumulate yet an unknown extent of complexity due to the nature of these documents that I was yet to uncover.

A binding solution, however, would provide me a way to create a "structural model", of sorts, that would insulate the consumer logic of the JSON data from this document complexity I was yet to uncover. Using Jackson, I was able to achieve my goal.

After my experience with Jackson during the project in 2015, I started to ponder what an ideal JSON binding solution would be. The first thought that came to mind is regarding XML. XML has the power of validation (via an XML Schema). XML validation allowed me to cleanly isolate the error checking of an XML document from the application logic, as XML validation is provided by validating XML parsers. Such an approach let me never have to worry about catching all the various kinds of content errors there could be in an XML document, because the validator would do that for me.

I imagined an analogous solution for JSON.

You mention OpenAPI, which is a solution that is designed for RESTful interfaces. OpenAPI provides a certain amount of "contract scope definition", but it's specific to RESTful interfaces. If you compare OpenAPI to the XML Schema Document, it's like comparing apples and cucumbers.

The solution I imagined was a JSON Schema Document that expressed most of the same structural and logical rules and constraints as are present in the XML Schema Document.

I started working toward this goal on a project I called: JJB. Using the JJB framework, I was able to create strongly-typed bindings to JSON Schemas that allowed me to define strict constraints unto the content. JJB enabled me to implement client/server application stacks very quickly and efficiently.

I almost stopped at JJB, but it had one fault: validation of arrays. In JSON, an array can be of mixed types. That is, it can be an array of booleans, such as: [true, false, true], or an array of booleans and strings, such as: [true, "hello", false, null]. With the JSON Schema model in JJB, I was not able to express a definition for members of an array. And therefore, I would not be able to validate contents of arrays.

In 2017, I decided to give this array problem a try. And to cut to the chase, the result is the JSONx project.

How is JSONx different from Jackson?

With Jackson, you define your "structural model" in code via the ObjectMapper. This places the complexity of Java<-->JSON binding rules into your application space. For small applications dealing with small JSON object, this is no problem. But for large applications with complex JSON structures, the ObjectMapper approach ends up requiring a lot of focus of your application to be put onto Jackson-specific complexity.

Also, the same logic applies to validation with Jackson. All of the validation spec must be inside your application space. For small projects, this is fine, but for large ones, such an approach would push all the errors to the surface of the application stack -- i.e. the errors are caught amidst the application logic. I would rather first validate the JSON document in its own layer, and then, if valid, engage the application logic. If not valid, then it's a problem the JSON producer has to deal with.

Jackson also provides a way to define a JSON schema, but the schema specification is, in my opinion, cumbersome and difficult to work with. With JSONx, the JSON Schema is defined in a XSD, which you declare in a XML file. Doing it this way gives you the power of edit-time validation of your JSON Schema (with a tool such as oXygen XML).

JSONx is implemented as an analogous solution to XSD. The JSON Schema Document in the JSONx Framework supports all content constraint rules that are available in the XSD spec (the XSD spec has a lot of logical processing also, which is not implemented in JSONx). The JSON Schema Document in the JSONx Framework allows definition of array constraints, including min/max-Occurs rules and "iteration". If you're interested, here's the doc snippet regarding this.

Where did the motivation for this project come from?

I love solving very complex problems. This project actually seemed a lot easier when I started it. By the time I was in the thick of it, the drive forward was purely from the thinking: "I cannot quit half-way".

Regarding testing:

The JSONx project is thoroughly tested, both via assertion libs and spec. This code here is a test suite that I built for JSONx. It models the rules for checking success and error cases during encoding and decoding of JSON for all individual facets of constraints provided by the JSON Schema Document and the JSONx Binding API. When the test suite runs, it creates random JSON documents that models the full entropy of variability as per the individual facets. The test suite is designed to create all success and all error cases possible, and to assert that the success cases pass, and the error cases produce the expected exceptions.

There is also a test suite specific to building the Java model of the JSD (JSON Schema Document). The model I'm referring to here is the model that provides JSON Schema<-->Java Binding generation and reverse generation. The SchemaTest does every forward and reverse parsing and marshaling from the XML form of the schema to the JSON form of the schema, and to and from Java source code as well.

I've already talked your ear off. If you're interested in the testing specifically, the best thing to do is to clone the repo and check it out yourself.

Thanks again for your interest!

P.S. I'm in the middle of writing an article that will talk about this project. Hoping it will be in the hands of the editors next week. If you're interested, I can forward you a link once it's published.