How does a non-null property wind up null in Kotlin? Let’s find out!

Review: How Kotlin Handles Nullable References

Kotlin’s nullable reference handling is great. It distinguishes String?, which is either null or a String, from String, which is always some String.
If you have:

data class Invitation(val placeName: String)

then you can trust that the property getter for placeName will never return null whenever you’re working with an Invitation.

That class declares, “The placeName property is a String that can’t be null.”

Need more background? Mark Allison will walk you through a concrete example in The Frontier screencast “Kotlin Nullability”.

Kotlin works hard to ensure this:

• At compile-time: During compilation, an obvious case where a null could wind up in a variable with non-null type triggers an error.

  If you try passing a null directly, the compiler will flag it as an error. The code:

  Invitation(null)
  

  yields the compiler error:

  error: null can not be a value of a non-null type String
  Invitation(null)
             ^
  

• At run-time: Kotlin also guards against null in the property setter. This lets it catch less obvious cases, like ones caused by Java not distinguishing null from not-null.
  So, if you try sneaking a null in by laundering it through the Java interop:

  val mostLikelyNull = System.getenv("not actually an environment variable")
  Invitation(mostLikelyNull)
  

  Your sneaky code will compile fine, but when run, it triggers an exception:

  java.lang.IllegalStateException: mostLikelyNull must not be null

Kotlin’s promise: No more defensive null-checks. No more lurking null pointer exceptions. It’s beautiful.
You try to write a null into a non-null property, Kotlin will shoot you down.

And Yet, a Wild Null Value Appears

That’s the theory. But I ran into a case where, all that aside, Kotlin’s “not null” guarantee wound up being violated in practice.

I’ve got a Room entity like so:

@Entity(tableName = "invitation")
data class Invitation(
  @SerializedName("name")
  @ColumnInfo(name = "device_name")
  val placeName: String
)

Room sees that placeName is a not-null String and not a maybe-null String?, and it generates a schema where the device_name column has a NOT NULL constraint.

But somehow, I wound up with a runtime exception where that constraint was violated:

android.database.sqlite.SQLiteConstraintException: NOT NULL constraint failed: invitation.device_name (code 1299)

My app asked Room to save an Invitation with a null placeName. Somehow, it got around all of Kotlin’s defenses!

It got worse: The exception left the database locked. The UI stopped updating. Database queries started piling up. Logcat showed reams of messages like:

W/SQLiteConnectionPool: The connection pool for database '/data/user/0/some.app.id.here/databases/database' has been unable to grant a connection to thread 20598 (RxCachedThreadScheduler-27) with flags 0x1 for 120.01101 seconds.
    Connections: 1 active, 0 idle, 0 available.

    Requests in progress:
      executeForCursorWindow started 127006ms ago - running, sql="SELECT * FROM invitation"

That request had started over two minutes ago!

In the end, Android had mercy, and put the poor app out of its misery:

    --------- beginning of crash
E/AndroidRuntime: FATAL EXCEPTION: pool-2-thread-2
    android.database.sqlite.SQLiteDatabaseLockedException: database is locked (code 5): retrycount exceeded

The exception that shouldn’t have been possible had left the database locked, and ultimately, the app had crashed.

How does a not-null property wind up null?

This data was read in from an API call. So the bogus data probably came from there. And, indeed, the corresponding field proved to be missing from the API response.

But why did this not error out at that point? How did an Invitation ever get created with a null placeName property in the first place? Kotlin told us that would be impossible, but exception logging doesn’t lie.

Where did things go wrong?

• Unchecked platform types?
• Retrofit2?
• Room?

Nope, it was Retrofit2’s little helper: Gson.

Gson treats “missing” as “null”

There’s one more actor in this drama: Gson.

Gson makes slurping in JSON as objects painless.

Gson’s “Finer Points with Objects” says:

This implementation handles nulls correctly.

• While serializing, a null field is omitted from the output.
• While deserializing, a missing entry in JSON results in setting the corresponding field in the object to its default value: null for object types, zero for numeric types, and false for booleans.

If you slurp in {}, Gson will apparently poke a null value into your not-null field. How?

What about when it’s not missing?

Well, let’s step back and ask: How did this work in the first place, when we didn’t encounter bogus data? Gson’s docs on “Writing an Instance Creator” say:

While deserializing an Object, Gson needs to create a default instance of the class. Well-behaved classes that are meant for serialization and deserialization should have a no-argument constructor.

• Doesn’t matter whether public or private

How does Gson handle poorly-behaved classes?

But this data class doesn’t have a no-args constructor. It’s not “well-behaved.” And yet, it was working fine up till now.

Gson expects either a no-args constructor (which our data class won’t provide) or a registered deserializer. This scenario has neither. How did this ever work in the first place? What’s handling the deserialization for us?

ReflectiveTypeAdapterFactory

Nosing around in the debugger shows GSON winds up using a ReflectiveTypeAdapterFactory, which relies on its ConstructorConstructor.

UnsafeAllocator

The factory ultimately falls back on sneaky, evil, unsafe allocation mechanisms rather than telling the developer to fix their code:

    // finally try unsafe
    return newUnsafeAllocator(type, rawType);

And UnsafeAllocator is documented to “[d]o sneaky things to allocate objects without invoking their constructors.” It has strategies to exploit Sun Java and the Dalvik VM pre- and post-Gingerbread. These let it build an object without providing any constructor args. On post-Gingerbread Android, it boils down to calling the (private, undocumented) method ObjectInputStream.newInstance().

Do a bad thing, then make it right

There’s the answer: Gson handles classes that are poorly behaved with regard to deserialization by doing a bad, bad thing. It sneaks behind their back and creates them using what amounts to a backdoor no-args constructor. All their fields start out as null.

Then, if it’s reading valid JSON, Gson makes it right: All the fields that need populating get populated. When it all works, no-one’s the wiser. And when it didn’t work in Java before widespread reliance on nullability annotation, it was probably still fine – null inhabits all types, and it’s not too terribly surprising when another one sneaks in.

For a Kotlin programmer, this is bad news.
Kotlin doesn’t check for nulls on read, only on write. Gson sneaking around the expected ways of building your object can leave a bomb waiting to go off in your codebase: An impossible scenario – a property declared as never null winding up null – happens, and the language ergonomics push back on trying to address that.

Working Around the Problem

To work around this, you write code that looks unnecessary: you null-check a property declared as never null. The compiler warns that the is-null branch will never be taken. You’re going to probably really want to listen to that warning, but if you do, you reintroduce a crasher. Paper that over with some comments, and maybe reduce the urge to “fix” it by tossing on a @Suppress("SENSELESS_COMPARISON").

The compiler warns, “Condition ‘invitation.placeName != null’ is always ‘true’”:

But luckily, it doesn’t optimize the branch away, because…

My debugger shows it ain’t. Thanks, Gson + under-specced backend!

Fixing the Problem

The fix is to make sure any classes you hand to Gson for deserialization either have a no-args constructor or have all their fields marked nullable. Don’t trust data from outside your app!

Use separate Entity classes with Room. Sanity-check your data after parsing, and handle when insanity comes knocking at the door with grace.

What about Moshi?

Would trading out Gson for Moshi have avoided this issue?

It turns out, it wouldn’t. But Moshi’s docs both call out the issue and suggest coping strategies. You’ll find this warning and advice in the README section “Default Values & Constructors”:

If the class doesn’t have a no-arguments constructor, Moshi can’t assign the field’s default value, even if it’s specified in the field declaration. Instead, the field’s default is always 0 for numbers, false for booleans, and null for references. […]

This is surprising and is a potential source of bugs! For this reason consider defining a no-arguments constructor in classes that you use with Moshi, using @SuppressWarnings(“unused”) to prevent it from being inadvertently deleted later […]. (emphasis added)