Encoding and Evolution: How Old Code and New Code Share Data

Encoding is translation from in memory objects to a byte sequence. Decoding is translation back from bytes to objects.

A relational database assumes one schema in force at one time. Schema on read databases allow mixed old and new formats.

Code also mixes. Server code uses rolling upgrade where a few nodes update at a time. Client code waits on users who may update late.

So old code, new code, old data, and new data coexist. Safe change has numbered requirements:

1. (1)New code can read old data. This is called backward compatibility.
2. (2)Old code can read new data. This is called forward compatibility.
3. (3)Old code that rewrites new data keeps fields it does not understand.
4. (4)Data written years ago remains readable after many code deploys.

Think of in memory data as structures with pointers for fast CPU access. Think of encoded data as a self contained byte sequence for files or networks.

A pointer means nothing to another process. So you must flatten structures to bytes on write. You must rebuild structures from bytes on read.

The dictionary for that rebuilding is the schema. Text formats like JSON, XML, and CSV include field names in every record. Tag based formats replace names with small numbers. Name based Avro omits both names and tags and relies on the reader having the right dictionary.

Thrift and Protocol Buffers require a schema before encoding. Example Thrift schema:

struct Person {
  1: required string       userName,
  2: optional i64          favoriteNumber,
  3: optional list<string> interests
}

Example Protocol Buffers schema:

message Person {
    required string user_name       = 1;
    optional int64  favorite_number = 2;
    repeated string interests       = 3;
}

A code generator turns this schema into classes for Java, C++, C#, and other languages.

Encoded data is a concatenation of fields. Each field has a field tag such as 1, 2, or 3. Each field has a type annotation. Each string or list has a length prefix. Unset fields are omitted.

Thrift BinaryProtocol writes type plus tag plus value and takes 59 bytes for Example 4-1. Thrift CompactProtocol packs type and tag delta into one byte and uses variable length integers and takes 34 bytes. Protocol Buffers uses a similar bit packing and takes 33 bytes. In all three, 1337 fits in two bytes instead of eight.

Schema resolution is simple. New tags can be added. Old code sees an unknown tag, reads the type annotation, and skips that many bytes. That preserves (2). New code still knows old tags. That preserves (1).

Failure cases:

• Add a field as required. New code reading old data fails the required check. This breaks (1).
• Remove a required field or reuse its tag. Old data with that tag becomes invalid or misread. This breaks (1) and (2).
• Change int32 to int64. New code reads old data by filling missing bits with zero. Old code reading new data may truncate a value that does not fit in 32 bits.
• Protocol Buffers allows optional to repeated evolution because repeated means the same tag appears many times. New code sees zero or one values from old data. Old code sees only the last value from new data. Thrift has a dedicated list type and does not allow that same evolution, but it supports nested lists.

Tags have a price. Tags must be assigned by hand and never reused. That is painful for dynamically generated schemas such as dumping a relational table where columns map to fields.

Avro removes tags. Avro schema in IDL:

record Person {
    string               userName;
    union { null, long } favoriteNumber = null;
    array<string>        interests;
}

Same schema in JSON form lists name and type for each field and gives favoriteNumber a default of null.

Avro encoding for Example 4-1 is 32 bytes. It is just values in schema order. A string is length plus UTF-8 bytes. An integer is variable length. Nothing marks field boundaries except the schema.

So decoding needs two schemas. The writer's schema is the version used to encode. The reader's schema is the version the reading code expects. The Avro library compares them side by side. This is called schema resolution. It matches fields by name, not by order. It ignores writer only fields. It fills reader only fields from defaults in the reader schema.

Rules to keep (1) and (2):

• You may only add or remove a field that has a default value.
• To allow null you must use a union type such as union {null, long}. Null alone is not a default unless null is a branch of the union.
• Changing a field name needs an alias. That change stays backward compatible but not forward compatible.
• Adding a union branch stays backward compatible but not forward compatible.

Price of the fix: the reader must learn the writer schema. Avro answers by context. Large files with many records store the writer schema once at the start in object container files. Databases with individually written records store a version number per record plus a registry of schemas, as in Espresso. Network connections negotiate the version once at setup, as in Avro RPC.

This runnable Python mirrors Figure 4-7. Old code knows only three fields. New data adds photoURL.

import json
import copy

# Data written by new code. Includes photoURL.
db_value = {
    "userName": "Martin",
    "favoriteNumber": 1337,
    "interests": ["hacking"],
    "photoURL": "http://example.com/martin.jpg"
}

# BAD: decode into narrow model object, then re-encode. Loses unknown field.
class PersonOld:
    def __init__(self, d):
        self.userName = d["userName"]
        self.favoriteNumber = d["favoriteNumber"]
        self.interests = d["interests"]
    def toJSON(self):
        return json.dumps({
            "userName": self.userName,
            "favoriteNumber": self.favoriteNumber,
            "interests": self.interests
        })

old = PersonOld(copy.deepcopy(db_value))
old.favoriteNumber = 42
print("BAD:", old.toJSON())  # photoURL is gone. Breaks (3).

# GOOD: keep unknown fields opaque and carry them forward.
def update_keep_unknown(record, new_fav):
    out = copy.deepcopy(record)  # keep all keys, even unknown ones
    out["favoriteNumber"] = new_fav
    return json.dumps(out)

print("GOOD:", update_keep_unknown(db_value, 42))  # photoURL survives.

Notes on the lines that matter:

• PersonOld.__init__ selects only known keys. That models old generated code.
• toJSON re-encodes only known keys. That is where (3) is lost.
• copy.deepcopy in the good path preserves unknown keys without interpreting them. That is what Thrift, Protocol Buffers, and Avro do at the format level when they skip and retain unknown fields.

• Language builtins such as Java Serializable, Ruby Marshal, Python pickle, and Kryo for Java: convenient for transient data. Avoid for durable data because they tie you to one language, allow instantiation of arbitrary classes on decode which creates security risk, treat versioning as an afterthought, and are often bloated and slow.
• JSON, XML, CSV for organization to organization interchange: good enough when agreement matters more than size. Watch numbers. XML and CSV cannot tell numbers from digit strings without external schema. JSON cannot tell integers from floats and loses precision past 2 to the power 53, which is why Twitter sends tweet IDs both as number and decimal string. Watch binary data. JSON and XML lack binary strings so binary is wrapped as Base64 text. Watch schemas. XML Schema and JSON Schema exist but many JSON tools skip them, so validation code moves into the app. CSV has no schema at all and vague escaping for commas and newlines.
• Binary JSON variants such as MessagePack, BSON, BJSON, UBJSON, BISON, Smile, and XML variants WBXML and Fast infoset: slightly smaller and faster but still include field names, so Example 4-1 only shrinks from 81 to 66 bytes in MessagePack. Useful in niches, not a big win.
• Thrift CompactProtocol and Protocol Buffers for internal services with stable schemas and code generation: omit field names, use tags 1, 2, 3, and reach 34 and 33 bytes. Good for statically typed languages like Java, C++, and C# where generated classes give type checking and autocomplete.
• Avro for Hadoop files, Espresso storage, and Pig analysis: reaches 32 bytes with no tags. Good for dynamically generated schemas from database tables and for dynamic languages like JavaScript, Ruby, and Python where code generation is unwanted. Files are self describing because they embed the writer schema.
• Archival and analytics copies: dump with the latest schema into Avro object container files, then optionally into column oriented Parquet for analytics. Source databases may hold mixed schema versions on disk but appear as one schema through evolution rules.
• Network and drivers: relational protocols with ODBC or JDBC drivers, SSL certificates with ASN.1 DER and X.509 tags, and web APIs over HTTP with URLs plus HTML for browsers or JSON for native apps and Ajax XMLHttpRequest clients. All need (1) and (2) because clients and servers update independently.