Structured Message Content¶

Structured Message Content (SMC) is the one event format of the Sobamail platform. Every event an app handles arrives as an SMC document: a request from the app's own user interface, a mutation replayed from another replica, an email from the outside world. It is also what rides inside a Structured Message, the email form that carries structured data between mailboxes.

SMC is a very simple format that can be applied to JSON-like serialization formats with loose schemas. It slightly extends the self-description features of such encodings with a functionality similar to XML Namespaces: every document says, in a globally unique way, what it is, so that the receiving side can decide who handles it before it looks at the content.

The envelope¶

Consider the following JSON document:

{value: 1}

It's simple and compact, but we don't know what this is. The above is not a valid SMC document. Let's add the missing pieces:

{
    namespace: "https://github.com/sobamail/counter/model/v1",
    name: "IncrementRequest",
    content: {value: 1}
}

That's much better! We have now a lot more information to work with.

An SMC document is a map with three keys:

That is the whole format. A response that reports a failure additionally carries an error key (see Errors), and the documents the runtime exchanges between replicas extend the envelope further (see Replication SMC); neither concerns an app that sends or receives objects.

The object key¶

The namespace and the name are concatenated in the form {namespace}name to form the object key. For the document above that is:

{https://github.com/sobamail/counter/model/v1}IncrementRequest

The braces are part of the string. This convention is sometimes called Clark notation, and it is what the platform routes on: when an event arrives, the runtime looks the key up to find the applications that declared an interest in it through the objects map of their Mutator module. An app that never listed a key never sees the event.

Because a namespace is a URI, there is no registry. Whoever controls a URL controls the vocabulary under it. The convention is to append /model/v1 to wherever the project lives, and to bump the version segment when the shape of the content changes in a way old handlers cannot read. Two objects with the same name in different namespaces are unrelated objects.

The platform's own objects live under https://sobamail.com/module/base/v1 (messages, rows, blobs, errors) and https://sobamail.com/module/appmanager/v1 (deployment and instantiation).

Content¶

SMC fixes the envelope and leaves the content to the owner of the namespace. In practice two shapes are used:

• A map, as in the counter example above. This is the natural shape for objects defined in JavaScript, where a class instance is serialized with its property names as keys.
• A positional list, used by the runtime's own objects. For example an AppInfoGet carries content: [appId] and the AppInfo that answers it carries [appId, appName, verName, icon, desc, root, assets, imports]. Positional lists are smaller on the wire and are what the C++ side of the platform prefers.

Values are restricted to what the serialization can carry: null, booleans, numbers, strings, binary buffers, arrays and maps of those. Nested SMC documents are fine; a Structured Message, for one, is an SMC document whose content holds the files of the email, each of which may hold another SMC document.

Encodings¶

SMC is defined over three encodings: JSON, MessagePack and XML. All carry the same document, and a Structured Message may carry any of the three.

JSON¶

The string-keyed form shown above, with Content-Type: application/json. Binary values have no native JSON representation, so objects that carry bytes (blobs, digests) are normally sent as MessagePack.

MessagePack¶

For MessagePack, the same string-keyed serialization works, but since MessagePack is a binary format, we can compress it further by ditching string keys with predefined alternatives:

{
    0: "https://github.com/sobamail/counter/model/v1",
    1: "IncrementRequest",
    3: {value: 1}
}

The above is also a valid SMC when encoded using the msgpack format, with Content-Type: application/msgpack. The following substitution table applies:

Senders may use either key style. The Sobamail runtime always emits the integer-keyed form, so a client reading a response finds the content under key 3. The integer keys apply to the envelope only; whatever is inside content keeps the keys its owner gave it.

The runtime's MessagePack encoder is canonical: a value has exactly one byte representation. Key order is part of a JavaScript object's value, since objects keep their assignment order and it survives a round trip, so {a: 1, b: 2} and {b: 2, a: 1} are two values with two encodings. This matters because a mutation is an SMC document hashed into the mutation graph, and those two would hash differently. See soba.pack.msgpack().

XML¶

The object key was designed to be an XML qualified name, and in the XML form it is exactly that, with Content-Type: application/xml. The document's root element is the object: its XML namespace is the SMC namespace, its local name is the SMC name, and its children are the content. The counter example reads:

<IncrementRequest xmlns="https://github.com/sobamail/counter/model/v1">
    <value>1</value>
</IncrementRequest>

A receiver forms the object key from the root element's namespace URI and local name, so a document whose root carries no namespace is not a valid SMC. Like JSON, XML has no native binary type, so objects that carry bytes go as MessagePack.

Errors¶

A failure is reported as an SMC document too. The error field carries a code, and the content is a StructuredError object from the base namespace. This is, for instance, what the app runtime answers when a request is refused:

{
    0: "https://sobamail.com/module/base/v1",
    1: "StructuredError",
    2: "Client.Validation",
    3: ["https://sobamail.com/module/base/v1", "StructuredError", 3, ["App 'x' carries 33554559 bytes of blobs, over the 33554432 byte deployment limit"]]
}

The content is a positional list: the namespace and name of the error, a numeric code, and a list of arguments. The arguments hold the text meant for a human.

Where SMC travels¶

The same document shape appears at three places in the platform. Only the first is a public contract: a Structured Message is email, and anything that can send and receive email can take part. The other two are inside the platform and may change between versions.

Inside an email: the Structured Message¶

A Structured Message is an ordinary MIME email that carries an SMC document as one of its attachments and points at it with a header:

Message-Structure: <content-id of the attachment>

The older spelling X-Message-Structure is accepted as well. The attachment is of type application/msgpack, application/json or application/xml and holds the SMC document. Everything else about the message is regular email: it has a sender, recipients, a subject, a date, and it passes through any SMTP server and lands in any IMAP mailbox. A client that knows nothing about Sobamail shows it as a message with an attachment. A client that does know reads the header, opens the attachment and routes the object by its key.

This is how data crosses mailbox boundaries. When an app instance is shared between users, each mutation travels to the other mailboxes as a Structured Message. The platform recognizes such a message by the object key {https://sobamail.com/module/base/v1}SMessage, as opposed to {https://sobamail.com/module/base/v1}Message for free-form mail. See Events.

Between an app's UI and its backend¶

An app's web interface talks to its Mutator by POSTing an SMC document to /api/v2/object as JSON or MessagePack, with the matching Content-Type. There are no methods, paths or headers to design: the object key says what the request is. The reply is an SMC document as well, emitted with integer keys. Reading a reply therefore looks like this:

const reply = decode(await response.arrayBuffer());
const content = reply[3];

The collaborative counter tutorial walks through a request and its reply end to end.

In the mutation log¶

A mutation is a Replication SMC document, described next: serialized as MessagePack, hashed, and placed in the instance's mutation graph. Replicas exchange these documents to converge. Within a mailbox they travel over the platform's own sync protocol; across mailboxes, as Structured Messages.

Replication SMC¶

When an app's backend records a change (through the soba.data functions), the runtime turns it into a document for the other replicas to replay. That document is a Replication SMC: the plain SMC of the object being recorded, extended with what the mutation graph needs to place it. Unlike plain SMC it is exclusively MessagePack, because it carries binary data (the digests and the replica uuid) that JSON and XML cannot hold natively; it is always integer-keyed, and it rides inside a Structured Message like any other SMC. Its keys:

The digests are what make the mutation log a Merkle DAG. A mutation is identified by its digest, and it names the mutations it depends on by their digests, so every edge of the graph is a hash of the node it points to. The parents travel as a single binary value whose length is a multiple of 64: an empty value means no parents, and the value holds at most 1024 digests. A mutation can have several parents: one for each happens-before relationship the backend's calls established, such as the row handle passed to delete() or update(), the row a foreign key or a last-write-wins constraint resolved to, or a handle from find(). A mutation with no parents is a root of the graph.

Well-known objects¶

Two of the objects a Replication SMC carries are defined by the runtime rather than by any app. They are so frequent that the document names them with a small integer under key 6 instead of a namespace and a name; a document carrying key 6 must not carry keys 0 or 1, and the deserializer rejects one that does.

Both only make sense inside the graph. A DeleteRow is a tombstone whose parents include the insertion it removes; an UpdateRow's parents include the row it changes, and its content is a commutative operation on one of that row's columns. Neither is an event an app sends or declares an interest in; the runtime emits them from soba.data.delete() and soba.data.update() and replays them on every replica.

Apps never build a Replication SMC by hand and never see one as an event: a replica replaying the graph hands the Mutator the object under content, as it would any other event. The layout is documented so that a reader of the raw mutation log, or of a Structured Message in a mailbox, can make sense of it.

A complete example¶

The counter app's IncrementRequest as its UI sends it, as JSON:

{
    "namespace": "https://github.com/sobamail/counter/model/v1",
    "name": "IncrementRequest",
    "content": {"value": 1}
}

The same document as the runtime would emit it, as MessagePack shown in JSON notation:

{
    0: "https://github.com/sobamail/counter/model/v1",
    1: "IncrementRequest",
    3: {"value": 1}
}

Its object key, and the entry the counter's Mutator declares in its objects map to receive it:

{https://github.com/sobamail/counter/model/v1}IncrementRequest