The Structured Message

This is the first post in a series on the ideas behind Sobamail, picking up where the Introducing Sobamail post left off. Each one takes a problem we had to solve before the platform could exist and explains what we built for it.

A problem as old as email itself

Email only carries opaque data — blobs held together by the MIME structure.

Still, a few standards that sit on top of email do give some blobs a meaning, one case at a time: calendar invitations have an attachment type of their own, and delivery status notifications have their own message type. Some providers also read a sliver of markup inside HTML mail. Everything else, from invoices to purchase orders, is handled at the endpoints, generally by the person reading the message.

The transport is already in place: every participant has an address, any server delivers to it, and the mailbox at the other end keeps what arrives until a program collects it. The one piece missing is a standard way of declaring what a message carries.

Sobamail adds that piece. A message can declare “here is an object, this is what it is, and this is what it contains.” Everything else we built sits on top of it.

What a structured message is

A structured message is an ordinary email with two additions: a designated attachment holding the data, and a header that names it.

Message-Structure: <content-id of the attachment>

The message still has a sender, recipients, a subject and a date. It still travels through any mail server and lands in any mailbox.

The attachment contains a document we call Structured Message Content, or SMC. It can be written as JSON or as MessagePack. In JSON it looks like this:

{
    "namespace": "https://github.com/sobamail/counter/model/v1",
    "name": "IncrementRequest",
    "content": {"value": 1}
}

The document has three fields. The namespace and the name together say what the content is. What it looks like inside is up to whoever owns the namespace; SMC says nothing about it, on purpose.

Why a namespace

On its own, {"value": 1} could represent a counter increment, a price, a quantity or a vote. Two programs can only exchange data if both agree on what fields in a given document mean, so the document has to carry a name both sides can look up.

XML solved this problem with namespaces: put your object names under a URI, and whoever owns that URI defines what they mean.

We drew heavily on that idea: an SMC namespace is still a URI, but an https:// URL in practice, which can give its definition a home. The namespace and the name together form the object’s key. Written in an informal notation (sometimes called Clark notation), the key looks like this:

{https://github.com/sobamail/counter/model/v1}IncrementRequest

That key is what the platform routes on: an app declares which keys it wants to see, and the platform hands it only those.

One format, all the way down

In Sobamail, every event is an SMC document.

When you click a button in a Sobamail app, the app’s web interface sends an SMC document to its backend, which runs in the client too, and the backend answers with another. Changes the backend records to the app’s data are SMC documents (we call them mutations), serialized and hashed into the app’s mutation log. If those changes have to reach a colleague who shares the app, the same documents (with additional fields) are wrapped in emails and sent as structured messages. App assets and link detracking info travel as signed SMC documents as well.

Where this leads

If all this reads like an elaborate way of attaching a JSON document to an email, you may be right: SMC on its own is not the interesting part. It carries one object over email and says nothing about how to validate a document or negotiate a version.

The interesting part is what happens between messages. Email is asynchronous: peers come and go, and messages arrive late, twice or out of order. For those messages to change application state that several people share, we also need to define the relations between them. The next post will explain how.

Read more

• The Structured Message Content reference covers SMC field by field, the JSON and MessagePack encodings and where the format appears in the platform.
• The collaborative counter tutorial sends and receives SMC documents end to end.
• The counter app is open source on GitHub.