What is Sobamail?¶

Sobamail is, first and foremost, an email solution built using a stack of well-known open-source technologies that support SMTP, IMAP, and POP3 as well as webmail.

On top of this foundation, Sobamail provides a distributed application platform that uses highly optimized yet email-compatible protocols for storage and transport of mutation events as well as app code, data and assets.

Sobamail's own email client, Sobamail Desktop, is the main interface to the Sobamail application platform. On top of conventional email capabilities, it also contains an application runtime based on V8, Chromium's JS engine and CEF, the Chromium Embedded Framework. Because the runtime is embedded in the native client, apps run locally on your machine rather than on a server you call out to.

The big picture¶

Everything below builds on four pillars. If you've read the 5-Minute Intro these will be familiar; if not, here they are in one breath:

• Your mailbox is your data store. One mailbox per user, identified by their email address, holding an email archive alongside application code and data.
• Apps are reactive. They don't run a server you call; they only wake up to handle incoming events, and every event is ultimately an email message.
• State is a replicated log of mutations. Apps append mutations rather than editing data in place, and those mutations sync across your mailbox replicas.
• An app is a Mutator plus a UI. A backend module that reacts to events, and a regular web frontend that talks to it.

The rest of this document fills in each of these.

Mailboxes¶

Each Sobamail user is assigned one primary email address, which also identifies their mailbox, which is their one and only data store.

Mailboxes contain:

• A folder hierarchy, where each folder contains zero or more message threads, comprised of one or more related messages, along with their metadata.
• Application instances, which contain a graph of mutations along with the latest app state that is the result of applying said mutations,
• Zero or more blobs used to organize and transfer immutable data like email attachments, application assets, etc. A single blob can be up to 1 GB in size, and its bytes live in your mailbox rather than as a link to a third-party host.

A mailbox has one or more replicas. These are typically on different devices that the user controls as well as the Sobamail infrastructure.

Mailbox replicas are identified by their randomly generated Universally Unique Identifiers (UUID v4).

Application Model¶

Applications are identified by their application ids, which are domain-name-like strings just like the ones mobile applications have. An application can have multiple versions, which are identified by cryptographic hashes of their components as well as a human-readable version name¹.

In Sobamail, applications need to be instantiated in order to be useful. Application instances are identified by an instance id². Each instance has its own independent data store tied to a given mailbox replica.

Sobamail's application model is a bit different from conventional platforms:

• In desktop platforms, apps typically have one instance per user, and one version per device.
• In mobile platforms, apps typically have one instance and one version per device.

In both, apps are local-only and it is the apps' responsibility to implement synchronization logic (or not).

In Sobamail, apps can have an infinite number of instances. Instances can be local to a single mailbox replica (developer mode) or synchronized across all replicas of one (private mode) or more (shared mode) mailboxes.

In other words, a given list of users can share a single app instance. This allows multiple users to collaborate on the same instance of, for example, a to-do list app.

Events¶

Sobamail apps are reactive. This means that they are only invoked to handle incoming events. Each event is identified by an object key -- a globally unique string built from a namespace and a name. We'll see exactly how object keys look in a moment.

Outside events arrive exclusively in the form of email messages. Sobamail can handle two types of email messages:

• Message, identified by object key: {https://sobamail.com/module/base/v1}Message. This is just regular email -- it has body, attachments, subject, etc.
• Structured Message, identified by object key: {https://sobamail.com/module/base/v1}SMessage

Structured messages are also regular email messages, but they additionally contain a X-Message-Structure or Message-Structure header.

Message-Structure header in structured messages contain the content id of the attachment that contains the Structured Message Content (SMC).

The SMC is the main event format of Sobamail -- It's a JSON or a msgpack document that obeys the following format:

{
  "namespace": "https://example.com/some/ns/v1",
  "name": "Example",
  "content": ["anything", "can", {"go": "here"}]
}

The namespace and name are concatenated in the form of {namespace}name to form the object key ({https://example.com/some/ns/v1}Example in our example). This key is in turn used to look up which applications are interested in processing the event at hand.

A couple of details about the object key format: the braces are part of the string, the namespace must conform to the XML Namespace spec, and Name must be a valid JavaScript class name. This "{Namespace}Name" convention is sometimes called Clark notation.

Outside events always arrive as email, but they aren't the only source of events: a request from an app's own UI is delivered to its backend as a message too -- just locally rather than over the network. The backend handles both through the same mechanism.

Mutations¶

Events come in two flavors, and the runtime treats them differently:

• Replicated events are mutations: they modify application state. Because they must be applied identically on every replica -- with no chance to coordinate with instances that may be offline -- only strictly monotonic operations are allowed, which avoids data races between instances.
• Non-replicated events don't change replicated state directly. They can run arbitrary queries against the underlying SQLite database over a read-only connection, and they can emit mutations (i.e. replicated events). These are typically used to respond to requests arriving over email or from the web interface.

This distinction extends to the runtime API. Because a mutation is replayed independently by every replica, any runtime function reachable while processing one must be deterministic: given the same mutation log, all replicas must converge on the same state. Functions whose results can vary between runs -- those that read wall clocks, generate random values, read the live SQLite database, or expose instance and account identity -- are therefore non-deterministic and may only be called from non-replicated events; calling one while processing a mutation raises an exception. The runtime reference flags each such function with an [NR] marker.

Processing a mutation ultimately results in one or more INSERT or DELETE statements to application tables (those defined by soba.schema.table calls), followed by metadata updates to system tables (whose names start with an underscore) so that other replicas replay the same event efficiently. The runtime supports the UPDATE statement only for commutative operations such as in-place addition or multiplication, which every replica can apply in any order and still converge on the same state. Other updates are emulated using consecutive DELETE and INSERT operations.

Like all Sobamail events, a mutation obeys the SMC format. It is serialized as msgpack, after which it is hashed and takes its place in the mutation graph that other replicas replay.

Because a mutation is identified by this hash, mutations are content-addressed and deduplicated: two mutations with byte-identical contents collapse into a single node in the graph. This is what makes replay idempotent, but it has a consequence worth internalizing -- if you genuinely intend two distinct operations that happen to carry identical contents (say, two separate +1 increments of the same counter column), they would dedupe into one and you would silently lose an operation. To keep them distinct you must vary their contents, typically by attaching a fresh value such as a UUID. Note that the sources of such entropy (random values, wall clocks) are [NR]: the distinguishing value can only be minted in a non-replicated context and then carried into the mutation -- it cannot be generated while a mutation is being replayed. This is why, for example, the counter app mints its nonce when handling the incoming request rather than inside the mutation itself.

Application Components¶

Sobamail apps have two distinct components:

• A Mutator module that defines how the Application reacts to incoming events. The Mutator can import other modules which are identified using a cryptographic hash of the imported source code text.
• A User Interface package that provides a local web interface to the application. This part is just a zip file that contains a regular frontend app built using HTML/CSS/Javascript. GUI packages are identified by the cryptographic hash value of this said zip file and are distributed as part of app deployments. When a user launches the GUI for a given app, they are shown the GUI version tied to the primary mutator version.

The Mutator Module¶

The Mutator module (aka the backend or the root module) is the heart of a Sobamail application. Written as a modern JavaScript module, each mutator module has to export a default class that contains a process() method along with metadata like objects of interest, application id, name, version, etc.

The process() method serves as the main entry point to the event handler (just like the main() function in C). It is only invoked to handle incoming events which arrive exclusively in the form of email messages.

When an email arrives at a mailbox replica, it is "broadcast" to all applications. Applications need to express interest in a given object type to be able to react to it. This is done by adding object keys to the objects map in the root module.

The runtime then hands the Mutator either kind of event described above: a replicated event is a mutation it must apply deterministically, while a non-replicated event may read the database and emit new mutations in response.

Imports¶

Mutators can contain one or more import statements -- top-level statements that take a string in the URI format.

The first import statement of a mutator module must be the import statement that designates the computer where the module will run on.

In Sobamail, import statements are comprised of two parts: The namespace and the cryptographic hash corresponding to the desired module (found in the query string). Since any app can import any module from another app, the apps running in the Sobamail platform actually form one giant codebase.

This reduces barriers to app interoperability -- as long as apps export relevant classes, any app can use data types of any other app.

As of now, the Sobamail computer does not allow sharing of actual data between app instances, but that's on the roadmap.

The User Interface¶

The UI layer in Sobamail is a regular frontend application which can be built using regular JS tools and frameworks. Sobamail uses Chromium Embedded Framework to display messages and run app UI. As of now, your GUI code needs to be compatible with Chrome 117.

The UI communicates with the Mutator module exclusively through an RPC interface that uses the SMC format. This is not a RESTful interface -- one can define objects that go back and forth but it's not possible to send additional metadata like headers or methods like GET or PUT. If you'd rather keep the REST mindset, you can just add headers or method properties to your objects.

Sobamail prefers msgpack as object encoding since it supports full-width integers and binary data, but JSON is also supported.

Version Progression¶

Sobamail Apps typically start their life as a single instance in developer mode. When in this mode, no replication is performed and all module hash comparisons return true. This is to provide the app developer a hassle-free environment to work in.

As part of the deployment process, all components of an application are hashed and frozen, and the app is now ready to be instantiated with all replication features enabled.

Once an app version is deployed, it is set in stone. A mutation generated by a particular version of an application is processed by the same version in all replicas. This prevents app states from diverging because of behavioral differences between different app versions, assuming the schema changes were backwards-compatible with all the older versions.

An instance has a primary version that is used to produce mutations. Older versions are kept around to replay mutations that are produced by older modules in other mailbox replicas.

Just like apps, the Sobamail runtime is also versioned. Runtime versions are called computers. Apps need to declare the computer version to run on in their first import statement. As of now, the Sobamail platform only has 1 computer called Replicated-2 whose import identifier is: "soba://computer/R2"

The R2 is still in experimental mode and can exhibit subtle behavioral differences between versions. If this doesn't work for you, do not hesitate to get in touch with us.

Replication¶

Applications in Sobamail operate in a truly distributed manner:

• Each mailbox replica has its own independent copy of the application instance.
• Applications share mutations automatically as replicas come online.

How those mutations travel depends on where they are headed:

• Within mailbox boundaries -- when state is replicated between replicas of the same mailbox, an optimized protocol is used to synchronize quickly, ensuring fast and efficient convergence.
• Across mailbox boundaries -- when state is replicated between different mailboxes (shared instances), each event travels as a Structured Message: a regular MIME document carrying a Message-Structure or X-Message-Structure header that points to the attachment holding the encoded mutation (XML, JSON, or MessagePack).

As mentioned above, to prevent state divergence due to version progression, all events are propagated with the mutator that produced them. This means that once installed, mutator modules never get deleted as long as the instance that has mutations produced by these modules is alive.

Two special system applications keep the mailbox itself in sync. Each mailbox has exactly one instance of each:

1. The Application Manager: Manages application lifetime (modules, assets, etc.)
2. The Mailbox Manager: Manages folders, messages and their metadata.

Both use the facilities of the Sobamail runtime to synchronize mailbox replicas. They are automatically replicated across all replicas of a user's mailbox, private to the mailbox owner (no replication across mailbox boundaries), and restricted to one instance per mailbox (i.e. they are singletons).

Email Compatibility¶

The POP3 or IMAP protocols can be used to access the mailbox replica data kept in Sobamail servers. App instances can be exposed as folders, and mutations as structured messages.

Next Steps¶

That's about it!

The best way to internalize the Sobamail concepts is to get your hands dirty! Head over to Instantiating Your First App to see how to implement a Sobamail app!