2-GEM - Generic Enterprise Messaging Framework

This document describes the Generic Enterprise Messaging (GEM) framework for building messaging middleware stacks. GEM is a framework of protocols, an abstracted stack. It is designed to solve a broad range of problems encountered while building AMQP and related technologies.

Name: gro.pqma.ikiw|MEG-2#gro.pqma.ikiw|MEG-2
Editor: Pieter Hintjens <moc.xitami|hp#moc.xitami|hp>
Contributors: Tony Garnock-Jones <ten.tfihsl|gynot#ten.tfihsl|gynot>, Matthias Radestock <ten.tfihsl|saihttam#ten.tfihsl|saihttam>, Alexis Richardson <moc.tfevisehoc|nosdrahcir.sixela#moc.tfevisehoc|nosdrahcir.sixela>
State: draft

License

This Specification is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This Specification is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, see <http://www.gnu.org/licenses>.

Change Process

This document is governed by the gro.pqma.ikiw|SSOC-1#gro.pqma.ikiw|SSOC-1 specification.

Goals

The goals of the GEM framework are:

To define a clear abstract "stack" of protocols that interoperate to deliver functionality that covers AMQP.
To provide the basis for a clean refactoring of the AMQP/0.91 specification's functionality.
To deliver small specifications that can be implemented by small teams.
To reuse and improve existing AMQP terminology whenever possible.
To optimally support both high-performance brokerless and high-reliability brokered messaging.

Architecture

General design

GEM defines a physical network of nodes that send each other unreliable datagram blobs.
On top of the physical network, it builds an isomorphic blameable network of applications that exchange blameable messages.
On top of that, it builds a bound network subset of applications that have consumer relationships to one another.
Applications implement interfaces that accept a set of commands.
Applications send each other commands, which can include content blobs.
The physical network may be virtualized on top of a lower-level application network.

Deconstructing the AMQ model

The AMQ exchange-binding-queue model has been the basis for the successful modeling of various messaging scenarios. Newer knowledge shows us ways to improve it through refactoring the semantics of each entity.

We deconstruct the AMQ model as follows:

The AMQ queue entity provides shared queues for workload distribution, and private queues for data distribution and request-response scenarios. In fact these two types of queue have entirely separate semantics. Private queues are better held on recipient nodes than on a central broker, i.e. at the receiving, not the sending, end of a connection. GEM therefore specifies that every application has an automatic, unnamed private queue for incoming messages.

The AMQ exchange entity is defined as having no storage. However this is inaccurate: as asynchronous objects, exchanges must at least have a queue for incoming messages. GEM therefore specifies that exchanges, as recipients, each have an automatic unnamed private queue for incoming messages.

The AMQ model defines a special entity (shared queue) for a queue that feeds a distribution algorithm. GEM states that shared queues are a type of exchange, and that all exchanges are composed of a queue plus a distribution algorithm (the "workload" algorithm to implement shared queues).

The AMQ model specifies that any message storage in the broker must happen in a queue. GEM specifies that any automatic private queue may be backed to disk, and thus exchanges (and any other application) can be fed by persistent messages.

The AMQ binding entity connects a broker queue to an broker exchange. GEM specifies that bindings are elastic and stretch between the automatic private queue at a receiving application, to an exchange anywhere on the network.

The consequences of this are:

Exchanges become applications that accept binding requests.
Consumers are replaced by bindings.
Exchanges can be placed anywhere on the network.
Brokers become clouds of exchanges.

Layers

The GEM framework defines these individual layers, each solving precisely one problem:

The Addressing Layer, which turns a node name into a physical network address. This layer provides some level of trust, including authentication and identities needed for transport-level encryption.
The Transport Layer, which allows nodes to send each other datagrams. Nodes send and receive datagrams, which are blobs with no semantics. Nodes have no other semantics. This layer does framing (i.e. wire level separation of datagrams) and transport-level encryption. Datagrams are not ordered.
The Control Layer, which allows nodes to send each other datagrams about the transport layer state, especially in case of overload or other problems.
The Contract Layer, in which each node becomes an application, and applications exchange messages and acknowledgments. An application can send a message, assigning blame to the recipient, and the recipient can acknowledge the message and thus take blame for it. Applications can also send messages without assigning blame to the recipient, in which case the recipient's acknowledgment does not accept blame for the message.
The Typing Model, in which messages are typed, and the different types are defined and explained. This may do framing if necessary.
The Command Layer, which specifies how typed, blameable messages carry commands and contents.
The Data Definition Layer, which specifies the encoding and decoding of commands to and from messages.
The Interface Layer, which defines how applications accept commands. The Interface Layer defines a schema language for defining specific interfaces and their commands, called methods.
The Interface Specification Layer, which defines the standard interfaces used in GEM applications, each defined as a specification.
The Application Specification Layer, which defines the standard applications used in a GEM network, each defined as a specification.

Specifications

A layer is not a specification. A layer is an placeholder for arbitrary future specifications that implement the layer and cooperate with specifications in layers above and below.

Stacks

A stack is a bundle of layers implemented by a single specification. Note that specifications can refer to one another, so a stack can be and often is a "cover sheet" for other more detailed specifications.

We expect or imagine these stacks:

A stack that implements the Typing Layer on top of TCP/IP, giving a disconnected blameable datagram layer.
A stack that implements the Command Layer on top of TCP/IP, providing typed message frames.
A stack that implements AMQP/0.91.
A stack that implements the Typing Layer on top of XMPP.