kea/doc/design/ipc-high.txt

The IPC protocol
================

While the cc-protocol.txt describes the low-level primitives, here we
describe how the whole IPC should work and how to use it.

Assumptions
-----------

We assume the low-level protocol keeps ordering of messages. That is,
if A sends messages 1 and 2 to B, they get delivered in the same order
as they were sent. However, if A sends message 1 to B and 2 to C, the
order in which get them or the order in which they answer is not
defined.

We also assume that the delivery is reliable. If B gets a message from
A, it can be sure that all previous messages were delivered too. If A
sends a message to B, B either gets the message or either A or B is
disconnected during the attempt.

Also, we expect the messages don't get damaged or modified on their
way.

On unrecoverable error (errors like EINTR or short read/write are
recoverable, since there's clear way how to continue without losing
any messages, errors like connection reset are unrecoverable), the
client should abort completely. If it deems better to reconnect, it
must assume anything might have happened during the time and start
communication from scratch, discarding any knowledge gathered from the
previous connection (configuration, addresses of other clients, etc).

Addressing
----------

We can specify the recipient in two different ways:

 * Directly. Each connected client has an unique address. A message
   addressed to that address is sent only to the one client.
 * By a group. A client might subscribe to any number of groups.
   When a message is sent to the group, all clients subscribed to the
   group receive it. It is legal to send to an empty group.

[NOTE]
If it is possible a group may contain multiple recipients, it is
discouraged to send messages expecting an answer addressed to the
group. It is not known how many answers are to come. See below for
details on one-to-many communication.

Feedback from the IPC system
----------------------------

The IPC system generates some additional information to aid the
communicating clients.

Undeliverable notification::
  If the client requests it (by a per-message flag) and the set of
  recipients specified is empty (either because the connection
  ID/lname is not connected or because the addressed group is empty),
  an answer message is sent from the daemon to notify it about
  the situation. However, since the recipient still can take a long
  time to answer (if it exists), clients that need high availability
  should not wait for the answer in blocking way.
Notifications about connections and disconnections::
  The system generates notification about following events:
  * Client connected (sent with the lname of the client)
  * Client disconnected (sent with the lname of the client)
  * Client subscribed (sent with the name of group and lname of
    client)
  * Client unsubscribed (sent with the name of group and lname of
    client)
List of group members:
  The daemon provides a command to list lnames of clients subscribed
  to given group, and lnames of all connections.

Communication paradigms
-----------------------

Event notifications
~~~~~~~~~~~~~~~~~~~

Sometimes, an event that may be interesting to other parts of the
system happens. The originating module may not know what other modules
are interested in that kind of event, nor it may know if any at all
wants to know that. With such event, the originating module does not
need any feedback.

For each kind or family of notifications, there's a group. Everybody
interested in that family of notifications subscribes to the group.
When the event happens, it is sent (broadcasted) to the group, without
requiring an answer.

[[NOTE]]
A care should be taken to avoid race conditions. Imagine one module
provides some kind of state (let's say it's the configuration manager
and the configuration is the shared state). The other modules are
using notifications to update their copy when the configuration
changes (eg. when the configuration changes, the configuration manager
sends a notification with description of the change).

The correct order is to first subscribe to the notifications and then
request the whole configuration. If it was done the other way around,
there would be a short time between the request and the subscription
when an update to the state could happen without the module noticing.

With first subscribing, the notification could come before the initial
version is known or arrive even when the initial version already
includes the change, but these are possible to handle, while the
missing update is not.

One-to-one RPC call
~~~~~~~~~~~~~~~~~~~

Sometimes, a process needs to call remote function (or command) in
other process. An example could be asking the configuration manager
for the current configuration or asking it to change it, asking single
process to terminate, etc.

It may be that the group is a singleton group (eg. the command
manager, there must be exactly one in a running system, and is used
just as a stable name for the process) or an lname received by means
of other communication (like a previous subscribe notification).

A command message (containing the parameters, name of the command,
etc) is sent, with the want-answer flag set. The other side processes
the command and sends a result or error back.

If the recipient does not exist, the daemon sends an error right away.

There are still two ways this may fail to provide an answer:

 * The receiving module reads the command, but does not provide an
   answer. Clearly, such module is broken. There should be some (long)
   timeout for this situation, and loud logging to get it fixed.
 * The receiving module terminated at the exact time when daemon tried
   to send to it, or crashed handling the command. Therefore the
   sender listens for disconnect or unsubscription notifications
   (depending on if it was sent by lname or group name) and if the
   recipient disconnects, the sender knows it should not expect the
   answer any more.

An asynchronous waiting for the answer is preferred.

One-to-many RPC call
~~~~~~~~~~~~~~~~~~~~

Sometimes it is needed to send a command to bunch of modules at once,
usually all members of a group that can contain any number of clients.

This would be done by requesting the members of the group from the
daemon and then sending a one-to-one RPC call to each of them,
tracking them separately.

[NOTE]
It might happen the list of group members changes between the time it
was requested and the time the commands are sent. If a client gets
disconnected, the sender gets an undeliverable error back from the
daemon.  If anything else happens (the client unsubscribes, connects,
subscribes), it must explicitly synchronise to the state anyway,
because we could have sent the commands before the change actually
happened and it would look the same to the client.

[WARNING]
It would look better to first request the list of group members and
then send the command to the group, and use the list to track the
answers only. But that is prone to race conditions ‒ if there's any
change between the request for the member list and sending the
command, the actual recipients don't match the list and the server
could get more answers than expected or could wait for answer of a
module that no longer exists.

Known limitations
-----------------

It is meant mostly as signalling protocol. Sending millions of
messages or messages of several tens of megabytes is probably a bad
idea. While there's no architectural limitation with regards of the
number of transferred messages or their sizes, the code is not
optimised and it would probably be very slow.

We currently expect the system not to be at heavy load. Therefore, we
expect the daemon to keep up with clients sending messages. The
libraries write in blocking mode, which is no problem if the
expectation is true, as the write buffers will generally be empty and
the write wouldn't block, but if it turns out it is not the case, we
might need to reconsider.