kea/doc/design/ipc-high.txt

The IPC protocol
================

While the cc-protocol.txt describes the low-level primitives, here we
describe how the whole IPC should work and how to use it.

Assumptions
-----------

We assume the low-level protocol keeps ordering of messages. That is,
if A sends messages 1 and 2 to B, they get delivered in the same order
as they were sent. However, if A sends message 1 to B and 2 to C, the
order in which get them or the order in which they answer is not
defined.

We also assume that the delivery is reliable. If B gets a message from
A, it can be sure that all previous messages were delivered too. If A
sends a message to B, B either gets the message or either A or B is
disconnected during the attempt.

Also, we expect the messages don't get damaged or modified on their
way.

Addressing
----------

We can specify the recipient in two different ways:

 * Directly. Each connected client has an unique address. A message
   addressed to that addressed is sent only to the one client.
 * By a group. A client might subscribe to any number of groups.
   When a message is sent to the group, all clients subscribed to the
   group receive it. It is legal to send to an empty group.

Feedback from the IPC system
----------------------------

The IPC system generates some additional information to aid the
communicating clients.

Undeliverable notification::
  If the client requests it (by a per-message flag) and the set of
  recipients specified is empty (either because the connection
  ID/lname is not connected or because the addressed group is empty),
  an answer message is sent from the MSGQ daemon to notify it about
  the situation.
Notifications about connections and disconnections::
  The system generates notification about following events:
  * Client with given lname connected
  * Client with given lname disconnected
  * Client with given lname subscribed to given group
  * Client with given lname unsubscribed from given group
List of group members:
  The MSGQ provides a command to list members of given group and list
  of all connections.

Communication paradigms
-----------------------

Event notifications
~~~~~~~~~~~~~~~~~~~

Sometimes, an event that may be interesting to other parts of the
system happens. The originating module may not know what other modules
are interested in that kind of event, nor it may know if any at all
wants to know that. With such event, the originating module does not
need any feedback.

For each kind or family of notifications, there's a group. Everybody
interested in that family of notifications subscribes to the group.
When the event happens, it is sent (broadcasted) to the group, without
requiring an answer.

[[NOTE]]
To avoid race conditions on start up, it is important to first
subscribe to the group and then load the initial state (for which
change the notification would be), not the other way around. The other
way, the state could be loaded, and then, before subscribing, an event
could happen and be unnoticed. On the other hand, we may still receive
event for change to the state we already loaded (which should be
generally safe), but not lose an update.

Examples of these kinds could be change to a zone data (so it would
get reloaded by every process that has the data),
connection/disconnection notification from msgq, or change to
configuration data.

It would be the recipients responsibility to handle the notification,
or at least, produce an error log message. In these situations, the
originator can not reasonably handle error cases anyway (the zone data
has been written), so if something does not work, log is everything we
can do.

One-to-one RPC call
~~~~~~~~~~~~~~~~~~~

Sometimes, a process needs to call remote function (or command) in
other process. An example could be asking the configuration manager
for the current configuration or asking it to change it, asking single
process to terminate, etc.

It may be that the group is a singleton group (eg. the command
manager, there must be exactly one in a running system, and is used
just as a stable name for the process) or an lname received by means
of other communication (like a previous subscribe notification).

A command message (containing the parameters, name of the command,
etc) is sent, with the want-answer flag set. The other side processes
the command and sends a result or error back.

If the recipient does not exist, the msgq sends an error right away.

There are still two ways this may fail to provide an answer:

 * The receiving module reads the command, but does not provide an
   answer. Clearly, such module is broken. There should be some (long)
   timeout for this situation, and loud logging to get it fixed.
 * The receiving module terminated at the exact time when msgq tried
   to send to it, or crashed handling the command. Therefore the
   sender listens for disconnect or unsubscription notifications
   (depending on if it was sent by lname or group name) and if the
   recipient disconnects, the sender knows it should not expect the
   answer any more.

An asynchronous waiting for the answer is preferred.

One-to-many RPC call
~~~~~~~~~~~~~~~~~~~~

Sometimes it is needed to send a command to bunch of modules at once,
usually all members of a group that can contain any number of clients.

This would be done by requesting the members of the group from msgq
and then sending a one-to-one RPC call to each of them, tracking them
separately.

[NOTE]
It might happen the list of group members changes between the time it
was requested and the time the commands are sent. If a client gets
disconnected, the sender gets an undeliverable error back from msgq.
If anything else happens (the client unsubscribes, connects,
subscribes), it must explicitly synchronise to the state anyway,
because we could have sent the commands before the change actually
happened and it would look the same to the client.

[WARNING]
It would look better to first request the list of group members and
then send the command to the group, and use the list to track the
answers only. But that is prone to race conditions ‒ if there's any
change between the request for the member list and sending the
command, the actual recipients don't match the list and the server
could get more answers than expected or could wait for answer of a
module that no longer exists.

Known limitations
-----------------

It is meant mostly as signalling protocol. Sending millions of
messages or messages of several tens of megabytes is probably a bad
idea. While there's no architectural limitation with regards of the
number of transferred messages or their sizes, the code is not
optimised and it would probably be very slow.