[5036] Doc updated.

doc/guide/config.xml

@@ -24,9 +24,36 @@
       <para>Configuration files for DHCPv4, DHCPv6 and DDNS modules are defined
       in an extended JSON format. Basic JSON is defined in <ulink
       url="http://tools.ietf.org/html/rfc4627">RFC 4627</ulink>.  Kea components
-      use a slightly modified form of JSON in that they allow shell-style
-      comments in the file: lines with the hash (#) character in the first column
-      are comment lines and are ignored.</para>
+      use an extended JSON with additional features allowed:
+      in that they allow shell-style
+      <itemizedlist>
+        <listitem>
+          <simpara>shell comments: any text after the hash (#)
+          character is ignored. Dhcp6 allows # in any column, while
+          Dhcp4 and Ddns require hash to be in the first
+          column.</simpara>
+        </listitem>
+        <listitem>
+          <simpara>C comments: any text after the double slashes (//)
+          character is ignored. We're in a process of
+          migrating the configuation parsers and currently only Dhcp6
+          supports this feature.</simpara>
+        </listitem>
+        <listitem>
+          <simpara>Multiline comments: any text between /* and */ is
+          ignored. This commenting can span multiple lines. We're in a
+          process of migrating the configuation parsers and currently
+          only Dhcp6 supports this feature.</simpara>
+        </listitem>
+        <listitem>
+          <simpara>File inclusion: JSON files can include other JSON
+          files. This can be done by using &lt;?include
+          "file.json"?&gt;. We're in a process of migrating the
+          configuation parsers and currently only Dhcp6 supports this
+          feature.</simpara>
+        </listitem>
+      </itemizedlist>
+      </para>
 
       <para>The configuration file consists of a single object (often colloquially
       called a map) started with a curly bracket. It comprises the "Dhcp4", "Dhcp6",

src/bin/dhcp6/dhcp6.dox

@@ -24,6 +24,18 @@ component implementation.
 
 @section dhcpv6ConfigParser Configuration Parsers in DHCPv6
 
+Three minutes overview. If you are here only to learn absolute minimum about
+the new parser, here's how you use it:
+
+@code
+ // The following code:
+ json = isc::data::Element::fromJSONFile(file_name, true);
+
+ // can be replaced with this:
+ Parser6Context parser;
+ json = parser.parseFile(file_name, Parser6Context::PARSER_DHCP6);
+@endcode
+
 The common configuration parsers for the DHCP servers are located in the
 src/lib/dhcpsrv/parsers/ directory. Parsers specific to the DHCPv6 component
 are located in the src/bin/dhcp6/json_config_parser.cc. These parsers derive
@@ -102,6 +114,7 @@ PHASE 1: replace isc::data::fromJSON with bison-based parser. This will allow
    the JSON structures also allows us to continue using existing parsers.
    Furthermore, it is possible to implement default values at this level
    as simply inserting extra JSON structures in places that are necessary.
+   This part is covered by ticket 5036.
 
 PHASE 2: simplify existing parsers by getting rid of the build/commit split.
    Get rid of the inheritance contexts. Essentially the parser should
@@ -114,16 +127,17 @@ PHASE 2: simplify existing parsers by getting rid of the build/commit split.
    from the existing parsers  and implemented in the bison parser.
    It should return extra JSON elements. The details are TBD, but there is
    one example for setting up an renew-timer value on the subnet level that
-   is ihnerited from the global ("Dhcp6") level. This phase is still a bit
-   loosely defined.
-
-There is now a fully working prototype for phase 1. It introduces bison
-based parser. It is essentially defined in two files: dhcp6_lexer.ll,
-which defines regular expressions that are used on the input (be it
-a file or a string in memory). In essence, this code is being called
-repetively and each time it returns a token. This repeats until
-either the parsing is complete or syntax error is encountered. For
-example, for the following text:
+   is ihnerited from the global ("Dhcp6") level. This phase is covered by
+   ticket 5039.
+
+The code change for 5036 introduces bison based parser. It is
+essentially defined in two files: dhcp6_lexer.ll, which defines
+regular expressions that are used on the input (be it a file or a
+string in memory). In essence, this code is being called repetively
+and each time it returns a token. This repeats until either the
+parsing is complete or syntax error is encountered. For example, for
+the following text:
+
 @code
 {
    "Dhcp6":
@@ -176,7 +190,7 @@ to the notation, it's very powerful and easy to extend. The first line defines
 that dhcp6_object consists of certain tokens (DHCP6, COLON and LCURLY_BRACKET)
 followed by 'global_params' expression, followed by RCURLY_BRACKET.
 
-The global_params is defined recursively. It can either be a single 'global_param'
+The 'global_params' is defined recursively. It can either be a single 'global_param'
 expression, or (a shorter) global_params followed by a comma and global_param.
 Bison will apply this and will be able to parse comma separated lists of
 arbitrary lengths.
@@ -223,48 +237,93 @@ is removed from the stack. At the end of parsing, there should be a single
 element on the stack as the top-level parsing (syntax_map) only inserts the
 MapElement object, but does not remove it.
 
+@section dhcpv6ConfigSubParser Parsing Partial Configuation in DHCPv6
+
 One another important capability required is the ability to parse not only the
 whole configuration, but a subset of it. This is done by introducing articifical
-tokens (TOPLEVEL_GENERIC_JSON and TOPLEVEL_DHCP6). The Parse6Context::parse() method
-takes one parameter that specifies, whether the data to be parsed is expected
-to have a generic JSON or the whole configuration (DHCP6). This is only a
-proof-of-concept, but similar approach can be implemented to parse only subnets,
-host reservations, options or basically any other elements. For example, to add
-the ability to parse only pools, the following could be added:
+tokens (e.g. TOPLEVEL_JSON and TOPLEVEL_DHCP6). For complete list of available
+starting contexts, see @ref isc::dhcp::Parser6Context::ParserType. The
+Parse6Context::parse() method takes one parameter that specifies, whether the
+data to be parsed is expected to have a generic JSON or the whole configuration
+(DHCP6). This is only a proof-of-concept, but similar approach can be implemented
+to parse only subnets, host reservations, options or basically any other elements.
+For example, to add the ability to parse only pools, the following could be added:
 
 @code
-start: TOPLEVEL_GENERIC_JSON map2
-| TOPLEVEL_DHCP6 syntax_map
-| TOPLEVEL_POOL pool_entry;
+start: TOPLEVEL_GENERIC_JSON sub_json
+| TOPLEVEL_DHCP6 sub_dhcp6
+| TOPLEVEL_POOL6 sub_pool6;
 @endcode
 
-The code on branch trac5014 contains the code defintion and the Kea-dhcp6 updated
+The code on trac5036 branch contains the code defintion and the Kea-dhcp6 updated
 to use that new parser. I'm sure that parser does not cover 100% of all parameters,
 but so far it is able to load all examples from doc/example/kea6. It is also
 able to parser # comments (bash style, starting at the beginning or middle of
-the line), // comments (C++ style, can start anywhere) or /* */ comments (C style,
+the line), // comments (C++ style, can start anywhere) or / * * / comments (C style,
 can span multiple lines).
 
 This parser is currently used. See configure() method in kea_controller.cc.
 
-There are several new unit-tests written. They're not super-extensive, but
-they do cover the essentials: basic types, maps and lists encapsulating
-each other in various combinations, bash, C, C++ comments. There's one
-particularly useful unit-test called ParserTest.file. It loads all the
-examples we have.
+There are several new unit-tests written, but the code mostly reuses existing
+one to verify that existing functionality was not compromised. There are
+several new interesting ones, though. ParserTest.file loads all the
+config file examples we have in doc/examples/kea6. This ensures that the
+parser is able to load them and also checks that our examples are sane.
+
+@section dhcp6ParserIncludes Config File Includes
 
-The parser currently does not support file includes, but that's easy to
-implement in bison-based parsers.
+The new parser provides an ability to include files. The syntax was chosen
+to look similar to how Apache includes PHP scripts in HTML code. This
+particular syntax was chosen to emphasize that the inclusion directive
+is an additional feature and not really a part of JSON syntax.
 
-The parser's ability to parse generic JSON is somewhat fragile, because
-it's more of a proof of concept rather than a solid capability. The issue
-comes from the fact that if the generic json contains specific tokens that
-are defined in DHCP6 nomenclature (e.g. "renew-timer"), it will interpret
-it as RENEW_TIMER token rather than as STRING token. This can be solved
-by having separate grammar for generic JSON if we need it. It's way
-beyond the scope of proof-of-concept, though.
+To include one file from another, user the following syntax:
+
+@code
+{
+    "Dhcp6": {
+        "interfaces-config": {
+            "interfaces": [ "*" ]},
+        "preferred-lifetime": 3000,
+        "rebind-timer": 2000,
+        "renew-timer": 1000,
+        <?include "subnets.json"?>
+        "valid-lifetime": 4000
+     }
+}
+@endcode
 
-Details of the refactor of the classes derived from DhcpConfigParser is TBD.
+The inclusion is implemented as a stack of files. Typically, when a single
+file is parsed, the files_ (a vector of strings) and sfiles_ (a vector of FILE*)
+both contain a single entry. However, when lexer detects &lt;?include "filename.json?&gt;,
+it calls @ref isc::dhcp::Parser6Context::includeFile method. Up to ten
+nesting levels are supported. This arbitrarily chosen limit is a protection
+against recursive inclusions.
+
+@section dhcp6ParserConflicts Avoiding syntactical conflicts in parsers
+
+Syntactic parser has a powerful ability to not only parse the string and
+check if it's a valid JSON syntax, but also check that the resulting structures
+match expected syntax (if subnet6 are really an array, not a map, if
+timers are expressed as integers, not as strings etc.).
+
+
+However, there are times when we need to parse a string as arbitrary JSON.
+For example, if we're in Dhcp6 and the config contains entries for DhcpDdns
+or Dhcp4. If we were to use naive approach, the lexer would go through
+that content and most likely find some tokens that are also used in Dhcp6.
+for example 'renew-timer' would be detected and the parser would complain
+that it was not expected. To avoid this problem, parser context was
+introduced. When the syntactic parser expects certain type of content,
+it calls @ref isc::dhcp::Parser6Context::enter() method to indicate what
+type of content is expected. For example, when Dhcp6 parser discovers
+uninteresting content like Dhcp4, it enters NO_KEYWORD mode. In this
+mode, everything is parsed as generic maps, lists, integers, booleans
+or strings. This results in generic JSON structures without any syntax
+checking.
+
+Details of the refactor of the classes derived from DhcpConfigParser are
+documented in http://kea.isc.org/wiki/SimpleParser.
 
 @section dhcpv6ConfigInherit DHCPv6 Configuration Inheritance