Merge branch 'master' into trac2353

Conflicts:
	src/bin/bind10/bind10_src.py.in

ChangeLog

@@ -1,3 +1,141 @@
+519.	[bug]		muks
+	Fixed a problem in inmem NSEC lookup which caused assert failures
+	in some cases.
+	(Trac #2504, git 835553eb309d100b062051f7ef18422d2e8e3ae4)
+
+518.	[func]		stephen
+	Extend DHCP MySQL backend to handle IPv4 addresses.
+	(Trac #2404, git ce7db48d3ff5d5aad12b1da5e67ae60073cb2607)
+
+517.	[func]		stephen
+	Added IOAddress::toBytes() to get byte representation of address.
+	Also added convenience methods for V4/V6 address determination.
+	(Trac #2396, git c23f87e8ac3ea781b38d688f8f7b58539f85e35a)
+
+516.	[bug]		marcin
+	Fixed 'make distcheck' failure when running perfdhcp unit tests.
+	The unit tests used to read files from the folder specified
+	with the path relative to current folder, thus when the test was
+	run from a different folder the files could not be found.
+	(Trac #2479, git 4e8325e1b309f1d388a3055ec1e1df98c377f383)
+
+515.	[bug]		jinmei
+	The in-memory data source now accepts an RRSIG provided without
+	a covered RRset in loading.  A subsequent query for its owner name
+	of the covered type would generally result in NXRRSET; if the
+	covered RRset is of type NSEC3, the corresponding NSEC3 processing
+	would result in SERVFAIL.
+	(Trac #2420, git 6744c100953f6def5500bcb4bfc330b9ffba0f5f)
+
+514.	[bug]		jelte
+	b10-msgq now handles socket errors more gracefully when sending data
+	to clients. It no longer exits with 'broken pipe' errors, and is
+	also better at resending data on temporary error codes from send().
+	(Trac #2398, git 9f6b45ee210a253dca608848a58c824ff5e0d234)
+
+513.	[func]		marcin
+	Implemented the OptionCustom class for DHCPv4 and DHCPv6.
+	This class represents an option which has a defined
+	structure: a set of data fields of specific types and order.
+	It is used to represent those options that can't be
+	represented by any other specialized class.
+	(Trac #2312, git 28d885b457dda970d9aecc5de018ec1120143a10)
+
+512.	[func]		jelte
+	Added a new tool b10-certgen, to check and update the self-signed
+	SSL certificate used by b10-cmdctl. The original certificate
+	provided has been removed, and a fresh one is generated upon first
+	build. See the b10-certgen manpage for information on how to update
+	existing installed certificates.
+	(Trac #1044, git 510773dd9057ccf6caa8241e74a7a0b34ca971ab)
+
+511.	[bug]		stephen
+	Fixed a race condition in the DHCP tests whereby the test program
+	spawned a subprocess and attempted to read (without waiting) from
+	the interconnecting pipe before the subprocess had written
+	anything.  The lack of output was being interpreted as a test
+	failure.
+	(Trac #2410, git f53e65cdceeb8e6da4723730e4ed0a17e4646579)
+
+510.	[func]		marcin
+	DHCP option instances can be created using a collection of strings.
+	Each string represents a value of a particular data field within
+	an option. The data field values, given as strings, are validated
+	against the actual types of option fields specified in the options
+	definitions.
+	(Trac #2490, git 56cfd6612fcaeae9acec4a94e1e5f1a88142c44d)
+
+509.	[func]		muks
+	Log messages now include the pid of the process that logged the
+	message.
+	(Trac #1745, git fc8bbf3d438e8154e7c2bdd322145a7f7854dc6a)
+
+508.	[bug]		stephen
+	Split the DHCP library into two directories, each with its own
+	Makefile.  This properly solves the problem whereby a "make"
+	operation with multiple threads could fail because of the
+	dependencies between two libraries in the same directory.
+	(Trac #2475, git 834fa9e8f5097c6fd06845620f68547a97da8ff8)
+
+bind10-devel-20121115 released on November 15, 2012
+
+507.	[doc]		jelte
+	Added a chapter about the use of the bindctl command tool to
+	to the BIND 10 guide.
+	(Trac #2305, git c4b0294b5bf4a9d32fb18ab62ca572f492788d72)
+
+506.	[security]		jinmei
+	Fixed a use-after-free case in handling DNAME record with the
+	in-memory data source.  This could lead to a crash of b10-auth
+	if it serves a zone containing a DNAME RR from the in-memory
+	data source.  This bug was introduced at bind10-devel-20120927.
+	(Trac #2471, git 2b1793ac78f972ddb1ae2fd092a7f539902223ff)
+
+505.	[bug]		jelte
+	Fixed a bug in b10-xfrin where a wrong call was made during the
+	final check of a TSIG-signed transfer, incorrectly rejecting the
+	transfer.
+	(Trac #2464, git eac81c0cbebee72f6478bdb5cda915f5470d08e1)
+
+504.	[bug]*		naokikambe
+	Fixed an XML format viewed from b10-stats-httpd. Regarding
+	per-zone counters as zones of Xfrout, a part of the item
+	values wasn't an exact XML format. A zone name can be
+	specified in URI as
+	/bind10/statistics/xml/Xfrout/zones/example.org/xfrreqdone.
+	XSD and XSL formats are also changed to constant ones due
+	to these changes.
+	(Trac #2298, git 512d2d46f3cb431bcdbf8d90af27bff8874ba075)
+
+503.	[func]		Stephen
+	Add initial version of a MySQL backend for the DHCP code.  This
+	implements the basic IPv6 lease access functions - add lease, delete
+	lease and update lease.  The backend is enabled by specifying
+	--with-dhcp-mysql on the "configure" command line: without this
+	switch, the MySQL code is not compiled, so leaving BIND 10 able to
+	be built on systems without MySQL installed.
+	(Trac #2342, git c7defffb89bd0f3fdd7ad2437c78950bcb86ad37)
+
+502.	[func]		vorner
+	TTLs can be specified with units as well as number of seconds now.
+	This allows specifications like "1D3H".
+	(Trac #2384, git 44c321c37e17347f33ced9d0868af0c891ff422b)
+
+501.	[func]		tomek
+	Added DHCPv6 allocation engine, now used in the processing of DHCPv6
+	messages.
+	(Trac #2414, git b3526430f02aa3dc3273612524d23137b8f1fe87)
+
+500.	[bug]		jinmei
+	Corrected the autoconf example in the examples directory so it can
+	use the configured path to Boost to check availability of the BIND 10
+	library.  Previously the sample configure script could fail if
+	Boost is installed in an uncommon place.  Also, it now provides a
+	helper m4 function and example usage for embedding the library
+	path to executable (using linker options like -Wl,-R) to help
+	minimize post-build hassles.
+	(Trac #2356, git 36514ddc884c02a063e166d44319467ce6fb1d8f)
+
 499.	[func]		team
 	The b10-auth 'loadzone' command now uses the internal thread
 	introduced in 495 to (re)load a zone in the background, so that
@@ -8,8 +146,9 @@
 	Implemented DHCPv6 option values configuration using configuration
 	manager. In order to set values for data fields carried by the
 	particular option, user specifies the string of hexadecimal digits
-	that is in turn converted to binary data and stored into option buffer.
-	More user friendly way of option content specification is planned.
+	that is in turn converted to binary data and stored into option
+	buffer. More user friendly way of option content specification is
+	planned.
 	(Trac #2318, git e75c686cd9c14f4d6c2a242a0a0853314704fee9)
 
 497.	[bug]		jinmei

Makefile.am

@@ -1,4 +1,4 @@
-ACLOCAL_AMFLAGS = -I m4macros ${ACLOCAL_FLAGS}
+ACLOCAL_AMFLAGS = -I m4macros -I examples/m4 ${ACLOCAL_FLAGS}
 # ^^^^^^^^ This has to be the first line and cannot come later in this
 # Makefile.am due to some bork in some versions of autotools.
 
@@ -20,6 +20,13 @@ dist_doc_DATA = AUTHORS COPYING ChangeLog README
 
 .PHONY: check-valgrind check-valgrind-suppress
 
+install-exec-hook:
+	-@echo -e "\033[1;33m" # switch to yellow color text
+	@echo "NOTE: BIND 10 does not automatically start DNS services when it is run"
+	@echo "      in its default configuration. Please see the Guide for information"
+	@echo "      on how to configure these services to be started automatically."
+	-@echo -e "\033[m" # switch back to normal
+
 check-valgrind:
 if HAVE_VALGRIND
 	@VALGRIND_COMMAND="$(VALGRIND) -q --gen-suppressions=all --track-origins=yes --num-callers=48 --leak-check=full --fullpath-after=" \

README

@@ -1,16 +1,11 @@
 
-This is the source for the development version of BIND 10.
+This is the source for the BIND 10 suite.
 
 BIND is the popular implementation of a DNS server, developer
 interfaces, and DNS tools. BIND 10 is a rewrite of BIND 9 and ISC
 DHCP. BIND 10 is written in C++ and Python and provides a modular
 environment for serving, maintaining, and developing DNS and DHCP.
 
-BIND10-devel is new development leading up to the production
-BIND 10 release. It contains prototype code and experimental
-interfaces. Nevertheless it is ready to use now for testing the
-new BIND 10 infrastructure ideas.
-
 This release includes the bind10 master process, b10-msgq message
 bus, b10-auth authoritative DNS server (with SQLite3 and in-memory
 backends), b10-resolver recursive or forwarding DNS server, b10-cmdctl
@@ -62,3 +57,8 @@ For operating system specific tips see the wiki at:
        http://bind10.isc.org/wiki/SystemSpecificNotes
 
 Please see the wiki and the doc/ directory for various documentation.
+
+The BIND 10 suite is started by running "bind10". Note that the default
+configuration does not start any DNS or DHCP services.  Please see the
+Guide for information on how to configure these services to be started
+automatically.

configure.ac

@@ -64,25 +64,9 @@ AM_CONDITIONAL(USE_CLANGPP, test "X${CLANGPP}" = "Xyes")
 
 # Linker options
 
-# check -R and -Wl,-R rather than gcc specific -rpath to be as portable
-# as possible.
-AC_MSG_CHECKING([whether -R flag is available in linker])
-LDFLAGS_SAVED="$LDFLAGS"
-LDFLAGS="$LDFLAGS -R/usr/lib"
-AC_TRY_LINK([],[],
-    [ AC_MSG_RESULT(yes)
-        rpath_flag=-R
-    ],[ AC_MSG_RESULT(no)
-        AC_MSG_CHECKING([whether -Wl,-R flag is available in linker])
-        LDFLAGS="$LDFLAGS_SAVED -Wl,-R"
-        AC_TRY_LINK([], [],
-            [ AC_MSG_RESULT(yes)
-                rpath_flag=-Wl,-R
-            ],[ AC_MSG_RESULT(no)
-                 rpath_flag=no
-            ])
-    ])
-LDFLAGS=$LDFLAGS_SAVED
+# check -R, "-Wl,-R" or -rpath (we share the AX function defined in
+#  examples/m4)
+AX_ISC_RPATH
 
 # Compiler dependent settings: define some mandatory CXXFLAGS here.
 # We also use a separate variable B10_CXXFLAGS.  This will (and should) be
@@ -332,10 +316,10 @@ fi
 # modules, we embed the path to the modules when possible.  We do this even
 # when the path is known in the common operational environment (e.g. when
 # it's stored in a common "hint" file) for simplicity.
-if test $rpath_flag != no; then
+if test "x$ISC_RPATH_FLAG" != "x"; then
 	python_rpath=
 	for flag in ${PYTHON_LDFLAGS}; do
-		python_rpath="${python_rpath} `echo $flag | sed -ne "s/^\(\-L\)/${rpath_flag}/p"`"
+		python_rpath="${python_rpath} `echo $flag | sed -ne "s/^\(\-L\)/${ISC_RPATH_FLAG}/p"`"
 	done
 	PYTHON_LDFLAGS="${PYTHON_LDFLAGS} ${python_rpath}"
 fi
@@ -701,10 +685,10 @@ for flag in ${BOTAN_LIBS}; do
 done
 
 # See python_rpath for some info on why we do this
-if test $rpath_flag != no; then
+if test "x$ISC_RPATH_FLAG" != "x"; then
     BOTAN_RPATH=
     for flag in ${BOTAN_LIBS}; do
-            BOTAN_RPATH="${BOTAN_RPATH} `echo $flag | sed -ne "s/^\(\-L\)/${rpath_flag}/p"`"
+            BOTAN_RPATH="${BOTAN_RPATH} `echo $flag | sed -ne "s/^\(\-L\)/${ISC_RPATH_FLAG}/p"`"
     done
 AC_SUBST(BOTAN_RPATH)
 
@@ -721,7 +705,6 @@ fi
 AC_SUBST(BOTAN_LDFLAGS)
 AC_SUBST(BOTAN_LIBS)
 AC_SUBST(BOTAN_INCLUDES)
-
 # Even though chances are high we already performed a real compilation check
 # in the search for the right (pkg)config data, we try again here, to
 # be sure.
@@ -749,6 +732,60 @@ AC_LINK_IFELSE(
 CPPFLAGS=$CPPFLAGS_SAVED
 LIBS=$LIBS_SAVED
 
+# Check for MySql.  The path to the mysql_config program is given with
+# the --with-mysql-config (default to /usr/bin/mysql-config).  By default,
+# the software is not built with MySQL support enabled.
+mysql_config="no"
+AC_ARG_WITH([dhcp-mysql],
+  AC_HELP_STRING([--with-dhcp-mysql=PATH],
+    [path to the MySQL 'mysql_config' script (MySQL is used for the DHCP database)]),
+    [mysql_config="$withval"])
+
+if test "${mysql_config}" = "yes" ; then
+    MYSQL_CONFIG="/usr/bin/mysql_config"
+elif test "${mysql_config}" != "no" ; then
+    MYSQL_CONFIG="${withval}"
+fi
+
+if test "$MYSQL_CONFIG" != "" ; then
+    if test -d "$MYSQL_CONFIG" -o ! -x "$MYSQL_CONFIG" ; then
+        AC_MSG_ERROR([--with-dhcp-mysql should point to a mysql_config program])
+    fi
+
+    MYSQL_CPPFLAGS=`$MYSQL_CONFIG --cflags`
+    MYSQL_LIBS=`$MYSQL_CONFIG --libs`
+
+    AC_SUBST(MYSQL_CPPFLAGS)
+    AC_SUBST(MYSQL_LIBS)
+
+    # Check that a simple program using MySQL functions can compile and link.
+    CPPFLAGS_SAVED="$CPPFLAGS"
+    LIBS_SAVED="$LIBS"
+
+    CPPFLAGS="$MYSQL_CPPFLAGS $CPPFLAGS"
+    LIBS="$MYSQL_LIBS $LIBS"
+
+    AC_LINK_IFELSE(
+            [AC_LANG_PROGRAM([#include <mysql.h>],
+                             [MYSQL mysql_handle;
+                              (void) mysql_init(&mysql_handle);
+                             ])],
+            [AC_MSG_RESULT([checking for MySQL headers and library... yes])],
+            [AC_MSG_RESULT([checking for MySQL headers and library... no])
+             AC_MSG_ERROR([Needs MySQL library])]
+    )
+
+    CPPFLAGS=$CPPFLAGS_SAVED
+    LIBS=$LIBS_SAVED
+
+    # Note that MYSQL is present in the config.h file
+    AC_DEFINE([HAVE_MYSQL], [1], [MySQL is present])
+fi
+
+# ... and at the shell level, so Makefile.am can take action depending on this.
+AM_CONDITIONAL(HAVE_MYSQL, test "$MYSQL_CONFIG" != "")
+
+
 # Check for log4cplus
 log4cplus_path="yes"
 AC_ARG_WITH([log4cplus],
@@ -866,11 +903,12 @@ AC_SUBST(MULTITHREADING_FLAG)
 #
 GTEST_LDFLAGS=
 GTEST_LDADD=
-# TODO: set DISTCHECK_GTEST_CONFIGURE_FLAG for --with-gtest too
 DISTCHECK_GTEST_CONFIGURE_FLAG=
 
 if test "x$enable_gtest" = "xyes" ; then
 
+    DISTCHECK_GTEST_CONFIGURE_FLAG="--with-gtest=$gtest_path"
+
     if test -n "$with_gtest_source" ; then
 
           if test "x$GTEST_SOURCE" = "xyes" ; then
@@ -1226,6 +1264,8 @@ AC_CONFIG_FILES([Makefile
                  src/lib/dns/benchmarks/Makefile
                  src/lib/dhcp/Makefile
                  src/lib/dhcp/tests/Makefile
+                 src/lib/dhcpsrv/Makefile
+                 src/lib/dhcpsrv/tests/Makefile
                  src/lib/exceptions/Makefile
                  src/lib/exceptions/tests/Makefile
                  src/lib/datasrc/Makefile
@@ -1269,7 +1309,7 @@ AC_CONFIG_FILES([Makefile
                  tests/tools/badpacket/tests/Makefile
                  tests/tools/perfdhcp/Makefile
                  tests/tools/perfdhcp/tests/Makefile
-                 tests/tools/perfdhcp/templates/Makefile
+                 tests/tools/perfdhcp/tests/testdata/Makefile
                  dns++.pc
                ])
 AC_OUTPUT([doc/version.ent
@@ -1437,6 +1477,18 @@ dnl includes too
                  ${LOG4CPLUS_LIBS}
   SQLite:        $SQLITE_CFLAGS
                  $SQLITE_LIBS
+END
+
+# Avoid confusion on DNS/DHCP and only mention MySQL if it
+# were specified on the command line.
+if test "$MYSQL_CPPFLAGS" != "" ; then
+cat >> config.report << END
+  MySQL:         $MYSQL_CPPFLAGS
+                 $MYSQL_LIBS
+END
+fi
+
+cat >> config.report << END
 
 Features:
   $enable_features

doc/Doxyfile

@@ -580,8 +580,8 @@ INPUT                  = ../src/lib/exceptions ../src/lib/cc \
     ../src/lib/testutils ../src/lib/cache ../src/lib/server_common/ \
     ../src/bin/sockcreator/ ../src/lib/util/ ../src/lib/util/io/ \
     ../src/lib/util/threads/ ../src/lib/resolve ../src/lib/acl \
-    ../src/lib/statistics ../src/bin/dhcp6 ../src/lib/dhcp ../src/bin/dhcp4 \
-    ../tests/tools/perfdhcp devel
+    ../src/lib/statistics ../src/bin/dhcp6 ../src/lib/dhcp ../src/lib/dhcpsrv \
+    ../src/bin/dhcp4 ../tests/tools/perfdhcp devel
 
 # This tag can be used to specify the character encoding of the source files
 # that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is

doc/devel/mainpage.dox

@@ -33,6 +33,7 @@
  *   - @subpage leasemgr
  *   - @subpage cfgmgr
  *   - @subpage allocengine
+ * - @subpage dhcp-database-backends
  * - @subpage perfdhcpInternals
  *
  * @section misc Miscellaneous topics

examples/README

@@ -30,3 +30,18 @@ to the configure.ac file:
 sinclude(m4/ax_boost_include.m4)
 sinclude(m4/ax_isc_bind10.m4)
 (and same for other m4 files as they are added under m4/)
+
+On some systems, espeically if you have installed the BIND 10
+libraries in an uncommon path, programs linked with the BIND 10
+library may not work at run time due to the "missing" shared library.
+Normally, you should be able to avoid this problem by making sure
+to invoking the program explicitly specifying the path to the library,
+e.g., "LD_LIBRARY_PATH=/usr/local/lib/bind10 ./my_bind10_app", or
+you may not even notice the issue if you have installed BIND 10
+library in a common library path on your system (sometimes you may
+still need to run ldconfig(8) beforehand).  Another option is to embed
+the path to the library in your program.  While this approach is
+controversial, and some people rather choose the alternatives, we
+provide a helper tool in case you want to use this option: see the
+lines using BIND10_RPATH in the sample configure.ac file of this
+directory.

examples/configure.ac

@@ -14,7 +14,21 @@ AC_LANG([C++])
 # Checks for BIND 10 headers and libraries
 AX_ISC_BIND10
 
-# For the example host program, we require the BIND 10 DNS library
+# We use -R, -rpath etc so the resulting program will be more likekly to
+# "just work" by default.  Embedding a specific library path is a controversial
+# practice, though; if you don't like it you can remove the following setting.
+if test "x$BIND10_RPATH" != "x"; then
+   LDFLAGS="$LDFLAGS $BIND10_RPATH"
+fi
+
+# For the example host program, we require some socket API library
+# and the BIND 10 DNS library.
+
+# In practice, these are specific to Solaris, but wouldn't do any harm for
+# others except for the checking overhead.
+AC_SEARCH_LIBS(inet_pton, [nsl])
+AC_SEARCH_LIBS(recvfrom, [socket])
+
 if test "x$BIND10_DNS_LIB" = "x"; then
    AC_MSG_ERROR([unable to find BIND 10 DNS library needed to build 'host'])
 fi

examples/m4/ax_boost_include.m4

@@ -34,7 +34,7 @@ if test -z "$with_boost_include"; then
 		fi
 	done
 fi
-CPPFLAGS_SAVES="$CPPFLAGS"
+CPPFLAGS_SAVED="$CPPFLAGS"
 if test "${boost_include_path}" ; then
 	BOOST_CPPFLAGS="-I${boost_include_path}"
 	CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS"
@@ -57,7 +57,7 @@ AC_TRY_COMPILE([
  CPPFLAGS_BOOST_THREADCONF="-DBOOST_DISABLE_THREADS=1"],
 [AC_MSG_RESULT(yes)])
 
-CPPFLAGS="$CPPFLAGS_SAVES $CPPFLAGS_BOOST_THREADCONF"
+CPPFLAGS="$CPPFLAGS_SAVED $CPPFLAGS_BOOST_THREADCONF"
 AC_SUBST(BOOST_CPPFLAGS)
 
 AC_LANG_RESTORE

examples/m4/ax_isc_bind10.m4

@@ -1,4 +1,4 @@
-dnl @synopsis AX_BIND10
+dnl @synopsis AX_ISC_BIND10
 dnl
 dnl @summary figure out how to build C++ programs using ISC BIND 10 libraries
 dnl
@@ -20,9 +20,18 @@ dnl Checks for other BIND 10 module libraries are option, as not all
 dnl applications need all libraries.  The main configure.ac can handle any
 dnl missing library as fatal by checking whether the corresponding
 dnl BIND10_xxx_LIB is defined.
+dnl
+dnl In addition, it sets the BIND10_RPATH variable to a usable linker option
+dnl to embed the path to the BIND 10 library to the programs that are to be
+dnl linked with the library.  If the developer wants to use the option,
+dnl it can be used as follows:
+dnl if test "x$BIND10_RPATH" != "x"; then
+dnl     LDFLAGS="$LDFLAGS $BIND10_RPATH"
+dnl fi
 
 AC_DEFUN([AX_ISC_BIND10], [
 AC_REQUIRE([AX_BOOST_INCLUDE])
+AC_REQUIRE([AX_ISC_RPATH])
 AC_LANG_SAVE
 AC_LANG([C++])
 
@@ -42,19 +51,20 @@ if test "$bind10_inc_path" = "no"; then
 	fi
    done
 fi
-CPPFLAGS_SAVES="$CPPFLAGS"
+CPPFLAGS_SAVED="$CPPFLAGS"
+CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS" # boost headers will be used in buffer.h
 if test "${bind10_inc_path}" != "no"; then
    BIND10_CPPFLAGS="-I${bind10_inc_path}"
    CPPFLAGS="$CPPFLAGS $BIND10_CPPFLAGS"
 fi
 AC_CHECK_HEADERS([util/buffer.h],,
-  AC_MSG_ERROR([Missing a commonly used BIND 10 header files]))
-CPPFLAGS="$CPPFLAGS_SAVES"
+  AC_MSG_ERROR([Missing a commonly used BIND 10 header file]))
+CPPFLAGS="$CPPFLAGS_SAVED"
 AC_SUBST(BIND10_CPPFLAGS)
 
 # Check for BIND10 libraries
 CPPFLAGS_SAVED="$CPPFLAGS"
-CPPFLAGS="$CPPFLAGS $BIND10_CPPFLAGS"
+CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS $BIND10_CPPFLAGS"
 
 AC_ARG_WITH(bind10-lib,
   AS_HELP_STRING([--with-bind10-lib=PATH],
@@ -70,21 +80,25 @@ fi
 # make sure we have buildable libraries
 AC_MSG_CHECKING([for BIND 10 common library])
 BIND10_COMMON_LIB="-lb10-util -lb10-exceptions"
-LDFLAGS="$LDFLAGS $BIND10_LDFLAGS"
+LDFLAGS_SAVED="$LDFLAGS"
+LDFLAGS_CHECK_COMMON="$LDFLAGS $BIND10_LDFLAGS"
+LIBS_SAVED="$LIBS"
 LIBS="$LIBS $BIND10_COMMON_LIB"
 for d in $bind10_lib_dirs
 do
-  LDFLAGS_SAVED="$LDFLAGS"
-  LDFLAGS="$LDFLAGS -L$d"
+  LDFLAGS="$LDFLAGS_CHECK_COMMON -L$d"
   AC_TRY_LINK([
 #include <util/buffer.h>
 ],[
 isc::util::OutputBuffer buffer(0);
-], [BIND10_LDFLAGS="-L${d}"])
+], [BIND10_LDFLAGS="-L${d}"
+    if test "x$ISC_RPATH_FLAG" != "x"; then
+       BIND10_RPATH="${ISC_RPATH_FLAG}${d}"
+    fi
+    ])
   if test "x$BIND10_LDFLAGS" != "x"; then
      break
   fi
-  LDFLAGS="$LDFLAGS_SAVED"
 done
 if test "x$BIND10_LDFLAGS" != "x"; then
   AC_MSG_RESULT(yes)
@@ -94,7 +108,7 @@ else
 fi
 
 # restore LIBS once at this point
-LIBS="$LIBS_SAVES"
+LIBS="$LIBS_SAVED"
 
 AC_SUBST(BIND10_LDFLAGS)
 AC_SUBST(BIND10_COMMON_LIB)
@@ -111,12 +125,12 @@ isc::dns::RRType rrtype(1);
 ], [BIND10_DNS_LIB="-lb10-dns++"
     AC_MSG_RESULT(yes)],
    [AC_MSG_RESULT(no)])
-LIBS="$LIBS_SAVES"
+LIBS="$LIBS_SAVED"
 AC_SUBST(BIND10_DNS_LIB)
 
 # Restore other flags
 CPPFLAGS="$CPPFLAGS_SAVED"
-LDFLAGS="$LDFLAGS_SAVES"
+LDFLAGS="$LDFLAGS_SAVED"
 
 AC_LANG_RESTORE
 ])dnl AX_ISC_BIND10

examples/m4/ax_isc_rpath.m4

@@ -0,0 +1,46 @@
+dnl @synopsis AX_ISC_RPATH
+dnl
+dnl @summary figure out whether and which "rpath" linker option is available
+dnl
+dnl This macro checks if the linker supports an option to embed a path
+dnl to a runtime library (often installed in an uncommon place), such as
+dnl gcc's -rpath option.  If found, it sets the ISC_RPATH_FLAG variable to
+dnl the found option flag.  The main configure.ac can use it as follows:
+dnl if test "x$ISC_RPATH_FLAG" != "x"; then
+dnl     LDFLAGS="$LDFLAGS ${ISC_RPATH_FLAG}/usr/local/lib/some_library"
+dnl fi
+
+AC_DEFUN([AX_ISC_RPATH], [
+
+# We'll tweak both CXXFLAGS and CCFLAGS so this function will work whichever
+# language is used in the main script.  Note also that it's not LDFLAGS;
+# technically this is a linker flag, but we've noticed $LDFLAGS can be placed
+# where the compiler could interpret it as a compiler option, leading to
+# subtle failure mode.  So, in the check below using the compiler flag is
+# safer (in the actual Makefiles the flag should be set in LDFLAGS).
+CXXFLAGS_SAVED="$CXXFLAGS"
+CXXFLAGS="$CXXFLAGS -Wl,-R/usr/lib"
+CCFLAGS_SAVED="$CCFLAGS"
+CCFLAGS="$CCFLAGS -Wl,-R/usr/lib"
+
+# check -Wl,-R and -R rather than gcc specific -rpath to be as portable
+# as possible.  -Wl,-R seems to be safer, so we try it first.  In some cases
+# -R is not actually recognized but AC_TRY_LINK doesn't fail due to that.
+AC_MSG_CHECKING([whether -Wl,-R flag is available in linker])
+AC_TRY_LINK([],[],
+    [ AC_MSG_RESULT(yes)
+        ISC_RPATH_FLAG=-Wl,-R
+    ],[ AC_MSG_RESULT(no)
+        AC_MSG_CHECKING([whether -R flag is available in linker])
+	CXXFLAGS="$CXXFLAGS_SAVED -R"
+	CCFLAGS="$CCFLAGS_SAVED -R"
+        AC_TRY_LINK([], [],
+            [ AC_MSG_RESULT([yes; note that -R is more sensitive about the position in option arguments])
+                ISC_RPATH_FLAG=-R
+            ],[ AC_MSG_RESULT(no) ])
+    ])
+
+CXXFLAGS=$CXXFLAGS_SAVED
+CCFLAGS=$CCFLAGS_SAVED
+
+])dnl AX_ISC_RPATH

src/bin/auth/auth_messages.mes

@@ -47,10 +47,15 @@ available. It is issued during server startup is an indication that
 the initialization is proceeding normally.
 
 % AUTH_CONFIG_LOAD_FAIL load of configuration failed: %1
-An attempt to configure the server with information from the configuration
-database during the startup sequence has failed. (The reason for
-the failure is given in the message.) The server will continue its
-initialization although it may not be configured in the desired way.
+An attempt to configure the server with information from the
+configuration database during the startup sequence has failed.  The
+server will continue its initialization although it may not be
+configured in the desired way.  The reason for the failure is given in
+the message.  One common reason is that the server failed to acquire a
+socket bound to a privileged port (53 for DNS).  In that case the
+reason in the log message should show something like "permission
+denied", and the solution would be to restart BIND 10 as a super
+(root) user.
 
 % AUTH_CONFIG_UPDATE_FAIL update of configuration failed: %1
 At attempt to update the configuration the server with information
@@ -93,6 +98,16 @@ This debug message is issued when the separate thread for maintaining data
 source clients successfully loaded the named zone of the named class as a
 result of the 'loadzone' command.
 
+% AUTH_DATASRC_CLIENTS_BUILDER_LOAD_ZONE_NOCACHE skipped loading zone %1/%2 due to no in-memory cache
+This debug message is issued when the separate thread for maintaining data
+source clients received a command to reload a zone but skipped it because
+the specified zone is not loaded in-memory (but served from an underlying
+data source).  This could happen if the loadzone command is manually issued
+by a user but the zone name is misspelled, but a more likely cause is
+that the command is sent from another BIND 10 module (such as xfrin or DDNS).
+In the latter case it can be simply ignored because there is no need
+for explicit reloading.
+
 % AUTH_DATASRC_CLIENTS_BUILDER_RECONFIGURE_CONFIG_ERROR Error in data source configuration: %1
 The thread for maintaining data source clients has received a command to
 reconfigure, but the parameter data (the new configuration) contains an

src/bin/auth/auth_srv.cc

@@ -651,9 +651,10 @@ AuthSrvImpl::processNormalQuery(const IOMessage& io_message, Message& message,
         local_edns->setUDPSize(AuthSrvImpl::DEFAULT_LOCAL_UDPSIZE);
         message.setEDNS(local_edns);
     }
-    // Get access to data source client list through the holder and keep the
-    // holder until the processing and rendering is done to avoid inter-thread
-    // race.
+
+    // Get access to data source client list through the holder and keep
+    // the holder until the processing and rendering is done to avoid
+    // race with any other thread(s) such as the background loader.
     auth::DataSrcClientsMgr::Holder datasrc_holder(datasrc_clients_mgr_);
 
     try {
@@ -688,6 +689,9 @@ AuthSrvImpl::processNormalQuery(const IOMessage& io_message, Message& message,
     return (true);
     // The message can contain some data from the locked resource. But outside
     // this method, we touch only the RCode of it, so it should be safe.
+
+    // Lock on datasrc_clients_mgr_ acquired by datasrc_holder is
+    // released here upon its deletion.
 }
 
 bool

src/bin/auth/datasrc_clients_mgr.h

@@ -581,6 +581,9 @@ DataSrcClientsBuilderBase<MutexType, CondVarType>::doLoadZone(
     try {
         boost::shared_ptr<datasrc::memory::ZoneWriter> zwriter =
             getZoneWriter(*client_list, rrclass, origin);
+        if (!zwriter) {
+            return;
+        }
 
         zwriter->load(); // this can take time but doesn't cause a race
         {   // install() can cause a race and must be in a critical section
@@ -614,8 +617,14 @@ DataSrcClientsBuilderBase<MutexType, CondVarType>::getZoneWriter(
     datasrc::ConfigurableClientList& client_list,
     const dns::RRClass& rrclass, const dns::Name& origin)
 {
-    const datasrc::ConfigurableClientList::ZoneWriterPair writerpair =
-        client_list.getCachedZoneWriter(origin);
+    // getCachedZoneWriter() could get access to an underlying data source
+    // that can cause a race condition with the main thread using that data
+    // source for lookup.  So we need to protect the access here.
+    datasrc::ConfigurableClientList::ZoneWriterPair writerpair;
+    {
+        typename MutexType::Locker locker(*map_mutex_);
+        writerpair = client_list.getCachedZoneWriter(origin);
+    }
 
     switch (writerpair.first) {
     case datasrc::ConfigurableClientList::ZONE_SUCCESS:
@@ -626,8 +635,10 @@ DataSrcClientsBuilderBase<MutexType, CondVarType>::getZoneWriter(
                   << "/" << rrclass << ": not found in any configured "
                   "data source.");
     case datasrc::ConfigurableClientList::ZONE_NOT_CACHED:
-        isc_throw(InternalCommandError, "failed to load zone " << origin
-                  << "/" << rrclass << ": not served from memory");
+        LOG_DEBUG(auth_logger, DBG_AUTH_OPS,
+                  AUTH_DATASRC_CLIENTS_BUILDER_LOAD_ZONE_NOCACHE)
+            .arg(origin).arg(rrclass);
+        break;                  // return NULL below
     case datasrc::ConfigurableClientList::CACHE_DISABLED:
         // This is an internal error. Auth server must have the cache
         // enabled.
@@ -636,8 +647,6 @@ DataSrcClientsBuilderBase<MutexType, CondVarType>::getZoneWriter(
                   "is somehow disabled");
     }
 
-    // all cases above should either return or throw, but some compilers
-    // still need a return statement
     return (boost::shared_ptr<datasrc::memory::ZoneWriter>());
 }
 } // namespace datasrc_clientmgr_internal

src/bin/auth/query.cc

@@ -410,9 +410,9 @@ Query::process(datasrc::ClientList& client_list,
              */
             assert(db_context->rrset->getRdataCount() > 0);
             // Get the data of DNAME
+            RdataIteratorPtr rit = db_context->rrset->getRdataIterator();
             const rdata::generic::DNAME& dname(
-                dynamic_cast<const rdata::generic::DNAME&>(
-                db_context->rrset->getRdataIterator()->getCurrent()));
+                dynamic_cast<const rdata::generic::DNAME&>(rit->getCurrent()));
             // The yet unmatched prefix dname
             const Name prefix(qname_->split(0, qname_->getLabelCount() -
                 db_context->rrset->getName().getLabelCount()));

src/bin/auth/tests/Makefile.am

@@ -50,6 +50,7 @@ run_unittests_SOURCES += config_syntax_unittest.cc
 run_unittests_SOURCES += command_unittest.cc
 run_unittests_SOURCES += common_unittest.cc
 run_unittests_SOURCES += query_unittest.cc
+run_unittests_SOURCES += query_inmemory_unittest.cc
 run_unittests_SOURCES += statistics_unittest.cc
 run_unittests_SOURCES += test_datasrc_clients_mgr.h test_datasrc_clients_mgr.cc
 run_unittests_SOURCES += datasrc_clients_builder_unittest.cc

src/bin/auth/tests/auth_srv_unittest.cc

@@ -225,7 +225,7 @@ createBuiltinVersionResponse(const qid_t qid, vector<uint8_t>& data) {
     message.setHeaderFlag(Message::HEADERFLAG_AA);
     RRsetPtr rrset_version = RRsetPtr(new RRset(version_name, RRClass::CH(),
                                                 RRType::TXT(), RRTTL(0)));
-    rrset_version->addRdata(generic::TXT(PACKAGE_STRING));
+    rrset_version->addRdata(generic::TXT("\"" PACKAGE_STRING "\""));
     message.addRRset(Message::SECTION_ANSWER, rrset_version);
 
     RRsetPtr rrset_version_ns = RRsetPtr(new RRset(apex_name, RRClass::CH(),

src/bin/auth/tests/datasrc_clients_builder_unittest.cc

@@ -308,8 +308,12 @@ TEST_F(DataSrcClientsBuilderTest, loadZone) {
                                    "{\"class\": \"IN\","
                                    " \"origin\": \"test1.example\"}"));
     EXPECT_TRUE(builder.handleCommand(loadzone_cmd));
-    EXPECT_EQ(1, map_mutex.lock_count); // we should have acquired the lock
-    EXPECT_EQ(1, map_mutex.unlock_count); // and released it.
+
+    // loadZone involves two critical sections: one for getting the zone
+    // writer, and one for actually updating the zone data.  So the lock/unlock
+    // count should be incremented by 2.
+    EXPECT_EQ(2, map_mutex.lock_count);
+    EXPECT_EQ(2, map_mutex.unlock_count);
 
     newZoneChecks(clients_map, rrclass);
 }
@@ -381,7 +385,10 @@ TEST_F(DataSrcClientsBuilderTest,
               find(Name("example.org")).finder_->
               find(Name("example.org"), RRType::SOA())->code);
 
-    // attempt of reloading a zone but in-memory cache is disabled.
+    // attempt of reloading a zone but in-memory cache is disabled.  In this
+    // case the command is simply ignored.
+    const size_t orig_lock_count = map_mutex.lock_count;
+    const size_t orig_unlock_count = map_mutex.unlock_count;
     const ConstElementPtr config2(Element::fromJSON("{"
         "\"IN\": [{"
         "    \"type\": \"sqlite3\","
@@ -390,11 +397,13 @@ TEST_F(DataSrcClientsBuilderTest,
         "    \"cache-zones\": [\"example.org\"]"
         "}]}"));
     clients_map = configureDataSource(config2);
-    EXPECT_THROW(builder.handleCommand(
+    builder.handleCommand(
                      Command(LOADZONE, Element::fromJSON(
                                  "{\"class\": \"IN\","
-                                 " \"origin\": \"example.org\"}"))),
-                 TestDataSrcClientsBuilder::InternalCommandError);
+                                 " \"origin\": \"example.org\"}")));
+    // Only one mutex was needed because there was no actual reload.
+    EXPECT_EQ(orig_lock_count + 1, map_mutex.lock_count);
+    EXPECT_EQ(orig_unlock_count + 1, map_mutex.unlock_count);
 
     // basically impossible case: in-memory cache is completely disabled.
     // In this implementation of manager-builder, this should never happen,

src/bin/auth/tests/query_inmemory_unittest.cc

@@ -0,0 +1,123 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#include <dns/name.h>
+#include <dns/message.h>
+#include <dns/rcode.h>
+#include <dns/opcode.h>
+
+#include <cc/data.h>
+
+#include <datasrc/client_list.h>
+
+#include <auth/query.h>
+
+#include <testutils/dnsmessage_test.h>
+
+#include <gtest/gtest.h>
+
+#include <string>
+
+using namespace isc::dns;
+using namespace isc::auth;
+using namespace isc::testutils;
+using isc::datasrc::ConfigurableClientList;
+using std::string;
+
+namespace {
+
+// The DNAME to do tests against
+const char* const dname_txt =
+    "dname.example.com. 3600 IN DNAME "
+    "somethinglong.dnametarget.example.com.\n";
+// This is not inside the zone, this is created at runtime
+const char* const synthetized_cname_txt =
+    "www.dname.example.com. 3600 IN CNAME "
+    "www.somethinglong.dnametarget.example.com.\n";
+
+// This is a subset of QueryTest using (subset of) the same test data, but
+// with the production in-memory data source.  Both tests should be eventually
+// unified to avoid duplicates.
+class InMemoryQueryTest : public ::testing::Test {
+protected:
+    InMemoryQueryTest() : list(RRClass::IN()), response(Message::RENDER) {
+        response.setRcode(Rcode::NOERROR());
+        response.setOpcode(Opcode::QUERY());
+        list.configure(isc::data::Element::fromJSON(
+                           "[{\"type\": \"MasterFiles\","
+                           "  \"cache-enable\": true, "
+                           "  \"params\": {\"example.com\": \"" +
+                           string(TEST_OWN_DATA_DIR "/example.zone") +
+                           "\"}}]"), true);
+    }
+
+    ConfigurableClientList list;
+    Message response;
+    Query query;
+};
+
+// A wrapper to check resulting response message commonly used in
+// tests below.
+// check_origin needs to be specified only when the authority section has
+// an SOA RR.  The interface is not generic enough but should be okay
+// for our test cases in practice.
+void
+responseCheck(Message& response, const isc::dns::Rcode& rcode,
+              unsigned int flags, const unsigned int ancount,
+              const unsigned int nscount, const unsigned int arcount,
+              const char* const expected_answer,
+              const char* const expected_authority,
+              const char* const expected_additional,
+              const Name& check_origin = Name::ROOT_NAME())
+{
+    // In our test cases QID, Opcode, and QDCOUNT should be constant, so
+    // we don't bother the test cases specifying these values.
+    headerCheck(response, response.getQid(), rcode, Opcode::QUERY().getCode(),
+                flags, 0, ancount, nscount, arcount);
+    if (expected_answer != NULL) {
+        rrsetsCheck(expected_answer,
+                    response.beginSection(Message::SECTION_ANSWER),
+                    response.endSection(Message::SECTION_ANSWER),
+                    check_origin);
+    }
+    if (expected_authority != NULL) {
+        rrsetsCheck(expected_authority,
+                    response.beginSection(Message::SECTION_AUTHORITY),
+                    response.endSection(Message::SECTION_AUTHORITY),
+                    check_origin);
+    }
+    if (expected_additional != NULL) {
+        rrsetsCheck(expected_additional,
+                    response.beginSection(Message::SECTION_ADDITIONAL),
+                    response.endSection(Message::SECTION_ADDITIONAL));
+    }
+}
+
+/*
+ * Test a query under a domain with DNAME. We should get a synthetized CNAME
+ * as well as the DNAME.
+ *
+ * TODO: Once we have CNAME chaining, check it works with synthetized CNAMEs
+ * as well. This includes tests pointing inside the zone, outside the zone,
+ * pointing to NXRRSET and NXDOMAIN cases (similarly as with CNAME).
+ */
+TEST_F(InMemoryQueryTest, DNAME) {
+    query.process(list, Name("www.dname.example.com"), RRType::A(),
+                  response);
+
+    responseCheck(response, Rcode::NOERROR(), AA_FLAG, 2, 0, 0,
+        (string(dname_txt) + synthetized_cname_txt).c_str(),
+        NULL, NULL);
+}
+}

src/bin/auth/tests/testdata/Makefile.am

@@ -21,6 +21,7 @@ EXTRA_DIST += simpleresponse_fromWire.spec
 EXTRA_DIST += spec.spec
 
 EXTRA_DIST += example.com
+EXTRA_DIST += example.zone
 EXTRA_DIST += example.sqlite3
 
 .spec.wire:

src/bin/auth/tests/testdata/example.zone

@@ -0,0 +1,121 @@
+;;
+;; This is a complete (but crafted and somewhat broken) zone file used
+;; in query tests.
+;;
+
+example.com. 3600 IN SOA . . 0 0 0 0 0
+example.com. 3600 IN NS glue.delegation.example.com.
+example.com. 3600 IN NS noglue.example.com.
+example.com. 3600 IN NS example.net.
+example.com. 3600 IN DS 57855 5 1 B6DCD485719ADCA18E5F3D48A2331627FDD3 636B
+glue.delegation.example.com. 3600 IN A 192.0.2.153
+glue.delegation.example.com. 3600 IN AAAA 2001:db8::53
+noglue.example.com. 3600 IN A 192.0.2.53
+delegation.example.com. 3600 IN NS glue.delegation.example.com.
+delegation.example.com. 3600 IN NS noglue.example.com.
+delegation.example.com. 3600 IN NS cname.example.com.
+delegation.example.com. 3600 IN NS example.org.
+;; Borrowed from the RFC4035
+delegation.example.com. 3600 IN DS 57855 5 1 B6DCD485719ADCA18E5F3D48A2331627FDD3 636B
+mx.example.com. 3600 IN MX 10 www.example.com.
+mx.example.com. 3600 IN MX 20 mailer.example.org.
+mx.example.com. 3600 IN MX 30 mx.delegation.example.com.
+www.example.com. 3600 IN A 192.0.2.80
+cname.example.com. 3600 IN CNAME www.example.com.
+cnamenxdom.example.com. 3600 IN CNAME nxdomain.example.com.
+;; CNAME Leading out of zone
+cnameout.example.com. 3600 IN CNAME www.example.org.
+;; The DNAME to do tests against
+dname.example.com. 3600 IN DNAME somethinglong.dnametarget.example.com.
+;; Some data at the dname node (allowed by RFC 2672)
+dname.example.com. 3600 IN A 192.0.2.5
+;; The rest of data won't be referenced from the test cases.
+cnamemailer.example.com. 3600 IN CNAME www.example.com.
+cnamemx.example.com. 3600 IN MX 10 cnamemailer.example.com.
+mx.delegation.example.com. 3600 IN A 192.0.2.100
+;; Wildcards
+*.wild.example.com. 3600 IN A 192.0.2.7
+*.wild.example.com. 3600 IN NSEC www.example.com. A NSEC RRSIG
+*.cnamewild.example.com. 3600 IN CNAME www.example.org.
+*.cnamewild.example.com. 3600 IN NSEC delegation.example.com. CNAME NSEC RRSIG
+;; Wildcard_nxrrset
+*.uwild.example.com. 3600 IN A 192.0.2.9
+*.uwild.example.com. 3600 IN NSEC www.uwild.example.com. A NSEC RRSIG
+www.uwild.example.com. 3600 IN A 192.0.2.11
+www.uwild.example.com. 3600 IN NSEC *.wild.example.com. A NSEC RRSIG
+;; Wildcard empty
+b.*.t.example.com. 3600 IN A 192.0.2.13
+b.*.t.example.com. 3600 IN NSEC *.uwild.example.com. A NSEC RRSIG
+t.example.com. 3600 IN A 192.0.2.15
+t.example.com. 3600 IN NSEC b.*.t.example.com. A NSEC RRSIG
+;; Used in NXDOMAIN proof test.  We are going to test some unusual case where
+;; the best possible wildcard is below the "next domain" of the NSEC RR that
+;; proves the NXDOMAIN, i.e.,
+;; mx.example.com. (exist)
+;; (.no.example.com. (qname, NXDOMAIN)
+;; ).no.example.com. (exist)
+;; *.no.example.com. (best possible wildcard, not exist)
+).no.example.com. 3600 IN AAAA 2001:db8::53
+;; NSEC records.
+example.com. 3600 IN NSEC cname.example.com. NS SOA NSEC RRSIG
+mx.example.com. 3600 IN NSEC ).no.example.com. MX NSEC RRSIG
+).no.example.com. 3600 IN NSEC nz.no.example.com. AAAA NSEC RRSIG
+;; We'll also test the case where a single NSEC proves both NXDOMAIN and the
+;; non existence of wildcard.  The following records will be used for that
+;; test.
+;; ).no.example.com. (exist, whose NSEC proves everything)
+;; *.no.example.com. (best possible wildcard, not exist)
+;; nx.no.example.com. (NXDOMAIN)
+;; nz.no.example.com. (exist)
+nz.no.example.com. 3600 IN AAAA 2001:db8::5300
+nz.no.example.com. 3600 IN NSEC noglue.example.com. AAAA NSEC RRSIG
+noglue.example.com. 3600 IN NSEC nonsec.example.com. A
+
+;; NSEC for the normal NXRRSET case
+www.example.com. 3600 IN NSEC example.com. A NSEC RRSIG
+
+;; Authoritative data without NSEC
+nonsec.example.com. 3600 IN A 192.0.2.0
+
+;; NSEC3 RRs.  You may also need to add mapping to MockZoneFinder::hash_map_.
+0p9mhaveqvm6t7vbl5lop2u3t2rp3tom.example.com. 3600 IN NSEC3 1 1 12 aabbccdd 2t7b4g4vsa5smi47k61mv5bv1a22bojr NS SOA NSEC3PARAM RRSIG
+0p9mhaveqvm6t7vbl5lop2u3t2rp3tom.example.com. 3600 IN RRSIG NSEC3 5 3 3600 20000101000000 20000201000000 12345 example.com. FAKEFAKEFAKE
+q04jkcevqvmu85r014c7dkba38o0ji5r.example.com. 3600 IN NSEC3 1 1 12 aabbccdd r53bq7cc2uvmubfu5ocmm6pers9tk9en A RRSIG
+q04jkcevqvmu85r014c7dkba38o0ji5r.example.com. 3600 IN RRSIG NSEC3 5 3 3600 20000101000000 20000201000000 12345 example.com. FAKEFAKEFAKE
+
+;; NSEC3 for wild.example.com (used in wildcard tests, will be added on
+;; demand not to confuse other tests)
+ji6neoaepv8b5o6k4ev33abha8ht9fgc.example.com. 3600 IN NSEC3 1 1 12 aabbccdd r53bq7cc2uvmubfu5ocmm6pers9tk9en
+
+;; NSEC3 for cnamewild.example.com (used in wildcard tests, will be added on
+;; demand not to confuse other tests)
+k8udemvp1j2f7eg6jebps17vp3n8i58h.example.com. 3600 IN NSEC3 1 1 12 aabbccdd r53bq7cc2uvmubfu5ocmm6pers9tk9en
+
+;; NSEC3 for *.uwild.example.com (will be added on demand not to confuse
+;; other tests)
+b4um86eghhds6nea196smvmlo4ors995.example.com. 3600 IN NSEC3 1 1 12 aabbccdd r53bq7cc2uvmubfu5ocmm6pers9tk9en A RRSIG
+;; NSEC3 for uwild.example.com. (will be added on demand)
+t644ebqk9bibcna874givr6joj62mlhv.example.com. 3600 IN NSEC3 1 1 12 aabbccdd r53bq7cc2uvmubfu5ocmm6pers9tk9en A RRSIG
+
+;; (Secure) delegation data; Delegation with DS record
+signed-delegation.example.com. 3600 IN NS ns.example.net.
+signed-delegation.example.com. 3600 IN DS 12345 8 2 764501411DE58E8618945054A3F620B36202E115D015A7773F4B78E0F952CECA
+
+;; (Secure) delegation data; Delegation without DS record (and both NSEC
+;; and NSEC3 denying its existence)
+unsigned-delegation.example.com. 3600 IN NS ns.example.net.
+unsigned-delegation.example.com. 3600 IN NSEC unsigned-delegation-optout.example.com. NS RRSIG NSEC
+;; This one will be added on demand
+q81r598950igr1eqvc60aedlq66425b5.example.com. 3600 IN NSEC3 1 1 12 aabbccdd 0p9mhaveqvm6t7vbl5lop2u3t2rp3tom NS RRSIG
+
+;; Delegation without DS record, and no direct matching NSEC3 record
+unsigned-delegation-optout.example.com. 3600 IN NS ns.example.net.
+unsigned-delegation-optout.example.com. 3600 IN NSEC *.uwild.example.com. NS RRSIG NSEC
+
+;; (Secure) delegation data; Delegation where the DS lookup will raise an
+;; exception.
+bad-delegation.example.com. 3600 IN NS ns.example.net.
+
+;; Delegation from an unsigned parent.  There's no DS, and there's no NSEC
+;; or NSEC3 that proves it.
+nosec-delegation.example.com. 3600 IN NS ns.nosec.example.net.

src/bin/bind10/bind10_messages.mes

@@ -82,6 +82,21 @@ the boss process will try to force them).
 A debug message. The configurator is about to perform one task of the plan it
 is currently executing on the named component.
 
+% BIND10_CONNECTING_TO_CC_FAIL failed to connect to configuration/command channel; try -v to see output from msgq
+The boss process tried to connect to the communication channel for
+commands and configuration updates during initialization, but it
+failed.  This is a fatal startup error, and process will soon
+terminate after some cleanup.  There can be several reasons for the
+failure, but the most likely cause is that the msgq daemon failed to
+start, and the most likely cause of the msgq failure is that it
+doesn't have a permission to create a socket file for the
+communication.  To confirm that, you can see debug messages from msgq
+by starting BIND 10 with the -v command line option.  If it indicates
+permission problem for msgq, make sure the directory where the socket
+file is to be created is writable for the msgq process.  Note that if
+you specify the -u option to change process users, the directory must
+be writable for that user.
+
 % BIND10_INVALID_STATISTICS_DATA invalid specification of statistics data specified
 An error was encountered when the boss module specified
 statistics data which is invalid for the boss specification file.
@@ -94,11 +109,6 @@ and continue running as the specified user, but the user is unknown.
 The boss module was not able to start every process it needed to start
 during startup, and will now kill the processes that did get started.
 
-% BIND10_KILL_PROCESS killing process %1
-The boss module is sending a kill signal to process with the given name,
-as part of the process of killing all started processes during a failed
-startup, as described for BIND10_KILLING_ALL_PROCESSES
-
 % BIND10_LOST_SOCKET_CONSUMER consumer %1 of sockets disconnected, considering all its sockets closed
 A connection from one of the applications which requested a socket was
 closed. This means the application has terminated, so all the sockets it was

src/bin/bind10/bind10_src.py.in

@@ -337,11 +337,7 @@ class BoB:
             each one.  It then clears that list.
         """
         logger.info(BIND10_KILLING_ALL_PROCESSES)
-
-        for pid in self.components:
-            logger.info(BIND10_KILL_PROCESS, self.components[pid].name())
-            self.components[pid].kill(True)
-        self.components = {}
+        self.__kill_children(True)
 
     def _read_bind10_config(self):
         """
@@ -446,6 +442,7 @@ class BoB:
 
             # if we have been trying for "a while" give up
             if (time.time() - cc_connect_start) > self.msgq_timeout:
+                logger.error(BIND10_CONNECTING_TO_CC_FAIL)
                 raise CChannelConnectError("Unable to connect to c-channel after 5 seconds")
 
             # try to connect, and if we can't wait a short while
@@ -1169,6 +1166,21 @@ def main():
 
     options = parse_args()
 
+    # Announce startup.  Making this is the first log message.
+    try:
+        logger.info(BIND10_STARTING, VERSION)
+    except RuntimeError as e:
+        sys.stderr.write('ERROR: failed to write the initial log: %s\n' %
+                         str(e))
+        sys.stderr.write("""\
+TIP: if this is about permission error for a lock file, check if the directory
+of the file is writable for the user of the bind10 process; often you need
+to start bind10 as a super user.  Also, if you specify the -u option to
+change the user and group, the directory must be writable for the group,
+and the created lock file must be writable for that user.
+""")
+        sys.exit(1)
+
     # Check user ID.
     setuid = None
     setgid = None
@@ -1201,9 +1213,6 @@ def main():
             logger.fatal(BIND10_INVALID_USER, options.user)
             sys.exit(1)
 
-    # Announce startup.
-    logger.info(BIND10_STARTING, VERSION)
-
     # Create wakeup pipe for signal handlers
     wakeup_pipe = os.pipe()
     signal.set_wakeup_fd(wakeup_pipe[1])

src/bin/bindctl/bindcmd.py

@@ -133,7 +133,18 @@ class BindCmdInterpreter(Cmd):
         return digest
 
     def run(self):
-        '''Parse commands from user and send them to cmdctl. '''
+        '''Parse commands from user and send them to cmdctl.'''
+
+        # Show helper warning about a well known issue.  We only do this
+        # when stdin is attached to a terminal, because otherwise it doesn't
+        # matter and is just noisy, and could even be harmful if the output
+        # is processed by a script that expects a specific format.
+        if my_readline == sys.stdin.readline and sys.stdin.isatty():
+            sys.stdout.write("""\
+WARNING: Python readline module isn't available, so the command line editor
+         (including command history management) does not work.  See BIND 10
+         guide for more details.\n\n""")
+
         try:
             if not self.login_to_cmdctl():
                 return 1

src/bin/bindctl/moduleinfo.py

@@ -221,7 +221,8 @@ class ModuleInfo:
 
     def module_help(self):
         """Prints the help info for this module to stdout"""
-        print("Module ", self, "\nAvailable commands:")
+        print("Module " + str(self))
+        print("Available commands:")
         for k in self.commands.values():
             n = k.get_name()
             if len(n) >= CONST_BINDCTL_HELP_INDENT_WIDTH:

src/bin/bindctl/tests/bindctl_test.py

@@ -364,16 +364,24 @@ class TestConfigCommands(unittest.TestCase):
         socket_err_output = io.StringIO()
         sys.stdout = socket_err_output
         self.assertEqual(1, self.tool.run())
-        self.assertEqual("Failed to send request, the connection is closed\n",
-                         socket_err_output.getvalue())
+
+        # First few lines may be some kind of heading, or a warning that
+        # Python readline is unavailable, so we do a sub-string check.
+        self.assertIn("Failed to send request, the connection is closed",
+                      socket_err_output.getvalue())
+
         socket_err_output.close()
 
         # validate log message for http.client.CannotSendRequest
         cannot_send_output = io.StringIO()
         sys.stdout = cannot_send_output
         self.assertEqual(1, self.tool.run())
-        self.assertEqual("Can not send request, the connection is busy\n",
-                         cannot_send_output.getvalue())
+
+        # First few lines may be some kind of heading, or a warning that
+        # Python readline is unavailable, so we do a sub-string check.
+        self.assertIn("Can not send request, the connection is busy",
+                      cannot_send_output.getvalue())
+
         cannot_send_output.close()
 
     def test_apply_cfg_command_int(self):

src/bin/cfgmgr/local_plugins/.gitignore

@@ -0,0 +1 @@
+/datasrc.spec

src/bin/cmdctl/.gitignore

@@ -1,6 +1,10 @@
+/b10-certgen
+/b10-certgen.1
 /b10-cmdctl
+/b10-cmdctl.8
+/cmdctl-certfile.pem
+/cmdctl-keyfile.pem
 /cmdctl.py
 /cmdctl.spec
 /cmdctl.spec.pre
 /run_b10-cmdctl.sh
-/b10-cmdctl.8

src/bin/cmdctl/Makefile.am

@@ -4,6 +4,8 @@ pkglibexecdir = $(libexecdir)/@PACKAGE@
 
 pkglibexec_SCRIPTS = b10-cmdctl
 
+bin_PROGRAMS = b10-certgen
+
 nodist_pylogmessage_PYTHON = $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py
 pylogmessagedir = $(pyexecdir)/isc/log_messages/
 
@@ -25,15 +27,18 @@ CLEANFILES= b10-cmdctl cmdctl.pyc cmdctl.spec
 CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py
 CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.pyc
 
-man_MANS = b10-cmdctl.8
-DISTCLEANFILES = $(man_MANS)
-EXTRA_DIST += $(man_MANS) b10-cmdctl.xml cmdctl_messages.mes
+man_MANS = b10-cmdctl.8 b10-certgen.1
+DISTCLEANFILES = $(man_MANS) cmdctl-certfile.pem cmdctl-keyfile.pem
+EXTRA_DIST += $(man_MANS) b10-certgen.xml b10-cmdctl.xml cmdctl_messages.mes
 
 if GENERATE_DOCS
 
 b10-cmdctl.8: b10-cmdctl.xml
 	@XSLTPROC@ --novalid --xinclude --nonet -o $@ http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl $(srcdir)/b10-cmdctl.xml
 
+b10-certgen.1: b10-certgen.xml
+	@XSLTPROC@ --novalid --xinclude --nonet -o $@ http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl $(srcdir)/b10-certgen.xml
+
 else
 
 $(man_MANS):
@@ -54,12 +59,23 @@ b10-cmdctl: cmdctl.py $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py
 	$(SED) "s|@@PYTHONPATH@@|@pyexecdir@|" cmdctl.py >$@
 	chmod a+x $@
 
+b10_certgen_SOURCES = b10-certgen.cc
+b10_certgen_CXXFLAGS = $(BOTAN_INCLUDES)
+b10_certgen_LDFLAGS = $(BOTAN_LIBS)
+
+# Generate the initial certificates immediately
+cmdctl-certfile.pem: b10-certgen
+	./b10-certgen -q -w
+
+cmdctl-keyfile.pem: b10-certgen
+	./b10-certgen -q -w
+
 if INSTALL_CONFIGURATIONS
 
 # Below we intentionally use ${INSTALL} -m 640 instead of $(INSTALL_DATA)
 # because these file will contain sensitive information.
 install-data-local:
-	$(mkinstalldirs) $(DESTDIR)/@sysconfdir@/@PACKAGE@   
+	$(mkinstalldirs) $(DESTDIR)/@sysconfdir@/@PACKAGE@
 	for f in $(CMDCTL_CONFIGURATIONS) ; do	\
 	  if test ! -f $(DESTDIR)$(sysconfdir)/@PACKAGE@/$$f; then	\
 	    ${INSTALL} -m 640 $(srcdir)/$$f $(DESTDIR)$(sysconfdir)/@PACKAGE@/ ;	\

src/bin/cmdctl/b10-certgen.cc

@@ -0,0 +1,429 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#include <botan/botan.h>
+#include <botan/x509self.h>
+#include <botan/x509stor.h>
+#include <botan/rsa.h>
+#include <botan/dsa.h>
+#include <botan/data_src.h>
+using namespace Botan;
+
+#include <iostream>
+#include <fstream>
+#include <memory>
+#include <getopt.h>
+
+// For cleaner 'does not exist or is not readable' output than
+// botan provides
+#include <unistd.h>
+#include <errno.h>
+
+// This is a simple tool that creates a self-signed PEM certificate
+// for use with BIND 10. It creates a simple certificate for initial
+// setup. Currently, all values are hardcoded defaults. For future
+// versions, we may want to add more options for administrators.
+
+// It will create a PEM file containing a certificate with the following
+// values:
+// common name: localhost
+// organization: BIND10
+// country code: US
+
+// Additional error return codes; these are specifically
+// chosen to be distinct from validation error codes as
+// provided by Botan. Their main use is to distinguish
+// error cases in the unit tests.
+const int DECODING_ERROR = 100;
+const int BAD_OPTIONS = 101;
+const int READ_ERROR = 102;
+const int WRITE_ERROR = 103;
+const int UNKNOWN_ERROR = 104;
+const int NO_SUCH_FILE = 105;
+const int FILE_PERMISSION_ERROR = 106;
+
+void
+usage() {
+    std::cout << "Usage: b10-certgen [OPTION]..." << std::endl;
+    std::cout << "Validate, create, or update a self-signed certificate for "
+                 "use with b10-cmdctl" << std::endl;
+    std::cout << "" << std::endl;
+    std::cout << "Options:" << std::endl;
+    std::cout << "-c, --certfile=FILE\t\tfile to read or store the certificate"
+              << std::endl;
+    std::cout << "-f, --force\t\t\toverwrite existing certficate even if it"
+              << std::endl <<"\t\t\t\tis valid" << std::endl;
+    std::cout << "-h, --help\t\t\tshow this help" << std::endl;
+    std::cout << "-k, --keyfile=FILE\t\tfile to store the generated private key"
+              << std::endl;
+    std::cout << "-w, --write\t\t\tcreate a new certificate if the given file"
+              << std::endl << "\t\t\t\tdoes not exist, or if is is not valid"
+              << std::endl;
+    std::cout << "-q, --quiet\t\t\tprint no output when creating or validating"
+              << std::endl;
+}
+
+/// \brief Returns true if the given file exists
+///
+/// \param filename The file to check
+/// \return true if file exists
+bool
+fileExists(const std::string& filename) {
+    return (access(filename.c_str(), F_OK) == 0);
+}
+
+/// \brief Returns true if the given file exists and is readable
+///
+/// \param filename The file to check
+/// \return true if file exists and is readable
+bool
+fileIsReadable(const std::string& filename) {
+    return (access(filename.c_str(), R_OK) == 0);
+}
+
+/// \brief Returns true if the given file exists and is writable
+///
+/// \param filename The file to check
+/// \return true if file exists and is writable
+bool
+fileIsWritable(const std::string& filename) {
+    return (access(filename.c_str(), W_OK) == 0);
+}
+
+/// \brief Helper function for readable error output;
+///
+/// Returns string representation of X509 result code
+/// This does not appear to be provided by Botan itself
+///
+/// \param code An \c X509_Code instance
+/// \return A human-readable c string
+const char*
+X509CodeToString(const X509_Code& code) {
+    // note that this list provides more than we would
+    // need in this context, it is just the enum from
+    // the source code of Botan.
+    switch (code) {
+    case VERIFIED:
+        return ("verified");
+    case UNKNOWN_X509_ERROR:
+        return ("unknown x509 error");
+    case CANNOT_ESTABLISH_TRUST:
+        return ("cannot establish trust");
+    case CERT_CHAIN_TOO_LONG:
+        return ("certificate chain too long");
+    case SIGNATURE_ERROR:
+        return ("signature error");
+    case POLICY_ERROR:
+        return ("policy error");
+    case INVALID_USAGE:
+        return ("invalid usage");
+    case CERT_FORMAT_ERROR:
+        return ("certificate format error");
+    case CERT_ISSUER_NOT_FOUND:
+        return ("certificate issuer not found");
+    case CERT_NOT_YET_VALID:
+        return ("certificate not yet valid");
+    case CERT_HAS_EXPIRED:
+        return ("certificate has expired");
+    case CERT_IS_REVOKED:
+        return ("certificate has been revoked");
+    case CRL_FORMAT_ERROR:
+        return ("crl format error");
+    case CRL_NOT_YET_VALID:
+        return ("crl not yet valid");
+    case CRL_HAS_EXPIRED:
+        return ("crl has expired");
+    case CA_CERT_CANNOT_SIGN:
+        return ("CA cert cannot sign");
+    case CA_CERT_NOT_FOR_CERT_ISSUER:
+        return ("CA certificate not for certificate issuer");
+    case CA_CERT_NOT_FOR_CRL_ISSUER:
+        return ("CA certificate not for crl issuer");
+    default:
+        return ("Unknown X509 code");
+    }
+}
+
+class CertificateTool {
+public:
+    CertificateTool(bool quiet) : quiet_(quiet) {}
+
+    int
+    createKeyAndCertificate(const std::string& key_file_name,
+                            const std::string& cert_file_name) {
+        try {
+            AutoSeeded_RNG rng;
+
+            // Create and store a private key
+            print("Creating key file " + key_file_name);
+            RSA_PrivateKey key(rng, 2048);
+            std::ofstream key_file(key_file_name.c_str());
+            if (!key_file.good()) {
+                print(std::string("Error writing to ") + key_file_name +
+                      ": " + std::strerror(errno));
+                return (WRITE_ERROR);
+            }
+            key_file << PKCS8::PEM_encode(key, rng, "");
+            if (!key_file.good()) {
+                print(std::string("Error writing to ") + key_file_name +
+                      ": " + std::strerror(errno));
+                return (WRITE_ERROR);
+            }
+            key_file.close();
+
+            // Certificate options, currently hardcoded.
+            // For a future version we may want to make these
+            // settable.
+            X509_Cert_Options opts;
+            opts.common_name = "localhost";
+            opts.organization = "UNKNOWN";
+            opts.country = "XX";
+
+            opts.CA_key();
+
+            print("Creating certificate file " + cert_file_name);
+
+            // The exact call changed aftert 1.8, adding the
+            // hash function option
+#if BOTAN_VERSION_CODE >= BOTAN_VERSION_CODE_FOR(1,9,0)
+            X509_Certificate cert =
+            X509::create_self_signed_cert(opts, key, "SHA-256", rng);
+#else
+            X509_Certificate cert =
+            X509::create_self_signed_cert(opts, key, rng);
+#endif
+
+            std::ofstream cert_file(cert_file_name.c_str());
+            if (!cert_file.good()) {
+                print(std::string("Error writing to ") + cert_file_name +
+                      ": " + std::strerror(errno));
+                return (WRITE_ERROR);
+            }
+            cert_file << cert.PEM_encode();
+            if (!cert_file.good()) {
+                print(std::string("Error writing to ") + cert_file_name +
+                      ": " + std::strerror(errno));
+                return (WRITE_ERROR);
+            }
+            cert_file.close();
+        } catch(std::exception& e) {
+            std::cout << "Error creating key or certificate: " << e.what()
+                      << std::endl;
+            return (UNKNOWN_ERROR);
+        }
+        return (0);
+    }
+
+    int
+    validateCertificate(const std::string& certfile) {
+        // Since we are dealing with a self-signed certificate here, we
+        // also use the certificate to check itself; i.e. we add it
+        // as a trusted certificate, then validate the certficate itself.
+        //const X509_Certificate cert(certfile);
+        try {
+            X509_Store store;
+            DataSource_Stream in(certfile);
+            store.add_trusted_certs(in);
+
+            const X509_Code result = store.validate_cert(certfile);
+
+            if (result == VERIFIED) {
+                print(certfile + " is valid");
+            } else {
+                print(certfile + " failed to verify: " +
+                      X509CodeToString(result));
+            }
+            return (result);
+        } catch (const Botan::Decoding_Error& bde) {
+            print(certfile + " failed to verify: " + bde.what());
+            return (DECODING_ERROR);
+        } catch (const Botan::Stream_IO_Error& bsie) {
+            print(certfile + " not read: " + bsie.what());
+            return (READ_ERROR);
+        }
+    }
+
+    /// \brief Runs the tool
+    ///
+    /// \param create_cert  Create certificate if true, validate if false.
+    ///                     Does nothing if certificate exists and is valid.
+    /// \param force_create Create new certificate even if it is valid.
+    /// \param certfile     Certificate file to read to or write from.
+    /// \param keyfile      Key file to write if certificate is created.
+    ///                     Ignored if create_cert is false
+    /// \return zero on success, non-zero on failure
+    int
+    run(bool create_cert, bool force_create, const std::string& certfile,
+        const std::string& keyfile)
+    {
+        if (create_cert) {
+            // Unless force is given, only create it if the current
+            // one is not OK
+
+            // First do some basic permission checks; both files
+            // should either not exist, or be both readable
+            // and writable
+            // The checks are done one by one so all errors can
+            // be enumerated in one go
+            if (fileExists(certfile)) {
+                if (!fileIsReadable(certfile)) {
+                    print(certfile + " not readable: " + std::strerror(errno));
+                    create_cert = false;
+                }
+                if (!fileIsWritable(certfile)) {
+                    print(certfile + " not writable: " + std::strerror(errno));
+                    create_cert = false;
+                }
+            }
+            // The key file really only needs write permissions (for
+            // b10-certgen that is)
+            if (fileExists(keyfile)) {
+                if (!fileIsWritable(keyfile)) {
+                    print(keyfile + " not writable: " + std::strerror(errno));
+                    create_cert = false;
+                }
+            }
+            if (!create_cert) {
+                print("Not creating new certificate, "
+                      "check file permissions");
+                return (FILE_PERMISSION_ERROR);
+            }
+
+            // If we reach this, we know that if they exist, we can both
+            // read and write them, so now it's up to content checking
+            // and/or force_create
+
+            if (force_create || !fileExists(certfile) ||
+                validateCertificate(certfile) != VERIFIED) {
+                return (createKeyAndCertificate(keyfile, certfile));
+            } else {
+                print("Not creating a new certificate (use -f to force)");
+            }
+        } else {
+            if (!fileExists(certfile)) {
+                print(certfile + ": " + std::strerror(errno));
+                return (NO_SUCH_FILE);
+            }
+            if (!fileIsReadable(certfile)) {
+                print(certfile + " not readable: " + std::strerror(errno));
+                return (FILE_PERMISSION_ERROR);
+            }
+            int result = validateCertificate(certfile);
+            if (result != 0) {
+                print("Running with -w would overwrite the certificate");
+            }
+            return (result);
+        }
+        return (0);
+    }
+private:
+    /// Prints the message to stdout unless quiet_ is true
+    void print(const std::string& msg) {
+        if (!quiet_) {
+            std::cout << msg << std::endl;
+        }
+    }
+
+    bool quiet_;
+};
+
+int
+main(int argc, char* argv[])
+{
+    Botan::LibraryInitializer init;
+
+    // create or check certificate
+    bool create_cert = false;
+    // force creation even if not necessary
+    bool force_create = false;
+    // don't print any output
+    bool quiet = false;
+
+    // default certificate file
+    std::string certfile("cmdctl-certfile.pem");
+    // default key file
+    std::string keyfile("cmdctl-keyfile.pem");
+
+    // whether or not the above values have been
+    // overridden (used in command line checking)
+    bool certfile_default = true;
+    bool keyfile_default = true;
+
+    // It would appear some environments insist on
+    // char* here (Sunstudio on Solaris), so we const_cast
+    // them to get rid of compiler warnings.
+    const struct option long_options[] = {
+        { const_cast<char*>("certfile"), required_argument, NULL, 'c' },
+        { const_cast<char*>("force"), no_argument, NULL, 'f' },
+        { const_cast<char*>("help"), no_argument, NULL, 'h' },
+        { const_cast<char*>("keyfile"), required_argument, NULL, 'k' },
+        { const_cast<char*>("write"), no_argument, NULL, 'w' },
+        { const_cast<char*>("quiet"), no_argument, NULL, 'q' },
+        { NULL, 0, NULL, 0 }
+    };
+
+    int opt, option_index;
+    while ((opt = getopt_long(argc, argv, "c:fhk:wq", long_options,
+                              &option_index)) != -1) {
+        switch (opt) {
+            case 'c':
+                certfile = optarg;
+                certfile_default = false;
+                break;
+            case 'f':
+                force_create = true;
+                break;
+            case 'h':
+                usage();
+                return (0);
+                break;
+            case 'k':
+                keyfile = optarg;
+                keyfile_default = false;
+                break;
+            case 'w':
+                create_cert = true;
+                break;
+            case 'q':
+                quiet = true;
+                break;
+            default:
+                // A message will have already been output about the error.
+                return (BAD_OPTIONS);
+        }
+    }
+
+    if (optind < argc) {
+        std::cout << "Error: extraneous arguments" << std::endl << std::endl;
+        usage();
+        return (BAD_OPTIONS);
+    }
+
+    // Some sanity checks on option combinations
+    if (create_cert && (certfile_default ^ keyfile_default)) {
+        std::cout << "Error: keyfile and certfile must both be specified "
+                     "if one of them is when calling b10-certgen in write "
+                     "mode." << std::endl;
+        return (BAD_OPTIONS);
+    }
+    if (!create_cert && !keyfile_default) {
+        std::cout << "Error: keyfile is not used when not in write mode"
+                  << std::endl;
+        return (BAD_OPTIONS);
+    }
+
+    // Initialize the tool and perform the appropriate action(s)
+    CertificateTool tool(quiet);
+    return (tool.run(create_cert, force_create, certfile, keyfile));
+}

src/bin/cmdctl/b10-certgen.xml

@@ -0,0 +1,214 @@
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+               "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
+	       [<!ENTITY mdash "&#8212;">]>
+<!--
+ - Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+ -
+ - Permission to use, copy, modify, and/or distribute this software for any
+ - purpose with or without fee is hereby granted, provided that the above
+ - copyright notice and this permission notice appear in all copies.
+ -
+ - THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+ - REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+ - AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+ - INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+ - LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+ - OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ - PERFORMANCE OF THIS SOFTWARE.
+-->
+
+<refentry>
+
+  <refentryinfo>
+    <date>November 15, 2012</date>
+  </refentryinfo>
+
+  <refmeta>
+    <refentrytitle>b10-certgen</refentrytitle>
+    <manvolnum>1</manvolnum>
+    <refmiscinfo>BIND10</refmiscinfo>
+  </refmeta>
+
+  <refnamediv>
+    <refname>b10-certgen</refname>
+    <refpurpose>X509 Certificate generation tool for use with b10-cmdctl</refpurpose>
+  </refnamediv>
+
+  <docinfo>
+    <copyright>
+      <year>2012</year>
+      <holder>Internet Systems Consortium, Inc. ("ISC")</holder>
+    </copyright>
+  </docinfo>
+
+  <refsynopsisdiv>
+    <cmdsynopsis>
+      <command>b10-certgen</command>
+        <group choice="opt">
+          <arg choice="[OPTION]..."><option>-</option></arg>
+        </group>
+    </cmdsynopsis>
+  </refsynopsisdiv>
+
+  <refsect1>
+    <title>DESCRIPTION</title>
+    <para>The <command>b10-certgen</command> tool validates, creates, or
+      updates a self-signed X509 certificate for use in b10-cmdctl.
+    </para>
+
+    <para>
+      The connection between <command>bindctl</command> and
+      <command>b10-cmdctl</command> is done over HTTPS, and therefore
+      <command>b10-cmdctl</command> needs a certificate. Since these
+      certificates have expiry dates, they also need to be regenerated at
+      some point.
+
+      There are many tools to do so, but for ease of use, <command>
+      b10-certgen</command> can create a simple self-signed certificate.
+
+      By default, it will not create anything, but it will merely check an
+      existing certificate (if not specified, cmdctl-certfile.pem, in the
+      current working directory). And print whether it is valid, and
+      whether it would update if the option '-w' is given.
+
+      With that option, the certificate could then be replaced by a newly
+      created one. If the certificate is still valid, it would still not
+      be overwritten (however, if it is found to be invalid, for example
+      because it has expired, it would create a new one).
+
+      A new certificate is always created if the certificate file does
+      not exist, or if creation is forced (with the -f option).
+    </para>
+  </refsect1>
+
+  <refsect1>
+    <title>ARGUMENTS</title>
+
+    <para>The arguments are as follows:</para>
+
+    <variablelist>
+
+      <varlistentry>
+        <term>
+          <option>-c <replaceable>file</replaceable></option>,
+          <option>--certfile=<replaceable>file</replaceable></option>
+        </term>
+        <listitem>
+          <para>
+            File to read the certificate from, or write the certificate to.
+            If <option>-w</option> and <option>-c</option> are used,
+            <option>-k</option> is mandatory as well.
+          </para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term>
+          <option>-f</option>,
+          <option>--force</option>
+        </term>
+        <listitem>
+          <para>
+            Force updating of certificate when <option>-w</option> is used,
+            even if the existing certificate is still valid.
+          </para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term>
+          <option>-h</option>,
+          <option>--help</option>
+        </term>
+        <listitem>
+          <para>
+            Print the command line arguments and exit.
+          </para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term>
+          <option>-k <replaceable>file</replaceable></option>,
+          <option>--keyfile=<replaceable>file</replaceable></option>
+        </term>
+        <listitem>
+          <para>
+            File to write the private key to. This option is only valid when <option>-w</option> is used, and if this option is used, <option>-c</option> is mandatory as well.
+          </para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term>
+          <option>-w</option>,
+          <option>--write</option>
+        </term>
+        <listitem>
+          <para>
+            Check the given certificate file. If it does not exist, a new
+            private key and certificate are created. If it does exist, the
+            certificate is validated. If it is not valid (for instance
+            because it has expired), it is overwritten with a newly created
+            certificate. If it is valid, nothing happens (use
+            <option>-f</option> to force an update in that case).
+          </para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term>
+          <option>-q</option>,
+          <option>--quiet</option>
+        </term>
+        <listitem>
+          <para>
+            Don't print informational messages (only command-line errors are
+            printed). Useful in scripts when only the return code is needed.
+          </para>
+        </listitem>
+      </varlistentry>
+    </variablelist>
+  </refsect1>
+
+  <refsect1>
+    <title>SEE ALSO</title>
+    <para>
+      <citerefentry>
+        <refentrytitle>b10-cmdctl</refentrytitle><manvolnum>8</manvolnum>
+      </citerefentry>,
+      <citetitle>BIND 10 Guide</citetitle>.
+    </para>
+  </refsect1>
+
+  <refsect1>
+    <title>HISTORY</title>
+    <para>
+      The <command>b10-certgen</command> tool was first implemented
+      in November 2012 for the ISC BIND 10 project.
+    </para>
+  </refsect1>
+
+  <refsect1>
+    <title>EXAMPLE</title>
+    <para>
+      To update an expired certificate in BIND 10 that has been installed to
+      /usr/local:
+      <screen>
+$> cd /usr/local/etc/bind10-devel/
+
+$> b10-certgen
+cmdctl-certfile.pem failed to verify: certificate has expired
+Running with -w would overwrite the certificate
+
+$> b10-certgen --write
+cmdctl-certfile.pem failed to verify: certificate has expired
+Creating key file cmdctl-keyfile.pem
+Creating certificate file cmdctl-certfile.pem
+
+$> b10-certgen --write
+cmdctl-certfile.pem is valid
+Not creating a new certificate (use -f to force)
+      </screen>
+    </para>
+  </refsect1>
+</refentry><!--
+ - Local variables:
+ - mode: sgml
+ - End:
+-->

src/bin/cmdctl/cmdctl-keyfile.pem

@@ -1,15 +0,0 @@
------BEGIN RSA PRIVATE KEY-----
-MIICXAIBAAKBgQDpICWxJGKMvUhLFPbf5n8ZWogqjYcQqqoHqHVRHYjyiey6FZdt
-ZkY2s1gYh0G0NXtimlIgic+vEcFe7vdmyKntW7DYDaqAj0KrED7RKAj8324jNbSJ
-HtLP4evvJep3vsoNtTvNuceQJ46vukxyxgg3DuC9kVqPuD8CZ1Rq4ATyiwIDAQAB
-AoGBAOJlOtV+DUq6Y2Ou91VXRiU8GzKgAQP5iWgoe84Ljbxkn4XThBxVD2j94Fbp
-u7AjpDCMx6cbzpoo9w6XqaGizAmAehIfTE3eFYs74N/FM09Wg2OSDyxMY0jgyECU
-A4ukjlPwcGDbmgbmlY3i+FVHp+zCgtZEsMC1IAosMac1BoX5AkEA/lrXWaVtH8bo
-mut3GBaXvubZMdaUr0BUd5a9q+tt4dQcKG1kFqgCNKhNhBIcpiMVcz+jGmOuopNA
-8dnUGqv3FQJBAOqiJ54ZvOTWNDpJIe02wIXRxRmc1xhHFCqYP23KxBVrAcTYB19J
-lesov/hEbnGLCbKS/naZJ1zrTImUPNRLqx8CQCzDtA7U7GWhTiKluioFH+O7IRKC
-X1yQh80cPHlbT9VkzSfYSLssCmdWD35k6aHbntTPqFbmoD+AhveJjKi9BxkCQDwX
-1c+/RcrSNcQr0N2hZUOgyztZGRnlsnuKTMyA3yGhK23P6mt0PEpjQG+Ej0jTVGOB
-FF0pspQwy4R9C+tPif8CQH36NNlXBfVNmT7kDtyLmaE6pID0vY9duX56BJbU1R0x
-SQ8/LcfJagk6gvp08OyYCPA+WZ7u/bas9R/nMTCLivc=
------END RSA PRIVATE KEY-----

src/bin/cmdctl/tests/Makefile.am

@@ -1,6 +1,9 @@
 PYCOVERAGE_RUN=@PYCOVERAGE_RUN@
-PYTESTS = cmdctl_test.py
+PYTESTS = cmdctl_test.py b10-certgen_test.py
 EXTRA_DIST = $(PYTESTS)
+EXTRA_DIST += testdata/expired-certfile.pem
+EXTRA_DIST += testdata/mangled-certfile.pem
+EXTRA_DIST += testdata/noca-certfile.pem
 
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
@@ -9,10 +12,12 @@ if SET_ENV_LIBRARY_PATH
 LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
+CLEANFILES = test-keyfile.pem test-certfile.pem
+
 # test using command-line arguments, so use check-local target instead of TESTS
 check-local:
 if ENABLE_PYTHON_COVERAGE
-	touch $(abs_top_srcdir)/.coverage 
+	touch $(abs_top_srcdir)/.coverage
 	rm -f .coverage
 	${LN_S} $(abs_top_srcdir)/.coverage .coverage
 endif

src/bin/cmdctl/tests/b10-certgen_test.py

@@ -0,0 +1,254 @@
+# Copyright (C) 2012  Internet Systems Consortium.
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
+# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
+# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
+# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
+# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+# Note: the main code is in C++, but what we are mostly testing is
+# options and behaviour (output/file creation, etc), which is easier
+# to test in python.
+
+import unittest
+import os
+from subprocess import call
+import subprocess
+import ssl
+import stat
+
+def run(command):
+    """
+    Small helper function that returns a tuple of (rcode, stdout, stderr) after
+    running the given command (an array of command and arguments, as passed on
+    to subprocess).
+    """
+    subp = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = subp.communicate()
+    return (subp.returncode, stdout, stderr)
+
+class FileDeleterContext:
+    """
+    Simple Context Manager that deletes a given set of files when the context
+    is left.
+    """
+    def __init__(self, files):
+        self.files = files
+
+    def __enter__(self):
+        pass
+
+    def __exit__(self, type, value, traceback):
+        for f in self.files:
+            if os.path.exists(f):
+                os.unlink(f)
+
+class FilePermissionContext:
+    """
+    Simple Context Manager that temporarily modifies file permissions for
+    a given file
+    """
+    def __init__(self, f, unset_flags = [], set_flags = []):
+        """
+        Initialize file permission context.
+        See the stat module for possible flags to set or unset.
+        The flags are changed when the context is entered (i.e.
+        you can create the context first without any change)
+        The flags are changed back when the context is left.
+
+        Parameters:
+        f: string, file to change permissions for
+        unset_flags: list of flags to unset
+        set_flags: list of flags to set
+        """
+        self.file = f
+        self.orig_mode = os.stat(f).st_mode
+        new_mode = self.orig_mode
+        for flag in unset_flags:
+            new_mode = new_mode & ~flag
+        for flag in set_flags:
+            new_mode = new_mode | flag
+        self.new_mode = new_mode
+
+    def __enter__(self):
+        os.chmod(self.file, self.new_mode)
+
+    def __exit__(self, type, value, traceback):
+        os.chmod(self.file, self.orig_mode)
+
+def read_file_data(filename):
+    """
+    Simple text file reader that returns its contents as an array
+    """
+    with open(filename) as f:
+        return f.readlines()
+
+class TestCertGenTool(unittest.TestCase):
+    TOOL = '../b10-certgen'
+
+    def run_check(self, expected_returncode, expected_stdout, expected_stderr, command):
+        """
+        Runs the given command, and checks return code, and outputs (if provided).
+        Arguments:
+        expected_returncode, return code of the command
+        expected_stdout, (multiline) string that is checked agains stdout.
+                         May be None, in which case the check is skipped.
+        expected_stderr, (multiline) string that is checked agains stderr.
+                         May be None, in which case the check is skipped.
+        """
+        (returncode, stdout, stderr) = run(command)
+        self.assertEqual(expected_returncode, returncode, " ".join(command))
+        if expected_stdout is not None:
+            self.assertEqual(expected_stdout, stdout.decode())
+        if expected_stderr is not None:
+            self.assertEqual(expected_stderr, stderr.decode())
+
+    def validate_certificate(self, expected_result, certfile):
+        """
+        Validate a certificate, using the quiet option of the tool; it runs
+        the check option (-c) for the given base name of the certificate (-f
+        <certfile>), and compares the return code to the given
+        expected_result value
+        """
+        self.run_check(expected_result, '', '',
+                       [self.TOOL, '-q', '-c', certfile])
+        # Same with long options
+        self.run_check(expected_result, '', '',
+                       [self.TOOL, '--quiet', '--certfile', certfile])
+
+
+    def test_basic_creation(self):
+        """
+        Tests whether basic creation with no arguments (except output
+        file name) successfully creates a key and certificate
+        """
+        keyfile = 'test-keyfile.pem'
+        certfile = 'test-certfile.pem'
+        command = [ self.TOOL, '-q', '-w', '-c', certfile, '-k', keyfile ]
+        self.creation_helper(command, certfile, keyfile)
+        # Do same with long options
+        command = [ self.TOOL, '--quiet', '--write', '--certfile=' + certfile, '--keyfile=' + keyfile ]
+        self.creation_helper(command, certfile, keyfile)
+
+    def creation_helper(self, command, certfile, keyfile):
+        """
+        Helper method for test_basic_creation.
+        Performs the actual checks
+        """
+        with FileDeleterContext([keyfile, certfile]):
+            self.assertFalse(os.path.exists(keyfile))
+            self.assertFalse(os.path.exists(certfile))
+            self.run_check(0, '', '', command)
+            self.assertTrue(os.path.exists(keyfile))
+            self.assertTrue(os.path.exists(certfile))
+
+            # Validate the certificate that was just created
+            self.validate_certificate(0, certfile)
+
+            # When run with the same options, it should *not* create it again,
+            # as the current certificate should still be valid
+            certdata = read_file_data(certfile)
+            keydata = read_file_data(keyfile)
+
+            self.run_check(0, '', '', command)
+
+            self.assertEqual(certdata, read_file_data(certfile))
+            self.assertEqual(keydata, read_file_data(keyfile))
+
+            # but if we add -f, it should force a new creation
+            command.append('-f')
+            self.run_check(0, '', '', command)
+            self.assertNotEqual(certdata, read_file_data(certfile))
+            self.assertNotEqual(keydata, read_file_data(keyfile))
+
+    def test_check_bad_certificates(self):
+        """
+        Tests a few pre-created certificates with the -c option
+        """
+        if ('CMDCTL_SRC_PATH' in os.environ):
+            path = os.environ['CMDCTL_SRC_PATH'] + "/tests/testdata/"
+        else:
+            path = "testdata/"
+        self.validate_certificate(10, path + 'expired-certfile.pem')
+        self.validate_certificate(100, path + 'mangled-certfile.pem')
+        self.validate_certificate(17, path + 'noca-certfile.pem')
+
+    def test_bad_options(self):
+        """
+        Tests some combinations of commands that should fail.
+        """
+        # specify -c but not -k
+        self.run_check(101,
+                       'Error: keyfile and certfile must both be specified '
+                       'if one of them is when calling b10-certgen in write '
+                       'mode.\n',
+                       '', [self.TOOL, '-w', '-c', 'foo'])
+        self.run_check(101,
+                       'Error: keyfile and certfile must both be specified '
+                       'if one of them is when calling b10-certgen in write '
+                       'mode.\n',
+                       '', [self.TOOL, '-w', '-k', 'foo'])
+        self.run_check(101,
+                       'Error: keyfile is not used when not in write mode\n',
+                       '', [self.TOOL, '-k', 'foo'])
+        # Extraneous argument
+        self.run_check(101, None, None, [self.TOOL, 'foo'])
+        # No such file
+        self.run_check(105, None, None, [self.TOOL, '-c', 'foo'])
+
+    def test_permissions(self):
+        """
+        Test some combinations of correct and bad permissions.
+        """
+        keyfile = 'mod-keyfile.pem'
+        certfile = 'mod-certfile.pem'
+        command = [ self.TOOL, '-q', '-w', '-c', certfile, '-k', keyfile ]
+        # Delete them at the end
+        with FileDeleterContext([keyfile, certfile]):
+            # Create the two files first
+            self.run_check(0, '', '', command)
+            self.validate_certificate(0, certfile)
+
+            # Make the key file unwritable
+            with FilePermissionContext(keyfile, unset_flags = [stat.S_IWUSR]):
+                self.run_check(106, '', '', command)
+                # Should have no effect on validation
+                self.validate_certificate(0, certfile)
+
+            # Make the cert file unwritable
+            with FilePermissionContext(certfile, unset_flags = [stat.S_IWUSR]):
+                self.run_check(106, '', '', command)
+                # Should have no effect on validation
+                self.validate_certificate(0, certfile)
+
+            # Make the key file unreadable (this should not matter)
+            with FilePermissionContext(keyfile, unset_flags = [stat.S_IRUSR]):
+                self.run_check(0, '', '', command)
+
+                # unreadable key file should also not have any effect on
+                # validation
+                self.validate_certificate(0, certfile)
+
+            # Make the cert file unreadable (this should matter)
+            with FilePermissionContext(certfile, unset_flags = [stat.S_IRUSR]):
+                self.run_check(106, '', '', command)
+
+                # Unreadable cert file should also fail validation
+                self.validate_certificate(106, certfile)
+
+        # Not directly a permission problem, but trying to check or create
+        # in a nonexistent directory returns different error codes
+        self.validate_certificate(105, 'fakedir/cert')
+        self.run_check(103, '', '', [ self.TOOL, '-q', '-w', '-c',
+                                      'fakedir/cert', '-k', 'fakedir/key' ])
+
+if __name__== '__main__':
+    unittest.main()
+

src/bin/cmdctl/tests/testdata/mangled-certfile.pem

@@ -0,0 +1,21 @@
+-----BEGIN CERTIFICATE-----
+MIIDhzCCAvCgAwIBAgIJALwngNFik7ONMA0GCSqGSIb3DQEBBQUAMIGKMQswCQYD
+VQQGEwJjbjEQMA4GA1UECBMHYmVpamluZzEQMA4GA1UEBxMHYmVpamluZzEOMAwG
+A1UEChMFY25uaWMxDjAMBgNVBAsTBWNubmljMRMwEQYDVQQDEwp6aGFuZ2xpa3Vu
+MSIwIAYJKoZIhvcNAQkBFhN6aGFuZ2xpa3VuQGNubmljLmNuMB4XDTEwMDEwNzEy
+NDcxOFoXDTExMDEwNzEyNDcxOFowgYoxCzAJBgNVBAYTAmNuMRAwDgYDVQQIEwdi
+ZWlqaW5nMraWDgYDVQQHEwdiZWlqaW5nMQ4wDAYDVQQKEwVjbm5pYzEOMAwGA1UE
+CxMFY25uaWMxeZaRBgNVBAMTCnpoYW5nbGlrdW4xIjAgBgkqhkiG9w0BCQEWE3po
+YW5nbGlrdW5AY25UAwMuY24wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAOkg
+JbEkYoy9SEsU9t/mfxLAICqNhxCqqgeodVEdiPKJ7LoVl21mRjazWBiHQbQ1e2Ka
+UiCJz68RwV7u92bIqe1bsNgNqoCPQqsQPtEoCPzfbiM1tIke0s/h6+8l6ne+yg21
+O825x5Anjq+6THLGCDcO4L2RWo+4PwJnVGrgBPKLAgMBAAGjgfIwge8wHQYDVR0O
+BBYEFJKM/O0ViGlwtb3JEci/DLTO/7DaMIG/BgNVHSMEgbcwgbSAFJKM/O0ViGlw
+tb3JEci/DLTO/7DaoYGQpIGNMIGKMQswCQYDVQQGEwJjbjEQMA4GA1UECBMHYmVp
+amluZzEQMA4GA1UEBxMHYmVpamluZzEOMAwGA1UEChMFY25uaWMxDjAMBgNVBAsT
+BWNubmljMRMwEQYDVQQDEwp6aGFuZ2xpa3VuMSIwIAYJKoZIhvcNAQkBFhN6aGFu
+Z2xpa3VuQGNubmljLmNuggkAvCeA0WKTs40wDAYDVR0TBAUwAwEB/zANBgkqhkiG
+9w0BAQUFAAOBgQBh5N6isMAQAFFD+pbfpppjQlO4vUNcEdzPdeuBFaf9CsX5ZdxV
+jmn1ZuGm6kRzqUPwPSxvCIAY0wuSu1g7YREPAZ3XBVwcg6262iGOA6n7E+nv5PLz
+EuZ1oUg+IfykUIoflKH6xZB4MyPL+EgkMT+i9BrngaXHXF8tEO30YppMiA==
+-----END CERTIFICATE-----

src/bin/cmdctl/tests/testdata/noca-certfile.pem

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDBjCCAe6gAwIBAgIRALUIj3nnW5uDE/+fglPvUDwwDQYJKoZIhvcNAQELBQAw
+HjELMAkGA1UEBhMCVVMxDzANBgNVBAMTBkJJTkQxMDAeFw0xMjExMTQxMjQ5MjVa
+Fw0xMzExMTQxMjQ5MjVaMB4xCzAJBgNVBAYTAlVTMQ8wDQYDVQQDEwZCSU5EMTAw
+ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDkIOPfs3Aw9kNDu1JqA2w3
+84/n9oUgAwAlHVmuJv7ZDw1MDaIKHjsh3DW09z+nv67GVksI7pFtAw5O4mnTDxpa
+JT0NKzhvYGfe8VdV/hWDogTIdk1QBJNZ2/id8z0h8z5001sARXPf+4mHBJslenH3
+YtZs22BG5RBLULtZ/2Nr7JkdfLlc6D5PCoDG22r1OiFkYVdCWfLDjisVIbSYPBtY
+BlKAIrvbmOtWcaGM+vQAhl0T5N8WRCKhaQH0DEmzQNckkYd7rSECo57KYiuvOdzp
+d+3bWTgGGy2ff0o3LZypv0O5s0TDC2H6hYtN4bUbcChUJbFu9b5sVZaOEVZtUsyD
+AgMBAAGjPzA9MAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQW
+BBSqGzsEDNs9E7gBL5pD6XVAwUo4DTANBgkqhkiG9w0BAQsFAAOCAQEAMTNB8NCU
+dnLFZ0jNpvecbECkX/OWGlBYU4/CsoNiibwp4CtUYS2A4NFVjWAyuzLSHhRQi0vJ
+CCWLpKL4VTkaDN5Oft42iUhvEXMnriJqpfXHnjCiBwFFSPl5WKfMIaRNK+tF4zbB
+F+FGNEEmYG3t/ni82orDLq4oy+7CoQwzZNzj5yoV6q7O9kLR9OMPNwJrc27A4erB
+7VMRZslSrNA4uA6YhMZl8iEvO1H801ct0zTxawrCihPOZOCSLew35xjztO7d3YH8
+YavOu5kzeu7AgZ2n75H/qU47ZgBjbonn9Osvrct+RIwZuWTB2bDML8JhNaZCq0aA
+TDBC0QWqIYypLg==
+-----END CERTIFICATE-----

src/bin/dhcp4/ctrl_dhcp4_srv.cc

@@ -14,21 +14,20 @@
 
 #include <config.h>
 
-#include <cassert>
-#include <iostream>
-
 #include <asiolink/asiolink.h>
 #include <cc/data.h>
 #include <cc/session.h>
-#include <cc/session.h>
 #include <config/ccsession.h>
+#include <dhcp/iface_mgr.h>
 #include <dhcp4/ctrl_dhcp4_srv.h>
 #include <dhcp4/dhcp4_log.h>
 #include <dhcp4/spec_config.h>
-#include <dhcp/iface_mgr.h>
 #include <exceptions/exceptions.h>
 #include <util/buffer.h>
 
+#include <cassert>
+#include <iostream>
+
 using namespace isc::asiolink;
 using namespace isc::cc;
 using namespace isc::config;

src/bin/dhcp4/ctrl_dhcp4_srv.h

@@ -15,11 +15,11 @@
 #ifndef CTRL_DHCPV4_SRV_H
 #define CTRL_DHCPV4_SRV_H
 
-#include <dhcp4/dhcp4_srv.h>
 #include <asiolink/asiolink.h>
+#include <cc/data.h>
 #include <cc/session.h>
 #include <config/ccsession.h>
-#include <cc/data.h>
+#include <dhcp4/dhcp4_srv.h>
 
 namespace isc {
 namespace dhcp {

src/bin/dhcp4/dhcp4_log.cc

@@ -14,7 +14,7 @@
 
 /// Defines the logger used by the top-level component of b10-dhcp4.
 
-#include "dhcp4_log.h"
+#include <dhcp4/dhcp4_log.h>
 
 namespace isc {
 namespace dhcp {

src/bin/dhcp4/dhcp4_log.h

@@ -15,8 +15,8 @@
 #ifndef DHCP4_LOG_H
 #define DHCP4_LOG_H
 
-#include <log/macros.h>
 #include <log/logger_support.h>
+#include <log/macros.h>
 #include <dhcp4/dhcp4_messages.h>
 
 namespace isc {

src/bin/dhcp4/dhcp4_messages.mes

@@ -42,17 +42,17 @@ server is about to open sockets on the specified port.
 The IPv4 DHCP server has received a packet that it is unable to
 interpret. The reason why the packet is invalid is included in the message.
 
-% DHCP4_PACKET_RECEIVE_FAIL error on attempt to receive packet: %1
-The IPv4 DHCP server tried to receive a packet but an error
-occured during this attempt. The reason for the error is included in
-the message.
-
 % DHCP4_PACKET_RECEIVED %1 (type %2) packet received on interface %3
 A debug message noting that the server has received the specified type of
 packet on the specified interface.  Note that a packet marked as UNKNOWN
 may well be a valid DHCP packet, just a type not expected by the server
 (e.g. it will report a received OFFER packet as UNKNOWN).
 
+% DHCP4_PACKET_RECEIVE_FAIL error on attempt to receive packet: %1
+The IPv4 DHCP server tried to receive a packet but an error
+occured during this attempt. The reason for the error is included in
+the message.
+
 % DHCP4_PACKET_SEND_FAIL failed to send DHCPv4 packet: %1
 This error is output if the IPv4 DHCP server fails to send an assembled
 DHCP message to a client. The reason for the error is included in the

src/bin/dhcp4/dhcp4_srv.cc

@@ -12,13 +12,13 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
+#include <asiolink/io_address.h>
 #include <dhcp/dhcp4.h>
-#include <dhcp/pkt4.h>
 #include <dhcp/iface_mgr.h>
-#include <dhcp4/dhcp4_srv.h>
-#include <dhcp4/dhcp4_log.h>
-#include <asiolink/io_address.h>
 #include <dhcp/option4_addrlst.h>
+#include <dhcp/pkt4.h>
+#include <dhcp4/dhcp4_log.h>
+#include <dhcp4/dhcp4_srv.h>
 
 using namespace isc;
 using namespace isc::asiolink;

src/bin/dhcp4/dhcp4_srv.h

@@ -15,10 +15,12 @@
 #ifndef DHCPV4_SRV_H
 #define DHCPV4_SRV_H
 
-#include <boost/noncopyable.hpp>
 #include <dhcp/dhcp4.h>
 #include <dhcp/pkt4.h>
 #include <dhcp/option.h>
+
+#include <boost/noncopyable.hpp>
+
 #include <iostream>
 
 namespace isc {

src/bin/dhcp4/main.cc

@@ -13,14 +13,15 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
-#include <iostream>
-
-#include <boost/lexical_cast.hpp>
 
 #include <dhcp4/ctrl_dhcp4_srv.h>
 #include <dhcp4/dhcp4_log.h>
 #include <log/logger_support.h>
 
+#include <boost/lexical_cast.hpp>
+
+#include <iostream>
+
 using namespace isc::dhcp;
 using namespace std;
 

src/bin/dhcp4/tests/ctrl_dhcp4_srv_unittest.cc

@@ -13,16 +13,18 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
-#include <iostream>
+
+#include <config/ccsession.h>
+#include <dhcp/dhcp4.h>
+#include <dhcp4/ctrl_dhcp4_srv.h>
+
+#include <gtest/gtest.h>
+
 #include <fstream>
+#include <iostream>
 #include <sstream>
 
 #include <arpa/inet.h>
-#include <gtest/gtest.h>
-
-#include <dhcp/dhcp4.h>
-#include <dhcp4/ctrl_dhcp4_srv.h>
-#include <config/ccsession.h>
 
 using namespace std;
 using namespace isc;

src/bin/dhcp4/tests/dhcp4_srv_unittest.cc

@@ -13,17 +13,19 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
-#include <iostream>
-#include <fstream>
 #include <sstream>
 
-#include <arpa/inet.h>
-#include <gtest/gtest.h>
-
+#include <asiolink/io_address.h>
 #include <dhcp/dhcp4.h>
-#include <dhcp4/dhcp4_srv.h>
 #include <dhcp/option.h>
-#include <asiolink/io_address.h>
+#include <dhcp4/dhcp4_srv.h>
+
+#include <gtest/gtest.h>
+
+#include <fstream>
+#include <iostream>
+
+#include <arpa/inet.h>
 
 using namespace std;
 using namespace isc;

src/bin/dhcp4/tests/dhcp4_test.py

@@ -45,11 +45,30 @@ class TestDhcpv4Daemon(unittest.TestCase):
     def tearDown(self):
         pass
 
+    def readPipe(self, pipe_fd):
+        """
+        Reads bytes from a pipe and returns a character string.  If nothing is
+        read, or if there is an error, an empty string is returned.
+
+        pipe_fd - Pipe file descriptor to read
+        """
+        try:
+            data = os.read(pipe_fd, 16384)
+            # Make sure we have a string
+            if (data is None):
+                data = ""
+            else:
+                data = str(data)
+        except OSError:
+            data = ""
+
+        return data
+
     def runCommand(self, params, wait=1):
         """
-        This method runs dhcp4 and returns a tuple: (returncode, stdout, stderr)
+        This method runs a command and returns a tuple: (returncode, stdout, stderr)
         """
-        ## @todo: Convert this into generic method and reuse it in dhcp6
+        ## @todo: Convert this into generic method and reuse it in dhcp4 and dhcp6
 
         print("Running command: %s" % (" ".join(params)))
 
@@ -89,46 +108,48 @@ class TestDhcpv4Daemon(unittest.TestCase):
         fl = fcntl.fcntl(fd, fcntl.F_GETFL)
         fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
 
-        # There's potential problem if b10-dhcp4 prints out more
-        # than 16kB of text
-        try:
-            output = os.read(self.stdout_pipes[0], 16384)
-        except OSError:
-            print("No data available from stdout")
-            output = ""
-
-        # read can return None. Make sure we have a string
-        if (output is None):
-            output = ""
-
-        try:
-            error = os.read(self.stderr_pipes[0], 16384)
-        except OSError:
-            print("No data available on stderr")
-            error = ""
-
-        # read can return None. Make sure we have a string
-        if (error is None):
-            error = ""
-
-
-        try:
-            if (not pi.process.poll()):
-                # let's be nice at first...
+        # As we don't know how long the subprocess will take to start and
+        # produce output, we'll loop and sleep for 250 ms between each
+        # iteration.  To avoid an infinite loop, we'll loop for a maximum
+        # of five seconds: that should be enough.
+        for count in range(20):
+            # Read something from stderr and stdout (these reads don't block).
+            output = self.readPipe(self.stdout_pipes[0])
+            error  = self.readPipe(self.stderr_pipes[0])
+
+            # If the process has already exited, or if it has output something,
+            # quit the loop now.
+            if pi.process.poll() is not None or len(error) > 0 or len(output) > 0:
+                break
+
+            # Process still running, try again in 250 ms.
+            time.sleep(0.25)
+
+        # Exited loop, kill the process if it is still running
+        if pi.process.poll() is None:
+            try:
                 pi.process.terminate()
-        except OSError:
-            print("Ignoring failed kill attempt. Process is dead already.")
+            except OSError:
+                print("Ignoring failed kill attempt. Process is dead already.")
 
         # call this to get returncode, process should be dead by now
         rc = pi.process.wait()
 
         # Clean up our stdout/stderr munging.
         os.dup2(self.stdout_old, sys.stdout.fileno())
+        os.close(self.stdout_old)
         os.close(self.stdout_pipes[0])
 
         os.dup2(self.stderr_old, sys.stderr.fileno())
+        os.close(self.stderr_old)
         os.close(self.stderr_pipes[0])
 
+        # Free up resources (file descriptors) from the ProcessInfo object
+        # TODO: For some reason, this gives an error if the process has ended,
+        #       although it does cause all descriptors still allocated to the
+        #       object to be freed.
+        pi = None
+
         print ("Process finished, return code=%d, stdout=%d bytes, stderr=%d bytes"
                % (rc, len(output), len(error)) )
 

src/bin/dhcp4/tests/dhcp4_unittests.cc

@@ -12,10 +12,10 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <stdio.h>
-#include <gtest/gtest.h>
 #include <log/logger_support.h>
 
+#include <gtest/gtest.h>
+
 int
 main(int argc, char* argv[]) {
 

src/bin/dhcp6/Makefile.am

@@ -64,7 +64,7 @@ b10_dhcp6_LDADD += $(top_builddir)/src/lib/util/libb10-util.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/asiolink/libb10-asiolink.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/log/libb10-log.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/dhcp/libb10-dhcp++.la
-b10_dhcp6_LDADD += $(top_builddir)/src/lib/dhcp/libb10-dhcpsrv.la
+b10_dhcp6_LDADD += $(top_builddir)/src/lib/dhcpsrv/libb10-dhcpsrv.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/config/libb10-cfgclient.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/cc/libb10-cc.la
 

src/bin/dhcp6/config_parser.cc

@@ -12,26 +12,30 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <stdint.h>
-#include <iostream>
-#include <vector>
-#include <map>
-#include <boost/foreach.hpp>
-#include <boost/shared_ptr.hpp>
-#include <boost/scoped_ptr.hpp>
-#include <boost/lexical_cast.hpp>
-#include <boost/algorithm/string.hpp>
-#include <util/encode/hex.h>
 #include <asiolink/io_address.h>
 #include <cc/data.h>
 #include <config/ccsession.h>
-#include <log/logger_support.h>
-#include <dhcp/triplet.h>
-#include <dhcp/pool.h>
-#include <dhcp/subnet.h>
-#include <dhcp/cfgmgr.h>
+#include <dhcp/libdhcp++.h>
 #include <dhcp6/config_parser.h>
 #include <dhcp6/dhcp6_log.h>
+#include <dhcpsrv/cfgmgr.h>
+#include <dhcpsrv/pool.h>
+#include <dhcpsrv/subnet.h>
+#include <dhcpsrv/triplet.h>
+#include <log/logger_support.h>
+#include <util/encode/hex.h>
+
+#include <boost/algorithm/string.hpp>
+#include <boost/foreach.hpp>
+#include <boost/lexical_cast.hpp>
+#include <boost/scoped_ptr.hpp>
+#include <boost/shared_ptr.hpp>
+
+#include <iostream>
+#include <map>
+#include <vector>
+
+#include <stdint.h>
 
 using namespace std;
 using namespace isc::data;
@@ -40,13 +44,13 @@ using namespace isc::asiolink;
 namespace isc {
 namespace dhcp {
 
-/// @brief auxiliary type used for storing element name and its parser
+/// @brief an auxiliary type used for storing an element name and its parser
 typedef pair<string, ConstElementPtr> ConfigPair;
 
 /// @brief a factory method that will create a parser for a given element name
 typedef DhcpConfigParser* ParserFactory(const std::string& config_id);
 
-/// @brief a collection of factories that creates parsers for specified element names
+/// @brief a collection of factories that create parsers for specified element names
 typedef std::map<std::string, ParserFactory*> FactoryMap;
 
 /// @brief a collection of elements that store uint32 values (e.g. renew-timer = 900)
@@ -57,12 +61,14 @@ typedef std::map<string, string> StringStorage;
 
 /// @brief a collection of pools
 ///
-/// That type is used as intermediate storage, when pools are parsed, but there is
+/// This type is used as intermediate storage, when pools are parsed, but there is
 /// no subnet object created yet to store them.
 typedef std::vector<Pool6Ptr> PoolStorage;
 
-/// @brief Collection of options.
-typedef std::vector<OptionPtr> OptionStorage;
+/// @brief Collection of option descriptors. This container allows searching for
+/// options using the option code or persistency flag. This is useful when merging
+/// existing options with newly configured options.
+typedef Subnet::OptionContainer OptionStorage;
 
 /// @brief Global uint32 parameters that will be used as defaults.
 Uint32Storage uint32_defaults;
@@ -75,9 +81,9 @@ OptionStorage option_defaults;
 
 /// @brief a dummy configuration parser
 ///
-/// It is a debugging parser. It does not configure anything,
+/// This is a debugging parser. It does not configure anything,
 /// will accept any configuration and will just print it out
-/// on commit. Useful for debugging existing configurations and
+/// on commit.  Useful for debugging existing configurations and
 /// adding new ones.
 class DebugParser : public DhcpConfigParser {
 public:
@@ -104,7 +110,7 @@ public:
 
     /// @brief pretends to apply the configuration
     ///
-    /// This is a method required by base class. It pretends to apply the
+    /// This is a method required by the base class. It pretends to apply the
     /// configuration, but in fact it only prints the parameter out.
     ///
     /// See \ref DhcpConfigParser class for details.
@@ -168,7 +174,7 @@ public:
             // Parsing the value as a int64 value allows to
             // check if the provided value is within the range
             // of uint32_t (is not negative or greater than
-            // maximal uint32_t value.
+            // maximal uint32_t value).
             int64value = boost::lexical_cast<int64_t>(value->str());
         } catch (const boost::bad_lexical_cast&) {
             parse_error = true;
@@ -192,19 +198,20 @@ public:
                       << " as unsigned 32-bit integer.");
         }
 
-        storage_->insert(pair<string, uint32_t>(param_name_, value_));
+        // If a given parameter already exists in the storage we override
+        // its value. If it doesn't we insert a new element.
+        (*storage_)[param_name_] = value_;
     }
 
     /// @brief does nothing
     ///
-    /// This method is required for all parser. The value itself
+    /// This method is required for all parsers. The value itself
     /// is not commited anywhere. Higher level parsers are expected to
     /// use values stored in the storage, e.g. renew-timer for a given
     /// subnet is stored in subnet-specific storage. It is not commited
     /// here, but is rather used by \ref Subnet6Parser when constructing
     /// the subnet.
-    virtual void commit() {
-    }
+    virtual void commit() { }
 
     /// @brief factory that constructs Uint32Parser objects
     ///
@@ -236,9 +243,9 @@ protected:
 /// @brief Configuration parser for string parameters
 ///
 /// This class is a generic parser that is able to handle any string
-/// parameter. By default it stores the value in external global container
+/// parameter. By default it stores the value in an external global container
 /// (string_defaults). If used in smaller scopes (e.g. to parse parameters
-/// in subnet config), it can be pointed to a different storage, using
+/// in subnet config), it can be pointed to a different storage, using the
 /// setStorage() method. This class follows the parser interface, laid out
 /// in its base class, \ref DhcpConfigParser.
 ///
@@ -255,26 +262,27 @@ public:
 
     /// @brief parses parameter value
     ///
-    /// Parses configuration entry and stored it in storage. See
+    /// Parses configuration entry and stores it in storage. See
     /// \ref setStorage() for details.
     ///
     /// @param value pointer to the content of parsed values
     virtual void build(ConstElementPtr value) {
         value_ = value->str();
         boost::erase_all(value_, "\"");
-        storage_->insert(pair<string, string>(param_name_, value_));
+        // If a given parameter already exists in the storage we override
+        // its value. If it doesn't we insert a new element.
+        (*storage_)[param_name_] = value_;
     }
 
     /// @brief does nothing
     ///
-    /// This method is required for all parser. The value itself
+    /// This method is required for all parsers. The value itself
     /// is not commited anywhere. Higher level parsers are expected to
     /// use values stored in the storage, e.g. renew-timer for a given
     /// subnet is stored in subnet-specific storage. It is not commited
     /// here, but is rather used by its parent parser when constructing
     /// an object, e.g. the subnet.
-    virtual void commit() {
-    }
+    virtual void commit() { }
 
     /// @brief factory that constructs StringParser objects
     ///
@@ -365,7 +373,7 @@ protected:
 /// and stored in chosen PoolStorage container.
 ///
 /// As there are no default values for pool, setStorage() must be called
-/// before build(). Otherwise exception will be thrown.
+/// before build(). Otherwise an exception will be thrown.
 ///
 /// It is useful for parsing Dhcp6/subnet6[X]/pool parameters.
 class PoolParser : public DhcpConfigParser {
@@ -392,7 +400,7 @@ public:
         BOOST_FOREACH(ConstElementPtr text_pool, pools_list->listValue()) {
 
             // That should be a single pool representation. It should contain
-            // text is form prefix/len or first - last. Note that spaces
+            // text in the form prefix/len or first - last. Note that spaces
             // are allowed
             string txt = text_pool->stringValue();
 
@@ -411,7 +419,7 @@ public:
                     // start with the first character after /
                     string prefix_len = txt.substr(pos + 1);
 
-                    // It is lexical cast to int and then downcast to uint8_t.
+                    // It is lexically cast to int and then downcast to uint8_t.
                     // Direct cast to uint8_t (which is really an unsigned char)
                     // will result in interpreting the first digit as output
                     // value and throwing exception if length is written on two
@@ -460,7 +468,7 @@ public:
 
     /// @brief does nothing.
     ///
-    /// This method is required for all parser. The value itself
+    /// This method is required for all parsers. The value itself
     /// is not commited anywhere. Higher level parsers (for subnet) are expected
     /// to use values stored in the storage.
     virtual void commit() {}
@@ -475,7 +483,7 @@ public:
 protected:
     /// @brief pointer to the actual Pools storage
     ///
-    /// That is typically a storage somewhere in Subnet parser
+    /// This is typically a storage somewhere in Subnet parser
     /// (an upper level parser).
     PoolStorage* pools_;
 };
@@ -484,11 +492,11 @@ protected:
 ///
 /// This parser parses configuration entries that specify value of
 /// a single option. These entries include option name, option code
-/// and data carried by the option. If parsing is successful than
+/// and data carried by the option. If parsing is successful than an
 /// instance of an option is created and added to the storage provided
 /// by the calling class.
 ///
-/// @todo This class parses and validates option name. However it is
+/// @todo This class parses and validates the option name. However it is
 /// not used anywhere util support for option spaces is implemented
 /// (see tickets #2319, #2314). When option spaces are implemented
 /// there will be a way to reference the particular option using
@@ -500,7 +508,9 @@ public:
     ///
     /// Class constructor.
     OptionDataParser(const std::string&)
-        : options_(NULL) { }
+        : options_(NULL),
+          // initialize option to NULL ptr
+          option_descriptor_(false) { }
 
     /// @brief Parses the single option data.
     ///
@@ -558,11 +568,37 @@ public:
         createOption();
     }
 
-    /// @brief Does nothing.
+    /// @brief Commits option value.
     ///
-    /// This function does noting because option data is committed
-    /// by a higher level parser.
-    virtual void commit() { }
+    /// This function adds a new option to the storage or replaces an existing option
+    /// with the same code.
+    ///
+    /// @throw isc::InvalidOperation if failed to set pointer to storage or failed
+    /// to call build() prior to commit. If that happens data in the storage
+    /// remain un-modified.
+    virtual void commit() {
+        if (options_ == NULL) {
+            isc_throw(isc::InvalidOperation, "Parser logic error: storage must be set before "
+                      "commiting option data.");
+        } else  if (!option_descriptor_.option) {
+            // Before we can commit the new option should be configured. If it is not
+            // than somebody must have called commit() before build().
+            isc_throw(isc::InvalidOperation, "Parser logic error: no option has been configured and"
+                      " thus there is nothing to commit. Has build() been called?");
+        }
+        uint16_t opt_type = option_descriptor_.option->getType();
+        Subnet::OptionContainerTypeIndex& idx = options_->get<1>();
+        // Try to find options with the particular option code in the main
+        // storage. If found, remove these options because they will be
+        // replaced with new one.
+        Subnet::OptionContainerTypeRange range =
+            idx.equal_range(opt_type);
+        if (std::distance(range.first, range.second) > 0) {
+            idx.erase(range.first, range.second);
+        }
+        // Append new option to the main storage.
+        options_->push_back(option_descriptor_);
+    }
 
     /// @brief Set storage for the parser.
     ///
@@ -583,11 +619,11 @@ private:
     ///
     /// Creates an instance of an option and adds it to the provided
     /// options storage. If the option data parsed by \ref build function
-    /// are invalid or insufficient it emits exception.
+    /// are invalid or insufficient this function emits an exception.
     ///
     /// @warning this function does not check if options_ storage pointer
-    /// is intitialized but this is not needed here because it is checked in
-    /// \ref build function.
+    /// is intitialized but this check is not needed here because it is done
+    /// in the \ref build function.
     ///
     /// @throw Dhcp6ConfigError if parameters provided in the configuration
     /// are invalid.
@@ -604,7 +640,7 @@ private:
             isc_throw(Dhcp6ConfigError, "Parser error: value of 'code' must not"
                       << " exceed " << std::numeric_limits<uint16_t>::max());
         }
-        // Check the option name has been specified, is non-empty and does not
+        // Check that the option name has been specified, is non-empty and does not
         // contain spaces.
         // @todo possibly some more restrictions apply here?
         std::string option_name = getStringParam("name");
@@ -616,7 +652,11 @@ private:
                       << " spaces");
         }
 
+        // Get option data from the configuration database ('data' field).
+        // Option data is specified by the user as case insensitive string
+        // of hexadecimal digits for each option.
         std::string option_data = getStringParam("data");
+        // Transform string of hexadecimal digits into binary format.
         std::vector<uint8_t> binary;
         try {
             util::encode::decodeHex(option_data, binary);
@@ -624,16 +664,51 @@ private:
             isc_throw(Dhcp6ConfigError, "Parser error: option data is not a valid"
                       << " string of hexadecimal digits: " << option_data);
         }
-
-        // Create the actual option.
-        // @todo Currently we simply create dhcp::Option instance here but we will
-        // need to use dedicated factory functions once the option definitions are
-        // created for all options.
-        OptionPtr option(new Option(Option::V6, static_cast<uint16_t>(option_code),
-                                    binary));
-
-        // If option is created succesfully, add it to the storage.
-        options_->push_back(option);
+        // Get all existing DHCPv6 option definitions. The one that matches
+        // our option will be picked and used to create it.
+        OptionDefContainer option_defs = LibDHCP::getOptionDefs(Option::V6);
+        // Get search index #1. It allows searching for options definitions
+        // using option type value.
+        const OptionDefContainerTypeIndex& idx = option_defs.get<1>();
+        // Get all option definitions matching option code we want to create.
+        const OptionDefContainerTypeRange& range = idx.equal_range(option_code);
+        size_t num_defs = std::distance(range.first, range.second);
+        OptionPtr option;
+        // Currently we do not allow duplicated definitions and if there are
+        // any duplicates we issue internal server error.
+        if (num_defs > 1) {
+            isc_throw(Dhcp6ConfigError, "Internal error: currently it is not"
+                      << " supported to initialize multiple option definitions"
+                      << " for the same option code. This will be supported once"
+                      << " there option spaces are implemented.");
+        } else if (num_defs == 0) {
+            // @todo We have a limited set of option definitions intiialized at the moment.
+            // In the future we want to initialize option definitions for all options.
+            // Consequently an error will be issued if an option definition does not exist
+            // for a particular option code. For now it is ok to create generic option
+            // if definition does not exist.
+            OptionPtr option(new Option(Option::V6, static_cast<uint16_t>(option_code),
+                                        binary));
+            // The created option is stored in option_descriptor_ class member until the
+            // commit stage when it is inserted into the main storage. If an option with the
+            // same code exists in main storage already the old option is replaced.
+            option_descriptor_.option = option;
+            option_descriptor_.persistent = false;
+        } else {
+            // We have exactly one option definition for the particular option code
+            // use it to create the option instance.
+            const OptionDefinitionPtr& def = *(range.first);
+            try {
+                OptionPtr option = def->optionFactory(Option::V6, option_code, binary);
+                Subnet::OptionDescriptor desc(option, false);
+                option_descriptor_.option = option;
+                option_descriptor_.persistent = false;
+            } catch (const isc::Exception& ex) {
+                isc_throw(Dhcp6ConfigError, "Parser error: option data does not match"
+                          << " option definition (code " << option_code << "): "
+                          << ex.what());
+            }
+        }
     }
 
     /// @brief Get a parameter from the strings storage.
@@ -669,9 +744,11 @@ private:
     /// Pointer to options storage. This storage is provided by
     /// the calling class and is shared by all OptionDataParser objects.
     OptionStorage* options_;
+    /// Option descriptor holds newly configured option.
+    Subnet::OptionDescriptor option_descriptor_;
 };
 
-/// @brief Parser for option data values with ina subnet.
+/// @brief Parser for option data values within a subnet.
 ///
 /// This parser iterates over all entries that define options
 /// data for a particular subnet and creates a collection of options.
@@ -683,10 +760,10 @@ public:
     /// @brief Constructor.
     ///
     /// Unless otherwise specified, parsed options will be stored in
-    /// a global option containers (option_default). That storage location
+    /// a global option container (option_default). That storage location
     /// is overriden on a subnet basis.
     OptionDataListParser(const std::string&)
-        : options_(&option_defaults) { }
+        : options_(&option_defaults), local_options_() { }
 
     /// @brief Parses entries that define options' data for a subnet.
     ///
@@ -700,9 +777,11 @@ public:
             boost::shared_ptr<OptionDataParser> parser(new OptionDataParser("option-data"));
             // options_ member will hold instances of all options thus
             // each OptionDataParser takes it as a storage.
-            parser->setStorage(options_);
-            // Build the instance of a singkle option.
+            parser->setStorage(&local_options_);
+            // Build the instance of a single option.
             parser->build(option_value);
+            // Store a parser as it will be used to commit.
+            parsers_.push_back(parser);
         }
     }
 
@@ -714,11 +793,18 @@ public:
     }
 
 
-    /// @brief Does nothing.
+    /// @brief Commit all option values.
     ///
-    /// @todo Currently this function does nothing but in the future
-    /// we may need to extend it to commit at this level.
-    void commit() { }
+    /// This function invokes commit for all option values.
+    void commit() {
+        BOOST_FOREACH(ParserPtr parser, parsers_) {
+            parser->commit();
+        }
+        // Parsing was successful and we have all configured
+        // options in local storage. We can now replace old values
+        // with new values.
+        std::swap(local_options_, *options_);
+    }
 
     /// @brief Create DhcpDataListParser object
     ///
@@ -729,8 +815,15 @@ public:
         return (new OptionDataListParser(param_name));
     }
 
+    /// Intermediate option storage. This storage is used by
+    /// lower level parsers to add new options.  Values held
+    /// in this storage are assigned to main storage (options_)
+    /// if overall parsing was successful.
+    OptionStorage local_options_;
     /// Pointer to options instances storage.
     OptionStorage* options_;
+    /// Collection of parsers;
+    ParserCollection parsers_;
 };
 
 /// @brief this class parses a single subnet
@@ -742,8 +835,8 @@ public:
 
     /// @brief constructor
     Subnet6ConfigParser(const std::string& ) {
-        // The parameter should always be "subnet", but we don't check here
-        // against it in case some wants to reuse this parser somewhere.
+        // The parameter should always be "subnet", but we don't check
+        // against that here in case some wants to reuse this parser somewhere.
     }
 
     /// @brief parses parameter value
@@ -754,8 +847,8 @@ public:
         BOOST_FOREACH(ConfigPair param, subnet->mapValue()) {
             ParserPtr parser(createSubnet6ConfigParser(param.first));
             // The actual type of the parser is unknown here. We have to discover
-            // parser type here to invoke corresponding setStorage function on it.
-            // We discover parser type by trying to cast the parser to various
+            // the parser type here to invoke the corresponding setStorage function
+            // on it.  We discover parser type by trying to cast the parser to various
             // parser types and checking which one was successful. For this one
             // a setStorage and build methods are invoked.
 
@@ -792,6 +885,10 @@ public:
     /// created in other parsers are used here and added to newly created Subnet6
     /// objects. Subnet6 are then added to DHCP CfgMgr.
     void commit() {
+        // Invoke commit on all sub-data parsers.
+        BOOST_FOREACH(ParserPtr parser, parsers_) {
+            parser->commit();
+        }
 
         StringStorage::const_iterator it = string_values_.find("subnet");
         if (it == string_values_.end()) {
@@ -833,33 +930,33 @@ public:
         const Subnet::OptionContainerTypeIndex& idx = options.get<1>();
 
         // Add subnet specific options.
-        BOOST_FOREACH(OptionPtr option, options_) {
-            Subnet::OptionContainerTypeRange range = idx.equal_range(option->getType());
+        BOOST_FOREACH(Subnet::OptionDescriptor desc, options_) {
+            Subnet::OptionContainerTypeRange range = idx.equal_range(desc.option->getType());
             if (std::distance(range.first, range.second) > 0) {
                 LOG_WARN(dhcp6_logger, DHCP6_CONFIG_OPTION_DUPLICATE)
-                    .arg(option->getType()).arg(addr.toText());
+                    .arg(desc.option->getType()).arg(addr.toText());
             }
-            subnet->addOption(option);
+            subnet->addOption(desc.option);
         }
 
         // Check all global options and add them to the subnet object if
         // they have been configured in the global scope. If they have been
         // configured in the subnet scope we don't add global option because
-        // the one configured in the subnet scope always takes precedense.
-        BOOST_FOREACH(OptionPtr option, option_defaults) {
+        // the one configured in the subnet scope always takes precedence.
+        BOOST_FOREACH(Subnet::OptionDescriptor desc, option_defaults) {
             // Get all options specified locally in the subnet and having
             // code equal to global option's code.
-            Subnet::OptionContainerTypeRange range = idx.equal_range(option->getType());
+            Subnet::OptionContainerTypeRange range = idx.equal_range(desc.option->getType());
             // @todo: In the future we will be searching for options using either
-            // option code or namespace. Currently we have only the option
+            // an option code or namespace. Currently we have only the option
             // code available so if there is at least one option found with the
-            // specific code we don't add globally configured option.
+            // specific code we don't add the globally configured option.
             // @todo with this code the first globally configured option
             // with the given code will be added to a subnet. We may
-            // want to issue warning about dropping configuration of
-            // global option if one already exsist.
+            // want to issue a warning about dropping the configuration of
+            // a global option if one already exsists.
             if (std::distance(range.first, range.second) == 0) {
-                subnet->addOption(option);
+                subnet->addOption(desc.option);
             }
         }
 
@@ -870,9 +967,9 @@ private:
 
     /// @brief Set storage for a parser and invoke build.
     ///
-    /// This helper method casts the provided parser pointer to specified
-    /// type. If cast is successful it sets the corresponding storage for
-    /// this parser, invokes build on it and save the parser.
+    /// This helper method casts the provided parser pointer to the specified
+    /// type. If the cast is successful it sets the corresponding storage for
+    /// this parser, invokes build on it and saves the parser.
     ///
     /// @tparam T parser type to which parser argument should be cast.
     /// @tparam Y storage type for the specified parser type.
@@ -944,7 +1041,7 @@ private:
 
     /// @brief returns value for a given parameter (after using inheritance)
     ///
-    /// This method implements inheritance. For a given parameter name, it first
+    /// This method implements inheritance.  For a given parameter name, it first
     /// checks if there is a global value for it and overwrites it with specific
     /// value if such value was defined in subnet.
     ///
@@ -990,7 +1087,7 @@ private:
     ParserCollection parsers_;
 };
 
-/// @brief this class parses list of subnets
+/// @brief this class parses a list of subnets
 ///
 /// This is a wrapper parser that handles the whole list of Subnet6
 /// definitions. It iterates over all entries and creates Subnet6ConfigParser
@@ -1006,7 +1103,7 @@ public:
 
     /// @brief parses contents of the list
     ///
-    /// Iterates over all entries on the list and creates Subnet6ConfigParser
+    /// Iterates over all entries on the list and creates a Subnet6ConfigParser
     /// for each entry.
     ///
     /// @param subnets_list pointer to a list of IPv6 subnets
@@ -1115,12 +1212,6 @@ configureDhcp6Server(Dhcpv6Srv& , ConstElementPtr config_set) {
                   "Null pointer is passed to configuration parser");
     }
 
-    /// Reset global storage. Containers being reset below may contain
-    /// data from the previous configuration attempts.
-    option_defaults.clear();
-    uint32_defaults.clear();
-    string_defaults.clear();
-
     /// @todo: append most essential info here (like "2 new subnets configured")
     string config_details;
 

src/bin/dhcp6/config_parser.h

@@ -12,13 +12,14 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <string>
-#include <exceptions/exceptions.h>
-#include <cc/data.h>
-
 #ifndef DHCP6_CONFIG_PARSER_H
 #define DHCP6_CONFIG_PARSER_H
 
+#include <cc/data.h>
+#include <exceptions/exceptions.h>
+
+#include <string>
+
 namespace isc {
 namespace dhcp {
 

src/bin/dhcp6/ctrl_dhcp6_srv.cc

@@ -14,22 +14,22 @@
 
 #include <config.h>
 
-#include <cassert>
-#include <iostream>
-
 #include <asiolink/asiolink.h>
 #include <cc/data.h>
 #include <cc/session.h>
 #include <cc/session.h>
 #include <config/ccsession.h>
+#include <dhcp/iface_mgr.h>
+#include <dhcp6/config_parser.h>
 #include <dhcp6/ctrl_dhcp6_srv.h>
 #include <dhcp6/dhcp6_log.h>
 #include <dhcp6/spec_config.h>
-#include <dhcp6/config_parser.h>
-#include <dhcp/iface_mgr.h>
 #include <exceptions/exceptions.h>
 #include <util/buffer.h>
 
+#include <cassert>
+#include <iostream>
+
 using namespace isc::asiolink;
 using namespace isc::cc;
 using namespace isc::config;
@@ -42,6 +42,7 @@ using namespace std;
 namespace isc {
 namespace dhcp {
 
+
 ControlledDhcpv6Srv* ControlledDhcpv6Srv::server_ = NULL;
 
 ConstElementPtr
@@ -149,8 +150,8 @@ void ControlledDhcpv6Srv::disconnectSession() {
     IfaceMgr::instance().set_session_socket(IfaceMgr::INVALID_SOCKET, NULL);
 }
 
-ControlledDhcpv6Srv::ControlledDhcpv6Srv(uint16_t port /*= DHCP6_SERVER_PORT*/)
-    :Dhcpv6Srv(port), cc_session_(NULL), config_session_(NULL) {
+ControlledDhcpv6Srv::ControlledDhcpv6Srv(uint16_t port, const char* dbconfig)
+    : Dhcpv6Srv(port, dbconfig), cc_session_(NULL), config_session_(NULL) {
     server_ = this; // remember this instance for use in callback
 }
 

src/bin/dhcp6/ctrl_dhcp6_srv.h

@@ -15,11 +15,11 @@
 #ifndef CTRL_DHCPV6_SRV_H
 #define CTRL_DHCPV6_SRV_H
 
-#include <dhcp6/dhcp6_srv.h>
 #include <asiolink/asiolink.h>
+#include <cc/data.h>
 #include <cc/session.h>
 #include <config/ccsession.h>
-#include <cc/data.h>
+#include <dhcp6/dhcp6_srv.h>
 
 namespace isc {
 namespace dhcp {
@@ -41,7 +41,9 @@ public:
     /// @brief Constructor
     ///
     /// @param port UDP port to be opened for DHCP traffic
-    ControlledDhcpv6Srv(uint16_t port = DHCP6_SERVER_PORT);
+    /// @param dbconfig Lease manager database configuration string
+    ControlledDhcpv6Srv(uint16_t port = DHCP6_SERVER_PORT,
+                        const char* dbconfig = "type=memfile");
 
     /// @brief Destructor.
     ~ControlledDhcpv6Srv();

src/bin/dhcp6/dhcp6_log.cc

@@ -14,7 +14,7 @@
 
 /// Defines the logger used by the top-level component of b10-dhcp6.
 
-#include "dhcp6_log.h"
+#include <dhcp6/dhcp6_log.h>
 
 namespace isc {
 namespace dhcp {

src/bin/dhcp6/dhcp6_log.h

@@ -15,9 +15,9 @@
 #ifndef DHCP6_LOG_H
 #define DHCP6_LOG_H
 
-#include <log/macros.h>
-#include <log/logger_support.h>
 #include <dhcp6/dhcp6_messages.h>
+#include <log/logger_support.h>
+#include <log/macros.h>
 
 namespace isc {
 namespace dhcp {

src/bin/dhcp6/dhcp6_messages.mes

@@ -26,10 +26,61 @@ to establish a session with the BIND 10 control channel.
 A debug message listing the command (and possible arguments) received
 from the BIND 10 control system by the IPv6 DHCP server.
 
+% DHCP6_CONFIG_COMPLETE DHCPv6 server has completed configuration: %1
+This is an informational message announcing the successful processing of a
+new configuration. it is output during server startup, and when an updated
+configuration is committed by the administrator.  Additional information
+may be provided.
+
+% DHCP6_CONFIG_LOAD_FAIL failed to load configuration: %1
+This critical error message indicates that the initial DHCPv6
+configuration has failed. The server will start, but nothing will be
+served until the configuration has been corrected.
+
+% DHCP6_CONFIG_NEW_SUBNET a new subnet has been added to configuration: %1
+This is an informational message reporting that the configuration has
+been extended to include the specified subnet.
+
+% DHCP6_CONFIG_OPTION_DUPLICATE multiple options with the code: %1 added to the subnet: %2
+This warning message is issued on attempt to configure multiple options with the
+same option code for the particular subnet. Adding multiple options is uncommon
+for DHCPv6, yet it is not prohibited.
+
+% DHCP6_CONFIG_START DHCPv6 server is processing the following configuration: %1
+This is a debug message that is issued every time the server receives a
+configuration. That happens start up and also when a server configuration
+change is committed by the administrator.
+
 % DHCP6_CONFIG_UPDATE updated configuration received: %1
 A debug message indicating that the IPv6 DHCP server has received an
 updated configuration from the BIND 10 configuration system.
 
+% DHCP6_DB_BACKEND_STARTED lease database started (type: %1, name: %2)
+This informational message is printed every time DHCPv6 is started.
+It indicates what database backend type is being to store lease and
+other information.
+
+% DHCP6_LEASE_ADVERT lease %1 advertised (client duid=%2, iaid=%3)
+This debug message indicates that the server successfully advertised
+a lease. It is up to the client to choose one server out of othe advertised
+and continue allocation with that server. This is a normal behavior and
+indicates successful operation.
+
+% DHCP6_LEASE_ADVERT_FAIL failed to advertise a lease for client duid=%1, iaid=%2
+This message indicates that the server failed to advertise (in response to
+received SOLICIT) a lease for a given client. There may be many reasons for
+such failure. Each specific failure is logged in a separate log entry.
+
+% DHCP6_LEASE_ALLOC lease %1 has been allocated (client duid=%2, iaid=%3)
+This debug message indicates that the server successfully granted (in
+response to client's REQUEST message) a lease. This is a normal behavior
+and incicates successful operation.
+
+% DHCP6_LEASE_ALLOC_FAIL failed to grant a lease for client duid=%1, iaid=%2
+This message indicates that the server failed to grant (in response to
+received REQUEST) a lease for a given client. There may be many reasons for
+such failure. Each specific failure is logged in a separate log entry.
+
 % DHCP6_NOT_RUNNING IPv6 DHCP server is not running
 A warning message is issued when an attempt is made to shut down the
 IPv6 DHCP server but it is not running.
@@ -38,6 +89,18 @@ IPv6 DHCP server but it is not running.
 During startup the IPv6 DHCP server failed to detect any network
 interfaces and is therefore shutting down.
 
+% DHCP6_NO_SUBNET_DEF_OPT failed to find subnet for address %1 when adding default options
+This warning message indicates that when attempting to add default options
+to a response, the server found that it was not configured to support
+the subnet from which the DHCPv6 request was received.  The packet has
+been ignored.
+
+% DHCP6_NO_SUBNET_REQ_OPT failed to find subnet for address %1 when adding requested options
+This warning message indicates that when attempting to add requested
+options to a response, the server found that it was not configured
+to support the subnet from which the DHCPv6 request was received.
+The packet has been ignored.
+
 % DHCP6_OPEN_SOCKET opening sockets on port %1
 A debug message issued during startup, this indicates that the IPv6 DHCP
 server is about to open sockets on the specified port.
@@ -45,17 +108,17 @@ server is about to open sockets on the specified port.
 % DHCP6_PACKET_PARSE_FAIL failed to parse incoming packet
 The IPv6 DHCP server has received a packet that it is unable to interpret.
 
-% DHCP6_PACKET_RECEIVE_FAIL error on attempt to receive packet: %1
-The IPv6 DHCP server tried to receive a packet but an error
-occured during this attempt. The reason for the error is included in
-the message.
-
-% DHCP6_PACKET_RECEIVED %1 (type %2) packet received
+% DHCP6_PACKET_RECEIVED %1 packet received
 A debug message noting that the server has received the specified type
 of packet.  Note that a packet marked as UNKNOWN may well be a valid
 DHCP packet, just a type not expected by the server (e.g. it will report
 a received OFFER packet as UNKNOWN).
 
+% DHCP6_PACKET_RECEIVE_FAIL error on attempt to receive packet: %1
+The IPv6 DHCP server tried to receive a packet but an error
+occured during this attempt. The reason for the error is included in
+the message.
+
 % DHCP6_PACKET_SEND_FAIL failed to send DHCPv6 packet: %1
 This error is output if the IPv6 DHCP server fails to send an assembled
 DHCP message to a client. The reason for the error is included in the
@@ -66,10 +129,15 @@ This error is output if the server failed to assemble the data to be
 returned to the client into a valid packet.  The reason is most likely
 to be to a programming error: please raise a bug report.
 
-% DHCP6_QUERY_DATA received packet length %1, data length %2, data is <%3>
+% DHCP6_PROCESS_IA_NA_REQUEST server is processing IA_NA option (duid=%1, iaid=%2, hint=%3)
+This is a debug message that indicates a processing of received IA_NA
+option. It may optionally contain an address that may be used by the server
+as a hint for possible requested address.
+
+% DHCP6_QUERY_DATA received packet length %1, data length %2, data is %3
 A debug message listing the data received from the client or relay.
 
-% DHCP6_RESPONSE_DATA responding with packet type %1 data is <%2>
+% DHCP6_RESPONSE_DATA responding with packet type %1 data is %2
 A debug message listing the data returned to the client.
 
 % DHCP6_SERVER_FAILED server failed: %1
@@ -110,27 +178,14 @@ This is a debug message issued during the IPv6 DHCP server startup.
 It lists some information about the parameters with which the server
 is running.
 
-% DHCP6_CONFIG_LOAD_FAIL failed to load configuration: %1
-This critical error message indicates that the initial DHCPv6
-configuration has failed. The server will start, but nothing will be
-served until the configuration has been corrected.
-
-% DHCP6_CONFIG_START DHCPv6 server is processing the following configuration: %1
-This is a debug message that is issued every time the server receives a
-configuration. That happens start up and also when a server configuration
-change is committed by the administrator.
-
-% DHCP6_CONFIG_NEW_SUBNET A new subnet has been added to configuration: %1
-This is an informational message reporting that the configuration has
-been extended to include the specified subnet.
-
-% DHCP6_CONFIG_COMPLETE DHCPv6 server has completed configuration: %1
-This is an informational message announcing the successful processing of a
-new configuration. it is output during server startup, and when an updated
-configuration is committed by the administrator.  Additional information
-may be provided.
-
-% DHCP6_CONFIG_OPTION_DUPLICATE multiple options with the code: %1 added to the subnet: %2
-This warning message is issued on attempt to configure multiple options with the
-same option code for the particular subnet. Adding multiple options is uncommon
-for DHCPv6, yet it is not prohibited.
+% DHCP6_SUBNET_SELECTED the %1 subnet was selected for client assignment
+This is a debug message informing that a given subnet was selected. It will
+be used for address and option assignment. This is one of the early steps
+in the processing of incoming client message.
+
+% DHCP6_SUBNET_SELECTION_FAILED failed to select a subnet for incoming packet, src=%1 type=%2
+This warning message is output when a packet was received from a subnet for
+which the DHCPv6 server has not been configured. The cause is most likely due
+to a misconfiguration of the server. The packet processing will continue, but
+the response will only contain generic configuration parameters and no
+addresses or prefixes.

src/bin/dhcp6/dhcp6_srv.cc

@@ -12,72 +12,86 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <stdlib.h>
-#include <time.h>
+#include <config.h>
 
 #include <asiolink/io_address.h>
-#include <dhcp6/dhcp6_log.h>
-#include <dhcp6/dhcp6_srv.h>
 #include <dhcp/dhcp6.h>
+#include <dhcp/duid.h>
 #include <dhcp/iface_mgr.h>
+#include <dhcp/libdhcp++.h>
 #include <dhcp/option6_addrlst.h>
-#include <dhcp/option6_iaaddr.h>
 #include <dhcp/option6_ia.h>
+#include <dhcp/option6_iaaddr.h>
+#include <dhcp/option6_iaaddr.h>
+#include <dhcp/option6_int_array.h>
 #include <dhcp/pkt6.h>
+#include <dhcp6/dhcp6_log.h>
+#include <dhcp6/dhcp6_srv.h>
+#include <dhcpsrv/cfgmgr.h>
+#include <dhcpsrv/lease_mgr.h>
+#include <dhcpsrv/lease_mgr_factory.h>
+#include <dhcpsrv/subnet.h>
 #include <exceptions/exceptions.h>
 #include <util/io_utilities.h>
 #include <util/range_utilities.h>
 
+#include <boost/foreach.hpp>
+
+#include <stdlib.h>
+#include <time.h>
+
 using namespace isc;
 using namespace isc::asiolink;
 using namespace isc::dhcp;
 using namespace isc::util;
 using namespace std;
 
-const std::string HARDCODED_LEASE = "2001:db8:1::1234:abcd";
-const uint32_t HARDCODED_T1 = 1500; // in seconds
-const uint32_t HARDCODED_T2 = 2600; // in seconds
-const uint32_t HARDCODED_PREFERRED_LIFETIME = 3600; // in seconds
-const uint32_t HARDCODED_VALID_LIFETIME = 7200; // in seconds
-const std::string HARDCODED_DNS_SERVER = "2001:db8:1::1";
+namespace isc {
+namespace dhcp {
 
-Dhcpv6Srv::Dhcpv6Srv(uint16_t port) {
-    if (port == 0) {
-        // used for testing purposes. Some tests, e.g. configuration parser,
-        // require Dhcpv6Srv object, but they don't really need it to do
-        // anything. This speed up and simplifies the tests.
-        return;
-    }
+Dhcpv6Srv::Dhcpv6Srv(uint16_t port, const char* dbconfig)
+    : alloc_engine_(), serverid_(), shutdown_(true) {
 
     LOG_DEBUG(dhcp6_logger, DBG_DHCP6_START, DHCP6_OPEN_SOCKET).arg(port);
 
-    // First call to instance() will create IfaceMgr (it's a singleton)
-    // it may throw something if things go wrong
+    // Initialize objects required for DHCP server operation.
     try {
-
-        if (IfaceMgr::instance().countIfaces() == 0) {
-            LOG_ERROR(dhcp6_logger, DHCP6_NO_INTERFACES);
-            shutdown_ = true;
-            return;
+        // Port 0 is used for testing purposes. It means that the server should
+        // not open any sockets at all. Some tests, e.g. configuration parser,
+        // require Dhcpv6Srv object, but they don't really need it to do
+        // anything. This speed up and simplifies the tests.
+        if (port > 0) {
+            if (IfaceMgr::instance().countIfaces() == 0) {
+                LOG_ERROR(dhcp6_logger, DHCP6_NO_INTERFACES);
+                return;
+            }
+            IfaceMgr::instance().openSockets6(port);
         }
 
-        IfaceMgr::instance().openSockets6(port);
-
         setServerID();
 
-        /// @todo: instantiate LeaseMgr here once it is imlpemented.
+        // Instantiate LeaseMgr
+        LeaseMgrFactory::create(dbconfig);
+        LOG_INFO(dhcp6_logger, DHCP6_DB_BACKEND_STARTED)
+            .arg(LeaseMgrFactory::instance().getType())
+            .arg(LeaseMgrFactory::instance().getName());
+
+        // Instantiate allocation engine
+        alloc_engine_.reset(new AllocEngine(AllocEngine::ALLOC_ITERATIVE, 100));
 
     } catch (const std::exception &e) {
         LOG_ERROR(dhcp6_logger, DHCP6_SRV_CONSTRUCT_ERROR).arg(e.what());
-        shutdown_ = true;
         return;
     }
 
+    // All done, so can proceed
     shutdown_ = false;
 }
 
 Dhcpv6Srv::~Dhcpv6Srv() {
     IfaceMgr::instance().closeSockets();
+
+    LeaseMgrFactory::destroy();
 }
 
 void Dhcpv6Srv::shutdown() {
@@ -87,7 +101,12 @@ void Dhcpv6Srv::shutdown() {
 
 bool Dhcpv6Srv::run() {
     while (!shutdown_) {
-        /// @todo: calculate actual timeout once we have lease database
+        /// @todo: calculate actual timeout to the next event (e.g. lease
+        /// expiration) once we have lease database. The idea here is that
+        /// it is possible to do everything in a single process/thread.
+        /// For now, we are just calling select for 1000 seconds. There
+        /// were some issues reported on some systems when calling select()
+        /// with too large values. Unfortunately, I don't recall the details.
         int timeout = 1000;
 
         // client's message and server's response
@@ -107,10 +126,9 @@ bool Dhcpv6Srv::run() {
                 continue;
             }
             LOG_DEBUG(dhcp6_logger, DBG_DHCP6_DETAIL, DHCP6_PACKET_RECEIVED)
-                      .arg(serverReceivedPacketName(query->getType()))
-                      .arg(query->getType());
+                      .arg(serverReceivedPacketName(query->getType()));
             LOG_DEBUG(dhcp6_logger, DBG_DHCP6_DETAIL_DATA, DHCP6_QUERY_DATA)
-                      .arg(query->getType())
+                      .arg(static_cast<int>(query->getType()))
                       .arg(query->getBuffer().getLength())
                       .arg(query->toText());
 
@@ -196,7 +214,7 @@ void Dhcpv6Srv::setServerID() {
 
     const IfaceMgr::IfaceCollection& ifaces = IfaceMgr::instance().getIfaces();
 
-    // let's find suitable interface
+    // Let's find suitable interface.
     for (IfaceMgr::IfaceCollection::const_iterator iface = ifaces.begin();
          iface != ifaces.end(); ++iface) {
         // All the following checks could be merged into one multi-condition
@@ -217,17 +235,17 @@ void Dhcpv6Srv::setServerID() {
             continue;
         }
 
-        // let's don't use loopback
+        // Let's don't use loopback.
         if (iface->flag_loopback_) {
             continue;
         }
 
-        // let's skip downed interfaces. It is better to use working ones.
+        // Let's skip downed interfaces. It is better to use working ones.
         if (!iface->flag_up_) {
             continue;
         }
 
-        // some interfaces (like lo on Linux) report 6-bytes long
+        // Some interfaces (like lo on Linux) report 6-bytes long
         // MAC adress 00:00:00:00:00:00. Let's not use such weird interfaces
         // to generate DUID.
         if (isRangeZero(iface->getMac(), iface->getMac() + iface->getMacLen())) {
@@ -243,37 +261,37 @@ void Dhcpv6Srv::setServerID() {
         seconds -= DUID_TIME_EPOCH;
 
         OptionBuffer srvid(8 + iface->getMacLen());
-        writeUint16(DUID_LLT, &srvid[0]);
+        writeUint16(DUID::DUID_LLT, &srvid[0]);
         writeUint16(HWTYPE_ETHERNET, &srvid[2]);
         writeUint32(static_cast<uint32_t>(seconds), &srvid[4]);
-        memcpy(&srvid[0]+8, iface->getMac(), iface->getMacLen());
+        memcpy(&srvid[0] + 8, iface->getMac(), iface->getMacLen());
 
         serverid_ = OptionPtr(new Option(Option::V6, D6O_SERVERID,
                                          srvid.begin(), srvid.end()));
         return;
     }
 
-    // if we reached here, there are no suitable interfaces found.
+    // If we reached here, there are no suitable interfaces found.
     // Either interface detection is not supported on this platform or
     // this is really weird box. Let's use DUID-EN instead.
     // See Section 9.3 of RFC3315 for details.
 
     OptionBuffer srvid(12);
-    writeUint16(DUID_EN, &srvid[0]);
+    writeUint16(DUID::DUID_EN, &srvid[0]);
     writeUint32(ENTERPRISE_ID_ISC, &srvid[2]);
 
     // Length of the identifier is company specific. I hereby declare
     // ISC "standard" of 6 bytes long pseudo-random numbers.
     srandom(time(NULL));
-    fillRandom(&srvid[6],&srvid[12]);
+    fillRandom(&srvid[6], &srvid[12]);
 
     serverid_ = OptionPtr(new Option(Option::V6, D6O_SERVERID,
                                      srvid.begin(), srvid.end()));
 }
 
 void Dhcpv6Srv::copyDefaultOptions(const Pkt6Ptr& question, Pkt6Ptr& answer) {
-    // add client-id
-    boost::shared_ptr<Option> clientid = question->getOption(D6O_CLIENTID);
+    // Add client-id.
+    OptionPtr clientid = question->getOption(D6O_CLIENTID);
     if (clientid) {
         answer->addOption(clientid);
     }
@@ -281,52 +299,221 @@ void Dhcpv6Srv::copyDefaultOptions(const Pkt6Ptr& question, Pkt6Ptr& answer) {
     // TODO: Should throw if there is no client-id (except anonymous INF-REQUEST)
 }
 
-void Dhcpv6Srv::appendDefaultOptions(const Pkt6Ptr& /*question*/, Pkt6Ptr& answer) {
-    // TODO: question is currently unused, but we need it at least to know
-    // message type we are answering
-
+void Dhcpv6Srv::appendDefaultOptions(const Pkt6Ptr& question, Pkt6Ptr& answer) {
     // add server-id
     answer->addOption(getServerID());
+
+    // Get the subnet object. It holds options to be sent to the client
+    // that belongs to the particular subnet.
+    Subnet6Ptr subnet = CfgMgr::instance().getSubnet6(question->getRemoteAddr());
+    // Warn if subnet is not supported and quit.
+    if (!subnet) {
+        LOG_WARN(dhcp6_logger, DHCP6_NO_SUBNET_DEF_OPT)
+            .arg(question->getRemoteAddr().toText());
+        return;
+    }
+
 }
 
+void Dhcpv6Srv::appendRequestedOptions(const Pkt6Ptr& question, Pkt6Ptr& answer) {
+    // Get the subnet for a particular address.
+    Subnet6Ptr subnet = CfgMgr::instance().getSubnet6(question->getRemoteAddr());
+    if (!subnet) {
+        LOG_WARN(dhcp6_logger, DHCP6_NO_SUBNET_REQ_OPT)
+            .arg(question->getRemoteAddr().toText());
+        return;
+    }
+
+    // Client requests some options using ORO option. Try to
+    // get this option from client's message.
+    boost::shared_ptr<Option6IntArray<uint16_t> > option_oro =
+        boost::dynamic_pointer_cast<Option6IntArray<uint16_t> >(question->getOption(D6O_ORO));
+    // Option ORO not found. Don't do anything then.
+    if (!option_oro) {
+        return;
+    }
+    // Get the list of options that client requested.
+    const std::vector<uint16_t>& requested_opts = option_oro->getValues();
+    // Get the list of options configured for a subnet.
+    const Subnet::OptionContainer& options = subnet->getOptions();
+    const Subnet::OptionContainerTypeIndex& idx = options.get<1>();
+    // Try to match requested options with those configured for a subnet.
+    // If match is found, append configured option to the answer message.
+    BOOST_FOREACH(uint16_t opt, requested_opts) {
+        const Subnet::OptionContainerTypeRange& range = idx.equal_range(opt);
+        BOOST_FOREACH(Subnet::OptionDescriptor desc, range) {
+            answer->addOption(desc.option);
+        }
+    }
+}
 
-void Dhcpv6Srv::appendRequestedOptions(const Pkt6Ptr& /*question*/, Pkt6Ptr& answer) {
-    // TODO: question is currently unused, but we need to extract ORO from it
-    // and act on its content. Now we just send DNS-SERVERS option.
+OptionPtr Dhcpv6Srv::createStatusCode(uint16_t code, const std::string& text) {
 
-    // add dns-servers option
-    boost::shared_ptr<Option> dnsservers(new Option6AddrLst(D6O_NAME_SERVERS,
-                                         IOAddress(HARDCODED_DNS_SERVER)));
-    answer->addOption(dnsservers);
+    // @todo: Implement Option6_StatusCode and rewrite this code here
+    vector<uint8_t> data(text.c_str(), text.c_str() + text.length());
+    data.insert(data.begin(), static_cast<uint8_t>(code % 256));
+    data.insert(data.begin(), static_cast<uint8_t>(code >> 8));
+    OptionPtr status(new Option(Option::V6, D6O_STATUS_CODE, data));
+    return (status);
+}
+
+Subnet6Ptr Dhcpv6Srv::selectSubnet(const Pkt6Ptr& question) {
+    Subnet6Ptr subnet = CfgMgr::instance().getSubnet6(question->getRemoteAddr());
+
+    return (subnet);
 }
 
 void Dhcpv6Srv::assignLeases(const Pkt6Ptr& question, Pkt6Ptr& answer) {
-    /// TODO Rewrite this once LeaseManager is implemented.
-
-    // answer client's IA (this is mostly a dummy,
-    // so let's answer only first IA and hope there is only one)
-    boost::shared_ptr<Option> ia_opt = question->getOption(D6O_IA_NA);
-    if (ia_opt) {
-        // found IA
-        Option* tmp = ia_opt.get();
-        Option6IA* ia_req = dynamic_cast<Option6IA*>(tmp);
-        if (ia_req) {
-            boost::shared_ptr<Option6IA>
-                ia_rsp(new Option6IA(D6O_IA_NA, ia_req->getIAID()));
-            ia_rsp->setT1(HARDCODED_T1);
-            ia_rsp->setT2(HARDCODED_T2);
-            boost::shared_ptr<Option6IAAddr>
-                addr(new Option6IAAddr(D6O_IAADDR,
-                                       IOAddress(HARDCODED_LEASE),
-                                       HARDCODED_PREFERRED_LIFETIME,
-                                       HARDCODED_VALID_LIFETIME));
-            ia_rsp->addOption(addr);
-            answer->addOption(ia_rsp);
+
+    // We need to allocate addresses for all IA_NA options in the client's
+    // question (i.e. SOLICIT or REQUEST) message.
+
+    // We need to select a subnet the client is connected in.
+    Subnet6Ptr subnet = selectSubnet(question);
+    if (subnet) {
+        // This particular client is out of luck today. We do not have
+        // information about the subnet he is connected to. This likely means
+        // misconfiguration of the server (or some relays). We will continue to
+        // process this message, but our response will be almost useless: no
+        // addresses or prefixes, no subnet specific configuration etc. The only
+        // thing this client can get is some global information (like DNS
+        // servers).
+        LOG_DEBUG(dhcp6_logger, DBG_DHCP6_DETAIL_DATA, DHCP6_SUBNET_SELECTED)
+            .arg(subnet->toText());
+    } else {
+        // perhaps this should be logged on some higher level? This is most likely
+        // configuration bug.
+        LOG_DEBUG(dhcp6_logger, DBG_DHCP6_BASIC, DHCP6_SUBNET_SELECTION_FAILED);
+    }
+
+    // @todo: We should implement Option6Duid some day, but we can do without it
+    // just fine for now
+
+    // Let's find client's DUID. Client is supposed to include its client-id
+    // option almost all the time (the only exception is an anonymous inf-request,
+    // but that is mostly a theoretical case). Our allocation engine needs DUID
+    // and will refuse to allocate anything to anonymous clients.
+    DuidPtr duid;
+    OptionPtr opt_duid = question->getOption(D6O_CLIENTID);
+    if (opt_duid) {
+        duid = DuidPtr(new DUID(opt_duid->getData()));
+    }
+
+    // Now that we have all information about the client, let's iterate over all
+    // received options and handle IA_NA options one by one and store our
+    // responses in answer message (ADVERTISE or REPLY).
+    //
+    // @todo: expand this to cover IA_PD and IA_TA once we implement support for
+    // prefix delegation and temporary addresses.
+    for (Option::OptionCollection::iterator opt = question->options_.begin();
+         opt != question->options_.end(); ++opt) {
+        switch (opt->second->getType()) {
+        case D6O_IA_NA: {
+            OptionPtr answer_opt = handleIA_NA(subnet, duid, question,
+                                   boost::dynamic_pointer_cast<Option6IA>(opt->second));
+            if (answer_opt) {
+                answer->addOption(answer_opt);
+            }
+            break;
         }
+        default:
+            break;
+        }
+    }
+}
+
+OptionPtr Dhcpv6Srv::handleIA_NA(const Subnet6Ptr& subnet, const DuidPtr& duid, Pkt6Ptr question,
+                                 boost::shared_ptr<Option6IA> ia) {
+    // If there is no subnet selected for handling this IA_NA, the only thing to do left is
+    // to say that we are sorry, but the user won't get an address. As a convenience, we
+    // use a different status text to indicate that (compare to the same status code,
+    // but different wording below)
+    if (!subnet) {
+        // Create empty IA_NA option with IAID matching the request.
+        boost::shared_ptr<Option6IA> ia_rsp(new Option6IA(D6O_IA_NA, ia->getIAID()));
+
+        // Insert status code NoAddrsAvail.
+        ia_rsp->addOption(createStatusCode(STATUS_NoAddrsAvail, "Sorry, no subnet available."));
+        return (ia_rsp);
+    }
+
+    // Check if the client sent us a hint in his IA_NA. Clients may send an
+    // address in their IA_NA options as a suggestion (e.g. the last address
+    // they used before).
+    boost::shared_ptr<Option6IAAddr> hintOpt = boost::dynamic_pointer_cast<Option6IAAddr>
+                                        (ia->getOption(D6O_IAADDR));
+    IOAddress hint("::");
+    if (hintOpt) {
+        hint = hintOpt->getAddress();
+    }
+
+    LOG_DEBUG(dhcp6_logger, DBG_DHCP6_DETAIL, DHCP6_PROCESS_IA_NA_REQUEST)
+        .arg(duid?duid->toText():"(no-duid)").arg(ia->getIAID())
+        .arg(hintOpt?hint.toText():"(no hint)");
+
+    // "Fake" allocation is processing of SOLICIT message. We pretend to do an
+    // allocation, but we do not put the lease in the database. That is ok,
+    // because we do not guarantee that the user will get that exact lease. If
+    // the user selects this server to do actual allocation (i.e. sends REQUEST)
+    // it should include this hint. That will help us during the actual lease
+    // allocation.
+    bool fake_allocation = false;
+    if (question->getType() == DHCPV6_SOLICIT) {
+        /// @todo: Check if we support rapid commit
+        fake_allocation = true;
     }
+
+    // Use allocation engine to pick a lease for this client. Allocation engine
+    // will try to honour the hint, but it is just a hint - some other address
+    // may be used instead. If fake_allocation is set to false, the lease will
+    // be inserted into the LeaseMgr as well.
+    Lease6Ptr lease = alloc_engine_->allocateAddress6(subnet, duid, ia->getIAID(),
+                                                      hint, fake_allocation);
+
+    // Create IA_NA that we will put in the response.
+    boost::shared_ptr<Option6IA> ia_rsp(new Option6IA(D6O_IA_NA, ia->getIAID()));
+
+    if (lease) {
+        // We have a lease! Let's wrap its content into IA_NA option
+        // with IAADDR suboption.
+        LOG_DEBUG(dhcp6_logger, DBG_DHCP6_DETAIL, fake_allocation?
+                  DHCP6_LEASE_ADVERT:DHCP6_LEASE_ALLOC)
+            .arg(lease->addr_.toText())
+            .arg(duid?duid->toText():"(no-duid)")
+            .arg(ia->getIAID());
+
+        ia_rsp->setT1(subnet->getT1());
+        ia_rsp->setT2(subnet->getT2());
+
+        boost::shared_ptr<Option6IAAddr>
+            addr(new Option6IAAddr(D6O_IAADDR,
+                                   lease->addr_,
+                                   lease->preferred_lft_,
+                                   lease->valid_lft_));
+        ia_rsp->addOption(addr);
+
+        // It would be possible to insert status code=0(success) as well,
+        // but this is considered waste of bandwidth as absence of status
+        // code is considered a success.
+    } else {
+        // Allocation engine did not allocate a lease. The engine logged
+        // cause of that failure. The only thing left is to insert
+        // status code to pass the sad news to the client.
+
+        LOG_DEBUG(dhcp6_logger, DBG_DHCP6_DETAIL, fake_allocation?
+                  DHCP6_LEASE_ADVERT_FAIL:DHCP6_LEASE_ALLOC_FAIL)
+            .arg(duid?duid->toText():"(no-duid)")
+            .arg(ia->getIAID())
+            .arg(subnet->toText());
+
+        ia_rsp->addOption(createStatusCode(STATUS_NoAddrsAvail,
+                          "Sorry, no address could be allocated."));
+    }
+    return (ia_rsp);
 }
 
 Pkt6Ptr Dhcpv6Srv::processSolicit(const Pkt6Ptr& solicit) {
+
     Pkt6Ptr advertise(new Pkt6(DHCPV6_ADVERTISE, solicit->getTransid()));
 
     copyDefaultOptions(solicit, advertise);
@@ -428,3 +615,6 @@ Dhcpv6Srv::serverReceivedPacketName(uint8_t type) {
     }
     return (UNKNOWN);
 }
+
+};
+};

src/bin/dhcp6/dhcp6_srv.h

@@ -15,23 +15,33 @@
 #ifndef DHCPV6_SRV_H
 #define DHCPV6_SRV_H
 
-#include <boost/noncopyable.hpp>
 #include <dhcp/dhcp6.h>
-#include <dhcp/pkt6.h>
+#include <dhcp/duid.h>
 #include <dhcp/option.h>
+#include <dhcp/option6_ia.h>
+#include <dhcp/option_definition.h>
+#include <dhcp/pkt6.h>
+#include <dhcpsrv/alloc_engine.h>
+#include <dhcpsrv/subnet.h>
+
+#include <boost/noncopyable.hpp>
+
 #include <iostream>
 
 namespace isc {
-
 namespace dhcp {
 /// @brief DHCPv6 server service.
 ///
-/// This singleton class represents DHCPv6 server. It contains all
+/// This class represents DHCPv6 server. It contains all
 /// top-level methods and routines necessary for server operation.
 /// In particular, it instantiates IfaceMgr, loads or generates DUID
 /// that is going to be used as server-identifier, receives incoming
 /// packets, processes them, manages leases assignment and generates
 /// appropriate responses.
+///
+/// @note Only one instance of this class is instantated as it encompasses
+///       the whole operation of the server.  Nothing, however, enforces the
+///       singleton status of the object.
 class Dhcpv6Srv : public boost::noncopyable {
 
 public:
@@ -47,12 +57,15 @@ public:
     /// old or create new DUID.
     ///
     /// @param port port on will all sockets will listen
-    Dhcpv6Srv(uint16_t port = DHCP6_SERVER_PORT);
+    /// @param dbconfig Lease manager configuration string.  The default
+    ///        of the "memfile" manager is used for testing.
+    Dhcpv6Srv(uint16_t port = DHCP6_SERVER_PORT,
+            const char* dbconfig = "type=memfile");
 
     /// @brief Destructor. Used during DHCPv6 service shutdown.
     virtual ~Dhcpv6Srv();
 
-    /// @brief Returns server-intentifier option
+    /// @brief Returns server-intentifier option.
     ///
     /// @return server-id option
     OptionPtr getServerID() { return serverid_; }
@@ -70,7 +83,7 @@ public:
     /// @brief Instructs the server to shut down.
     void shutdown();
 
-    /// @brief Return textual type of packet received by server
+    /// @brief Return textual type of packet received by server.
     ///
     /// Returns the name of valid packet received by the server (e.g. SOLICIT).
     /// If the packet is unknown - or if it is a valid DHCP packet but not one
@@ -147,7 +160,38 @@ protected:
     /// @param infRequest message received from client
     Pkt6Ptr processInfRequest(const Pkt6Ptr& infRequest);
 
-    /// @brief Copies required options from client message to server answer
+    /// @brief Creates status-code option.
+    ///
+    /// @param code status code value (see RFC3315)
+    /// @param text textual explanation (will be sent in status code option)
+    /// @return status-code option
+    OptionPtr createStatusCode(uint16_t code, const std::string& text);
+
+    /// @brief Selects a subnet for a given client's packet.
+    ///
+    /// @param question client's message
+    /// @return selected subnet (or NULL if no suitable subnet was found)
+    isc::dhcp::Subnet6Ptr selectSubnet(const Pkt6Ptr& question);
+
+    /// @brief Processes IA_NA option (and assigns addresses if necessary).
+    ///
+    /// Generates response to IA_NA. This typically includes selecting (and
+    /// allocating a lease in case of REQUEST) a lease and creating
+    /// IAADDR option. In case of allocation failure, it may contain
+    /// status code option with non-zero status, denoting cause of the
+    /// allocation failure.
+    ///
+    /// @param subnet subnet the client is connected to
+    /// @param duid client's duid
+    /// @param question client's message (typically SOLICIT or REQUEST)
+    /// @param ia pointer to client's IA_NA option (client's request)
+    /// @return IA_NA option (server's response)
+    OptionPtr handleIA_NA(const isc::dhcp::Subnet6Ptr& subnet,
+                          const isc::dhcp::DuidPtr& duid,
+                          isc::dhcp::Pkt6Ptr question,
+                          boost::shared_ptr<Option6IA> ia);
+
+    /// @brief Copies required options from client message to server answer.
     ///
     /// Copies options that must appear in any server response (ADVERTISE, REPLY)
     /// to client's messages (SOLICIT, REQUEST, RENEW, REBIND, DECLINE, RELEASE).
@@ -170,8 +214,6 @@ protected:
     /// @brief Appends requested options to server's answer.
     ///
     /// Appends options requested by client to the server's answer.
-    /// TODO: This method is currently a stub. It just appends DNS-SERVERS
-    /// option.
     ///
     /// @param question client's message
     /// @param answer server's message (options will be added here)
@@ -199,10 +241,17 @@ protected:
     ///         interfaces for new DUID generation are detected.
     void setServerID();
 
-    /// server DUID (to be sent in server-identifier option)
+private:
+    /// @brief Allocation Engine.
+    /// Pointer to the allocation engine that we are currently using
+    /// It must be a pointer, because we will support changing engines
+    /// during normal operation (e.g. to use different allocators)
+    boost::shared_ptr<AllocEngine> alloc_engine_;
+
+    /// Server DUID (to be sent in server-identifier option)
     boost::shared_ptr<isc::dhcp::Option> serverid_;
 
-    /// indicates if shutdown is in progress. Setting it to true will
+    /// Indicates if shutdown is in progress. Setting it to true will
     /// initiate server shutdown procedure.
     volatile bool shutdown_;
 };

src/bin/dhcp6/main.cc

@@ -13,14 +13,15 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
-#include <iostream>
-
-#include <boost/lexical_cast.hpp>
 
 #include <dhcp6/ctrl_dhcp6_srv.h>
 #include <dhcp6/dhcp6_log.h>
 #include <log/logger_support.h>
 
+#include <boost/lexical_cast.hpp>
+
+#include <iostream>
+
 using namespace isc::dhcp;
 using namespace std;
 
@@ -34,6 +35,16 @@ using namespace std;
 /// Dhcpv6Srv and other classes, see \ref dhcpv6Session.
 
 namespace {
+// @todo: Replace the next line by extraction from configuration parameters
+// This is the "dbconfig" string for the MySQL database.  It is likely
+// that a long-term solution will be to create the instance of the lease manager
+// somewhere other than the Dhcpv6Srv constructor, to give time to extract
+// the connection string from the configuration database.
+#ifdef HAVE_MYSQL
+const char* DBCONFIG = "type=mysql name=kea user=kea password=kea host=localhost";
+#else
+const char* DBCONFIG = "type=memfile";
+#endif
 
 const char* const DHCP6_NAME = "b10-dhcp6";
 
@@ -102,7 +113,7 @@ main(int argc, char* argv[]) {
 
     int ret = EXIT_SUCCESS;
     try {
-        ControlledDhcpv6Srv server(port_number);
+        ControlledDhcpv6Srv server(port_number, DBCONFIG);
         if (!stand_alone) {
             try {
                 server.establishSession();

src/bin/dhcp6/tests/Makefile.am

@@ -65,7 +65,7 @@ dhcp6_unittests_LDADD = $(GTEST_LDADD)
 dhcp6_unittests_LDADD += $(top_builddir)/src/lib/asiolink/libb10-asiolink.la
 dhcp6_unittests_LDADD += $(top_builddir)/src/lib/util/libb10-util.la
 dhcp6_unittests_LDADD += $(top_builddir)/src/lib/dhcp/libb10-dhcp++.la
-dhcp6_unittests_LDADD += $(top_builddir)/src/lib/dhcp/libb10-dhcpsrv.la
+dhcp6_unittests_LDADD += $(top_builddir)/src/lib/dhcpsrv/libb10-dhcpsrv.la
 dhcp6_unittests_LDADD += $(top_builddir)/src/lib/log/libb10-log.la
 dhcp6_unittests_LDADD += $(top_builddir)/src/lib/exceptions/libb10-exceptions.la
 dhcp6_unittests_LDADD += $(top_builddir)/src/lib/config/libb10-cfgclient.la

src/bin/dhcp6/tests/config_parser_unittest.cc

@@ -13,18 +13,23 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
-#include <iostream>
+
+#include <config/ccsession.h>
+#include <dhcp/libdhcp++.h>
+#include <dhcp/option6_ia.h>
+#include <dhcp6/config_parser.h>
+#include <dhcp6/dhcp6_srv.h>
+#include <dhcpsrv/cfgmgr.h>
+#include <dhcpsrv/subnet.h>
+
+#include <boost/foreach.hpp>
+#include <gtest/gtest.h>
+
 #include <fstream>
+#include <iostream>
 #include <sstream>
 
 #include <arpa/inet.h>
-#include <gtest/gtest.h>
-
-#include <dhcp6/dhcp6_srv.h>
-#include <dhcp6/config_parser.h>
-#include <config/ccsession.h>
-#include <dhcp/subnet.h>
-#include <dhcp/cfgmgr.h>
 
 using namespace std;
 using namespace isc;
@@ -37,16 +42,15 @@ namespace {
 
 class Dhcp6ParserTest : public ::testing::Test {
 public:
-    Dhcp6ParserTest()
-    :rcode_(-1) {
-        // Open port 0 means to not do anything at all. We don't want to
+    Dhcp6ParserTest() :rcode_(-1), srv_(0) {
+        // srv_(0) means to not open any sockets. We don't want to
         // deal with sockets here, just check if configuration handling
         // is sane.
-        srv_ = new Dhcpv6Srv(0);
     }
 
     ~Dhcp6ParserTest() {
-        delete srv_;
+        // Reset configuration database after each test.
+        resetConfiguration();
     };
 
     /// @brief Create the simple configuration with single option.
@@ -60,6 +64,24 @@ public:
     /// param value.
     std::string createConfigWithOption(const std::string& param_value,
                                        const std::string& parameter) {
+        std::map<std::string, std::string> params;
+        if (parameter == "name") {
+            params["name"] = param_value;
+            params["code"] = "80";
+            params["data"] = "AB CDEF0105";
+        } else if (parameter == "code") {
+            params["name"] = "option_foo";
+            params["code"] = param_value;
+            params["data"] = "AB CDEF0105";
+        } else if (parameter == "data") {
+            params["name"] = "option_foo";
+            params["code"] = "80";
+            params["data"] = param_value;
+        }
+        return (createConfigWithOption(params));
+    }
+
+    std::string createConfigWithOption(const std::map<std::string, std::string>& params) {
         std::ostringstream stream;
         stream << "{ \"interface\": [ \"all\" ],"
             "\"preferred-lifetime\": 3000,"
@@ -69,21 +91,21 @@ public:
             "    \"pool\": [ \"2001:db8:1::/80\" ],"
             "    \"subnet\": \"2001:db8:1::/64\", "
             "    \"option-data\": [ {";
-        if (parameter == "name") {
-            stream <<
-                "          \"name\": \"" << param_value << "\","
-                "          \"code\": 80,"
-                "          \"data\": \"AB CDEF0105\"";
-        } else if (parameter == "code") {
-            stream <<
-                "          \"name\": \"option_foo\","
-                "          \"code\": " << param_value << ","
-                "          \"data\": \"AB CDEF0105\"";
-        } else if (parameter == "data") {
-            stream <<
-                "          \"name\": \"option_foo\","
-                "          \"code\": 80,"
-                "          \"data\": \"" << param_value << "\"";
+        bool first = true;
+        typedef std::pair<std::string, std::string> ParamPair;
+        BOOST_FOREACH(ParamPair param, params) {
+            if (!first) {
+                stream << ", ";
+            } else {
+                first = false;
+            }
+            if (param.first == "name") {
+                stream << "\"name\": \"" << param.second << "\"";
+            } else if (param.first == "code") {
+                stream << "\"code\": " << param.second << "";
+            } else if (param.first == "data") {
+                stream << "\"data\": \"" << param.second << "\"";
+            }
         }
         stream <<
             "        } ]"
@@ -92,6 +114,51 @@ public:
         return (stream.str());
     }
 
+    /// @brief Reset configuration database.
+    ///
+    /// This function resets configuration data base by
+    /// removing all subnets and option-data. Reset must
+    /// be performed after each test to make sure that
+    /// contents of the database do not affect result of
+    /// subsequent tests.
+    void resetConfiguration() {
+        ConstElementPtr status;
+
+        string config = "{ \"interface\": [ \"all\" ],"
+            "\"preferred-lifetime\": 3000,"
+            "\"rebind-timer\": 2000, "
+            "\"renew-timer\": 1000, "
+            "\"valid-lifetime\": 4000, "
+            "\"subnet6\": [ ], "
+            "\"option-data\": [ ] }";
+
+        try {
+            ElementPtr json = Element::fromJSON(config);
+            status = configureDhcp6Server(srv_, json);
+        } catch (const std::exception& ex) {
+            FAIL() << "Fatal error: unable to reset configuration database"
+                   << " after the test. The following configuration was used"
+                   << " to reset database: " << std::endl
+                   << config << std::endl
+                   << " and the following error message was returned:"
+                   << ex.what() << std::endl;
+        }
+
+
+        // returned value should be 0 (configuration success)
+        if (!status) {
+            FAIL() << "Fatal error: unable to reset configuration database"
+                   << " after the test. Configuration function returned"
+                   << " NULL pointer" << std::endl;
+        }
+        comment_ = parseAnswer(rcode_, status);
+        if (rcode_ != 0) {
+            FAIL() << "Fatal error: unable to reset configuration database"
+                   << " after the test. Configuration function returned"
+                   << " error code " << rcode_ << std::endl;
+        }
+    }
+
     /// @brief Test invalid option parameter value.
     ///
     /// This test function constructs the simple configuration
@@ -107,7 +174,7 @@ public:
         ConstElementPtr x;
         std::string config = createConfigWithOption(param_value, parameter);
         ElementPtr json = Element::fromJSON(config);
-        EXPECT_NO_THROW(x = configureDhcp6Server(*srv_, json));
+        EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
         ASSERT_TRUE(x);
         comment_ = parseAnswer(rcode_, x);
         ASSERT_EQ(1, rcode_);
@@ -153,7 +220,7 @@ public:
         EXPECT_TRUE(memcmp(expected_data, data, expected_data_len));
     }
 
-    Dhcpv6Srv* srv_;
+    Dhcpv6Srv srv_;
 
     int rcode_;
     ConstElementPtr comment_;
@@ -166,7 +233,7 @@ TEST_F(Dhcp6ParserTest, version) {
 
     ConstElementPtr x;
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_,
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_,
                     Element::fromJSON("{\"version\": 0}")));
 
     // returned value must be 0 (configuration accepted)
@@ -181,7 +248,7 @@ TEST_F(Dhcp6ParserTest, bogusCommand) {
 
     ConstElementPtr x;
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_,
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_,
                     Element::fromJSON("{\"bogus\": 5}")));
 
     // returned value must be 1 (configuration parse error)
@@ -197,7 +264,7 @@ TEST_F(Dhcp6ParserTest, emptySubnet) {
 
     ConstElementPtr status;
 
-    EXPECT_NO_THROW(status = configureDhcp6Server(*srv_,
+    EXPECT_NO_THROW(status = configureDhcp6Server(srv_,
                     Element::fromJSON("{ \"interface\": [ \"all\" ],"
                                       "\"preferred-lifetime\": 3000,"
                                       "\"rebind-timer\": 2000, "
@@ -229,7 +296,7 @@ TEST_F(Dhcp6ParserTest, subnetGlobalDefaults) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(status = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(status = configureDhcp6Server(srv_, json));
 
     // check if returned status is OK
     ASSERT_TRUE(status);
@@ -268,7 +335,7 @@ TEST_F(Dhcp6ParserTest, subnetLocal) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(status = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(status = configureDhcp6Server(srv_, json));
 
     // returned value should be 0 (configuration success)
     ASSERT_TRUE(status);
@@ -301,7 +368,7 @@ TEST_F(Dhcp6ParserTest, poolOutOfSubnet) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(status = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(status = configureDhcp6Server(srv_, json));
 
     // returned value must be 2 (values error)
     // as the pool does not belong to that subnet
@@ -329,7 +396,7 @@ TEST_F(Dhcp6ParserTest, poolPrefixLen) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
 
     // returned value must be 1 (configuration parse error)
     ASSERT_TRUE(x);
@@ -371,7 +438,7 @@ TEST_F(Dhcp6ParserTest, optionDataDefaults) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
     ASSERT_TRUE(x);
     comment_ = parseAnswer(rcode_, x);
     ASSERT_EQ(0, rcode_);
@@ -446,7 +513,7 @@ TEST_F(Dhcp6ParserTest, optionDataInSingleSubnet) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
     ASSERT_TRUE(x);
     comment_ = parseAnswer(rcode_, x);
     ASSERT_EQ(0, rcode_);
@@ -512,7 +579,7 @@ TEST_F(Dhcp6ParserTest, optionDataInMultipleSubnets) {
 
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
     ASSERT_TRUE(x);
     comment_ = parseAnswer(rcode_, x);
     ASSERT_EQ(0, rcode_);
@@ -630,7 +697,7 @@ TEST_F(Dhcp6ParserTest, optionDataLowerCase) {
     std::string config = createConfigWithOption("0a0b0C0D", "data");
     ElementPtr json = Element::fromJSON(config);
 
-    EXPECT_NO_THROW(x = configureDhcp6Server(*srv_, json));
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
     ASSERT_TRUE(x);
     comment_ = parseAnswer(rcode_, x);
     ASSERT_EQ(0, rcode_);
@@ -658,4 +725,58 @@ TEST_F(Dhcp6ParserTest, optionDataLowerCase) {
     testOption(*range.first, 80, foo_expected, sizeof(foo_expected));
 }
 
+// Verify that specific option object is returned for standard
+// option which has dedicated option class derived from Option.
+TEST_F(Dhcp6ParserTest, stdOptionData) {
+    ConstElementPtr x;
+    std::map<std::string, std::string> params;
+    params["name"] = "OPTION_IA_NA";
+    // Option code 3 means OPTION_IA_NA.
+    params["code"] = "3";
+    params["data"] = "ABCDEF01 02030405 06070809";
+
+    std::string config = createConfigWithOption(params);
+    ElementPtr json = Element::fromJSON(config);
+
+    EXPECT_NO_THROW(x = configureDhcp6Server(srv_, json));
+    ASSERT_TRUE(x);
+    comment_ = parseAnswer(rcode_, x);
+    ASSERT_EQ(0, rcode_);
+
+    Subnet6Ptr subnet = CfgMgr::instance().getSubnet6(IOAddress("2001:db8:1::5"));
+    ASSERT_TRUE(subnet);
+    const Subnet::OptionContainer& options = subnet->getOptions();
+    ASSERT_EQ(1, options.size());
+
+    // Get the search index. Index #1 is to search using option code.
+    const Subnet::OptionContainerTypeIndex& idx = options.get<1>();
+
+    // Get the options for specified index. Expecting one option to be
+    // returned but in theory we may have multiple options with the same
+    // code so we get the range.
+    std::pair<Subnet::OptionContainerTypeIndex::const_iterator,
+              Subnet::OptionContainerTypeIndex::const_iterator> range =
+        idx.equal_range(D6O_IA_NA);
+    // Expect single option with the code equal to IA_NA option code.
+    ASSERT_EQ(1, std::distance(range.first, range.second));
+    // The actual pointer to the option is held in the option field
+    // in the structure returned.
+    OptionPtr option = range.first->option;
+    ASSERT_TRUE(option);
+    // Option object returned for here is expected to be Option6IA
+    // which is derived from Option. This class is dedicated to
+    // represent standard option IA_NA.
+    boost::shared_ptr<Option6IA> optionIA =
+        boost::dynamic_pointer_cast<Option6IA>(option);
+    // If cast is unsuccessful than option returned was of a
+    // differnt type than Option6IA. This is wrong.
+    ASSERT_TRUE(optionIA);
+    // If cast was successful we may use accessors exposed by
+    // Option6IA to validate that the content of this option
+    // has been set correctly.
+    EXPECT_EQ(0xABCDEF01, optionIA->getIAID());
+    EXPECT_EQ(0x02030405, optionIA->getT1());
+    EXPECT_EQ(0x06070809, optionIA->getT2());
+}
+
 };

src/bin/dhcp6/tests/ctrl_dhcp6_srv_unittest.cc

@@ -13,16 +13,18 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
+
+#include <dhcp/dhcp6.h>
+#include <dhcp6/ctrl_dhcp6_srv.h>
+#include <config/ccsession.h>
+
+#include <gtest/gtest.h>
+
 #include <iostream>
 #include <fstream>
 #include <sstream>
 
 #include <arpa/inet.h>
-#include <gtest/gtest.h>
-
-#include <dhcp/dhcp6.h>
-#include <dhcp6/ctrl_dhcp6_srv.h>
-#include <config/ccsession.h>
 
 using namespace std;
 using namespace isc;

src/bin/dhcp6/tests/dhcp6_srv_unittest.cc

@@ -13,73 +13,219 @@
 // PERFORMANCE OF THIS SOFTWARE.
 
 #include <config.h>
-#include <iostream>
-#include <fstream>
-#include <sstream>
-
-#include <arpa/inet.h>
-#include <gtest/gtest.h>
 
+#include <asiolink/io_address.h>
+#include <config/ccsession.h>
 #include <dhcp/dhcp6.h>
+#include <dhcp/duid.h>
+#include <dhcp/option.h>
+#include <dhcp/option6_addrlst.h>
 #include <dhcp/option6_ia.h>
+#include <dhcp/option6_iaaddr.h>
+#include <dhcp/option6_int_array.h>
+#include <dhcp6/config_parser.h>
 #include <dhcp6/dhcp6_srv.h>
+#include <dhcpsrv/cfgmgr.h>
+#include <dhcpsrv/lease_mgr.h>
+#include <dhcpsrv/lease_mgr_factory.h>
 #include <util/buffer.h>
 #include <util/range_utilities.h>
+
 #include <boost/scoped_ptr.hpp>
+#include <gtest/gtest.h>
+
+#include <fstream>
+#include <iostream>
+#include <sstream>
 
-using namespace std;
 using namespace isc;
+using namespace isc::asiolink;
+using namespace isc::config;
+using namespace isc::data;
 using namespace isc::dhcp;
 using namespace isc::util;
+using namespace std;
 
 // namespace has to be named, because friends are defined in Dhcpv6Srv class
 // Maybe it should be isc::test?
 namespace {
 
 class NakedDhcpv6Srv: public Dhcpv6Srv {
-    // "naked" Interface Manager, exposes internal fields
+    // "naked" Interface Manager, exposes internal members
 public:
-    NakedDhcpv6Srv():Dhcpv6Srv(DHCP6_SERVER_PORT + 10000) { }
+    NakedDhcpv6Srv(uint16_t port):Dhcpv6Srv(port) { }
 
-    boost::shared_ptr<Pkt6>
-    processSolicit(boost::shared_ptr<Pkt6>& request) {
-        return Dhcpv6Srv::processSolicit(request);
-    }
-    boost::shared_ptr<Pkt6>
-    processRequest(boost::shared_ptr<Pkt6>& request) {
-        return Dhcpv6Srv::processRequest(request);
-    }
+    using Dhcpv6Srv::processSolicit;
+    using Dhcpv6Srv::processRequest;
+    using Dhcpv6Srv::createStatusCode;
+    using Dhcpv6Srv::selectSubnet;
 };
 
 class Dhcpv6SrvTest : public ::testing::Test {
 public:
     // these are empty for now, but let's keep them around
-    Dhcpv6SrvTest() {
+    Dhcpv6SrvTest() : rcode_(-1) {
+        subnet_ = Subnet6Ptr(new Subnet6(IOAddress("2001:db8:1::"), 48, 1000,
+                                         2000, 3000, 4000));
+        pool_ = Pool6Ptr(new Pool6(Pool6::TYPE_IA, IOAddress("2001:db8:1:1::"), 64));
+        subnet_->addPool6(pool_);
+
+        CfgMgr::instance().addSubnet6(subnet_);
+    }
+
+    // Generate IA_NA option with specified parameters
+    boost::shared_ptr<Option6IA> generateIA(uint32_t iaid, uint32_t t1, uint32_t t2) {
+        boost::shared_ptr<Option6IA> ia =
+            boost::shared_ptr<Option6IA>(new Option6IA(D6O_IA_NA, iaid));
+        ia->setT1(t1);
+        ia->setT2(t2);
+        return (ia);
     }
+
+    // Generate client-id option
+    OptionPtr generateClientId(size_t duid_size = 32) {
+
+        OptionBuffer clnt_duid(duid_size);
+        for (int i = 0; i < duid_size; i++) {
+            clnt_duid[i] = 100 + i;
+        }
+
+        duid_ = DuidPtr(new DUID(clnt_duid));
+
+        return (OptionPtr(new Option(Option::V6, D6O_CLIENTID,
+                                     clnt_duid.begin(),
+                                     clnt_duid.begin() + duid_size)));
+    }
+
+    // Checks if server response (ADVERTISE or REPLY) includes proper server-id.
+    void checkServerId(const Pkt6Ptr& rsp, const OptionPtr& expected_srvid) {
+        // check that server included its server-id
+        OptionPtr tmp = rsp->getOption(D6O_SERVERID);
+        EXPECT_EQ(tmp->getType(), expected_srvid->getType() );
+        ASSERT_EQ(tmp->len(), expected_srvid->len() );
+        EXPECT_TRUE(tmp->getData() == expected_srvid->getData());
+    }
+
+    // Checks if server response (ADVERTISE or REPLY) includes proper client-id.
+    void checkClientId(const Pkt6Ptr& rsp, const OptionPtr& expected_clientid) {
+        // check that server included our own client-id
+        OptionPtr tmp = rsp->getOption(D6O_CLIENTID);
+        ASSERT_TRUE(tmp);
+        EXPECT_EQ(expected_clientid->getType(), tmp->getType());
+        ASSERT_EQ(expected_clientid->len(), tmp->len());
+
+        // check that returned client-id is valid
+        EXPECT_TRUE(expected_clientid->getData() == tmp->getData());
+    }
+
+    // Checks that server response (ADVERTISE or REPLY) contains proper IA_NA option
+    // It returns IAADDR option for each chaining with checkIAAddr method.
+    boost::shared_ptr<Option6IAAddr> checkIA_NA(const Pkt6Ptr& rsp, uint32_t expected_iaid,
+                                         uint32_t expected_t1, uint32_t expected_t2) {
+        OptionPtr tmp = rsp->getOption(D6O_IA_NA);
+        // Can't use ASSERT_TRUE() in method that returns something
+        if (!tmp) {
+            ADD_FAILURE() << "IA_NA option not present in response";
+            return (boost::shared_ptr<Option6IAAddr>());
+        }
+
+        boost::shared_ptr<Option6IA> ia = boost::dynamic_pointer_cast<Option6IA>(tmp);
+        EXPECT_EQ(expected_iaid, ia->getIAID() );
+        EXPECT_EQ(expected_t1, ia->getT1());
+        EXPECT_EQ(expected_t2, ia->getT2());
+
+        tmp = ia->getOption(D6O_IAADDR);
+        boost::shared_ptr<Option6IAAddr> addr = boost::dynamic_pointer_cast<Option6IAAddr>(tmp);
+        return (addr);
+    }
+
+    // Check that generated IAADDR option contains expected address.
+    void checkIAAddr(const boost::shared_ptr<Option6IAAddr>& addr,
+                     const IOAddress& expected_addr,
+                     uint32_t expected_preferred, uint32_t expected_valid) {
+
+        // Check that the assigned address is indeed from the configured pool.
+        // Note that when comparing addresses, we compare the textual
+        // representation. IOAddress does not support being streamed to
+        // an ostream, which means it can't be used in EXPECT_EQ.
+        EXPECT_TRUE(subnet_->inPool(addr->getAddress()));
+        EXPECT_EQ(expected_addr.toText(), addr->getAddress().toText());
+        EXPECT_EQ(addr->getPreferred(), subnet_->getPreferred());
+        EXPECT_EQ(addr->getValid(), subnet_->getValid());
+    }
+
+    // Basic checks for generated response (message type and transaction-id).
+    void checkResponse(const Pkt6Ptr& rsp, uint8_t expected_message_type,
+                       uint32_t expected_transid) {
+        ASSERT_TRUE(rsp);
+        EXPECT_EQ(expected_message_type, rsp->getType());
+        EXPECT_EQ(expected_transid, rsp->getTransid());
+    }
+
+    // Checks if the lease sent to client is present in the database
+    Lease6Ptr checkLease(const DuidPtr& duid, const OptionPtr& ia_na,
+                         boost::shared_ptr<Option6IAAddr> addr) {
+        boost::shared_ptr<Option6IA> ia = boost::dynamic_pointer_cast<Option6IA>(ia_na);
+
+        Lease6Ptr lease = LeaseMgrFactory::instance().getLease6(addr->getAddress());
+        if (!lease) {
+            cout << "Lease for " << addr->getAddress().toText()
+                 << " not found in the database backend.";
+            return (Lease6Ptr());
+        }
+
+        EXPECT_EQ(addr->getAddress().toText(), lease->addr_.toText());
+        EXPECT_TRUE(*lease->duid_ == *duid);
+        EXPECT_EQ(ia->getIAID(), lease->iaid_);
+        EXPECT_EQ(subnet_->getID(), lease->subnet_id_);
+
+        return (lease);
+    }
+
     ~Dhcpv6SrvTest() {
+        CfgMgr::instance().deleteSubnets6();
     };
+
+    // A subnet used in most tests
+    Subnet6Ptr subnet_;
+
+    // A pool used in most tests
+    Pool6Ptr pool_;
+
+    // A DUID used in most tests (typically as client-id)
+    DuidPtr duid_;
+
+    int rcode_;
+    ConstElementPtr comment_;
 };
 
+// Test verifies that the Dhcpv6_srv class can be instantiated. It checks a mode
+// without open sockets and with sockets opened on a high port (to not require
+// root privileges).
 TEST_F(Dhcpv6SrvTest, basic) {
     // srv has stubbed interface detection. It will read
     // interfaces.txt instead. It will pretend to have detected
     // fe80::1234 link-local address on eth0 interface. Obviously
     // an attempt to bind this socket will fail.
-    Dhcpv6Srv* srv = NULL;
+    boost::scoped_ptr<Dhcpv6Srv> srv;
+
     ASSERT_NO_THROW( {
+        // Skip opening any sockets
+        srv.reset(new Dhcpv6Srv(0));
+    });
+    srv.reset();
+    ASSERT_NO_THROW({
         // open an unpriviledged port
-        srv = new Dhcpv6Srv(DHCP6_SERVER_PORT + 10000);
+        srv.reset(new Dhcpv6Srv(DHCP6_SERVER_PORT + 10000));
     });
-
-    delete srv;
 }
 
+// Test checks that DUID is generated properly
 TEST_F(Dhcpv6SrvTest, DUID) {
-    // tests that DUID is generated properly
 
     boost::scoped_ptr<Dhcpv6Srv> srv;
     ASSERT_NO_THROW( {
-        srv.reset(new Dhcpv6Srv(DHCP6_SERVER_PORT + 10000));
+        srv.reset(new Dhcpv6Srv(0));
     });
 
     OptionPtr srvid = srv->getServerID();
@@ -101,7 +247,7 @@ TEST_F(Dhcpv6SrvTest, DUID) {
     uint16_t duid_type = data.readUint16();
     cout << "Duid-type=" << duid_type << endl;
     switch(duid_type) {
-    case DUID_LLT: {
+    case DUID::DUID_LLT: {
         // DUID must contain at least 6 bytes long MAC
         // + 8 bytes of fixed header
         EXPECT_GE(14, len);
@@ -125,7 +271,7 @@ TEST_F(Dhcpv6SrvTest, DUID) {
         EXPECT_TRUE(mac != zeros);
         break;
     }
-    case DUID_EN: {
+    case DUID::DUID_EN: {
         // there's not much we can check. Just simple
         // check if it is not all zeros
         vector<uint8_t> content(len-2);
@@ -133,7 +279,7 @@ TEST_F(Dhcpv6SrvTest, DUID) {
         EXPECT_FALSE(isRangeZero(content.begin(), content.end()));
         break;
     }
-    case DUID_LL: {
+    case DUID::DUID_LL: {
         // not supported yet
         cout << "Test not implemented for DUID-LL." << endl;
 
@@ -143,86 +289,467 @@ TEST_F(Dhcpv6SrvTest, DUID) {
         // and keep it despite hardware changes.
         break;
     }
-    case DUID_UUID: // not supported yet
+    case DUID::DUID_UUID: // not supported yet
     default:
         ADD_FAILURE() << "Not supported duid type=" << duid_type << endl;
         break;
     }
 }
 
-TEST_F(Dhcpv6SrvTest, Solicit_basic) {
+// This test checks if Option Request Option (ORO) is parsed correctly
+// and the requested options are actually assigned.
+TEST_F(Dhcpv6SrvTest, advertiseOptions) {
+    ConstElementPtr x;
+    string config = "{ \"interface\": [ \"all\" ],"
+        "\"preferred-lifetime\": 3000,"
+        "\"rebind-timer\": 2000, "
+        "\"renew-timer\": 1000, "
+        "\"subnet6\": [ { "
+        "    \"pool\": [ \"2001:db8:1::/64\" ],"
+        "    \"subnet\": \"2001:db8:1::/48\", "
+        "    \"option-data\": [ {"
+        "          \"name\": \"OPTION_DNS_SERVERS\","
+        "          \"code\": 23,"
+        "          \"data\": \"2001 0DB8 1234 FFFF 0000 0000 0000 0001"
+        "2001 0DB8 1234 FFFF 0000 0000 0000 0002\""
+        "        },"
+        "        {"
+        "          \"name\": \"OPTION_FOO\","
+        "          \"code\": 1000,"
+        "          \"data\": \"1234\""
+        "        } ]"
+        " } ],"
+        "\"valid-lifetime\": 4000 }";
+
+    ElementPtr json = Element::fromJSON(config);
+
     boost::scoped_ptr<NakedDhcpv6Srv> srv;
-    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv()) );
+    ASSERT_NO_THROW(srv.reset(new NakedDhcpv6Srv(0)));
 
-    // a dummy content for client-id
-    OptionBuffer clntDuid(32);
-    for (int i = 0; i < 32; i++) {
-        clntDuid[i] = 100 + i;
-    }
+    EXPECT_NO_THROW(x = configureDhcp6Server(*srv, json));
+    ASSERT_TRUE(x);
+    comment_ = parseAnswer(rcode_, x);
+
+    ASSERT_EQ(0, rcode_);
 
     Pkt6Ptr sol = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 1234));
+    sol->setRemoteAddr(IOAddress("fe80::abcd"));
+    sol->addOption(generateIA(234, 1500, 3000));
+    OptionPtr clientid = generateClientId();
+    sol->addOption(clientid);
+
+    // Pass it to the server and get an advertise
+    boost::shared_ptr<Pkt6> adv = srv->processSolicit(sol);
+
+    // check if we get response at all
+    ASSERT_TRUE(adv);
+
+    // We have not requested option with code 1000 so it should not
+    // be included in the response.
+    ASSERT_FALSE(adv->getOption(1000));
+    ASSERT_FALSE(adv->getOption(D6O_NAME_SERVERS));
+
+    // Let's now request option with code 1000.
+    // We expect that server will include this option in its reply.
+    boost::shared_ptr<Option6IntArray<uint16_t> >
+        option_oro(new Option6IntArray<uint16_t>(D6O_ORO));
+    // Create vector with two option codes.
+    std::vector<uint16_t> codes(2);
+    codes[0] = 1000;
+    codes[1] = D6O_NAME_SERVERS;
+    // Pass this code to option.
+    option_oro->setValues(codes);
+    // Append ORO to SOLICIT message.
+    sol->addOption(option_oro);
+
+    // Need to process SOLICIT again after requesting new option.
+    adv = srv->processSolicit(sol);
+    ASSERT_TRUE(adv);
+
+    OptionPtr tmp = adv->getOption(D6O_NAME_SERVERS);
+    ASSERT_TRUE(tmp);
+
+    boost::shared_ptr<Option6AddrLst> reply_nameservers =
+        boost::dynamic_pointer_cast<Option6AddrLst>(tmp);
+    ASSERT_TRUE(reply_nameservers);
+
+    Option6AddrLst::AddressContainer addrs = reply_nameservers->getAddresses();
+    ASSERT_EQ(2, addrs.size());
+    EXPECT_TRUE(addrs[0] == IOAddress("2001:db8:1234:FFFF::1"));
+    EXPECT_TRUE(addrs[1] == IOAddress("2001:db8:1234:FFFF::2"));
+
+    // There is a dummy option with code 1000 we requested from a server.
+    // Expect that this option is in server's response.
+    tmp = adv->getOption(1000);
+    ASSERT_TRUE(tmp);
+
+    // Check that the option contains valid data (from configuration).
+    std::vector<uint8_t> data = tmp->getData();
+    ASSERT_EQ(2, data.size());
+
+    const uint8_t foo_expected[] = {
+        0x12, 0x34
+    };
+    EXPECT_EQ(0, memcmp(&data[0], foo_expected, 2));
+
+    // more checks to be implemented
+}
+
+
+// There are no dedicated tests for Dhcpv6Srv::handleIA_NA and Dhcpv6Srv::assignLeases
+// as they are indirectly tested in Solicit and Request tests.
+
+// This test verifies that incoming SOLICIT can be handled properly, that an
+// ADVERTISE is generated, that the response has an address and that address
+// really belongs to the configured pool.
+//
+// This test sends a SOLICIT without any hint in IA_NA.
+//
+// constructed very simple SOLICIT message with:
+// - client-id option (mandatory)
+// - IA option (a request for address, without any addresses)
+//
+// expected returned ADVERTISE message:
+// - copy of client-id
+// - server-id
+// - IA that includes IAADDR
+TEST_F(Dhcpv6SrvTest, SolicitBasic) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    Pkt6Ptr sol = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 1234));
+    sol->setRemoteAddr(IOAddress("fe80::abcd"));
+    sol->addOption(generateIA(234, 1500, 3000));
+    OptionPtr clientid = generateClientId();
+    sol->addOption(clientid);
 
-    boost::shared_ptr<Option6IA> ia =
-        boost::shared_ptr<Option6IA>(new Option6IA(D6O_IA_NA, 234));
-    ia->setT1(1501);
-    ia->setT2(2601);
+    // Pass it to the server and get an advertise
+    Pkt6Ptr reply = srv->processSolicit(sol);
+
+    // check if we get response at all
+    checkResponse(reply, DHCPV6_ADVERTISE, 1234);
+
+    // check that IA_NA was returned and that there's an address included
+    boost::shared_ptr<Option6IAAddr> addr = checkIA_NA(reply, 234, subnet_->getT1(),
+                                                subnet_->getT2());
+
+    // Check that the assigned address is indeed from the configured pool
+    checkIAAddr(addr, addr->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+
+    // check DUIDs
+    checkServerId(reply, srv->getServerID());
+    checkClientId(reply, clientid);
+}
+
+// This test verifies that incoming SOLICIT can be handled properly, that an
+// ADVERTISE is generated, that the response has an address and that address
+// really belongs to the configured pool.
+//
+// This test sends a SOLICIT with IA_NA that contains a valid hint.
+//
+// constructed very simple SOLICIT message with:
+// - client-id option (mandatory)
+// - IA option (a request for address, with an address that belongs to the
+//              configured pool, i.e. is valid as hint)
+//
+// expected returned ADVERTISE message:
+// - copy of client-id
+// - server-id
+// - IA that includes IAADDR
+TEST_F(Dhcpv6SrvTest, SolicitHint) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    // Let's create a SOLICIT
+    Pkt6Ptr sol = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 1234));
+    sol->setRemoteAddr(IOAddress("fe80::abcd"));
+    boost::shared_ptr<Option6IA> ia = generateIA(234, 1500, 3000);
+
+    // with a valid hint
+    IOAddress hint("2001:db8:1:1::dead:beef");
+    ASSERT_TRUE(subnet_->inPool(hint));
+    OptionPtr hint_opt(new Option6IAAddr(D6O_IAADDR, hint, 300, 500));
+    ia->addOption(hint_opt);
     sol->addOption(ia);
+    OptionPtr clientid = generateClientId();
+    sol->addOption(clientid);
+
+    // Pass it to the server and get an advertise
+    Pkt6Ptr reply = srv->processSolicit(sol);
 
-    // Let's not send address in solicit yet
-    // boost::shared_ptr<Option6IAAddr> addr(new Option6IAAddr(D6O_IAADDR,
-    //    IOAddress("2001:db8:1234:ffff::ffff"), 5001, 7001));
-    // ia->addOption(addr);
-    // sol->addOption(ia);
+    // check if we get response at all
+    checkResponse(reply, DHCPV6_ADVERTISE, 1234);
+
+    OptionPtr tmp = reply->getOption(D6O_IA_NA);
+    ASSERT_TRUE(tmp);
 
-    // constructed very simple SOLICIT message with:
-    // - client-id option (mandatory)
-    // - IA option (a request for address, without any addresses)
+    // check that IA_NA was returned and that there's an address included
+    boost::shared_ptr<Option6IAAddr> addr = checkIA_NA(reply, 234, subnet_->getT1(),
+                                                subnet_->getT2());
 
-    // expected returned ADVERTISE message:
-    // - copy of client-id
-    // - server-id
-    // - IA that includes IAADDR
+    // check that we've got the address we requested
+    checkIAAddr(addr, hint, subnet_->getPreferred(), subnet_->getValid());
+
+    // check DUIDs
+    checkServerId(reply, srv->getServerID());
+    checkClientId(reply, clientid);
+}
 
-    OptionPtr clientid = OptionPtr(new Option(Option::V6, D6O_CLIENTID,
-                                              clntDuid.begin(),
-                                              clntDuid.begin() + 16));
+// This test verifies that incoming SOLICIT can be handled properly, that an
+// ADVERTISE is generated, that the response has an address and that address
+// really belongs to the configured pool.
+//
+// This test sends a SOLICIT with IA_NA that contains an invalid hint.
+//
+// constructed very simple SOLICIT message with:
+// - client-id option (mandatory)
+// - IA option (a request for address, with an address that does not
+//              belong to the configured pool, i.e. is valid as hint)
+//
+// expected returned ADVERTISE message:
+// - copy of client-id
+// - server-id
+// - IA that includes IAADDR
+TEST_F(Dhcpv6SrvTest, SolicitInvalidHint) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    // Let's create a SOLICIT
+    Pkt6Ptr sol = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 1234));
+    sol->setRemoteAddr(IOAddress("fe80::abcd"));
+    boost::shared_ptr<Option6IA> ia = generateIA(234, 1500, 3000);
+    IOAddress hint("2001:db8:1::cafe:babe");
+    ASSERT_FALSE(subnet_->inPool(hint));
+    OptionPtr hint_opt(new Option6IAAddr(D6O_IAADDR, hint, 300, 500));
+    ia->addOption(hint_opt);
+    sol->addOption(ia);
+    OptionPtr clientid = generateClientId();
     sol->addOption(clientid);
 
-    boost::shared_ptr<Pkt6> reply = srv->processSolicit(sol);
+    // Pass it to the server and get an advertise
+    Pkt6Ptr reply = srv->processSolicit(sol);
 
     // check if we get response at all
-    ASSERT_TRUE( reply != boost::shared_ptr<Pkt6>() );
+    checkResponse(reply, DHCPV6_ADVERTISE, 1234);
+
+    // check that IA_NA was returned and that there's an address included
+    boost::shared_ptr<Option6IAAddr> addr = checkIA_NA(reply, 234, subnet_->getT1(),
+                                                subnet_->getT2());
 
-    EXPECT_EQ( DHCPV6_ADVERTISE, reply->getType() );
-    EXPECT_EQ( 1234, reply->getTransid() );
+    // Check that the assigned address is indeed from the configured pool
+    checkIAAddr(addr, addr->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+    EXPECT_TRUE(subnet_->inPool(addr->getAddress()));
+
+    // check DUIDs
+    checkServerId(reply, srv->getServerID());
+    checkClientId(reply, clientid);
+}
+
+// This test checks that the server is offering different addresses to different
+// clients in ADVERTISEs. Please note that ADVERTISE is not a guarantee that such
+// and address will be assigned. Had the pool was very small and contained only
+// 2 addresses, the third client would get the same advertise as the first one
+// and this is a correct behavior. It is REQUEST that will fail for the third
+// client. ADVERTISE is basically saying "if you send me a request, you will
+// probably get an address like this" (there are no guarantees).
+TEST_F(Dhcpv6SrvTest, ManySolicits) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    Pkt6Ptr sol1 = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 1234));
+    Pkt6Ptr sol2 = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 2345));
+    Pkt6Ptr sol3 = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 3456));
+
+    sol1->setRemoteAddr(IOAddress("fe80::abcd"));
+    sol2->setRemoteAddr(IOAddress("fe80::1223"));
+    sol3->setRemoteAddr(IOAddress("fe80::3467"));
+
+    sol1->addOption(generateIA(1, 1500, 3000));
+    sol2->addOption(generateIA(2, 1500, 3000));
+    sol3->addOption(generateIA(3, 1500, 3000));
+
+    // different client-id sizes
+    OptionPtr clientid1 = generateClientId(12);
+    OptionPtr clientid2 = generateClientId(14);
+    OptionPtr clientid3 = generateClientId(16);
+
+    sol1->addOption(clientid1);
+    sol2->addOption(clientid2);
+    sol3->addOption(clientid3);
+
+    // Pass it to the server and get an advertise
+    Pkt6Ptr reply1 = srv->processSolicit(sol1);
+    Pkt6Ptr reply2 = srv->processSolicit(sol2);
+    Pkt6Ptr reply3 = srv->processSolicit(sol3);
+
+    // check if we get response at all
+    checkResponse(reply1, DHCPV6_ADVERTISE, 1234);
+    checkResponse(reply2, DHCPV6_ADVERTISE, 2345);
+    checkResponse(reply3, DHCPV6_ADVERTISE, 3456);
+
+    // check that IA_NA was returned and that there's an address included
+    boost::shared_ptr<Option6IAAddr> addr1 = checkIA_NA(reply1, 1, subnet_->getT1(),
+                                                subnet_->getT2());
+    boost::shared_ptr<Option6IAAddr> addr2 = checkIA_NA(reply2, 2, subnet_->getT1(),
+                                                subnet_->getT2());
+    boost::shared_ptr<Option6IAAddr> addr3 = checkIA_NA(reply3, 3, subnet_->getT1(),
+                                                subnet_->getT2());
+
+    // Check that the assigned address is indeed from the configured pool
+    checkIAAddr(addr1, addr1->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+    checkIAAddr(addr2, addr2->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+    checkIAAddr(addr3, addr3->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+
+    // check DUIDs
+    checkServerId(reply1, srv->getServerID());
+    checkServerId(reply2, srv->getServerID());
+    checkServerId(reply3, srv->getServerID());
+    checkClientId(reply1, clientid1);
+    checkClientId(reply2, clientid2);
+    checkClientId(reply3, clientid3);
+
+    // Finally check that the addresses offered are different
+    EXPECT_NE(addr1->getAddress().toText(), addr2->getAddress().toText());
+    EXPECT_NE(addr2->getAddress().toText(), addr3->getAddress().toText());
+    EXPECT_NE(addr3->getAddress().toText(), addr1->getAddress().toText());
+    cout << "Offered address to client1=" << addr1->getAddress().toText() << endl;
+    cout << "Offered address to client2=" << addr2->getAddress().toText() << endl;
+    cout << "Offered address to client3=" << addr3->getAddress().toText() << endl;
+}
+
+
+// This test verifies that incoming REQUEST can be handled properly, that a
+// REPLY is generated, that the response has an address and that address
+// really belongs to the configured pool.
+//
+// This test sends a REQUEST with IA_NA that contains a valid hint.
+//
+// constructed very simple REQUEST message with:
+// - client-id option (mandatory)
+// - IA option (a request for address, with an address that belongs to the
+//              configured pool, i.e. is valid as hint)
+//
+// expected returned REPLY message:
+// - copy of client-id
+// - server-id
+// - IA that includes IAADDR
+TEST_F(Dhcpv6SrvTest, RequestBasic) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    // Let's create a REQUEST
+    Pkt6Ptr req = Pkt6Ptr(new Pkt6(DHCPV6_REQUEST, 1234));
+    req->setRemoteAddr(IOAddress("fe80::abcd"));
+    boost::shared_ptr<Option6IA> ia = generateIA(234, 1500, 3000);
+
+    // with a valid hint
+    IOAddress hint("2001:db8:1:1::dead:beef");
+    ASSERT_TRUE(subnet_->inPool(hint));
+    OptionPtr hint_opt(new Option6IAAddr(D6O_IAADDR, hint, 300, 500));
+    ia->addOption(hint_opt);
+    req->addOption(ia);
+    OptionPtr clientid = generateClientId();
+    req->addOption(clientid);
+
+    // Pass it to the server and hope for a REPLY
+    Pkt6Ptr reply = srv->processRequest(req);
+
+    // check if we get response at all
+    checkResponse(reply, DHCPV6_REPLY, 1234);
 
     OptionPtr tmp = reply->getOption(D6O_IA_NA);
-    ASSERT_TRUE( tmp );
+    ASSERT_TRUE(tmp);
 
-    Option6IA* reply_ia = dynamic_cast<Option6IA*>(tmp.get());
-    EXPECT_EQ( 234, reply_ia->getIAID() );
+    // check that IA_NA was returned and that there's an address included
+    boost::shared_ptr<Option6IAAddr> addr = checkIA_NA(reply, 234, subnet_->getT1(),
+                                                subnet_->getT2());
 
-    // check that there's an address included
-    EXPECT_TRUE( reply_ia->getOption(D6O_IAADDR));
+    // check that we've got the address we requested
+    checkIAAddr(addr, hint, subnet_->getPreferred(), subnet_->getValid());
 
-    // check that server included our own client-id
-    tmp = reply->getOption(D6O_CLIENTID);
-    ASSERT_TRUE( tmp );
-    EXPECT_EQ(clientid->getType(), tmp->getType() );
-    ASSERT_EQ(clientid->len(), tmp->len() );
+    // check DUIDs
+    checkServerId(reply, srv->getServerID());
+    checkClientId(reply, clientid);
 
-    EXPECT_TRUE( clientid->getData() == tmp->getData() );
+    // check that the lease is really in the database
+    Lease6Ptr l = checkLease(duid_, reply->getOption(D6O_IA_NA), addr);
+    EXPECT_TRUE(l);
+    LeaseMgrFactory::instance().deleteLease6(addr->getAddress());
+}
 
-    // check that server included its server-id
-    tmp = reply->getOption(D6O_SERVERID);
-    EXPECT_EQ(tmp->getType(), srv->getServerID()->getType() );
-    ASSERT_EQ(tmp->len(),  srv->getServerID()->len() );
+// This test checks that the server is offering different addresses to different
+// clients in REQUEST. Please note that ADVERTISE is not a guarantee that such
+// and address will be assigned. Had the pool was very small and contained only
+// 2 addresses, the third client would get the same advertise as the first one
+// and this is a correct behavior. It is REQUEST that will fail for the third
+// client. ADVERTISE is basically saying "if you send me a request, you will
+// probably get an address like this" (there are no guarantees).
+TEST_F(Dhcpv6SrvTest, ManyRequests) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
 
-    EXPECT_TRUE(tmp->getData() == srv->getServerID()->getData());
+    Pkt6Ptr req1 = Pkt6Ptr(new Pkt6(DHCPV6_REQUEST, 1234));
+    Pkt6Ptr req2 = Pkt6Ptr(new Pkt6(DHCPV6_REQUEST, 2345));
+    Pkt6Ptr req3 = Pkt6Ptr(new Pkt6(DHCPV6_REQUEST, 3456));
 
-    // more checks to be implemented
+    req1->setRemoteAddr(IOAddress("fe80::abcd"));
+    req2->setRemoteAddr(IOAddress("fe80::1223"));
+    req3->setRemoteAddr(IOAddress("fe80::3467"));
+
+    req1->addOption(generateIA(1, 1500, 3000));
+    req2->addOption(generateIA(2, 1500, 3000));
+    req3->addOption(generateIA(3, 1500, 3000));
+
+    // different client-id sizes
+    OptionPtr clientid1 = generateClientId(12);
+    OptionPtr clientid2 = generateClientId(14);
+    OptionPtr clientid3 = generateClientId(16);
+
+    req1->addOption(clientid1);
+    req2->addOption(clientid2);
+    req3->addOption(clientid3);
+
+    // Pass it to the server and get an advertise
+    Pkt6Ptr reply1 = srv->processRequest(req1);
+    Pkt6Ptr reply2 = srv->processRequest(req2);
+    Pkt6Ptr reply3 = srv->processRequest(req3);
+
+    // check if we get response at all
+    checkResponse(reply1, DHCPV6_REPLY, 1234);
+    checkResponse(reply2, DHCPV6_REPLY, 2345);
+    checkResponse(reply3, DHCPV6_REPLY, 3456);
+
+    // check that IA_NA was returned and that there's an address included
+    boost::shared_ptr<Option6IAAddr> addr1 = checkIA_NA(reply1, 1, subnet_->getT1(),
+                                                subnet_->getT2());
+    boost::shared_ptr<Option6IAAddr> addr2 = checkIA_NA(reply2, 2, subnet_->getT1(),
+                                                subnet_->getT2());
+    boost::shared_ptr<Option6IAAddr> addr3 = checkIA_NA(reply3, 3, subnet_->getT1(),
+                                                subnet_->getT2());
+
+    // Check that the assigned address is indeed from the configured pool
+    checkIAAddr(addr1, addr1->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+    checkIAAddr(addr2, addr2->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+    checkIAAddr(addr3, addr3->getAddress(), subnet_->getPreferred(), subnet_->getValid());
+
+    // check DUIDs
+    checkServerId(reply1, srv->getServerID());
+    checkServerId(reply2, srv->getServerID());
+    checkServerId(reply3, srv->getServerID());
+    checkClientId(reply1, clientid1);
+    checkClientId(reply2, clientid2);
+    checkClientId(reply3, clientid3);
+
+    // Finally check that the addresses offered are different
+    EXPECT_NE(addr1->getAddress().toText(), addr2->getAddress().toText());
+    EXPECT_NE(addr2->getAddress().toText(), addr3->getAddress().toText());
+    EXPECT_NE(addr3->getAddress().toText(), addr1->getAddress().toText());
+    cout << "Assigned address to client1=" << addr1->getAddress().toText() << endl;
+    cout << "Assigned address to client2=" << addr2->getAddress().toText() << endl;
+    cout << "Assigned address to client3=" << addr3->getAddress().toText() << endl;
 }
 
+
 TEST_F(Dhcpv6SrvTest, serverReceivedPacketName) {
     // Check all possible packet types
     for (int itype = 0; itype < 256; ++itype) {
@@ -268,4 +795,41 @@ TEST_F(Dhcpv6SrvTest, serverReceivedPacketName) {
     }
 }
 
+// This test verifies if the status code option is generated properly.
+TEST_F(Dhcpv6SrvTest, StatusCode) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    // a dummy content for client-id
+    uint8_t expected[] = {0x0, 0x3, 0x41, 0x42, 0x43, 0x44, 0x45};
+    OptionBuffer exp(expected, expected + sizeof(expected));
+
+    OptionPtr status = srv->createStatusCode(3, "ABCDE");
+
+    EXPECT_TRUE(status->getData() == exp);
+}
+
+// This test verifies if the selectSubnet() method works as expected.
+TEST_F(Dhcpv6SrvTest, SelectSubnet) {
+    boost::scoped_ptr<NakedDhcpv6Srv> srv;
+    ASSERT_NO_THROW( srv.reset(new NakedDhcpv6Srv(0)) );
+
+    Pkt6Ptr pkt = Pkt6Ptr(new Pkt6(DHCPV6_SOLICIT, 1234));
+
+    // check that the packets originating from local addresses can be
+    pkt->setRemoteAddr(IOAddress("fe80::abcd"));
+    EXPECT_EQ(subnet_, srv->selectSubnet(pkt));
+
+    // packets originating from subnet A will select subnet A
+    pkt->setRemoteAddr(IOAddress("2001:db8:1::6789"));
+    EXPECT_EQ(subnet_, srv->selectSubnet(pkt));
+
+    // packets from a subnet that is not supported will not get
+    // a subnet
+    pkt->setRemoteAddr(IOAddress("3000::faf"));
+    EXPECT_FALSE(srv->selectSubnet(pkt));
+
+    /// @todo: expand this test once support for relays is implemented
+}
+
 }   // end of anonymous namespace

src/bin/dhcp6/tests/dhcp6_test.py

@@ -45,6 +45,25 @@ class TestDhcpv6Daemon(unittest.TestCase):
     def tearDown(self):
         pass
 
+    def readPipe(self, pipe_fd):
+        """
+        Reads bytes from a pipe and returns a character string.  If nothing is
+        read, or if there is an error, an empty string is returned.
+
+        pipe_fd - Pipe file descriptor to read
+        """
+        try:
+            data = os.read(pipe_fd, 16384)
+            # Make sure we have a string
+            if (data is None):
+                data = ""
+            else:
+                data = str(data)
+        except OSError:
+            data = ""
+
+        return data
+
     def runCommand(self, params, wait=1):
         """
         This method runs a command and returns a tuple: (returncode, stdout, stderr)
@@ -89,45 +108,48 @@ class TestDhcpv6Daemon(unittest.TestCase):
         fl = fcntl.fcntl(fd, fcntl.F_GETFL)
         fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
 
-        # There's potential problem if b10-dhcp4 prints out more
-        # than 16k of text
-        try:
-            output = os.read(self.stdout_pipes[0], 16384)
-        except OSError:
-            print("No data available from stdout")
-            output = ""
-
-        # read can return None. Make sure we have a string
-        if (output is None):
-            output = ""
-
-        try:
-            error = os.read(self.stderr_pipes[0], 16384)
-        except OSError:
-            print("No data available on stderr")
-            error = ""
-
-        # read can return None. Make sure we have a string
-        if (error is None):
-            error = ""
-
-        try:
-            if (not pi.process.poll()):
-                # let's be nice at first...
+        # As we don't know how long the subprocess will take to start and
+        # produce output, we'll loop and sleep for 250 ms between each
+        # iteration.  To avoid an infinite loop, we'll loop for a maximum
+        # of five seconds: that should be enough.
+        for count in range(20):
+            # Read something from stderr and stdout (these reads don't block).
+            output = self.readPipe(self.stdout_pipes[0])
+            error  = self.readPipe(self.stderr_pipes[0])
+
+            # If the process has already exited, or if it has output something,
+            # quit the loop now.
+            if pi.process.poll() is not None or len(error) > 0 or len(output) > 0:
+                break
+
+            # Process still running, try again in 250 ms.
+            time.sleep(0.25)
+
+        # Exited loop, kill the process if it is still running
+        if pi.process.poll() is None:
+            try:
                 pi.process.terminate()
-        except OSError:
-            print("Ignoring failed kill attempt. Process is dead already.")
+            except OSError:
+                print("Ignoring failed kill attempt. Process is dead already.")
 
         # call this to get returncode, process should be dead by now
         rc = pi.process.wait()
 
         # Clean up our stdout/stderr munging.
         os.dup2(self.stdout_old, sys.stdout.fileno())
+        os.close(self.stdout_old)
         os.close(self.stdout_pipes[0])
 
         os.dup2(self.stderr_old, sys.stderr.fileno())
+        os.close(self.stderr_old)
         os.close(self.stderr_pipes[0])
 
+        # Free up resources (file descriptors) from the ProcessInfo object
+        # TODO: For some reason, this gives an error if the process has ended,
+        #       although it does cause all descriptors still allocated to the
+        #       object to be freed.
+        pi = None
+
         print ("Process finished, return code=%d, stdout=%d bytes, stderr=%d bytes"
                % (rc, len(output), len(error)) )
 

src/bin/dhcp6/tests/dhcp6_unittests.cc

@@ -12,10 +12,10 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <stdio.h>
-#include <gtest/gtest.h>
 #include <log/logger_support.h>
 
+#include <gtest/gtest.h>
+
 int
 main(int argc, char* argv[]) {
 

src/bin/msgq/msgq.py.in

@@ -127,6 +127,7 @@ class MsgQ:
         self.subs = SubscriptionManager()
         self.lnames = {}
         self.sendbuffs = {}
+        self.running = False
 
     def setup_poller(self):
         """Set up the poll thing.  Internal function."""
@@ -178,6 +179,8 @@ class MsgQ:
             if os.path.exists(self.socket_file):
                 os.remove(self.socket_file)
             self.listen_socket.close()
+            sys.stderr.write("[b10-msgq] failed to setup listener on %s: %s\n"
+                             % (self.socket_file, str(e)))
             raise e
 
         if self.poller:
@@ -313,6 +316,8 @@ class MsgQ:
         elif cmd == 'ping':
             # Command for testing purposes
             self.process_command_ping(sock, routing, data)
+        elif cmd == 'stop':
+            self.stop()
         else:
             sys.stderr.write("[b10-msgq] Invalid command: %s\n" % cmd)
 
@@ -334,14 +339,34 @@ class MsgQ:
         self.send_prepared_msg(sock, self.preparemsg(env, msg))
 
     def __send_data(self, sock, data):
+        """
+        Send a piece of data to the given socket.
+        Parameters:
+        sock: The socket to send to
+        data: The list of bytes to send
+        Returns:
+        An integer or None. If an integer (which can be 0), it signals
+        the number of bytes sent. If None, the socket appears to have
+        been closed on the other end, and it has been killed on this
+        side too.
+        """
         try:
             # We set the socket nonblocking, MSG_DONTWAIT doesn't exist
             # on some OSes
             sock.setblocking(0)
             return sock.send(data)
         except socket.error as e:
-            if e.errno == errno.EAGAIN or e.errno == errno.EWOULDBLOCK:
+            if e.errno in [ errno.EAGAIN,
+                            errno.EWOULDBLOCK,
+                            errno.EINTR ]:
                 return 0
+            elif e.errno in [ errno.EPIPE,
+                              errno.ECONNRESET,
+                              errno.ENOBUFS ]:
+                print("[b10-msgq] " + errno.errorcode[e.errno] +
+                      " on send, dropping message and closing connection")
+                self.kill_socket(sock.fileno(), sock)
+                return None
             else:
                 raise e
         finally:
@@ -354,20 +379,12 @@ class MsgQ:
         if fileno in self.sendbuffs:
             amount_sent = 0
         else:
-            try:
-                amount_sent = self.__send_data(sock, msg)
-            except socket.error as sockerr:
-                # in the case the other side seems gone, kill the socket
-                # and drop the send action
-                if sockerr.errno == errno.EPIPE:
-                    print("[b10-msgq] SIGPIPE on send, dropping message " +
-                          "and closing connection")
-                    self.kill_socket(fileno, sock)
-                    return
-                else:
-                    raise
+            amount_sent = self.__send_data(sock, msg)
+            if amount_sent is None:
+                # Socket has been killed, drop the send
+                return
 
-        # Still something to send
+        # Still something to send, add it to outgoing queue
         if amount_sent < len(msg):
             now = time.clock()
             # Append it to buffer (but check the data go away)
@@ -392,17 +409,18 @@ class MsgQ:
         (_, msg) = self.sendbuffs[fileno]
         sock = self.sockets[fileno]
         amount_sent = self.__send_data(sock, msg)
-        # Keep the rest
-        msg = msg[amount_sent:]
-        if len(msg) == 0:
-            # If there's no more, stop requesting for write availability
-            if self.poller:
-                self.poller.register(fileno, select.POLLIN)
+        if amount_sent is not None:
+            # Keep the rest
+            msg = msg[amount_sent:]
+            if len(msg) == 0:
+                # If there's no more, stop requesting for write availability
+                if self.poller:
+                    self.poller.register(fileno, select.POLLIN)
+                else:
+                    self.delete_kqueue_socket(sock, True)
+                del self.sendbuffs[fileno]
             else:
-                self.delete_kqueue_socket(sock, True)
-            del self.sendbuffs[fileno]
-        else:
-            self.sendbuffs[fileno] = (time.clock(), msg)
+                self.sendbuffs[fileno] = (time.clock(), msg)
 
     def newlname(self):
         """Generate a unique connection identifier for this socket.
@@ -456,6 +474,7 @@ class MsgQ:
 
     def run(self):
         """Process messages.  Forever.  Mostly."""
+        self.running = True
 
         if self.poller:
             self.run_poller()
@@ -463,8 +482,10 @@ class MsgQ:
             self.run_kqueue()
 
     def run_poller(self):
-        while True:
+        while self.running:
             try:
+                # Poll with a timeout so that every once in a while,
+                # the loop checks for self.running.
                 events = self.poller.poll()
             except select.error as err:
                 if err.args[0] == errno.EINTR:
@@ -478,11 +499,15 @@ class MsgQ:
                 else:
                     if event & select.POLLOUT:
                         self.__process_write(fd)
-                    if event & select.POLLIN:
+                    elif event & select.POLLIN:
                         self.process_socket(fd)
+                    else:
+                        print("[b10-msgq] Error: Unknown even in run_poller()")
 
     def run_kqueue(self):
-        while True:
+        while self.running:
+            # Check with a timeout so that every once in a while,
+            # the loop checks for self.running.
             events = self.kqueue.control(None, 10)
             if not events:
                 raise RuntimeError('serve: kqueue returned no events')
@@ -500,6 +525,9 @@ class MsgQ:
                         self.kill_socket(event.ident,
                                          self.sockets[event.ident])
 
+    def stop(self):
+        self.running = False
+
     def shutdown(self):
         """Stop the MsgQ master."""
         if self.verbose:
@@ -543,9 +571,10 @@ if __name__ == "__main__":
 
     msgq = MsgQ(options.msgq_socket_file, options.verbose)
 
-    setup_result = msgq.setup()
-    if setup_result:
-        sys.stderr.write("[b10-msgq] Error on startup: %s\n" % setup_result)
+    try:
+        msgq.setup()
+    except Exception as e:
+        sys.stderr.write("[b10-msgq] Error on startup: %s\n" % str(e))
         sys.exit(1)
 
     try:

src/bin/msgq/tests/msgq_test.py

@@ -6,7 +6,10 @@ import socket
 import signal
 import sys
 import time
+import errno
+import threading
 import isc.cc
+import collections
 
 #
 # Currently only the subscription part and some sending is implemented...
@@ -112,6 +115,85 @@ class TestSubscriptionManager(unittest.TestCase):
         msgq = MsgQ("/does/not/exist")
         self.assertRaises(socket.error, msgq.setup)
 
+class DummySocket:
+    """
+    Dummy socket class.
+    This one does nothing at all, but some calls are used.
+    It is mainly intended to override the listen socket for msgq, which
+    we do not need in these tests.
+    """
+    def fileno():
+        return -1
+
+    def close():
+        pass
+
+class BadSocket:
+    """
+    Special socket wrapper class. Once given a socket in its constructor,
+    it completely behaves like that socket, except that its send() call
+    will only actually send one byte per call, and optionally raise a given
+    exception at a given time.
+    """
+    def __init__(self, real_socket, raise_on_send=0, send_exception=None):
+        """
+        Parameters:
+        real_socket: The actual socket to wrap
+        raise_on_send: integer. If higher than 0, and send_exception is
+                       not None, send_exception will be raised on the
+                       'raise_on_send'th call to send().
+        send_exception: if not None, this exception will be raised
+                        (if raise_on_send is not 0)
+        """
+        self.socket = real_socket
+        self.send_count = 0
+        self.raise_on_send = raise_on_send
+        self.send_exception = send_exception
+
+    # completely wrap all calls and member access
+    # (except explicitely overridden ones)
+    def __getattr__(self, name, *args):
+        attr = getattr(self.socket, name)
+        if isinstance(attr, collections.Callable):
+            def callable_attr(*args):
+                return attr.__call__(*args)
+            return callable_attr
+        else:
+            return attr
+
+    def send(self, data):
+        self.send_count += 1
+        if self.send_exception is not None and\
+           self.send_count == self.raise_on_send:
+            raise self.send_exception
+
+        if len(data) > 0:
+            return self.socket.send(data[:1])
+        else:
+            return 0
+
+class MsgQThread(threading.Thread):
+    """
+    Very simple thread class that runs msgq.run() when started,
+    and stores the exception that msgq.run() raises, if any.
+    """
+    def __init__(self, msgq):
+        threading.Thread.__init__(self)
+        self.msgq_ = msgq
+        self.caught_exception = None
+        self.lock = threading.Lock()
+
+    def run(self):
+        try:
+            self.msgq_.run()
+        except Exception as exc:
+            # Store the exception to make the test fail if necessary
+            self.caught_exception = exc
+
+    def stop(self):
+        self.msgq_.stop()
+
+
 class SendNonblock(unittest.TestCase):
     """
     Tests that the whole thing will not get blocked if someone does not read.
@@ -191,9 +273,6 @@ class SendNonblock(unittest.TestCase):
         msgq = MsgQ()
         # msgq.run needs to compare with the listen_socket, so we provide
         # a replacement
-        class DummySocket:
-            def fileno():
-                return -1
         msgq.listen_socket = DummySocket
         (queue, out) = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
         def run():
@@ -245,5 +324,137 @@ class SendNonblock(unittest.TestCase):
             data = data + data
         self.send_many(data)
 
+    def do_send(self, write, read, control_write, control_read,
+                expect_arrive=True, expect_send_exception=None):
+        """
+        Makes a msgq object that is talking to itself,
+        run it in a separate thread so we can use and
+        test run().
+        It is given two sets of connected sockets; write/read, and
+        control_write/control_read. The former may be throwing errors
+        and mangle data to test msgq. The second is mainly used to
+        send msgq the stop command.
+        (Note that the terms 'read' and 'write' are from the msgq
+        point of view, so the test itself writes to 'control_read')
+        Parameters:
+        write: a socket that is used to send the data to
+        read: a socket that is used to read the data from
+        control_write: a second socket for communication with msgq
+        control_read: a second socket for communication with msgq
+        expect_arrive: if True, the read socket is read from, and the data
+                       that is read is expected to be the same as the data
+                       that has been sent to the write socket.
+        expect_send_exception: if not None, this is the exception that is
+                               expected to be raised by msgq
+        """
+
+        # Some message and envelope data to send and check
+        env = b'{"env": "foo"}'
+        msg = b'{"msg": "bar"}'
+
+        msgq = MsgQ()
+        # Don't need a listen_socket
+        msgq.listen_socket = DummySocket
+        msgq.setup_poller()
+        msgq.register_socket(write)
+        msgq.register_socket(control_write)
+        # Queue the message for sending
+        msgq.sendmsg(write, env, msg)
+
+        # Run it in a thread
+        msgq_thread = MsgQThread(msgq)
+        # If we're done, just kill it
+        msgq_thread.start()
+
+        if expect_arrive:
+            (recv_env, recv_msg) = msgq.read_packet(read.fileno(),
+                read)
+            self.assertEqual(env, recv_env)
+            self.assertEqual(msg, recv_msg)
+
+        # Tell msgq to stop
+        msg = msgq.preparemsg({"type" : "stop"})
+        control_read.sendall(msg)
+
+        # Wait for thread to stop if it hasn't already.
+        # Put in a (long) timeout; the thread *should* stop, but if it
+        # does not, we don't want the test to hang forever
+        msgq_thread.join(60)
+        # Fail the test if it didn't stop
+        self.assertFalse(msgq_thread.isAlive(), "Thread did not stop")
+
+        # Check the exception from the thread, if any
+        # First, if we didn't expect it; reraise it (to make test fail and
+        # show the stacktrace for debugging)
+        if expect_send_exception is None:
+            if msgq_thread.caught_exception is not None:
+                raise msgq_thread.caught_exception
+        else:
+            # If we *did* expect it, fail it there was none
+            self.assertIsNotNone(msgq_thread.caught_exception)
+
+    def do_send_with_send_error(self, raise_on_send, send_exception,
+                                expect_answer=True,
+                                expect_send_exception=None):
+        """
+        Sets up two connected sockets, wraps the sender socket into a BadSocket
+        class, then performs a do_send() test.
+        Parameters:
+        raise_on_send: the byte at which send_exception should be raised
+                       (see BadSocket)
+        send_exception: the exception to raise (see BadSocket)
+        expect_answer: whether the send is expected to complete (and hence
+                       the read socket should get the message)
+        expect_send_exception: the exception msgq is expected to raise when
+                               send_exception is raised by BadSocket.
+        """
+        (write, read) = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
+        (control_write, control_read) = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
+        badwrite = BadSocket(write, raise_on_send, send_exception)
+        self.do_send(badwrite, read, control_write, control_read, expect_answer, expect_send_exception)
+        write.close()
+        read.close()
+        control_write.close()
+        control_read.close()
+
+    def test_send_raise_recoverable(self):
+        """
+        Test whether msgq survices a recoverable socket errors when sending.
+        Two tests are done: one where the error is raised on the 3rd octet,
+                            and one on the 23rd.
+        """
+        sockerr = socket.error
+        for err in [ errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR ]:
+            sockerr.errno = err
+            self.do_send_with_send_error(3, sockerr)
+            self.do_send_with_send_error(23, sockerr)
+
+    def test_send_raise_nonrecoverable(self):
+        """
+        Test whether msgq survives socket errors that are nonrecoverable
+        (for said socket that is, i.e. EPIPE etc).
+        Two tests are done: one where the error is raised on the 3rd octet,
+                            and one on the 23rd.
+        """
+        sockerr = socket.error
+        for err in [ errno.EPIPE, errno.ENOBUFS, errno.ECONNRESET ]:
+            sockerr.errno = err
+            self.do_send_with_send_error(3, sockerr, False)
+            self.do_send_with_send_error(23, sockerr, False)
+
+    def otest_send_raise_crash(self):
+        """
+        Test whether msgq does NOT survive on a general exception.
+        Note, perhaps it should; but we'd have to first discuss and decide
+        how it should recover (i.e. drop the socket and consider the client
+        dead?
+        It may be a coding problem in msgq itself, and we certainly don't
+        want to ignore those.
+        """
+        sockerr = Exception("just some general exception")
+        self.do_send_with_send_error(3, sockerr, False, sockerr)
+        self.do_send_with_send_error(23, sockerr, False, sockerr)
+
+
 if __name__ == '__main__':
     unittest.main()

src/bin/stats/stats-httpd-xsd.tpl

@@ -1,2 +1,97 @@
 <?xml version="1.0" encoding="UTF-8"?>
-$xsd_string
+<schema targetNamespace="$xsd_namespace"
+        xmlns="http://www.w3.org/2001/XMLSchema"
+        xmlns:bind10="$xsd_namespace">
+  <annotation>
+    <documentation>XML schema of statistics data in BIND 10</documentation>
+  </annotation>
+  <element name="statistics">
+    <annotation><documentation>A set of statistics data</documentation></annotation>
+    <complexType>
+      <sequence>
+        <element name="item" maxOccurs="unbounded" minOccurs="1">
+          <complexType>
+	    <attribute name="identifier" type="string" use="required">
+            <annotation>
+              <appinfo>Identifier</appinfo>
+              <documentation>Identifier of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="value" type="string" use="optional">
+            <annotation>
+              <appinfo>Value</appinfo>
+              <documentation>Value of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="owner" type="string" use="required">
+            <annotation>
+              <appinfo>Owner</appinfo>
+              <documentation>Owner module name</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="uri" type="anyURI" use="required">
+            <annotation>
+              <appinfo>URI</appinfo>
+              <documentation>URI of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="name" type="string" use="required">
+            <annotation>
+              <appinfo>Name</appinfo>
+              <documentation>Name of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="type" use="required">
+  	    	<annotation>
+  	    	  <appinfo>Type</appinfo>
+  	    	  <documentation>Type of item</documentation>
+  	    	</annotation>
+  	    	<simpleType>
+  	    	  <restriction base="token">
+  	    	    <enumeration value="boolean"/>
+  	    	    <enumeration value="integer"/>
+  	    	    <enumeration value="real"/>
+  	    	    <enumeration value="string"/>
+  	    	    <enumeration value="map"/>
+  	    	    <enumeration value="list"/>
+  	    	    <enumeration value="named_set"/>
+  	    	    <enumeration value="any"/>
+  	    	  </restriction>
+  	    	</simpleType>
+  	    </attribute>
+  	    <attribute name="description" type="string" use="optional">
+            <annotation>
+              <appinfo>Description</appinfo>
+              <documentation>Description of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="title" type="string" use="optional">
+            <annotation>
+              <appinfo>Title</appinfo>
+              <documentation>Title of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="optional" type="boolean" use="optional">
+            <annotation>
+              <appinfo>Optional</appinfo>
+              <documentation>The item is optional or not</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="default" type="string" use="optional">
+            <annotation>
+              <appinfo>Default</appinfo>
+              <documentation>Default of item</documentation>
+            </annotation>
+  	    </attribute>
+  	    <attribute name="format" type="string" use="optional">
+            <annotation>
+              <appinfo>Format</appinfo>
+              <documentation>Format of item value</documentation>
+            </annotation>
+  	    </attribute>
+          </complexType>
+        </element>
+      </sequence>
+    </complexType>
+  </element>
+</schema>

src/bin/stats/stats-httpd-xsl.tpl

@@ -30,5 +30,27 @@ td.title {
       </body>
     </html>
   </xsl:template>
-  $xsl_string
+  <xsl:template match="bind10:statistics">
+    <table>
+      <tr>
+	<th>Identifier</th><th>Value</th><th>Description</th>
+      </tr>
+      <xsl:for-each select="item">
+	<tr>
+	  <td>
+	    <xsl:element name="a">
+	      <xsl:attribute name="href"><xsl:value-of select="@uri" /></xsl:attribute>
+	      <xsl:value-of select="@identifier" />
+	    </xsl:element>
+	  </td>
+	  <td>
+	    <xsl:if test="@value"><xsl:value-of select="@value" /></xsl:if>
+	  </td>
+	  <td>
+	    <xsl:if test="@description"><xsl:value-of select="@description" /></xsl:if>
+	  </td>
+	</tr>
+      </xsl:for-each>
+    </table>
+  </xsl:template>
 </xsl:stylesheet>

src/bin/stats/stats_httpd.py.in

@@ -1,6 +1,6 @@
 #!@PYTHON@
 
-# Copyright (C) 2011  Internet Systems Consortium.
+# Copyright (C) 2011-2012  Internet Systems Consortium.
 #
 # Permission to use, copy, modify, and distribute this software for any
 # purpose with or without fee is hereby granted, provided that the above
@@ -30,6 +30,7 @@ import socket
 import string
 import xml.etree.ElementTree
 import urllib.parse
+import re
 
 import isc.cc
 import isc.config
@@ -64,14 +65,70 @@ XSL_TEMPLATE_LOCATION = BASE_LOCATION + os.sep + "stats-httpd-xsl.tpl"
 # These variables are paths part of URL.
 # eg. "http://${address}" + XXX_URL_PATH
 XML_URL_PATH = '/bind10/statistics/xml'
-XSD_URL_PATH = '/bind10/statistics/xsd'
-XSL_URL_PATH = '/bind10/statistics/xsl'
+XSD_URL_PATH = '/bind10/statistics.xsd'
+XSL_URL_PATH = '/bind10/statistics.xsl'
 # TODO: This should be considered later.
 XSD_NAMESPACE = 'http://bind10.isc.org/bind10'
+XMLNS_XSI = "http://www.w3.org/2001/XMLSchema-instance"
+
+# constant parameter of XML
+XML_ROOT_ELEMENT = 'bind10:statistics'
+XML_ROOT_ATTRIB = { 'xsi:schemaLocation' : '%s %s' % (XSD_NAMESPACE, XSD_URL_PATH),
+                    'xmlns:bind10' : XSD_NAMESPACE,
+                    'xmlns:xsi' : XMLNS_XSI }
 
 # Assign this process name
 isc.util.process.rename()
 
+def item_name_list(element, identifier):
+    """Return a list of strings. The strings are string expression of
+    the first argument element which is dict type. The second argument
+    identifier is a string for specifying the strings which are
+    returned from this method as a list. For example, if we specify as
+
+      item_name_list({'a': {'aa': [0, 1]}, 'b': [0, 1]}, 'a/aa'),
+
+    then it returns
+
+      ['a/aa', 'a/aa[0]', 'a/aa[1]'].
+
+    If an empty string is specified in the second argument, all
+    possible strings are returned as a list.  In that example if we
+    specify an empty string in the second argument, then it returns
+
+      ['a', 'a/aa', 'a/aa[0]', 'a/aa[1]', 'b', 'b[0]', 'b[1]'].
+
+    The key name of element which is in the first argument is sorted.
+    Even if we specify as
+
+      item_name_list({'xx': 0, 'a': 1, 'x': 2}, ''),
+
+    then it returns
+
+      ['a', 'x', 'xx'].
+
+    This method internally invokes isc.cc.data.find(). The arguments
+    of this method are passed to isc.cc.data.find(). So an exception
+    DataNotFoundError or DataTypeError might be raised via
+    isc.cc.data.find() depending on the arguments. See details of
+    isc.cc.data.find() for more information about exceptions"""
+    elem = isc.cc.data.find(element, identifier)
+    ret = []
+    ident = identifier
+    if ident:
+        ret.append(ident)
+    if type(elem) is dict:
+        if ident:
+            ident = ident + '/'
+        for key in sorted(elem.keys()):
+            idstr = '%s%s' % (ident, key)
+            ret += item_name_list(element, idstr)
+    elif type(elem) is list:
+        for i in range(0, len(elem)):
+            idstr = '%s[%d]' % (ident, i)
+            ret += item_name_list(element, idstr)
+    return ret
+
 class HttpHandler(http.server.BaseHTTPRequestHandler):
     """HTTP handler class for HttpServer class. The class inhrits the super
     class http.server.BaseHTTPRequestHandler. It implemets do_GET()
@@ -89,31 +146,37 @@ class HttpHandler(http.server.BaseHTTPRequestHandler):
             req_path = self.path
             req_path = urllib.parse.urlsplit(req_path).path
             req_path = urllib.parse.unquote(req_path)
-            req_path = os.path.normpath(req_path)
-            path_dirs = req_path.split('/')
-            path_dirs = [ d for d in filter(None, path_dirs) ]
-            req_path = '/'+"/".join(path_dirs)
-            module_name = None
-            item_name = None
-            # in case of /bind10/statistics/xxx/YYY/zzz
-            if len(path_dirs) >= 5:
-                item_name = path_dirs[4]
-            # in case of /bind10/statistics/xxx/YYY ...
-            if len(path_dirs) >= 4:
-                module_name = path_dirs[3]
-            if req_path == '/'.join([XML_URL_PATH] + path_dirs[3:5]):
-                body = self.server.xml_handler(module_name, item_name)
-            elif req_path == '/'.join([XSD_URL_PATH] + path_dirs[3:5]):
-                body = self.server.xsd_handler(module_name, item_name)
-            elif req_path == '/'.join([XSL_URL_PATH] + path_dirs[3:5]):
-                body = self.server.xsl_handler(module_name, item_name)
+            body = None
+            # In case that the requested path (req_path),
+            # e.g. /bind10/statistics/Auth/, is started with
+            # XML_URL_PATH + '/'
+            if req_path.find('%s/' % XML_URL_PATH) == 0:
+                # remove './' from the path if there is
+                req_path = os.path.normpath(req_path)
+                # get the strings tailing after XML_URL_PATH
+                req_path = req_path.lstrip('%s/' % XML_URL_PATH)
+                # remove empty dir names from the path if there are
+                path_dirs = req_path.split('/')
+                path_dirs = [ d for d in filter(None, path_dirs) ]
+                req_path = '/'.join(path_dirs)
+                # pass the complete requested path,
+                # e.g. Auth/queries.upd, to xml_handler()
+                body = self.server.xml_handler(req_path)
+            # In case that the requested path (req_path) is exactly
+            # matched with XSD_URL_PATH
+            elif req_path == XSD_URL_PATH:
+                body = self.server.xsd_handler()
+            # In case that the requested path (req_path) is exactly
+            # matched with XSL_URL_PATH
+            elif req_path == XSL_URL_PATH:
+                body = self.server.xsl_handler()
             else:
-                if req_path == '/' and 'Host' in self.headers.keys():
-                    # redirect to XML URL only when requested with '/'
+                if 'Host' in self.headers.keys() and \
+                        ( req_path == '/' or req_path == XML_URL_PATH ):
+                    # redirect to XML URL only when requested with '/' or XML_URL_PATH
+                    toloc = "http://%s%s/" % (self.headers.get('Host'), XML_URL_PATH)
                     self.send_response(302)
-                    self.send_header(
-                        "Location",
-                        "http://" + self.headers.get('Host') + XML_URL_PATH)
+                    self.send_header("Location", toloc)
                     self.end_headers()
                     return None
                 else:
@@ -126,7 +189,7 @@ class HttpHandler(http.server.BaseHTTPRequestHandler):
             self.send_error(404)
             logger.error(STATHTTPD_SERVER_DATAERROR, err)
             return None
-        except StatsHttpdError as err:
+        except Exception as err:
             self.send_error(500)
             logger.error(STATHTTPD_SERVER_ERROR, err)
             return None
@@ -134,6 +197,7 @@ class HttpHandler(http.server.BaseHTTPRequestHandler):
             self.send_response(200)
             self.send_header("Content-type", "text/xml")
             self.send_header("Content-Length", len(body))
+            self.send_header("Last-Modified", self.date_time_string(time.time()))
             self.end_headers()
             return body
 
@@ -436,71 +500,66 @@ class StatsHttpd:
                                   (err.__class__.__name__, err))
 
 
-    def xml_handler(self, module_name=None, item_name=None):
+    def xml_handler(self, path=''):
         """Requests the specified statistics data and specification by
         using the functions get_stats_data and get_stats_spec
         respectively and loads the XML template file and returns the
-        string of the XML document.The first argument is the module
-        name which owns the statistics data, the second argument is
-        one name of the statistics items which the the module
-        owns. The second argument cannot be specified when the first
-        argument is not specified."""
-
-        # TODO: Separate the following recursive function by type of
-        # the parameter. Because we should be sure what type there is
-        # when we call it recursively.
-        def stats_data2xml(stats_spec, stats_data, xml_elem):
-            """Internal use for xml_handler. Reads stats_data and
-            stats_spec specified as first and second arguments, and
-            modify the xml object specified as third
-            argument. xml_elem must be modified and always returns
-            None."""
-            # assumed started with module_spec or started with
-            # item_spec in statistics
-            if type(stats_spec) is dict:
-                # assumed started with module_spec
-                if 'item_name' not in stats_spec \
-                        and 'item_type' not in stats_spec:
-                    for module_name in stats_spec.keys():
-                        elem = xml.etree.ElementTree.Element(module_name)
-                        stats_data2xml(stats_spec[module_name],
-                                       stats_data[module_name], elem)
-                        xml_elem.append(elem)
-                # started with item_spec in statistics
-                else:
-                    elem = xml.etree.ElementTree.Element(stats_spec['item_name'])
-                    if stats_spec['item_type'] == 'map':
-                        stats_data2xml(stats_spec['map_item_spec'],
-                                       stats_data,
-                                       elem)
-                    elif stats_spec['item_type'] == 'list':
-                        for item in stats_data:
-                            stats_data2xml(stats_spec['list_item_spec'],
-                                           item, elem)
-                    else:
-                        elem.text = str(stats_data)
-                    xml_elem.append(elem)
-            # assumed started with stats_spec
-            elif type(stats_spec) is list:
-                for item_spec in stats_spec:
-                    stats_data2xml(item_spec,
-                                   stats_data[item_spec['item_name']],
-                                   xml_elem)
-
+        string of the XML document.The argument is a path in the
+        requested URI. For example, the path is assumed to be like
+        ${module_name}/${top_level_item_name}/${second_level_item_name}/..."""
+
+        dirs = [ d for d in path.split("/") if len(d) > 0 ]
+        module_name = None
+        item_name = None
+        if len(dirs) > 0:
+            module_name = dirs[0]
+        if len(dirs) > 1:
+            item_name = dirs[1]
+            # removed an index string when list-type value is
+            # requested. Because such a item name can be accept by the
+            # stats module currently.
+            item_name = re.sub('\[\d+\]$', '', item_name)
         stats_spec = self.get_stats_spec(module_name, item_name)
         stats_data = self.get_stats_data(module_name, item_name)
-        # make the path xxx/module/item if specified respectively
-        path_info = ''
-        if module_name is not None and item_name is not None:
-            path_info = '/' + module_name + '/' + item_name
-        elif module_name is not None:
-            path_info = '/' + module_name
+        path_list = []
+        try:
+            path_list = item_name_list(stats_data, path)
+        except isc.cc.data.DataNotFoundError as err:
+            raise StatsHttpdDataError(
+                "%s: %s" % (err.__class__.__name__, err))
+        item_list = []
+        for path in path_list:
+            dirs = path.split("/")
+            if len(dirs) < 2: continue
+            item = {}
+            item['identifier'] = path
+            value = isc.cc.data.find(stats_data, path)
+            if type(value) is bool:
+                value = str(value).lower()
+            if type(value) is dict or type(value) is list:
+                value = None
+            if value is not None:
+                item['value'] = str(value)
+            owner = dirs[0]
+            item['owner'] = owner
+            item['uri'] = urllib.parse.quote('%s/%s' % (XML_URL_PATH, path))
+            item_path = '/'.join(dirs[1:])
+            spec = isc.config.find_spec_part(stats_spec[owner], item_path)
+            for key in ['name', 'type', 'description', 'title', \
+                            'optional', 'default', 'format']:
+                value = spec.get('item_%s' % key)
+                if type(value) is bool:
+                    value = str(value).lower()
+                if type(value) is dict or type(value) is list:
+                    value = None
+                if value is not None:
+                    item[key] = str(value)
+            item_list.append(item)
         xml_elem = xml.etree.ElementTree.Element(
-            'bind10:statistics',
-            attrib={ 'xsi:schemaLocation' : XSD_NAMESPACE + ' ' + XSD_URL_PATH + path_info,
-                     'xmlns:bind10' : XSD_NAMESPACE,
-                     'xmlns:xsi' : "http://www.w3.org/2001/XMLSchema-instance" })
-        stats_data2xml(stats_spec, stats_data, xml_elem)
+            XML_ROOT_ELEMENT, attrib=XML_ROOT_ATTRIB)
+        for item in item_list:
+            item_elem = xml.etree.ElementTree.Element('item', attrib=item)
+            xml_elem.append(item_elem)
         # The coding conversion is tricky. xml..tostring() of Python 3.2
         # returns bytes (not string) regardless of the coding, while
         # tostring() of Python 3.1 returns a string.  To support both
@@ -512,313 +571,34 @@ class StatsHttpd:
         xml_string = str(xml.etree.ElementTree.tostring(xml_elem, encoding='utf-8'),
                          encoding='us-ascii')
         self.xml_body = self.open_template(XML_TEMPLATE_LOCATION).substitute(
-            xml_string=xml_string,
-            xsl_url_path=XSL_URL_PATH + path_info)
-        assert self.xml_body is not None
+            xml_string=xml_string, xsl_url_path=XSL_URL_PATH)
         return self.xml_body
 
-    def xsd_handler(self, module_name=None, item_name=None):
-        """Requests the specified statistics specification by using
-        the function get_stats_spec respectively and loads the XSD
-        template file and returns the string of the XSD document.The
-        first argument is the module name which owns the statistics
-        data, the second argument is one name of the statistics items
-        which the the module owns. The second argument cannot be
-        specified when the first argument is not specified."""
-
-        # TODO: Separate the following recursive function by type of
-        # the parameter. Because we should be sure what type there is
-        # when we call it recursively.
-        def stats_spec2xsd(stats_spec, xsd_elem):
-            """Internal use for xsd_handler. Reads stats_spec
-            specified as first arguments, and modify the xml object
-            specified as second argument. xsd_elem must be
-            modified. Always returns None with no exceptions."""
-            # assumed module_spec or one stats_spec
-            if type(stats_spec) is dict:
-                # assumed module_spec
-                if 'item_name' not in stats_spec:
-                    for mod in stats_spec.keys():
-                        elem = xml.etree.ElementTree.Element(
-                            "element", { "name" : mod })
-                        complextype = xml.etree.ElementTree.Element("complexType")
-                        alltag = xml.etree.ElementTree.Element("all")
-                        stats_spec2xsd(stats_spec[mod], alltag)
-                        complextype.append(alltag)
-                        elem.append(complextype)
-                        xsd_elem.append(elem)
-                # assumed stats_spec
-                else:
-                    if stats_spec['item_type'] == 'map':
-                        alltag = xml.etree.ElementTree.Element("all")
-                        stats_spec2xsd(stats_spec['map_item_spec'], alltag)
-                        complextype = xml.etree.ElementTree.Element("complexType")
-                        complextype.append(alltag)
-                        elem = xml.etree.ElementTree.Element(
-                            "element", attrib={ "name" : stats_spec["item_name"],
-                                                "minOccurs": "0" \
-                                                    if stats_spec["item_optional"] \
-                                                    else "1",
-                                                "maxOccurs": "unbounded" })
-                        elem.append(complextype)
-                        xsd_elem.append(elem)
-                    elif stats_spec['item_type'] == 'list':
-                        alltag = xml.etree.ElementTree.Element("sequence")
-                        stats_spec2xsd(stats_spec['list_item_spec'], alltag)
-                        complextype = xml.etree.ElementTree.Element("complexType")
-                        complextype.append(alltag)
-                        elem = xml.etree.ElementTree.Element(
-                            "element", attrib={ "name" : stats_spec["item_name"],
-                                                "minOccurs": "0" \
-                                                    if stats_spec["item_optional"] \
-                                                    else "1",
-                                                "maxOccurs": "1" })
-                        elem.append(complextype)
-                        xsd_elem.append(elem)
-                    else:
-                        # determine the datatype of XSD
-                        # TODO: Should consider other item_format types
-                        datatype = stats_spec["item_type"] \
-                            if stats_spec["item_type"].lower() != 'real' \
-                            else 'float'
-                        if "item_format" in stats_spec:
-                            item_format = stats_spec["item_format"]
-                            if datatype.lower() == 'string' \
-                                    and item_format.lower() == 'date-time':
-                                 datatype = 'dateTime'
-                            elif datatype.lower() == 'string' \
-                                    and (item_format.lower() == 'date' \
-                                             or item_format.lower() == 'time'):
-                                 datatype = item_format.lower()
-                        elem = xml.etree.ElementTree.Element(
-                            "element",
-                            attrib={
-                                'name' : stats_spec["item_name"],
-                                'type' : datatype,
-                                'minOccurs' : "0" \
-                                    if stats_spec["item_optional"] \
-                                    else "1",
-                                'maxOccurs' : "1"
-                                }
-                            )
-                        annotation = xml.etree.ElementTree.Element("annotation")
-                        appinfo = xml.etree.ElementTree.Element("appinfo")
-                        documentation = xml.etree.ElementTree.Element("documentation")
-                        if "item_title" in stats_spec:
-                            appinfo.text = stats_spec["item_title"]
-                        if "item_description" in stats_spec:
-                            documentation.text = stats_spec["item_description"]
-                        annotation.append(appinfo)
-                        annotation.append(documentation)
-                        elem.append(annotation)
-                        xsd_elem.append(elem)
-            # multiple stats_specs
-            elif type(stats_spec) is list:
-                for item_spec in stats_spec:
-                    stats_spec2xsd(item_spec, xsd_elem)
-
-        # for XSD
-        stats_spec = self.get_stats_spec(module_name, item_name)
-        alltag = xml.etree.ElementTree.Element("all")
-        stats_spec2xsd(stats_spec, alltag)
-        complextype = xml.etree.ElementTree.Element("complexType")
-        complextype.append(alltag)
-        documentation = xml.etree.ElementTree.Element("documentation")
-        documentation.text = "A set of statistics data"
-        annotation = xml.etree.ElementTree.Element("annotation")
-        annotation.append(documentation)
-        elem = xml.etree.ElementTree.Element(
-            "element", attrib={ 'name' : 'statistics' })
-        elem.append(annotation)
-        elem.append(complextype)
-        documentation = xml.etree.ElementTree.Element("documentation")
-        documentation.text = "XML schema of the statistics data in BIND 10"
-        annotation = xml.etree.ElementTree.Element("annotation")
-        annotation.append(documentation)
-        xsd_root = xml.etree.ElementTree.Element(
-            "schema",
-            attrib={ 'xmlns' : "http://www.w3.org/2001/XMLSchema",
-                     'targetNamespace' : XSD_NAMESPACE,
-                     'xmlns:bind10' : XSD_NAMESPACE })
-        xsd_root.append(annotation)
-        xsd_root.append(elem)
-        # The coding conversion is tricky. xml..tostring() of Python 3.2
-        # returns bytes (not string) regardless of the coding, while
-        # tostring() of Python 3.1 returns a string.  To support both
-        # cases transparently, we first make sure tostring() returns
-        # bytes by specifying utf-8 and then convert the result to a
-        # plain string (code below assume it).
-        # FIXME: Non-ASCII characters might be lost here. Consider how
-        # the whole system should handle non-ASCII characters.
-        xsd_string = str(xml.etree.ElementTree.tostring(xsd_root, encoding='utf-8'),
-                         encoding='us-ascii')
+    def xsd_handler(self):
+        """Loads the XSD template file, replaces the variable strings,
+        and returns the string of the XSD document."""
         self.xsd_body = self.open_template(XSD_TEMPLATE_LOCATION).substitute(
-            xsd_string=xsd_string)
-        assert self.xsd_body is not None
+            xsd_namespace=XSD_NAMESPACE)
         return self.xsd_body
 
-    def xsl_handler(self, module_name=None, item_name=None):
-        """Requests the specified statistics specification by using
-        the function get_stats_spec respectively and loads the XSL
-        template file and returns the string of the XSL document.The
-        first argument is the module name which owns the statistics
-        data, the second argument is one name of the statistics items
-        which the the module owns. The second argument cannot be
-        specified when the first argument is not specified."""
-
-        # TODO: Separate the following recursive function by type of
-        # the parameter. Because we should be sure what type there is
-        # when we call it recursively.
-        def stats_spec2xsl(stats_spec, xsl_elem, path=XML_URL_PATH):
-            """Internal use for xsl_handler. Reads stats_spec
-            specified as first arguments, and modify the xml object
-            specified as second argument. xsl_elem must be
-            modified. The third argument is a base path used for
-            making anchor tag in XSL. Always returns None with no
-            exceptions."""
-            # assumed module_spec or one stats_spec
-            if type(stats_spec) is dict:
-                # assumed module_spec
-                if 'item_name' not in stats_spec:
-                    table = xml.etree.ElementTree.Element("table")
-                    tr = xml.etree.ElementTree.Element("tr")
-                    th = xml.etree.ElementTree.Element("th")
-                    th.text = "Module Name"
-                    tr.append(th)
-                    th = xml.etree.ElementTree.Element("th")
-                    th.text = "Module Item"
-                    tr.append(th)
-                    table.append(tr)
-                    for mod in stats_spec.keys():
-                        foreach = xml.etree.ElementTree.Element(
-                            "xsl:for-each", attrib={ "select" : mod })
-                        tr = xml.etree.ElementTree.Element("tr")
-                        td = xml.etree.ElementTree.Element("td")
-                        a = xml.etree.ElementTree.Element(
-                            "a", attrib={ "href": urllib.parse.quote(path + "/" + mod) })
-                        a.text = mod
-                        td.append(a)
-                        tr.append(td)
-                        td = xml.etree.ElementTree.Element("td")
-                        stats_spec2xsl(stats_spec[mod], td,
-                                       path + "/" + mod)
-                        tr.append(td)
-                        foreach.append(tr)
-                        table.append(foreach)
-                    xsl_elem.append(table)
-                # assumed stats_spec
-                else:
-                    if stats_spec['item_type'] == 'map':
-                        table = xml.etree.ElementTree.Element("table")
-                        tr = xml.etree.ElementTree.Element("tr")
-                        th = xml.etree.ElementTree.Element("th")
-                        th.text = "Item Name"
-                        tr.append(th)
-                        th = xml.etree.ElementTree.Element("th")
-                        th.text = "Item Value"
-                        tr.append(th)
-                        table.append(tr)
-                        foreach = xml.etree.ElementTree.Element(
-                            "xsl:for-each", attrib={ "select" : stats_spec['item_name'] })
-                        tr = xml.etree.ElementTree.Element("tr")
-                        td = xml.etree.ElementTree.Element(
-                            "td",
-                            attrib={ "class" : "title",
-                                     "title" : stats_spec["item_description"] \
-                                         if "item_description" in stats_spec \
-                                         else "" })
-                        # TODO: Consider whether we should always use
-                        # the identical name "item_name" for the
-                        # user-visible name in XSL.
-                        td.text = stats_spec[ "item_title" if "item_title" in stats_spec else "item_name" ]
-                        tr.append(td)
-                        td = xml.etree.ElementTree.Element("td")
-                        stats_spec2xsl(stats_spec['map_item_spec'], td,
-                                       path + "/" + stats_spec["item_name"])
-                        tr.append(td)
-                        foreach.append(tr)
-                        table.append(foreach)
-                        xsl_elem.append(table)
-                    elif stats_spec['item_type'] == 'list':
-                        stats_spec2xsl(stats_spec['list_item_spec'], xsl_elem,
-                                       path + "/" + stats_spec["item_name"])
-                    else:
-                        xsl_valueof = xml.etree.ElementTree.Element(
-                            "xsl:value-of",
-                            attrib={'select': stats_spec["item_name"]})
-                        xsl_elem.append(xsl_valueof)
-
-            # multiple stats_specs
-            elif type(stats_spec) is list:
-                table = xml.etree.ElementTree.Element("table")
-                tr = xml.etree.ElementTree.Element("tr")
-                th = xml.etree.ElementTree.Element("th")
-                th.text = "Item Name"
-                tr.append(th)
-                th = xml.etree.ElementTree.Element("th")
-                th.text = "Item Value"
-                tr.append(th)
-                table.append(tr)
-                for item_spec in stats_spec:
-                    tr = xml.etree.ElementTree.Element("tr")
-                    td = xml.etree.ElementTree.Element(
-                        "td",
-                        attrib={ "class" : "title",
-                                 "title" : item_spec["item_description"] \
-                                     if "item_description" in item_spec \
-                                     else "" })
-                    # if the path length is equal to or shorter than
-                    # XML_URL_PATH + /Module/Item, add the anchor tag.
-                    if len(path.split('/')) <= len((XML_URL_PATH + '/Module/Item').split('/')):
-                        a = xml.etree.ElementTree.Element(
-                            "a", attrib={ "href": urllib.parse.quote(path + "/" + item_spec["item_name"]) })
-                        a.text = item_spec[ "item_title" if "item_title" in item_spec else "item_name" ]
-                        td.append(a)
-                    else:
-                        td.text = item_spec[ "item_title" if "item_title" in item_spec else "item_name" ]
-                    tr.append(td)
-                    td = xml.etree.ElementTree.Element("td")
-                    stats_spec2xsl(item_spec, td, path)
-                    tr.append(td)
-                    if item_spec['item_type'] == 'list':
-                        foreach = xml.etree.ElementTree.Element(
-                            "xsl:for-each", attrib={ "select" : item_spec['item_name'] })
-                        foreach.append(tr)
-                        table.append(foreach)
-                    else:
-                        table.append(tr)
-                xsl_elem.append(table)
-
-        # for XSL
-        stats_spec = self.get_stats_spec(module_name, item_name)
-        xsd_root = xml.etree.ElementTree.Element( # started with xml:template tag
-            "xsl:template",
-            attrib={'match': "bind10:statistics"})
-        stats_spec2xsl(stats_spec, xsd_root)
-        # The coding conversion is tricky. xml..tostring() of Python 3.2
-        # returns bytes (not string) regardless of the coding, while
-        # tostring() of Python 3.1 returns a string.  To support both
-        # cases transparently, we first make sure tostring() returns
-        # bytes by specifying utf-8 and then convert the result to a
-        # plain string (code below assume it).
-        # FIXME: Non-ASCII characters might be lost here. Consider how
-        # the whole system should handle non-ASCII characters.
-        xsl_string = str(xml.etree.ElementTree.tostring(xsd_root, encoding='utf-8'),
-                         encoding='us-ascii')
+    def xsl_handler(self):
+        """Loads the XSL template file, replaces the variable strings,
+        and returns the string of the XSL document."""
         self.xsl_body = self.open_template(XSL_TEMPLATE_LOCATION).substitute(
-            xsl_string=xsl_string,
             xsd_namespace=XSD_NAMESPACE)
-        assert self.xsl_body is not None
         return self.xsl_body
 
     def open_template(self, file_name):
         """It opens a template file, and it loads all lines to a
         string variable and returns string. Template object includes
-        the variable. Limitation of a file size isn't needed there."""
-        f = open(file_name, 'r')
-        lines = "".join(f.readlines())
-        f.close()
-        assert lines is not None
+        the variable. Limitation of a file size isn't needed there. XXXX"""
+        lines = None
+        try:
+            with open(file_name, 'r') as f:
+                lines = "".join(f.readlines())
+        except IOError as err:
+            raise StatsHttpdDataError(
+                "%s: %s" % (err.__class__.__name__, err))
         return string.Template(lines)
 
 if __name__ == "__main__":

src/bin/stats/stats_messages.mes

@@ -24,6 +24,10 @@ The stats module was unable to connect to the BIND 10 command and
 control bus. A likely problem is that the message bus daemon
 (b10-msgq) is not running. The stats module will now shut down.
 
+% STATS_RECEIVED_INVALID_STATISTICS_DATA received invalid statistics data from %1
+Invalid statistics data has been received from the module while
+polling and it has been discarded.
+
 % STATS_RECEIVED_NEW_CONFIG received new configuration: %1
 This debug message is printed when the stats module has received a
 configuration update from the configuration manager.
@@ -53,10 +57,6 @@ response indicating that it is running normally.
 An unknown command has been sent to the stats module. The stats module
 will respond with an error and the command will be ignored.
 
-% STATS_RECEIVED_INVALID_STATISTICS_DATA received invalid statistics data from %1
-Invalid statistics data has been received from the module while
-polling and it has been discarded.
-
 % STATS_SEND_STATISTICS_REQUEST requesting %1 to send statistics
 This debug message is printed when a request is sent to the module
 to send its data to the stats module.

src/bin/stats/tests/test_utils.py

@@ -1,3 +1,18 @@
+# Copyright (C) 2011-2012  Internet Systems Consortium.
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
+# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
+# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
+# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
+# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
 """
 Utilities and mock modules for unittests of statistics modules
 
@@ -16,6 +31,8 @@ import isc.config.cfgmgr
 import stats
 import stats_httpd
 
+CONST_BASETIME = (2011, 6, 22, 8, 14, 8, 2, 173, 0)
+
 class SignalHandler():
     """A signal handler class for deadlock in unittest"""
     def __init__(self, fail_handler, timeout=20):
@@ -222,7 +239,7 @@ class MockBoss:
   }
 }
 """
-    _BASETIME = (2011, 6, 22, 8, 14, 8, 2, 173, 0)
+    _BASETIME = CONST_BASETIME
 
     def __init__(self):
         self._started = threading.Event()
@@ -457,6 +474,11 @@ class MockAuth:
         return isc.config.create_answer(1, "Unknown Command")
 
 class MyStats(stats.Stats):
+
+    stats._BASETIME = CONST_BASETIME
+    stats.get_timestamp = lambda: time.mktime(CONST_BASETIME)
+    stats.get_datetime = lambda x=None: time.strftime("%Y-%m-%dT%H:%M:%SZ", CONST_BASETIME)
+
     def __init__(self):
         self._started = threading.Event()
         stats.Stats.__init__(self)

src/bin/xfrin/tests/xfrin_test.py

@@ -570,7 +570,7 @@ class TestXfrinIXFRAdd(TestXfrinState):
         # difference, starting with removing that SOA.
         self.conn._diff.add_data(self.ns_rrset) # put some dummy change
         self.conn._tsig_ctx = MockTSIGContext(TSIG_KEY)
-        self.conn._tsig_ctx.last_has_signature = lambda: False
+        self.conn._tsig_ctx.last_had_signature = lambda: False
         # First, push a starting SOA inside. This should be OK, nothing checked
         # yet.
         self.state.handle_rr(self.conn, self.begin_soa)
@@ -821,7 +821,7 @@ class TestAXFR(TestXfrinConnection):
         mock_ctx = MockTSIGContext(key)
         mock_ctx.error = error
         if not has_last_signature:
-            mock_ctx.last_has_signature = lambda: False
+            mock_ctx.last_had_signature = lambda: False
         return mock_ctx
 
     def __match_exception(self, expected_exception, expected_msg, expression):

src/bin/xfrin/xfrin.py.in

@@ -797,7 +797,7 @@ class XfrinConnection(asyncore.dispatcher):
         Check there's a signature at the last message.
         """
         if self._tsig_ctx is not None:
-            if not self._tsig_ctx.last_has_signature():
+            if not self._tsig_ctx.last_had_signature():
                 raise XfrinProtocolError('TSIG verify fail: no TSIG on last '+
                                          'message')
 

src/bin/xfrout/xfrout_messages.mes

@@ -107,10 +107,6 @@ received from the configuration manager.
 The xfrout daemon received a command on the command channel that
 NOTIFY packets should be sent for the given zone.
 
-% XFROUT_RECEIVED_GETSTATS_COMMAND received command to get statistics data
-The xfrout daemon received a command on the command channel that
-statistics data should be sent to the stats daemon.
-
 % XFROUT_PARSE_QUERY_ERROR error parsing query: %1
 There was a parse error while reading an incoming query. The parse
 error is shown in the log message. A remote client sent a packet we
@@ -151,6 +147,10 @@ given host. This is because of ACLs.  The %2 represents the IP
 address and port of the peer requesting the transfer, and the %3
 represents the zone name and class.
 
+% XFROUT_RECEIVED_GETSTATS_COMMAND received command to get statistics data
+The xfrout daemon received a command on the command channel that
+statistics data should be sent to the stats daemon.
+
 % XFROUT_RECEIVED_SHUTDOWN_COMMAND shutdown command received
 The xfrout daemon received a shutdown command from the command channel
 and will now shut down.

src/bin/zonemgr/zonemgr.py.in

@@ -193,7 +193,8 @@ class ZonemgrRefresh:
     def zone_handle_notify(self, zone_name_class, master):
         """Handle zone notify"""
         if (self._zone_not_exist(zone_name_class)):
-            logger.error(ZONEMGR_UNKNOWN_ZONE_NOTIFIED, zone_name_class[0], zone_name_class[1])
+            logger.error(ZONEMGR_UNKNOWN_ZONE_NOTIFIED, zone_name_class[0],
+                         zone_name_class[1], master)
             raise ZonemgrException("[b10-zonemgr] Notified zone (%s, %s) "
                                    "doesn't belong to zonemgr" % zone_name_class)
         self._set_zone_notifier_master(zone_name_class, master)

src/bin/zonemgr/zonemgr_messages.mes

@@ -138,7 +138,7 @@ zone, or, if this error appears without the administrator giving transfer
 commands, it can indicate an error in the program, as it should not have
 initiated transfers of unknown zones on its own.
 
-% ZONEMGR_UNKNOWN_ZONE_NOTIFIED notified zone %1 (class %2) is not known to the zone manager
+% ZONEMGR_UNKNOWN_ZONE_NOTIFIED notified zone %1/%2 from %3 is not known to the zone manager
 A NOTIFY was received but the zone that was the subject of the operation
 is not being managed by the zone manager.  This may indicate an error
 in the program (as the operation should not have been initiated if this

src/lib/Makefile.am

@@ -1,3 +1,3 @@
 SUBDIRS = exceptions util log cryptolink dns cc config acl xfr bench \
           asiolink asiodns nsas cache resolve testutils datasrc \
-          server_common python dhcp statistics
+          server_common python dhcp dhcpsrv statistics

src/lib/asiolink/io_address.cc

@@ -61,7 +61,7 @@ IOAddress::toText() const {
 }
 
 IOAddress
-IOAddress::from_bytes(short family, const uint8_t* data) {
+IOAddress::fromBytes(short family, const uint8_t* data) {
     if (data == NULL) {
         isc_throw(BadValue, "NULL pointer received.");
     } else
@@ -76,6 +76,21 @@ IOAddress::from_bytes(short family, const uint8_t* data) {
     return IOAddress(string(addr_str));
 }
 
+std::vector<uint8_t>
+IOAddress::toBytes() const {
+    if (asio_address_.is_v4()) {
+        const asio::ip::address_v4::bytes_type bytes4 =
+            asio_address_.to_v4().to_bytes();
+        return (std::vector<uint8_t>(bytes4.begin(), bytes4.end()));
+    }
+
+    // Not V4 address, so must be a V6 address (else we could never construct
+    // this object).
+    const asio::ip::address_v6::bytes_type bytes6 =
+        asio_address_.to_v6().to_bytes();
+    return (std::vector<uint8_t>(bytes6.begin(), bytes6.end()));
+}
+
 short
 IOAddress::getFamily() const {
     if (asio_address_.is_v4()) {

src/lib/asiolink/io_address.h

@@ -24,6 +24,7 @@
 
 #include <functional>
 #include <string>
+#include <vector>
 
 #include <exceptions/exceptions.h>
 
@@ -103,6 +104,19 @@ public:
     /// \return AF_INET for IPv4 or AF_INET6 for IPv6.
     short getFamily() const;
 
+    /// \brief Convenience function to check for an IPv4 address
+    ///
+    /// \return true if the address is a V4 address
+    bool isV4() const {
+        return (asio_address_.is_v4());
+    }
+
+    /// \brief Convenience function to check for an IPv6 address
+    ///
+    /// \return true if the address is a V6 address
+    bool isV6() const {
+        return (asio_address_.is_v6());
+    }
 
     /// \brief Creates an address from over wire data.
     ///
@@ -110,8 +124,13 @@ public:
     /// \param data pointer to first char of data
     ///
     /// \return Created IOAddress object
-    static IOAddress
-    from_bytes(short family, const uint8_t* data);
+    static IOAddress fromBytes(short family, const uint8_t* data);
+
+    /// \brief Return address as set of bytes
+    ///
+    /// \return Contents of the address as a set of bytes in network-byte
+    ///         order.
+    std::vector<uint8_t> toBytes() const;
 
     /// \brief Compare addresses for equality
     ///

src/lib/asiolink/tests/io_address_unittest.cc

@@ -18,7 +18,9 @@
 #include <asiolink/io_error.h>
 #include <asiolink/io_address.h>
 
+#include <algorithm>
 #include <cstring>
+#include <vector>
 
 using namespace isc::asiolink;
 
@@ -64,7 +66,7 @@ TEST(IOAddressTest, Family) {
     EXPECT_EQ(AF_INET6, IOAddress("2001:0DB8:0:0::0012").getFamily());
 }
 
-TEST(IOAddressTest, from_bytes) {
+TEST(IOAddressTest, fromBytes) {
     // 2001:db8:1::dead:beef
     uint8_t v6[] = {
         0x20, 0x01, 0x0d, 0xb8, 0x00, 0x01, 0, 0,
@@ -74,16 +76,55 @@ TEST(IOAddressTest, from_bytes) {
 
     IOAddress addr("::");
     EXPECT_NO_THROW({
-        addr = IOAddress::from_bytes(AF_INET6, v6);
+        addr = IOAddress::fromBytes(AF_INET6, v6);
     });
     EXPECT_EQ("2001:db8:1::dead:beef", addr.toText());
 
     EXPECT_NO_THROW({
-        addr = IOAddress::from_bytes(AF_INET, v4);
+        addr = IOAddress::fromBytes(AF_INET, v4);
     });
     EXPECT_EQ(addr.toText(), IOAddress("192.0.2.3").toText());
 }
 
+TEST(IOAddressTest, toBytesV4) {
+    // Address and network byte-order representation of the address.
+    const char* V4STRING = "192.0.2.1";
+    uint8_t V4[] = {0xc0, 0x00, 0x02, 0x01};
+
+    std::vector<uint8_t> actual = IOAddress(V4STRING).toBytes();
+    ASSERT_EQ(sizeof(V4), actual.size());
+    EXPECT_TRUE(std::equal(actual.begin(), actual.end(), V4));
+}
+
+TEST(IOAddressTest, toBytesV6) {
+    // Address and network byte-order representation of the address.
+    const char* V6STRING = "2001:db8:1::dead:beef";
+    uint8_t V6[] = {
+        0x20, 0x01, 0x0d, 0xb8, 0x00, 0x01, 0x00, 0x00,
+        0x00, 0x00, 0x00, 0x00, 0xde, 0xad, 0xbe, 0xef 
+    };
+
+    std::vector<uint8_t> actual = IOAddress(V6STRING).toBytes();
+    ASSERT_EQ(sizeof(V6), actual.size());
+    EXPECT_TRUE(std::equal(actual.begin(), actual.end(), V6));
+}
+
+TEST(IOAddressTest, isV4) {
+    const IOAddress address4("192.0.2.1");
+    const IOAddress address6("2001:db8:1::dead:beef");
+
+    EXPECT_TRUE(address4.isV4());
+    EXPECT_FALSE(address6.isV4());
+}
+
+TEST(IOAddressTest, isV6) {
+    const IOAddress address4("192.0.2.1");
+    const IOAddress address6("2001:db8:1::dead:beef");
+
+    EXPECT_FALSE(address4.isV6());
+    EXPECT_TRUE(address6.isV6());
+}
+
 TEST(IOAddressTest, uint32) {
     IOAddress addr1("192.0.2.5");
 

src/lib/datasrc/Makefile.am

@@ -26,6 +26,7 @@ lib_LTLIBRARIES = libb10-datasrc.la
 libb10_datasrc_la_SOURCES = data_source.h
 libb10_datasrc_la_SOURCES += rbnode_rrset.h
 libb10_datasrc_la_SOURCES += rbtree.h
+libb10_datasrc_la_SOURCES += exceptions.h
 libb10_datasrc_la_SOURCES += zonetable.h zonetable.cc
 libb10_datasrc_la_SOURCES += zone.h zone_finder.cc zone_finder_context.cc
 libb10_datasrc_la_SOURCES += result.h
@@ -35,6 +36,8 @@ libb10_datasrc_la_SOURCES += database.h database.cc
 libb10_datasrc_la_SOURCES += factory.h factory.cc
 libb10_datasrc_la_SOURCES += client_list.h client_list.cc
 libb10_datasrc_la_SOURCES += memory_datasrc.h memory_datasrc.cc
+libb10_datasrc_la_SOURCES += master_loader_callbacks.h
+libb10_datasrc_la_SOURCES += master_loader_callbacks.cc
 nodist_libb10_datasrc_la_SOURCES = datasrc_messages.h datasrc_messages.cc
 libb10_datasrc_la_LDFLAGS = -no-undefined -version-info 1:0:1
 

src/lib/datasrc/client_list.cc

@@ -14,12 +14,14 @@
 
 
 #include "client_list.h"
+#include "exceptions.h"
 #include "client.h"
 #include "factory.h"
 #include "memory/memory_client.h"
 #include "memory/zone_table_segment.h"
 #include "memory/zone_writer.h"
 #include "memory/zone_data_loader.h"
+#include "memory/zone_data_updater.h"
 #include "logger.h"
 #include <dns/masterload.h>
 #include <util/memory_segment_local.h>
@@ -37,6 +39,7 @@ using boost::shared_ptr;
 using boost::dynamic_pointer_cast;
 using isc::datasrc::memory::InMemoryClient;
 using isc::datasrc::memory::ZoneTableSegment;
+using isc::datasrc::memory::ZoneDataUpdater;
 
 namespace isc {
 namespace datasrc {
@@ -175,9 +178,9 @@ ConfigurableClientList::configure(const ConstElementPtr& config,
                         try {
                             cache->load(origin,
                                         paramConf->get(*it)->stringValue());
-                        } catch (const isc::dns::MasterLoadError& mle) {
-                            LOG_ERROR(logger, DATASRC_MASTERLOAD_ERROR)
-                                .arg(mle.what());
+                        } catch (const ZoneLoaderException& e) {
+                            LOG_ERROR(logger, DATASRC_LOAD_FROM_FILE_ERROR)
+                                .arg(origin).arg(e.what());
                         }
                     } else {
                         ZoneIteratorPtr iterator;
@@ -192,7 +195,12 @@ ConfigurableClientList::configure(const ConstElementPtr& config,
                             isc_throw(isc::Unexpected, "Got NULL iterator "
                                       "for zone " << origin);
                         }
-                        cache->load(origin, *iterator);
+                        try {
+                            cache->load(origin, *iterator);
+                        } catch (const ZoneLoaderException& e) {
+                            LOG_ERROR(logger, DATASRC_LOAD_FROM_ITERATOR_ERROR)
+                                .arg(origin).arg(e.what());
+                        }
                     }
                 }
             }

src/lib/datasrc/datasrc_messages.mes

@@ -305,10 +305,27 @@ Therefore, the zone will not be available for this process. If this is
 a problem, you should move the zone to some database backend (sqlite3, for
 example) and use it from there.
 
-% DATASRC_MASTERLOAD_ERROR %1
-An error was found in the zone data for a MasterFiles zone. The zone
-is not loaded. The specific error is shown in the message, and should
-be addressed.
+% DATASRC_LOAD_FROM_FILE_ERROR Error loading zone %1: %2
+An error was found in the zone data when it was being loaded from a
+file. The zone was not loaded. The specific error is shown in the
+message, and should be addressed.
+
+% DATASRC_LOAD_FROM_ITERATOR_ERROR Error loading zone %1: %2
+An error was found in the zone data when it was being loaded from
+another data source. The zone was not loaded. The specific error is
+shown in the message, and should be addressed.
+
+% DATASRC_MASTER_LOAD_ERROR %1:%2: Zone '%3/%4' contains error: %5
+There's an error in the given master file. The zone won't be loaded for
+this reason. Parsing might follow, so you might get further errors and
+warnings to fix everything at once. But in case the error is serious enough,
+the parser might just give up or get confused and generate false errors
+afterwards.
+
+% DATASRC_MASTER_LOAD_WARN %1:%2: Zone '%3/%4' has a potential problem: %5
+There's something suspicious in the master file. This is a warning only.
+It may be a problem or it may be harmless, but it should be checked.
+This problem does not stop the zone from being loaded.
 
 % DATASRC_MEM_ADD_RRSET adding RRset '%1/%2' into zone '%3'
 Debug information. An RRset is being added to the in-memory data source.

src/lib/datasrc/exceptions.h

@@ -0,0 +1,47 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#ifndef DATASRC_EXCEPTIONS_H
+#define DATASRC_EXCEPTIONS_H 1
+
+#include <exceptions/exceptions.h>
+
+namespace isc {
+namespace datasrc {
+
+/// Base class for a number of exceptions that are thrown while working
+/// with zones.
+struct ZoneException : public Exception {
+    ZoneException(const char* file, size_t line, const char* what) :
+        Exception(file, line, what)
+    {}
+};
+
+/// Base class for a number of exceptions that are thrown when zones are
+/// being loaded. This is a recoverable exception. It should be possible
+/// to skip the bad zone and continue loading/serving other zones.
+struct ZoneLoaderException : public ZoneException {
+    ZoneLoaderException(const char* file, size_t line, const char* what) :
+        ZoneException(file, line, what)
+    {}
+};
+
+} // namespace datasrc
+} // namespace isc
+
+#endif // DATASRC_EXCEPTIONS
+
+// Local Variables:
+// mode: c++
+// End:

src/lib/datasrc/master_loader_callbacks.cc

@@ -0,0 +1,73 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#include <datasrc/master_loader_callbacks.h>
+#include <datasrc/zone.h>
+#include <datasrc/logger.h>
+
+#include <dns/name.h>
+#include <dns/rrclass.h>
+#include <dns/rrset.h>
+
+#include <string>
+#include <boost/bind.hpp>
+
+namespace isc {
+namespace datasrc {
+
+namespace {
+
+void
+logError(const isc::dns::Name& name, const isc::dns::RRClass& rrclass,
+         bool* ok, const std::string& source, size_t line,
+         const std::string& reason)
+{
+    LOG_ERROR(logger, DATASRC_MASTER_LOAD_ERROR).arg(source).arg(line).
+        arg(name).arg(rrclass).arg(reason);
+    if (ok != NULL) {
+        *ok = false;
+    }
+}
+
+void
+logWarning(const isc::dns::Name& name, const isc::dns::RRClass& rrclass,
+         const std::string& source, size_t line, const std::string& reason)
+{
+    LOG_WARN(logger, DATASRC_MASTER_LOAD_WARN).arg(source).arg(line).
+        arg(name).arg(rrclass).arg(reason);
+}
+
+}
+
+isc::dns::MasterLoaderCallbacks
+createMasterLoaderCallbacks(const isc::dns::Name& name,
+                            const isc::dns::RRClass& rrclass, bool* ok)
+{
+    return (isc::dns::MasterLoaderCallbacks(boost::bind(&logError, name,
+                                                        rrclass, ok, _1, _2,
+                                                        _3),
+                                            boost::bind(&logWarning, name,
+                                                        rrclass, _1, _2, _3)));
+}
+
+isc::dns::AddRRsetCallback
+createMasterLoaderAddCallback(ZoneUpdater& updater) {
+    return (boost::bind(&ZoneUpdater::addRRset, &updater,
+                        // The callback provides a shared pointer, we
+                        // need the object. This bind unpacks the object.
+                        boost::bind(&isc::dns::RRsetPtr::operator*, _1)));
+}
+
+}
+}

src/lib/datasrc/master_loader_callbacks.h

@@ -0,0 +1,67 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#ifndef DATASRC_MASTER_LOADER_CALLBACKS_H
+#define DATASRC_MASTER_LOADER_CALLBACKS_H
+
+#include <dns/master_loader_callbacks.h>
+
+namespace isc {
+namespace dns {
+class Name;
+class RRClass;
+}
+namespace datasrc {
+
+class ZoneUpdater;
+
+/// \brief Create issue callbacks for MasterLoader
+///
+/// This will create set of callbacks for the MasterLoader that
+/// will be used to report any issues found in the zone data.
+///
+/// \param name Name of the zone. Used in logging.
+/// \param rrclass The class of the zone. Used in logging.
+/// \param ok If this is non-NULL and there are any errors during
+///     the loading, it is set to false. Otherwise, it is untouched.
+/// \return Set of callbacks to be passed to the master loader.
+/// \throw std::bad_alloc when allocation fails.
+isc::dns::MasterLoaderCallbacks
+createMasterLoaderCallbacks(const isc::dns::Name& name,
+                            const isc::dns::RRClass& rrclass, bool* ok);
+
+/// \brief Create a callback for MasterLoader to add RRsets.
+///
+/// This creates a callback that can be used by the MasterLoader to add
+/// loaded RRsets into a zone updater.
+///
+/// The zone updater should be opened in the replace mode no changes should
+/// have been done to it yet (but it is not checked). It is not commited
+/// automatically and it is up to the caller to commit the changes (or not).
+/// It must not be destroyed for the whole time of loading.
+///
+/// The function is mostly straightforward packing of the updater.addRRset
+/// into a boost::function, it is defined explicitly due to small technical
+/// annoyences around boost::bind application, so it can be reused.
+///
+/// \param updater The zone updater to use.
+/// \return The callback to be passed to MasterLoader.
+/// \throw std::bad_alloc when allocation fails.
+isc::dns::AddRRsetCallback
+createMasterLoaderAddCallback(ZoneUpdater& updater);
+
+}
+}
+
+#endif

src/lib/datasrc/memory/Makefile.am

@@ -27,6 +27,7 @@ libdatasrc_memory_la_SOURCES += memory_client.h memory_client.cc
 libdatasrc_memory_la_SOURCES += zone_writer.h
 libdatasrc_memory_la_SOURCES += zone_writer_local.h zone_writer_local.cc
 libdatasrc_memory_la_SOURCES += load_action.h
+libdatasrc_memory_la_SOURCES += util_internal.h
 
 nodist_libdatasrc_memory_la_SOURCES = memory_messages.h memory_messages.cc
 

src/lib/datasrc/memory/memory_client.h

@@ -87,11 +87,11 @@ public:
     /// current content. The masterfile parsing ability is kind of limited,
     /// see isc::dns::masterLoad.
     ///
-    /// This throws isc::dns::MasterLoadError if there is problem with loading
-    /// (missing file, malformed, it contains different zone, etc - see
-    /// isc::dns::masterLoad for details).
+    /// This throws isc::dns::MasterLoadError or AddError if there are
+    /// problems with loading (missing file, malformed data, unexpected
+    /// zone, etc. - see isc::dns::masterLoad for details).
     ///
-    /// In case of internal problems, OutOfZone, NullRRset or AssertError could
+    /// In case of internal problems, NullRRset or AssertError could
     /// be thrown, but they should not be expected. Exceptions caused by
     /// allocation may be thrown as well.
     ///

src/lib/datasrc/memory/rdataset.cc

@@ -122,8 +122,8 @@ RdataSet::create(util::MemorySegment& mem_sgmt, RdataEncoder& encoder,
 }
 
 void
-RdataSet::destroy(util::MemorySegment& mem_sgmt, RRClass rrclass,
-                  RdataSet* rdataset)
+RdataSet::destroy(util::MemorySegment& mem_sgmt, RdataSet* rdataset,
+                  RRClass rrclass)
 {
     const size_t data_len =
         RdataReader(rrclass, rdataset->type,

src/lib/datasrc/memory/rdataset.h

@@ -187,12 +187,12 @@ public:
     ///
     /// \param mem_sgmt The \c MemorySegment that allocated memory for
     /// \c node.
-    /// \param rrclass The RR class of the \c RdataSet to be destroyed.
     /// \param rdataset A non NULL pointer to a valid \c RdataSet object
+    /// \param rrclass The RR class of the \c RdataSet to be destroyed.
     /// that was originally created by the \c create() method (the behavior
     /// is undefined if this condition isn't met).
-    static void destroy(util::MemorySegment& mem_sgmt, dns::RRClass rrclass,
-                        RdataSet* rdataset);
+    static void destroy(util::MemorySegment& mem_sgmt, RdataSet* rdataset,
+                        dns::RRClass rrclass);
 
     /// \brief Find \c RdataSet of given RR type from a list (const version).
     ///
@@ -205,6 +205,11 @@ public:
     /// if not found in the entire list, it returns NULL.  The head pointer
     /// can be NULL, in which case this function will simply return NULL.
     ///
+    /// By default, this method ignores an RdataSet that only contains an
+    /// RRSIG (i.e., missing the covered RdataSet); if the optional
+    /// sigonly_ok parameter is explicitly set to true, it matches such
+    /// RdataSet and returns it if found.
+    ///
     /// \note This function is defined as a (static) class method to
     /// clarify its an operation for \c RdataSet objects and to make the
     /// name shorter.  But its implementation does not depend on private
@@ -215,10 +220,14 @@ public:
     /// \param rdata_head A pointer to \c RdataSet from which the search
     /// starts.  It can be NULL.
     /// \param type The RRType of \c RdataSet to find.
+    /// \param sigonly_ok Whether it should find an RdataSet that only has
+    /// RRSIG
     /// \return A pointer to the found \c RdataSet or NULL if none found.
     static const RdataSet*
-    find(const RdataSet* rdataset_head, const dns::RRType& type) {
-        return (find<const RdataSet>(rdataset_head, type));
+    find(const RdataSet* rdataset_head, const dns::RRType& type,
+         bool sigonly_ok = false)
+    {
+        return (find<const RdataSet>(rdataset_head, type, sigonly_ok));
     }
 
     /// \brief Find \c RdataSet of given RR type from a list (non const
@@ -227,8 +236,10 @@ public:
     /// This is similar to the const version, except it takes and returns non
     /// const pointers.
     static RdataSet*
-    find(RdataSet* rdataset_head, const dns::RRType& type) {
-        return (find<RdataSet>(rdataset_head, type));
+    find(RdataSet* rdataset_head, const dns::RRType& type,
+         bool sigonly_ok = false)
+    {
+        return (find<RdataSet>(rdataset_head, type, sigonly_ok));
     }
 
     typedef boost::interprocess::offset_ptr<RdataSet> RdataSetPtr;
@@ -347,12 +358,14 @@ private:
     // Shared by both mutable and immutable versions of find()
     template <typename RdataSetType>
     static RdataSetType*
-    find(RdataSetType* rdataset_head, const dns::RRType& type) {
+    find(RdataSetType* rdataset_head, const dns::RRType& type, bool sigonly_ok)
+    {
         for (RdataSetType* rdataset = rdataset_head;
              rdataset != NULL;
              rdataset = rdataset->getNext()) // use getNext() for efficiency
         {
-            if (rdataset->type == type) {
+            if (rdataset->type == type &&
+                (rdataset->getRdataCount() > 0 || sigonly_ok)) {
                 return (rdataset);
             }
         }

src/lib/datasrc/memory/util_internal.h

@@ -0,0 +1,57 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#ifndef DATASRC_MEMORY_UTIL_INTERNAL_H
+#define DATASRC_MEMORY_UTIL_INTERNAL_H 1
+
+#include <dns/rdataclass.h>
+#include <dns/rrset.h>
+#include <dns/rrtype.h>
+
+namespace isc {
+namespace datasrc {
+namespace memory {
+namespace detail {
+
+/// \brief Return the covered RR type of an RRSIG RRset.
+///
+/// This is a commonly used helper to extract the type covered field of an
+/// RRSIG RRset and return it in the form of an RRType object.
+///
+/// Normally, an empty RRSIG shouldn't be passed to this function, whether
+/// it comes from a master file or another data source iterator, but it could
+/// still happen in some buggy situations.  This function catches and rejects
+/// such cases.
+inline dns::RRType
+getCoveredType(const dns::ConstRRsetPtr& sig_rrset) {
+    dns::RdataIteratorPtr it = sig_rrset->getRdataIterator();
+    if (it->isLast()) {
+        isc_throw(isc::Unexpected,
+                  "Empty RRset is passed in-memory loader, name: "
+                  << sig_rrset->getName());
+    }
+    return (dynamic_cast<const dns::rdata::generic::RRSIG&>(it->getCurrent()).
+            typeCovered());
+}
+
+} // namespace detail
+} // namespace memory
+} // namespace datasrc
+} // namespace isc
+
+#endif // DATASRC_MEMORY_UTIL_INTERNAL_H
+
+// Local Variables:
+// mode: c++
+// End:

src/lib/datasrc/memory/zone_data.cc

@@ -49,7 +49,7 @@ rdataSetDeleter(RRClass rrclass, util::MemorySegment* mem_sgmt,
          rdataset = rdataset_next)
     {
         rdataset_next = rdataset->getNext();
-        RdataSet::destroy(*mem_sgmt, rrclass, rdataset);
+        RdataSet::destroy(*mem_sgmt, rdataset, rrclass);
     }
 }
 

src/lib/datasrc/memory/zone_data_loader.cc

@@ -16,6 +16,7 @@
 #include <datasrc/memory/zone_data_updater.h>
 #include <datasrc/memory/logger.h>
 #include <datasrc/memory/segment_object_holder.h>
+#include <datasrc/memory/util_internal.h>
 
 #include <dns/rdataclass.h>
 #include <dns/rrset.h>
@@ -35,6 +36,7 @@ namespace datasrc {
 namespace memory {
 
 using detail::SegmentObjectHolder;
+using detail::getCoveredType;
 
 namespace { // unnamed namespace
 
@@ -75,8 +77,6 @@ private:
     typedef NodeRRsets::value_type NodeRRsetsVal;
 
     // A helper to identify the covered type of an RRSIG.
-    static isc::dns::RRType getCoveredType
-        (const isc::dns::ConstRRsetPtr& sig_rrset);
     const isc::dns::Name& getCurrentName() const;
 
 private:
@@ -126,34 +126,17 @@ ZoneDataLoader::flushNodeRRsets() {
         updater_.add(val.second, sig_rrset);
     }
 
-    // Right now, we don't accept RRSIG without covered RRsets (this
-    // should eventually allowed, but to do so we'll need to update the
-    // finder).
-    if (!node_rrsigsets_.empty()) {
-        isc_throw(ZoneDataUpdater::AddError,
-                  "RRSIG is added without covered RRset for "
-                  << getCurrentName());
+    // Normally rrsigsets map should be empty at this point, but it's still
+    // possible that an RRSIG that don't has covered RRset is added; they
+    // still remain in the map.  We add them to the zone separately.
+    BOOST_FOREACH(NodeRRsetsVal val, node_rrsigsets_) {
+        updater_.add(ConstRRsetPtr(), val.second);
     }
 
     node_rrsets_.clear();
     node_rrsigsets_.clear();
 }
 
-RRType
-ZoneDataLoader::getCoveredType(const ConstRRsetPtr& sig_rrset) {
-    RdataIteratorPtr it = sig_rrset->getRdataIterator();
-    // Empty RRSIG shouldn't be passed either via a master file or
-    // another data source iterator, but it could still happen if the
-    // iterator has a bug.  We catch and reject such cases.
-    if (it->isLast()) {
-        isc_throw(isc::Unexpected,
-                  "Empty RRset is passed in-memory loader, name: "
-                  << sig_rrset->getName());
-    }
-    return (dynamic_cast<const generic::RRSIG&>(it->getCurrent()).
-            typeCovered());
-}
-
 const Name&
 ZoneDataLoader::getCurrentName() const {
     if (!node_rrsets_.empty()) {
@@ -207,7 +190,11 @@ void
 masterLoadWrapper(const char* const filename, const Name& origin,
                   const RRClass& zone_class, LoadCallback callback)
 {
-    masterLoad(filename, origin, zone_class, boost::bind(callback, _1));
+    try {
+        masterLoad(filename, origin, zone_class, boost::bind(callback, _1));
+    } catch (MasterLoadError& e) {
+        isc_throw(ZoneLoaderException, e.what());
+    }
 }
 
 // The installer called from the iterator version of loadZoneData().

src/lib/datasrc/memory/zone_data_loader.h

@@ -15,6 +15,7 @@
 #ifndef DATASRC_ZONE_DATA_LOADER_H
 #define DATASRC_ZONE_DATA_LOADER_H 1
 
+#include <datasrc/exceptions.h>
 #include <datasrc/memory/zone_data.h>
 #include <datasrc/iterator.h>
 #include <dns/name.h>
@@ -29,9 +30,9 @@ namespace memory {
 ///
 /// This is thrown if an empty zone would be created during
 /// \c loadZoneData().
-struct EmptyZone : public InvalidParameter {
+struct EmptyZone : public ZoneLoaderException {
     EmptyZone(const char* file, size_t line, const char* what) :
-        InvalidParameter(file, line, what)
+        ZoneLoaderException(file, line, what)
     {}
 };
 

src/lib/datasrc/memory/zone_data_updater.cc

@@ -12,12 +12,18 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
+#include <exceptions/exceptions.h>
+
 #include <datasrc/memory/zone_data_updater.h>
 #include <datasrc/memory/logger.h>
+#include <datasrc/memory/util_internal.h>
 #include <datasrc/zone.h>
 
 #include <dns/rdataclass.h>
 
+#include <cassert>
+#include <string>
+
 using namespace isc::dns;
 using namespace isc::dns::rdata;
 
@@ -25,6 +31,8 @@ namespace isc {
 namespace datasrc {
 namespace memory {
 
+using detail::getCoveredType;
+
 void
 ZoneDataUpdater::addWildcards(const Name& name) {
     Name wname(name);
@@ -99,9 +107,7 @@ ZoneDataUpdater::contextCheck(const AbstractRRset& rrset,
 
 void
 ZoneDataUpdater::validate(const isc::dns::ConstRRsetPtr rrset) const {
-    if (!rrset) {
-        isc_throw(NullRRset, "The rrset provided is NULL");
-    }
+    assert(rrset);
 
     if (rrset->getRdataCount() == 0) {
         isc_throw(AddError,
@@ -158,7 +164,7 @@ ZoneDataUpdater::validate(const isc::dns::ConstRRsetPtr rrset) const {
         LOG_ERROR(logger,
                   DATASRC_MEMORY_MEM_OUT_OF_ZONE).arg(rrset->getName()).
             arg(zone_name_);
-        isc_throw(OutOfZone,
+        isc_throw(AddError,
                   "The name " << rrset->getName() <<
                   " is not contained in zone " << zone_name_);
     }
@@ -241,31 +247,46 @@ ZoneDataUpdater::setupNSEC3(const ConstRRsetPtr rrset) {
 }
 
 void
-ZoneDataUpdater::addNSEC3(const ConstRRsetPtr rrset, const ConstRRsetPtr rrsig)
+ZoneDataUpdater::addNSEC3(const Name& name, const ConstRRsetPtr rrset,
+                          const ConstRRsetPtr rrsig)
 {
-    setupNSEC3<generic::NSEC3>(rrset);
+    if (rrset) {
+        setupNSEC3<generic::NSEC3>(rrset);
+    }
 
     NSEC3Data* nsec3_data = zone_data_.getNSEC3Data();
+    if (nsec3_data == NULL) {
+        // This is some tricky case: an RRSIG for NSEC3 is given without the
+        // covered NSEC3, and we don't even know any NSEC3 related data.
+        // This situation is not necessarily broken, but in our current
+        // implementation it's very difficult to deal with.  So we reject it;
+        // hopefully this case shouldn't happen in practice, at least unless
+        // zone is really broken.
+        assert(!rrset);
+        isc_throw(NotImplemented,
+                  "RRSIG for NSEC3 cannot be added - no known NSEC3 data");
+    }
 
     ZoneNode* node;
-    nsec3_data->insertName(mem_sgmt_, rrset->getName(), &node);
+    nsec3_data->insertName(mem_sgmt_, name, &node);
 
     RdataSet* rdataset = RdataSet::create(mem_sgmt_, encoder_, rrset, rrsig);
     RdataSet* old_rdataset = node->setData(rdataset);
     if (old_rdataset != NULL) {
-        RdataSet::destroy(mem_sgmt_, rrclass_, old_rdataset);
+        RdataSet::destroy(mem_sgmt_, old_rdataset, rrclass_);
     }
 }
 
 void
-ZoneDataUpdater::addRdataSet(const ConstRRsetPtr rrset,
+ZoneDataUpdater::addRdataSet(const Name& name, const RRType& rrtype,
+                             const ConstRRsetPtr rrset,
                              const ConstRRsetPtr rrsig)
 {
-    if (rrset->getType() == RRType::NSEC3()) {
-        addNSEC3(rrset, rrsig);
+    if (rrtype == RRType::NSEC3()) {
+        addNSEC3(name, rrset, rrsig);
     } else {
         ZoneNode* node;
-        zone_data_.insertName(mem_sgmt_, rrset->getName(), &node);
+        zone_data_.insertName(mem_sgmt_, name, &node);
 
         RdataSet* rdataset_head = node->getData();
 
@@ -273,13 +294,14 @@ ZoneDataUpdater::addRdataSet(const ConstRRsetPtr rrset,
         // fails and the exception is thrown, it may break strong
         // exception guarantee.  At the moment we prefer code simplicity
         // and don't bother to introduce complicated recovery code.
-        contextCheck(*rrset, rdataset_head);
+        if (rrset) { // this check is only for covered RRset, not RRSIG
+            contextCheck(*rrset, rdataset_head);
+        }
 
-        if (RdataSet::find(rdataset_head, rrset->getType()) != NULL) {
+        if (RdataSet::find(rdataset_head, rrtype, true) != NULL) {
             isc_throw(AddError,
                       "RRset of the type already exists: "
-                      << rrset->getName() << " (type: "
-                      << rrset->getType() << ")");
+                      << name << " (type: " << rrtype << ")");
         }
 
         RdataSet* rdataset_new = RdataSet::create(mem_sgmt_, encoder_,
@@ -289,23 +311,25 @@ ZoneDataUpdater::addRdataSet(const ConstRRsetPtr rrset,
 
         // Ok, we just put it in.
 
+        // Convenient (and more efficient) shortcut to check RRsets at origin
+        const bool is_origin = (node == zone_data_.getOriginNode());
+
         // If this RRset creates a zone cut at this node, mark the node
-        // indicating the need for callback in find().
-        if (rrset->getType() == RRType::NS() &&
-            rrset->getName() != zone_name_) {
+        // indicating the need for callback in find().  Note that we do this
+        // only when non RRSIG RRset of that type is added.
+        if (rrset && rrtype == RRType::NS() && !is_origin) {
             node->setFlag(ZoneNode::FLAG_CALLBACK);
             // If it is DNAME, we have a callback as well here
-        } else if (rrset->getType() == RRType::DNAME()) {
+        } else if (rrset && rrtype == RRType::DNAME()) {
             node->setFlag(ZoneNode::FLAG_CALLBACK);
         }
 
         // If we've added NSEC3PARAM at zone origin, set up NSEC3
         // specific data or check consistency with already set up
         // parameters.
-        if (rrset->getType() == RRType::NSEC3PARAM() &&
-            rrset->getName() == zone_name_) {
+        if (rrset && rrtype == RRType::NSEC3PARAM() && is_origin) {
             setupNSEC3<generic::NSEC3PARAM>(rrset);
-        } else if (rrset->getType() == RRType::NSEC()) {
+        } else if (rrset && rrtype == RRType::NSEC() && is_origin) {
             // If it is NSEC signed zone, we mark the zone as signed
             // (conceptually "signed" is a broader notion but our
             // current zone finder implementation regards "signed" as
@@ -319,27 +343,37 @@ void
 ZoneDataUpdater::add(const ConstRRsetPtr& rrset,
                      const ConstRRsetPtr& sig_rrset)
 {
-    // Validate input.  This will cause an exception to be thrown if the
-    // input RRset is empty.
-    validate(rrset);
+    // Validate input.
+    if (!rrset && !sig_rrset) {
+        isc_throw(NullRRset,
+                  "ZoneDataUpdater::add is given 2 NULL pointers");
+    }
+    if (rrset) {
+        validate(rrset);
+    }
     if (sig_rrset) {
         validate(sig_rrset);
     }
 
+    const Name& name = rrset ? rrset->getName() : sig_rrset->getName();
+    const RRType& rrtype = rrset ? rrset->getType() :
+        getCoveredType(sig_rrset);
+
     // OK, can add the RRset.
-    LOG_DEBUG(logger, DBG_TRACE_DATA, DATASRC_MEMORY_MEM_ADD_RRSET).
-        arg(rrset->getName()).arg(rrset->getType()).arg(zone_name_);
+    LOG_DEBUG(logger, DBG_TRACE_DATA, DATASRC_MEMORY_MEM_ADD_RRSET).arg(name).
+        arg(rrset ? rrtype.toText() : "RRSIG(" + rrtype.toText() + ")").
+        arg(zone_name_);
 
     // Add wildcards possibly contained in the owner name to the domain
     // tree.  This can only happen for the normal (non-NSEC3) tree.
     // Note: this can throw an exception, breaking strong exception
     // guarantee.  (see also the note for the call to contextCheck()
     // above).
-    if (rrset->getType() != RRType::NSEC3()) {
-        addWildcards(rrset->getName());
+    if (rrtype != RRType::NSEC3()) {
+        addWildcards(name);
     }
 
-    addRdataSet(rrset, sig_rrset);
+    addRdataSet(name, rrtype, rrset, sig_rrset);
 }
 
 } // namespace memory