Merge branch 'master' into trac878

Conflicts:
	ChangeLog

ChangeLog

@@ -1,4 +1,4 @@
-266.    [func]*		tomek
+293.    [func]*		tomek
 	b10-dhcp6: Implemented DHCPv6 echo server. It joins DHCPv6
 	multicast groups and listens to incoming DHCPv6 client messages.
 	Received messages are then echoed back to clients. This
@@ -8,6 +8,184 @@
 	and its address must be specified in interfaces.txt.
 	(Trac #878, git 3b1a604abf5709bfda7271fa94213f7d823de69d)
 
+292.	[func]		dvv
+	Implement the DLV rrtype according to RFC4431.
+	(Trac #1144, git d267c0511a07c41cd92e3b0b9ee9bf693743a7cf)
+
+291.    [func]          naokikambe
+	Statistics items are specified by each module's spec file.
+	Stats module can read these through the config manager. Stats
+	module and stats httpd report statistics data and statistics
+	schema by each module via both bindctl and HTTP/XML.
+	(Trac #928,#929,#930,#1175, git 054699635affd9c9ecbe7a108d880829f3ba229e)
+
+290.	[func]		jinmei
+	libdns++/pydnspp: added an option parameter to the "from wire"
+	methods of the Message class.  One option is defined,
+	PRESERVE_ORDER, which specifies the parser to handle each RR
+	separately, preserving the order, and constructs RRsets in the
+	message sections so that each RRset contains only one RR.
+	(Trac #1258, git c874cb056e2a5e656165f3c160e1b34ccfe8b302)
+
+289.	[func]*		jinmei
+	b10-xfrout: ACLs for xfrout can now be configured per zone basis.
+	A per zone ACl is part of a more general zone configuration.  A
+	quick example for configuring an ACL for zone "example.com" that
+	rejects any transfer request for that zone is as follows:
+	> config add Xfrout/zone_config
+	> config set Xfrout/zone_config[0]/origin "example.com"
+	> config add Xfrout/zone_config[0]/transfer_acl
+	> config set Xfrout/zone_config[0]/transfer_acl[0] {"action": "REJECT"}
+	The previous global ACL (query_acl) was renamed to transfer_acl,
+	which now works as the default ACL.  Note: backward compatibility
+	is not provided, so an existing configuration using query_acl
+	needs to be updated by hand.
+	Note: the per zone configuration framework is a temporary
+	workaround.  It will eventually be redesigned as a system wide
+	configuration.
+	(Trac #1165, git 698176eccd5d55759fe9448b2c249717c932ac31)
+
+288.    [bug]       stephen
+	Fixed problem whereby the order in which component files appeared in
+	rdataclass.cc was system dependent, leading to problems on some
+	systems where data types were used before the header file in which
+	they were declared was included.
+	(Trac #1202, git 4a605525cda67bea8c43ca8b3eae6e6749797450)
+
+287.	[bug]*		jinmei
+	Python script files for log messages (xxx_messages.py) should have
+	been installed under the "isc" package.  This fix itself should
+	be a transparent change without affecting existing configurations
+	or other operational practices, but you may want to clean up the
+	python files from the common directly (such as "site-packages").
+	(Trac #1101, git 0eb576518f81c3758c7dbaa2522bd8302b1836b3)
+
+286.    [func]		ocean
+	libdns++: Implement the HINFO rrtype support according to RFC1034,
+	and RFC1035.
+	(Trac #1112, git 12d62d54d33fbb1572a1aa3089b0d547d02924aa)
+
+285.	[bug]		jelte
+	sqlite3 data source: fixed a race condition on initial startup,
+	when the database has not been initialized yet, and multiple
+	processes are trying to do so, resulting in one of them failing.
+	(Trac #326, git 5de6f9658f745e05361242042afd518b444d7466)
+
+284.	[bug]		jerry
+	b10-zonemgr: zonemgr will not terminate on empty zones, it will
+	log a warning and try to do zone transfer for them.
+	(Trac #1153, git 0a39659638fc68f60b95b102968d7d0ad75443ea)
+
+283.    [bug]		zhanglikun
+	Make stats and boss processes wait for answer messages from each
+	other in block mode to avoid orphan answer messages, add an internal
+	command "getstats" to boss process for getting statistics data from
+	boss.
+	(Trac #519, git 67d8e93028e014f644868fede3570abb28e5fb43)
+
+282.    [func]		ocean
+	libdns++: Implement the NAPTR rrtype according to RFC2915,
+	RFC2168 and RFC3403.
+	(Trac #1130, git 01d8d0f13289ecdf9996d6d5d26ac0d43e30549c)
+
+bind10-devel-20110819 released on August 19, 2011
+
+281.	[func]		jelte
+	Added a new type for configuration data: "named set". This allows for
+	similar configuration as the current "list" type, but with strings
+	instead of indices as identifiers. The intended use is for instance
+	/foo/zones/example.org/bar instead of /foo/zones[2]/bar. Currently
+	this new type is not in use yet.
+	(Trac #926, git 06aeefc4787c82db7f5443651f099c5af47bd4d6)
+
+280.	[func]		jerry
+	libdns++: Implement the MINFO rrtype according to RFC1035.
+	(Trac #1113, git 7a9a19d6431df02d48a7bc9de44f08d9450d3a37)
+
+279.	[func]		jerry
+	libdns++: Implement the AFSDB rrtype according to RFC1183.
+	(Trac #1114, git ce052cd92cd128ea3db5a8f154bd151956c2920c)
+
+278.	[doc]		jelte
+	Add logging configuration documentation to the guide.
+	(Trac #1011, git 2cc500af0929c1f268aeb6f8480bc428af70f4c4)
+
+277.	[func]		jerry
+	libdns++: Implement the SRV rrtype according to RFC2782.
+	(Trac #1128, git 5fd94aa027828c50e63ae1073d9d6708e0a9c223)
+
+276.	[func]		stephen
+	Although the top-level loggers are named after the program (e.g.
+	b10-auth, b10-resolver), allow the logger configuration to omit the
+	"b10-" prefix and use just the module name.
+	(Trac #1003, git a01cd4ac5a68a1749593600c0f338620511cae2d)
+
+275.	[func]		jinmei
+	Added support for TSIG key matching in ACLs.  The xfrout ACL can
+	now refer to TSIG key names using the "key" attribute.  For
+	example, the following specifies an ACL that allows zone transfer
+	if and only if the request is signed with a TSIG of a key name
+	"key.example":
+	> config set Xfrout/query_acl[0] {"action": "ACCEPT", \
+	                                  "key": "key.example"}
+	(Trac #1104, git 9b2e89cabb6191db86f88ee717f7abc4171fa979)
+
+274.	[bug]		naokikambe
+	add unittests for functions xml_handler, xsd_handler and xsl_handler
+	respectively to make sure their behaviors are correct, regardless of
+	whether type which xml.etree.ElementTree.tostring() after Python3.2
+	returns is str or byte.
+	(Trac #1021, git 486bf91e0ecc5fbecfe637e1e75ebe373d42509b)
+
+273.	[func]		vorner
+	It is possible to specify ACL for the xfrout module. It is in the ACL
+	configuration key and has the usual ACL syntax. It currently supports
+	only the source address. Default ACL accepts everything.
+	(Trac #772, git 50070c824270d5da1db0b716db73b726d458e9f7)
+
+272.	[func]		jinmei
+	libdns++/pydnspp: TSIG signing now handles truncated DNS messages
+	(i.e. with TC bit on) with TSIG correctly.
+	(Trac #910, 8e00f359e81c3cb03c5075710ead0f87f87e3220)
+
+271.	[func]		stephen
+	Default logging for unit tests changed to severity DEBUG (level 99)
+	with the output routed to /dev/null.  This can be altered by setting
+	the B10_LOGGER_XXX environment variables.
+	(Trac #1024, git 72a0beb8dfe85b303f546d09986461886fe7a3d8)
+
+270.	[func]		jinmei
+	Added python bindings for ACLs using the DNS request as the
+	context.  They are accessible via the isc.acl.dns module.
+	(Trac #983, git c24553e21fe01121a42e2136d0a1230d75812b27)
+
+269.	[bug]		y-aharen
+	Modified IntervalTimerTest not to rely on the accuracy of the timer.
+	This fix addresses occasional failure of build tests.
+	(Trac #1016, git 090c4c5abac33b2b28d7bdcf3039005a014f9c5b)
+
+268.	[func]		stephen
+	Add environment variable to allow redirection of logging output during
+	unit tests.
+	(Trac #1071, git 05164f9d61006869233b498d248486b4307ea8b6)
+
+bind10-devel-20110705 released on July 05, 2011
+
+267.	[func]		tomek
+	Added a dummy module for DHCP6. This module does not actually
+	do anything at this point, and BIND 10 has no option for
+	starting it yet. It is included as a base for further
+	development.
+	(Trac #990, git 4a590df96a1b1d373e87f1f56edaceccb95f267d)
+
+266.	[func]		Multiple developers
+        Convert various error messages, debugging and other output
+        to the new logging interface, including for b10-resolver,
+        the resolver library, the CC library, b10-auth, b10-cfgmgr,
+        b10-xfrin, and b10-xfrout. This includes a lot of new
+        documentation describing the new log messages.
+        (Trac #738, #739, #742, #746, #759, #761, #762)
+
 265.	[func]*		jinmei
 	b10-resolver: Introduced ACL on incoming queries.  By default the
 	resolver accepts queries from ::1 and 127.0.0.1 and rejects all
@@ -62,7 +240,7 @@
 	Now builds and runs with Python 3.2
 	(Trac #710, git dae1d2e24f993e1eef9ab429326652f40a006dfb)
 
-257.	[bug]           y-aharen
+257.	[bug]		y-aharen
 	Fixed a bug an instance of IntervalTimerImpl may be destructed 
 	while deadline_timer is holding the handler. This fix addresses
 	occasional failure of IntervalTimerTest.destructIntervalTimer.
@@ -71,25 +249,25 @@
 256.	[bug]		jerry
 	src/bin/xfrin: update xfrin to check TSIG before other part of
 	incoming message.
-	(Trac955, git 261450e93af0b0406178e9ef121f81e721e0855c)
+	(Trac #955, git 261450e93af0b0406178e9ef121f81e721e0855c)
 
 255.	[func]		zhang likun
 	src/lib/cache:  remove empty code in lib/cache and the corresponding
 	suppression rule in	src/cppcheck-suppress.lst.
-	(Trac639, git 4f714bac4547d0a025afd314c309ca5cb603e212)
+	(Trac #639, git 4f714bac4547d0a025afd314c309ca5cb603e212)
 
 254.	[bug]		jinmei
 	b10-xfrout: failed to send notifies over IPv6 correctly.
-	(Trac964, git 3255c92714737bb461fb67012376788530f16e40)
+	(Trac #964, git 3255c92714737bb461fb67012376788530f16e40)
 
-253.    [func]		jelte
+253.	[func]		jelte
 	Add configuration options for logging through the virtual module
 	Logging.
-	(Trac 736, git 9fa2a95177265905408c51d13c96e752b14a0824)
+	(Trac #736, git 9fa2a95177265905408c51d13c96e752b14a0824)
 
-252.    [func]      	stephen
+252.	[func]		stephen
 	Add syslog as destination for logging.
-	(Trac976, git 31a30f5485859fd3df2839fc309d836e3206546e)
+	(Trac #976, git 31a30f5485859fd3df2839fc309d836e3206546e)
 
 251.	[bug]*		jinmei
 	Make sure bindctl private files are non readable to anyone except
@@ -98,38 +276,38 @@
 	group will have to be adjusted.  Also note that this change is
 	only effective for a fresh install; if these files already exist,
 	their permissions must be adjusted by hand (if necessary).
-	(Trac870, git 461fc3cb6ebabc9f3fa5213749956467a14ebfd4)
+	(Trac #870, git 461fc3cb6ebabc9f3fa5213749956467a14ebfd4)
 
-250.    [bug]           ocean
+250.	[bug]		ocean
 	src/lib/util/encode, in some conditions, the DecodeNormalizer's
 	iterator may reach the end() and when later being dereferenced
 	it will cause crash on some platform.
-	(Trac838, git 83e33ec80c0c6485d8b116b13045b3488071770f)
+	(Trac #838, git 83e33ec80c0c6485d8b116b13045b3488071770f)
 
-249.    [func]      	jerry
+249.	[func]		jerry
 	xfrout: add support for TSIG verification.
-	(Trac816, git 3b2040e2af2f8139c1c319a2cbc429035d93f217)
+	(Trac #816, git 3b2040e2af2f8139c1c319a2cbc429035d93f217)
 
-248.    [func]      	stephen
+248.	[func]		stephen
 	Add file and stderr as destinations for logging.
-	(Trac555, git 38b3546867425bd64dbc5920111a843a3330646b)
+	(Trac #555, git 38b3546867425bd64dbc5920111a843a3330646b)
 
-247.    [func]      	jelte
+247.	[func]		jelte
 	Upstream queries from the resolver now set EDNS0 buffer size.
-	(Trac834, git 48e10c2530fe52c9bde6197db07674a851aa0f5d)
+	(Trac #834, git 48e10c2530fe52c9bde6197db07674a851aa0f5d)
 
-246.    [func]      	stephen
+246.	[func]		stephen
 	Implement logging using log4cplus (http://log4cplus.sourceforge.net)
-	(Trac899, git 31d3f525dc01638aecae460cb4bc2040c9e4df10)
+	(Trac #899, git 31d3f525dc01638aecae460cb4bc2040c9e4df10)
 
-245.    [func]      	vorner
+245.	[func]		vorner
 	Authoritative server can now sign the answers using TSIG
 	(configured in tsig_keys/keys, list of strings like
 	"name:<base64-secret>:sha1-hmac"). It doesn't use them for
 	ACL yet, only verifies them and signs if the request is signed.
-	(Trac875, git fe5e7003544e4e8f18efa7b466a65f336d8c8e4d)
+	(Trac #875, git fe5e7003544e4e8f18efa7b466a65f336d8c8e4d)
 
-244.	[func] 		stephen
+244.	[func]		stephen
 	In unit tests, allow the choice of whether unhandled exceptions are
 	caught in the unit test program (and details printed) or allowed to
 	propagate to the default exception handler.  See the bind10-dev thread
@@ -139,7 +317,7 @@
 
 243.	[func]*		feng
 	Add optional hmac algorithm SHA224/384/812.
-	(Trac#782, git 77d792c9d7c1a3f95d3e6a8b721ac79002cd7db1)
+	(Trac #782, git 77d792c9d7c1a3f95d3e6a8b721ac79002cd7db1)
 
 bind10-devel-20110519 released on May 19, 2011
 
@@ -186,7 +364,7 @@ bind10-devel-20110519 released on May 19, 2011
 	stats module and stats-httpd module, and maybe with other
 	statistical modules in future. "stats.spec" has own configuration
 	and commands of stats module, if it requires.
-	(Trac#719, git a234b20dc6617392deb8a1e00eb0eed0ff353c0a)
+	(Trac #719, git a234b20dc6617392deb8a1e00eb0eed0ff353c0a)
 
 236.	[func]		jelte
 	C++ client side of configuration now uses BIND10 logging system.
@@ -229,13 +407,13 @@ bind10-devel-20110519 released on May 19, 2011
 	instead of '%s,%d', which allows us to cope better with
 	mismatched placeholders and allows reordering of them in
 	case of translation.
-	(Trac901, git 4903410e45670b30d7283f5d69dc28c2069237d6)
+	(Trac #901, git 4903410e45670b30d7283f5d69dc28c2069237d6)
 
 230.	[bug]		naokikambe
 	Removed too repeated verbose messages in two cases of:
 	 - when auth sends statistics data to stats
 	 - when stats receives statistics data from other modules
-	(Trac#620, git 0ecb807011196eac01f281d40bc7c9d44565b364)
+	(Trac #620, git 0ecb807011196eac01f281d40bc7c9d44565b364)
 
 229.	[doc]		jreed
 	Add manual page for b10-host.

README

@@ -8,10 +8,10 @@ for serving, maintaining, and developing DNS.
 BIND10-devel is new development leading up to the production
 BIND 10 release. It contains prototype code and experimental
 interfaces. Nevertheless it is ready to use now for testing the
-new BIND 10 infrastructure ideas. The Year 2 milestones of the
-five year plan are described here:
+new BIND 10 infrastructure ideas. The Year 3 goals of the five
+year plan are described here:
 
-	https://bind10.isc.org/wiki/Year2Milestones
+	http://bind10.isc.org/wiki/Year3Goals
 
 This release includes the bind10 master process, b10-msgq message
 bus, b10-auth authoritative DNS server (with SQLite3 and in-memory
@@ -67,8 +67,8 @@ e.g.,
 Operating-System specific tips:
 
 - FreeBSD
-  You may need to install a python binding for sqlite3 by hand.  A
-  sample procedure is as follows:
+  You may need to install a python binding for sqlite3 by hand.
+  A sample procedure is as follows:
   - add the following to /etc/make.conf
     PYTHON_VERSION=3.1
   - build and install the python binding from ports, assuming the top

configure.ac

@@ -2,7 +2,7 @@
 # Process this file with autoconf to produce a configure script.
 
 AC_PREREQ([2.59])
-AC_INIT(bind10-devel, 20110519, bind10-dev@isc.org)
+AC_INIT(bind10-devel, 20110809, bind10-dev@isc.org)
 AC_CONFIG_SRCDIR(README)
 AM_INIT_AUTOMAKE
 AC_CONFIG_HEADERS([config.h])
@@ -12,6 +12,12 @@ AC_PROG_CXX
 
 # Libtool configuration
 #
+
+# libtool cannot handle spaces in paths, so exit early if there is one
+if [ test `echo $PWD | grep -c ' '` != "0"  ]; then
+    AC_MSG_ERROR([BIND 10 cannot be built in a directory that contains spaces, because of libtool limitations. Please change the directory name, or use a symbolic link that does not contain spaces.])
+fi
+
 # On FreeBSD (and probably some others), clang++ does not meet an autoconf
 # assumption in identifying libtool configuration regarding shared library:
 # the configure script will execute "$CC -shared $CFLAGS/$CXXFLAGS -v" and
@@ -139,6 +145,26 @@ else
 	AC_SUBST(pkgpyexecdir)
 fi
 
+# We need to store the default pyexecdir in a separate variable so that
+# we can specify in Makefile.am the install directory of various BIND 10
+# python scripts and loadable modules; in Makefile.am we cannot replace
+# $(pyexecdir) using itself, e.g, this doesn't work:
+# pyexecdir = $(pyexecdir)/isc/some_module
+# The separate variable makes this setup possible as follows:
+# pyexecdir = $(PYTHON_SITEPKG_DIR)/isc/some_module
+PYTHON_SITEPKG_DIR=${pyexecdir}
+AC_SUBST(PYTHON_SITEPKG_DIR)
+
+# This will be commonly used in various Makefile.am's that need to generate
+# python log messages.
+PYTHON_LOGMSGPKG_DIR="\$(top_builddir)/src/lib/python/isc/log_messages"
+AC_SUBST(PYTHON_LOGMSGPKG_DIR)
+
+# This is python package paths commonly used in python tests.  See
+# README of log_messages for why it's included.
+COMMON_PYTHON_PATH="\$(abs_top_builddir)/src/lib/python/isc/log_messages:\$(abs_top_srcdir)/src/lib/python:\$(abs_top_builddir)/src/lib/python"
+AC_SUBST(COMMON_PYTHON_PATH)
+
 # Check for python development environments
 if test -x ${PYTHON}-config; then
 	PYTHON_INCLUDES=`${PYTHON}-config --includes`
@@ -260,6 +286,8 @@ B10_CXXFLAGS="-Wall -Wextra -Wwrite-strings -Woverloaded-virtual -Wno-sign-compa
 case "$host" in
 *-solaris*)
 	MULTITHREADING_FLAG=-pthreads
+	# In Solaris, IN6ADDR_ANY_INIT and IN6ADDR_LOOPBACK_INIT need -Wno-missing-braces
+	B10_CXXFLAGS="$B10_CXXFLAGS -Wno-missing-braces"
 	;;
 *)
 	MULTITHREADING_FLAG=-pthread
@@ -409,7 +437,7 @@ AC_ARG_WITH([botan],
   AC_HELP_STRING([--with-botan=PATH],
     [specify exact directory of Botan library]),
     [botan_path="$withval"])
-if test "${botan_path}" == "no" ; then
+if test "${botan_path}" = "no" ; then
     AC_MSG_ERROR([Need botan for libcryptolink])
 fi
 if test "${botan_path}" != "yes" ; then
@@ -482,7 +510,7 @@ AC_ARG_WITH([log4cplus],
   AC_HELP_STRING([--with-log4cplus=PATH],
     [specify exact directory of log4cplus library and headers]),
     [log4cplus_path="$withval"])
-if test "${log4cplus_path}" == "no" ; then
+if test "${log4cplus_path}" = "no" ; then
     AC_MSG_ERROR([Need log4cplus])
 elif test "${log4cplus_path}" != "yes" ; then
   LOG4CPLUS_INCLUDES="-I${log4cplus_path}/include"
@@ -789,12 +817,6 @@ AC_CONFIG_FILES([Makefile
                  src/bin/zonemgr/tests/Makefile
                  src/bin/stats/Makefile
                  src/bin/stats/tests/Makefile
-                 src/bin/stats/tests/isc/Makefile
-                 src/bin/stats/tests/isc/cc/Makefile
-                 src/bin/stats/tests/isc/config/Makefile
-                 src/bin/stats/tests/isc/util/Makefile
-                 src/bin/stats/tests/testdata/Makefile
-                 src/bin/stats/tests/http/Makefile
                  src/bin/usermgr/Makefile
                  src/bin/tests/Makefile
                  src/lib/Makefile
@@ -809,21 +831,30 @@ AC_CONFIG_FILES([Makefile
                  src/lib/cc/tests/Makefile
                  src/lib/python/Makefile
                  src/lib/python/isc/Makefile
+                 src/lib/python/isc/acl/Makefile
+                 src/lib/python/isc/acl/tests/Makefile
                  src/lib/python/isc/util/Makefile
                  src/lib/python/isc/util/tests/Makefile
                  src/lib/python/isc/datasrc/Makefile
                  src/lib/python/isc/datasrc/tests/Makefile
+                 src/lib/python/isc/dns/Makefile
                  src/lib/python/isc/cc/Makefile
                  src/lib/python/isc/cc/tests/Makefile
                  src/lib/python/isc/config/Makefile
                  src/lib/python/isc/config/tests/Makefile
                  src/lib/python/isc/log/Makefile
                  src/lib/python/isc/log/tests/Makefile
+                 src/lib/python/isc/log_messages/Makefile
+                 src/lib/python/isc/log_messages/work/Makefile
                  src/lib/python/isc/net/Makefile
                  src/lib/python/isc/net/tests/Makefile
                  src/lib/python/isc/notify/Makefile
                  src/lib/python/isc/notify/tests/Makefile
                  src/lib/python/isc/testutils/Makefile
+                 src/lib/python/isc/bind10/Makefile
+                 src/lib/python/isc/bind10/tests/Makefile
+                 src/lib/python/isc/xfrin/Makefile
+                 src/lib/python/isc/xfrin/tests/Makefile
                  src/lib/config/Makefile
                  src/lib/config/tests/Makefile
                  src/lib/config/tests/testdata/Makefile
@@ -839,6 +870,7 @@ AC_CONFIG_FILES([Makefile
                  src/lib/exceptions/tests/Makefile
                  src/lib/datasrc/Makefile
                  src/lib/datasrc/tests/Makefile
+                 src/lib/datasrc/tests/testdata/Makefile
                  src/lib/xfr/Makefile
                  src/lib/log/Makefile
                  src/lib/log/compiler/Makefile
@@ -856,6 +888,7 @@ AC_CONFIG_FILES([Makefile
                  src/lib/util/Makefile
                  src/lib/util/io/Makefile
                  src/lib/util/unittests/Makefile
+                 src/lib/util/python/Makefile
                  src/lib/util/pyunittests/Makefile
                  src/lib/util/tests/Makefile
                  src/lib/acl/Makefile
@@ -889,7 +922,7 @@ AC_OUTPUT([doc/version.ent
            src/bin/zonemgr/run_b10-zonemgr.sh
            src/bin/stats/stats.py
            src/bin/stats/stats_httpd.py
-           src/bin/bind10/bind10.py
+           src/bin/bind10/bind10_src.py
            src/bin/bind10/run_bind10.sh
            src/bin/bind10/tests/bind10_test.py
            src/bin/bindctl/run_bindctl.sh
@@ -913,17 +946,19 @@ AC_OUTPUT([doc/version.ent
            src/lib/python/isc/cc/tests/cc_test
            src/lib/python/isc/notify/tests/notify_out_test
            src/lib/python/isc/log/tests/log_console.py
+           src/lib/python/isc/log_messages/work/__init__.py
            src/lib/dns/gen-rdatacode.py
            src/lib/python/bind10_config.py
-           src/lib/dns/tests/testdata/gen-wiredata.py
            src/lib/cc/session_config.h.pre
            src/lib/cc/tests/session_unittests_config.h
            src/lib/log/tests/console_test.sh
            src/lib/log/tests/destination_test.sh
+           src/lib/log/tests/init_logger_test.sh
            src/lib/log/tests/local_file_test.sh
            src/lib/log/tests/severity_test.sh
            src/lib/log/tests/tempdir.h
            src/lib/util/python/mkpywrapper.py
+           src/lib/util/python/gen_wiredata.py
            src/lib/server_common/tests/data_path.h
            tests/system/conf.sh
            tests/system/glue/setup.sh
@@ -948,12 +983,13 @@ AC_OUTPUT([doc/version.ent
            chmod +x src/bin/msgq/run_msgq.sh
            chmod +x src/bin/msgq/tests/msgq_test
            chmod +x src/lib/dns/gen-rdatacode.py
-           chmod +x src/lib/dns/tests/testdata/gen-wiredata.py
-           chmod +x src/lib/log/tests/local_file_test.sh
            chmod +x src/lib/log/tests/console_test.sh
            chmod +x src/lib/log/tests/destination_test.sh
+           chmod +x src/lib/log/tests/init_logger_test.sh
+           chmod +x src/lib/log/tests/local_file_test.sh
            chmod +x src/lib/log/tests/severity_test.sh
            chmod +x src/lib/util/python/mkpywrapper.py
+           chmod +x src/lib/util/python/gen_wiredata.py
            chmod +x src/lib/python/isc/log/tests/log_console.py
            chmod +x tests/system/conf.sh
           ])

doc/Doxyfile

@@ -568,10 +568,10 @@ WARN_LOGFILE           =
 # directories like "/usr/src/myproject". Separate the files or directories
 # with spaces.
 
-INPUT                  = ../src/lib/cc ../src/lib/config \
-    ../src/lib/cryptolink ../src/lib/dns ../src/lib/datasrc \
-    ../src/bin/auth ../src/bin/resolver ../src/lib/bench \
-    ../src/lib/log ../src/lib/asiolink/ ../src/lib/nsas \
+INPUT                  = ../src/lib/exceptions ../src/lib/cc \
+    ../src/lib/config ../src/lib/cryptolink ../src/lib/dns ../src/lib/datasrc \
+    ../src/bin/auth ../src/bin/resolver ../src/lib/bench ../src/lib/log \
+    ../src/lib/log/compiler ../src/lib/asiolink/ ../src/lib/nsas \
     ../src/lib/testutils ../src/lib/cache ../src/lib/server_common/ \
     ../src/bin/sockcreator/ ../src/lib/util/ \
     ../src/lib/resolve ../src/lib/acl ../src/bin/dhcp6

ext/asio/asio/impl/error_code.ipp

@@ -11,6 +11,9 @@
 #ifndef ASIO_IMPL_ERROR_CODE_IPP
 #define ASIO_IMPL_ERROR_CODE_IPP
 
+// strerror() needs <cstring>
+#include <cstring>
+
 #if defined(_MSC_VER) && (_MSC_VER >= 1200)
 # pragma once
 #endif // defined(_MSC_VER) && (_MSC_VER >= 1200)

src/bin/auth/Makefile.am

@@ -50,12 +50,19 @@ b10_auth_SOURCES += command.cc command.h
 b10_auth_SOURCES += common.h common.cc
 b10_auth_SOURCES += statistics.cc statistics.h
 b10_auth_SOURCES += main.cc
+# This is a temporary workaround for #1206, where the InMemoryClient has been
+# moved to an ldopened library. We could add that library to LDADD, but that
+# is nonportable. When #1207 is done this becomes moot anyway, and the
+# specific workaround is not needed anymore, so we can then remove this
+# line again.
+b10_auth_SOURCES += ${top_srcdir}/src/lib/datasrc/memory_datasrc.cc
 
 nodist_b10_auth_SOURCES = auth_messages.h auth_messages.cc
 EXTRA_DIST += auth_messages.mes
 
 b10_auth_LDADD =  $(top_builddir)/src/lib/datasrc/libdatasrc.la
 b10_auth_LDADD += $(top_builddir)/src/lib/dns/libdns++.la
+b10_auth_LDADD += $(top_builddir)/src/lib/util/libutil.la
 b10_auth_LDADD += $(top_builddir)/src/lib/config/libcfgclient.la
 b10_auth_LDADD += $(top_builddir)/src/lib/cc/libcc.la
 b10_auth_LDADD += $(top_builddir)/src/lib/exceptions/libexceptions.la

src/bin/auth/auth.spec.pre.in

@@ -122,6 +122,24 @@
           }
         ]
       }
+    ],
+    "statistics": [
+      {
+        "item_name": "queries.tcp",
+        "item_type": "integer",
+        "item_optional": false,
+        "item_default": 0,
+        "item_title": "Queries TCP ",
+        "item_description": "A number of total query counts which all auth servers receive over TCP since they started initially"
+      },
+      {
+        "item_name": "queries.udp",
+        "item_type": "integer",
+        "item_optional": false,
+        "item_default": 0,
+        "item_title": "Queries UDP",
+        "item_description": "A number of total query counts which all auth servers receive over UDP since they started initially"
+      }
     ]
   }
 }

src/bin/auth/auth_config.cc

@@ -107,7 +107,7 @@ DatasourcesConfig::commit() {
     // server implementation details, and isn't scalable wrt the number of
     // data source types, and should eventually be improved.
     // Currently memory data source for class IN is the only possibility.
-    server_.setMemoryDataSrc(RRClass::IN(), AuthSrv::MemoryDataSrcPtr());
+    server_.setInMemoryClient(RRClass::IN(), AuthSrv::InMemoryClientPtr());
 
     BOOST_FOREACH(shared_ptr<AuthConfigParser> datasrc_config, datasources_) {
         datasrc_config->commit();
@@ -125,12 +125,12 @@ public:
     {}
     virtual void build(ConstElementPtr config_value);
     virtual void commit() {
-        server_.setMemoryDataSrc(rrclass_, memory_datasrc_);
+        server_.setInMemoryClient(rrclass_, memory_client_);
     }
 private:
     AuthSrv& server_;
     RRClass rrclass_;
-    AuthSrv::MemoryDataSrcPtr memory_datasrc_;
+    AuthSrv::InMemoryClientPtr memory_client_;
 };
 
 void
@@ -143,8 +143,8 @@ MemoryDatasourceConfig::build(ConstElementPtr config_value) {
     // We'd eventually optimize building zones (in case of reloading) by
     // selectively loading fresh zones.  Right now we simply check the
     // RR class is supported by the server implementation.
-    server_.getMemoryDataSrc(rrclass_);
-    memory_datasrc_ = AuthSrv::MemoryDataSrcPtr(new MemoryDataSrc());
+    server_.getInMemoryClient(rrclass_);
+    memory_client_ = AuthSrv::InMemoryClientPtr(new InMemoryClient());
 
     ConstElementPtr zones_config = config_value->get("zones");
     if (!zones_config) {
@@ -163,9 +163,10 @@ MemoryDatasourceConfig::build(ConstElementPtr config_value) {
             isc_throw(AuthConfigError, "Missing zone file for zone: "
                       << origin->str());
         }
-        shared_ptr<MemoryZone> new_zone(new MemoryZone(rrclass_,
+        shared_ptr<InMemoryZoneFinder> zone_finder(new
+                                                   InMemoryZoneFinder(rrclass_,
             Name(origin->stringValue())));
-        const result::Result result = memory_datasrc_->addZone(new_zone);
+        const result::Result result = memory_client_->addZone(zone_finder);
         if (result == result::EXIST) {
             isc_throw(AuthConfigError, "zone "<< origin->str()
                       << " already exists");
@@ -177,7 +178,7 @@ MemoryDatasourceConfig::build(ConstElementPtr config_value) {
          * need the load method to be split into some kind of build and
          * commit/abort parts.
          */
-        new_zone->load(file->stringValue());
+        zone_finder->load(file->stringValue());
     }
 }
 

src/bin/auth/auth_messages.mes

@@ -63,7 +63,7 @@ datebase data source, listing the file that is being accessed.
 
 % AUTH_DNS_SERVICES_CREATED DNS services created
 This is a debug message indicating that the component that will handling
-incoming queries for the authoritiative server (DNSServices) has been
+incoming queries for the authoritative server (DNSServices) has been
 successfully created. It is issued during server startup is an indication
 that the initialization is proceeding normally.
 
@@ -74,7 +74,7 @@ reason for the failure is given in the message.) The server will drop the
 packet.
 
 % AUTH_LOAD_TSIG loading TSIG keys
-This is a debug message indicating that the authoritiative server
+This is a debug message indicating that the authoritative server
 has requested the keyring holding TSIG keys from the configuration
 database. It is issued during server startup is an indication that the
 initialization is proceeding normally.
@@ -141,8 +141,8 @@ encountered an internal error whilst processing a received packet:
 the cause of the error is included in the message.
 
 The server will return a SERVFAIL error code to the sender of the packet.
-However, this message indicates a potential error in the server.
-Please open a bug ticket for this issue.
+This message indicates a potential error in the server.  Please open a
+bug ticket for this issue.
 
 % AUTH_RECEIVED_COMMAND command '%1' received
 This is a debug message issued when the authoritative server has received
@@ -209,7 +209,7 @@ channel.  It is issued during server startup is an indication that the
 initialization is proceeding normally.
 
 % AUTH_STATS_COMMS communication error in sending statistics data: %1
-An error was encountered when the authoritiative server tried to send data
+An error was encountered when the authoritative server tried to send data
 to the statistics daemon. The message includes additional information
 describing the reason for the failure.
 
@@ -257,4 +257,7 @@ request. The zone manager component has been informed of the request,
 but has returned an error response (which is included in the message). The
 NOTIFY request will not be honored.
 
+% AUTH_INVALID_STATISTICS_DATA invalid specification of statistics data specified
+An error was encountered when the authoritiative server specified
+statistics data which is invalid for the auth specification file.
 

src/bin/auth/auth_srv.cc

@@ -108,8 +108,8 @@ public:
     AbstractSession* xfrin_session_;
 
     /// In-memory data source.  Currently class IN only for simplicity.
-    const RRClass memory_datasrc_class_;
-    AuthSrv::MemoryDataSrcPtr memory_datasrc_;
+    const RRClass memory_client_class_;
+    AuthSrv::InMemoryClientPtr memory_client_;
 
     /// Hot spot cache
     isc::datasrc::HotCache cache_;
@@ -125,6 +125,10 @@ public:
 
     /// The TSIG keyring
     const shared_ptr<TSIGKeyRing>* keyring_;
+
+    /// Bind the ModuleSpec object in config_session_ with
+    /// isc:config::ModuleSpec::validateStatistics.
+    void registerStatisticsValidator();
 private:
     std::string db_file_;
 
@@ -139,13 +143,16 @@ private:
 
     /// Increment query counter
     void incCounter(const int protocol);
+
+    // validateStatistics
+    bool validateStatistics(isc::data::ConstElementPtr data) const;
 };
 
 AuthSrvImpl::AuthSrvImpl(const bool use_cache,
                          AbstractXfroutClient& xfrout_client) :
     config_session_(NULL),
     xfrin_session_(NULL),
-    memory_datasrc_class_(RRClass::IN()),
+    memory_client_class_(RRClass::IN()),
     statistics_timer_(io_service_),
     counters_(),
     keyring_(NULL),
@@ -290,7 +297,7 @@ makeErrorMessage(MessagePtr message, OutputBufferPtr buffer,
         message->toWire(renderer);
     }
     LOG_DEBUG(auth_logger, DBG_AUTH_MESSAGES, AUTH_SEND_ERROR_RESPONSE)
-              .arg(message->toText());
+              .arg(renderer.getLength()).arg(*message);
 }
 }
 
@@ -317,6 +324,7 @@ AuthSrv::setXfrinSession(AbstractSession* xfrin_session) {
 void
 AuthSrv::setConfigSession(ModuleCCSession* config_session) {
     impl_->config_session_ = config_session;
+    impl_->registerStatisticsValidator();
 }
 
 void
@@ -329,34 +337,34 @@ AuthSrv::getConfigSession() const {
     return (impl_->config_session_);
 }
 
-AuthSrv::MemoryDataSrcPtr
-AuthSrv::getMemoryDataSrc(const RRClass& rrclass) {
+AuthSrv::InMemoryClientPtr
+AuthSrv::getInMemoryClient(const RRClass& rrclass) {
     // XXX: for simplicity, we only support the IN class right now.
-    if (rrclass != impl_->memory_datasrc_class_) {
+    if (rrclass != impl_->memory_client_class_) {
         isc_throw(InvalidParameter,
                   "Memory data source is not supported for RR class "
                   << rrclass);
     }
-    return (impl_->memory_datasrc_);
+    return (impl_->memory_client_);
 }
 
 void
-AuthSrv::setMemoryDataSrc(const isc::dns::RRClass& rrclass,
-                          MemoryDataSrcPtr memory_datasrc)
+AuthSrv::setInMemoryClient(const isc::dns::RRClass& rrclass,
+                           InMemoryClientPtr memory_client)
 {
     // XXX: see above
-    if (rrclass != impl_->memory_datasrc_class_) {
+    if (rrclass != impl_->memory_client_class_) {
         isc_throw(InvalidParameter,
                   "Memory data source is not supported for RR class "
                   << rrclass);
-    } else if (!impl_->memory_datasrc_ && memory_datasrc) {
+    } else if (!impl_->memory_client_ && memory_client) {
         LOG_DEBUG(auth_logger, DBG_AUTH_OPS, AUTH_MEM_DATASRC_ENABLED)
                   .arg(rrclass);
-    } else if (impl_->memory_datasrc_ && !memory_datasrc) {
+    } else if (impl_->memory_client_ && !memory_client) {
         LOG_DEBUG(auth_logger, DBG_AUTH_OPS, AUTH_MEM_DATASRC_DISABLED)
                   .arg(rrclass);
     }
-    impl_->memory_datasrc_ = memory_datasrc;
+    impl_->memory_client_ = memory_client;
 }
 
 uint32_t
@@ -505,10 +513,10 @@ AuthSrvImpl::processNormalQuery(const IOMessage& io_message, MessagePtr message,
         // If a memory data source is configured call the separate
         // Query::process()
         const ConstQuestionPtr question = *message->beginQuestion();
-        if (memory_datasrc_ && memory_datasrc_class_ == question->getClass()) {
+        if (memory_client_ && memory_client_class_ == question->getClass()) {
             const RRType& qtype = question->getType();
             const Name& qname = question->getName();
-            auth::Query(*memory_datasrc_, qname, qtype, *message).process();
+            auth::Query(*memory_client_, qname, qtype, *message).process();
         } else {
             datasrc::Query query(*message, cache_, dnssec_ok);
             data_sources_.doQuery(query);
@@ -670,6 +678,22 @@ AuthSrvImpl::incCounter(const int protocol) {
     }
 }
 
+void
+AuthSrvImpl::registerStatisticsValidator() {
+    counters_.registerStatisticsValidator(
+        boost::bind(&AuthSrvImpl::validateStatistics, this, _1));
+}
+
+bool
+AuthSrvImpl::validateStatistics(isc::data::ConstElementPtr data) const {
+    if (config_session_ == NULL) {
+        return (false);
+    }
+    return (
+        config_session_->getModuleSpec().validateStatistics(
+            data, true));
+}
+
 ConstElementPtr
 AuthSrvImpl::setDbFile(ConstElementPtr config) {
     ConstElementPtr answer = isc::config::createAnswer();

src/bin/auth/auth_srv.h

@@ -17,7 +17,7 @@
 
 #include <string>
 
-// For MemoryDataSrcPtr below.  This should be a temporary definition until
+// For InMemoryClientPtr below.  This should be a temporary definition until
 // we reorganize the data source framework.
 #include <boost/shared_ptr.hpp>
 
@@ -39,7 +39,7 @@
 
 namespace isc {
 namespace datasrc {
-class MemoryDataSrc;
+class InMemoryClient;
 }
 namespace xfr {
 class AbstractXfroutClient;
@@ -133,7 +133,7 @@ public:
     /// If there is a data source installed, it will be replaced with the
     /// new one.
     ///
-    /// In the current implementation, the SQLite data source and MemoryDataSrc
+    /// In the current implementation, the SQLite data source and InMemoryClient
     /// are assumed.
     /// We can enable memory data source and get the path of SQLite database by
     /// the \c config parameter.  If we disabled memory data source, the SQLite
@@ -233,16 +233,16 @@ public:
     ///
     void setXfrinSession(isc::cc::AbstractSession* xfrin_session);
 
-    /// A shared pointer type for \c MemoryDataSrc.
+    /// A shared pointer type for \c InMemoryClient.
     ///
     /// This is defined inside the \c AuthSrv class as it's supposed to be
     /// a short term interface until we integrate the in-memory and other
     /// data source frameworks.
-    typedef boost::shared_ptr<isc::datasrc::MemoryDataSrc> MemoryDataSrcPtr;
+    typedef boost::shared_ptr<isc::datasrc::InMemoryClient> InMemoryClientPtr;
 
-    /// An immutable shared pointer type for \c MemoryDataSrc.
-    typedef boost::shared_ptr<const isc::datasrc::MemoryDataSrc>
-    ConstMemoryDataSrcPtr;
+    /// An immutable shared pointer type for \c InMemoryClient.
+    typedef boost::shared_ptr<const isc::datasrc::InMemoryClient>
+    ConstInMemoryClientPtr;
 
     /// Returns the in-memory data source configured for the \c AuthSrv,
     /// if any.
@@ -260,11 +260,11 @@ public:
     /// \param rrclass The RR class of the requested in-memory data source.
     /// \return A pointer to the in-memory data source, if configured;
     /// otherwise NULL.
-    MemoryDataSrcPtr getMemoryDataSrc(const isc::dns::RRClass& rrclass);
+    InMemoryClientPtr getInMemoryClient(const isc::dns::RRClass& rrclass);
 
     /// Sets or replaces the in-memory data source of the specified RR class.
     ///
-    /// As noted in \c getMemoryDataSrc(), some RR classes may not be
+    /// As noted in \c getInMemoryClient(), some RR classes may not be
     /// supported, in which case an exception of class \c InvalidParameter
     /// will be thrown.
     /// This method never throws an exception otherwise.
@@ -275,9 +275,9 @@ public:
     /// in-memory data source.
     ///
     /// \param rrclass The RR class of the in-memory data source to be set.
-    /// \param memory_datasrc A (shared) pointer to \c MemoryDataSrc to be set.
-    void setMemoryDataSrc(const isc::dns::RRClass& rrclass,
-                          MemoryDataSrcPtr memory_datasrc);
+    /// \param memory_datasrc A (shared) pointer to \c InMemoryClient to be set.
+    void setInMemoryClient(const isc::dns::RRClass& rrclass,
+                           InMemoryClientPtr memory_client);
 
     /// \brief Set the communication session with Statistics.
     ///

src/bin/auth/b10-auth.8

@@ -2,12 +2,12 @@
 .\"     Title: b10-auth
 .\"    Author: [FIXME: author] [see http://docbook.sf.net/el/author]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: March 8, 2011
+.\"      Date: August 11, 2011
 .\"    Manual: BIND10
 .\"    Source: BIND10
 .\"  Language: English
 .\"
-.TH "B10\-AUTH" "8" "March 8, 2011" "BIND10" "BIND10"
+.TH "B10\-AUTH" "8" "August 11, 2011" "BIND10" "BIND10"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -70,18 +70,6 @@ defines the path to the SQLite3 zone file when using the sqlite datasource\&. Th
 /usr/local/var/bind10\-devel/zone\&.sqlite3\&.
 .PP
 
-\fIlisten_on\fR
-is a list of addresses and ports for
-\fBb10\-auth\fR
-to listen on\&. The list items are the
-\fIaddress\fR
-string and
-\fIport\fR
-number\&. By default,
-\fBb10\-auth\fR
-listens on port 53 on the IPv6 (::) and IPv4 (0\&.0\&.0\&.0) wildcard addresses\&.
-.PP
-
 \fIdatasources\fR
 configures data sources\&. The list items include:
 \fItype\fR
@@ -114,6 +102,18 @@ In this development version, currently this is only used for the memory data sou
 .RE
 .PP
 
+\fIlisten_on\fR
+is a list of addresses and ports for
+\fBb10\-auth\fR
+to listen on\&. The list items are the
+\fIaddress\fR
+string and
+\fIport\fR
+number\&. By default,
+\fBb10\-auth\fR
+listens on port 53 on the IPv6 (::) and IPv4 (0\&.0\&.0\&.0) wildcard addresses\&.
+.PP
+
 \fIstatistics\-interval\fR
 is the timer interval in seconds for
 \fBb10\-auth\fR
@@ -164,6 +164,25 @@ immediately\&.
 \fBshutdown\fR
 exits
 \fBb10\-auth\fR\&. (Note that the BIND 10 boss process will restart this service\&.)
+.SH "STATISTICS DATA"
+.PP
+The statistics data collected by the
+\fBb10\-stats\fR
+daemon include:
+.PP
+auth\&.queries\&.tcp
+.RS 4
+Total count of queries received by the
+\fBb10\-auth\fR
+server over TCP since startup\&.
+.RE
+.PP
+auth\&.queries\&.udp
+.RS 4
+Total count of queries received by the
+\fBb10\-auth\fR
+server over UDP since startup\&.
+.RE
 .SH "FILES"
 .PP
 

src/bin/auth/b10-auth.xml

@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>March 8, 2011</date>
+    <date>August 11, 2011</date>
   </refentryinfo>
 
   <refmeta>
@@ -132,15 +132,6 @@
     </para>
 
     <para>
-      <varname>listen_on</varname> is a list of addresses and ports for
-      <command>b10-auth</command> to listen on.
-      The list items are the <varname>address</varname> string
-      and <varname>port</varname> number.
-      By default, <command>b10-auth</command> listens on port 53
-      on the IPv6 (::) and IPv4 (0.0.0.0) wildcard addresses.
-    </para>
-
-    <para>
       <varname>datasources</varname> configures data sources.
       The list items include:
       <varname>type</varname> to optionally choose the data source type
@@ -165,6 +156,15 @@
     </para>
 
     <para>
+      <varname>listen_on</varname> is a list of addresses and ports for
+      <command>b10-auth</command> to listen on.
+      The list items are the <varname>address</varname> string
+      and <varname>port</varname> number.
+      By default, <command>b10-auth</command> listens on port 53
+      on the IPv6 (::) and IPv4 (0.0.0.0) wildcard addresses.
+    </para>
+
+    <para>
       <varname>statistics-interval</varname> is the timer interval
       in seconds for <command>b10-auth</command> to share its
       statistics information to
@@ -209,6 +209,34 @@
   </refsect1>
 
   <refsect1>
+    <title>STATISTICS DATA</title>
+
+    <para>
+      The statistics data collected by the <command>b10-stats</command>
+      daemon include:
+    </para>
+
+    <variablelist>
+
+      <varlistentry>
+        <term>auth.queries.tcp</term>
+        <listitem><simpara>Total count of queries received by the
+          <command>b10-auth</command> server over TCP since startup.
+        </simpara></listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>auth.queries.udp</term>
+        <listitem><simpara>Total count of queries received by the
+          <command>b10-auth</command> server over UDP since startup.
+        </simpara></listitem>
+      </varlistentry>
+
+    </variablelist>
+
+  </refsect1>
+
+  <refsect1>
     <title>FILES</title>
     <para>
       <filename>/usr/local/var/bind10-devel/zone.sqlite3</filename>

src/bin/auth/benchmarks/Makefile.am

@@ -13,10 +13,17 @@ query_bench_SOURCES += ../auth_srv.h ../auth_srv.cc
 query_bench_SOURCES += ../auth_config.h ../auth_config.cc
 query_bench_SOURCES += ../statistics.h ../statistics.cc
 query_bench_SOURCES += ../auth_log.h ../auth_log.cc
+# This is a temporary workaround for #1206, where the InMemoryClient has been
+# moved to an ldopened library. We could add that library to LDADD, but that
+# is nonportable. When #1207 is done this becomes moot anyway, and the
+# specific workaround is not needed anymore, so we can then remove this
+# line again.
+query_bench_SOURCES += ${top_srcdir}/src/lib/datasrc/memory_datasrc.cc
 
 nodist_query_bench_SOURCES = ../auth_messages.h ../auth_messages.cc
 
 query_bench_LDADD = $(top_builddir)/src/lib/dns/libdns++.la
+query_bench_LDADD += $(top_builddir)/src/lib/util/libutil.la
 query_bench_LDADD += $(top_builddir)/src/lib/exceptions/libexceptions.la
 query_bench_LDADD += $(top_builddir)/src/lib/bench/libbench.la
 query_bench_LDADD += $(top_builddir)/src/lib/datasrc/libdatasrc.la

src/bin/auth/command.cc

@@ -136,19 +136,21 @@ public:
         // that doesn't block other server operations.
         // TODO: we may (should?) want to check the "last load time" and
         // the timestamp of the file and skip loading if the file isn't newer.
-        shared_ptr<MemoryZone> newzone(new MemoryZone(oldzone->getClass(),
-                                                      oldzone->getOrigin()));
-        newzone->load(oldzone->getFileName());
-        oldzone->swap(*newzone);
+        shared_ptr<InMemoryZoneFinder> zone_finder(
+            new InMemoryZoneFinder(old_zone_finder->getClass(),
+                                   old_zone_finder->getOrigin()));
+        zone_finder->load(old_zone_finder->getFileName());
+        old_zone_finder->swap(*zone_finder);
         LOG_DEBUG(auth_logger, DBG_AUTH_OPS, AUTH_LOAD_ZONE)
-                  .arg(newzone->getOrigin()).arg(newzone->getClass());
+                  .arg(zone_finder->getOrigin()).arg(zone_finder->getClass());
     }
 
 private:
-    shared_ptr<MemoryZone> oldzone; // zone to be updated with the new file.
+    // zone finder to be updated with the new file.
+    shared_ptr<InMemoryZoneFinder> old_zone_finder;
 
     // A helper private method to parse and validate command parameters.
-    // On success, it sets 'oldzone' to the zone to be updated.
+    // On success, it sets 'old_zone_finder' to the zone to be updated.
     // It returns true if everything is okay; and false if the command is
     // valid but there's no need for further process.
     bool validate(AuthSrv& server, isc::data::ConstElementPtr args) {
@@ -176,7 +178,7 @@ private:
         const RRClass zone_class = class_elem ?
             RRClass(class_elem->stringValue()) : RRClass::IN();
 
-        AuthSrv::MemoryDataSrcPtr datasrc(server.getMemoryDataSrc(zone_class));
+        AuthSrv::InMemoryClientPtr datasrc(server.getInMemoryClient(zone_class));
         if (datasrc == NULL) {
             isc_throw(AuthCommandError, "Memory data source is disabled");
         }
@@ -188,13 +190,14 @@ private:
         const Name origin(origin_elem->stringValue());
 
         // Get the current zone
-        const MemoryDataSrc::FindResult result = datasrc->findZone(origin);
+        const InMemoryClient::FindResult result = datasrc->findZone(origin);
         if (result.code != result::SUCCESS) {
             isc_throw(AuthCommandError, "Zone " << origin <<
                       " is not found in data source");
         }
 
-        oldzone = boost::dynamic_pointer_cast<MemoryZone>(result.zone);
+        old_zone_finder = boost::dynamic_pointer_cast<InMemoryZoneFinder>(
+            result.zone_finder);
 
         return (true);
     }

src/bin/auth/query.cc

@@ -19,7 +19,7 @@
 #include <dns/rcode.h>
 #include <dns/rdataclass.h>
 
-#include <datasrc/memory_datasrc.h>
+#include <datasrc/client.h>
 
 #include <auth/query.h>
 
@@ -31,14 +31,14 @@ namespace isc {
 namespace auth {
 
 void
-Query::getAdditional(const Zone& zone, const RRset& rrset) const {
+Query::getAdditional(ZoneFinder& zone, const RRset& rrset) const {
     RdataIteratorPtr rdata_iterator(rrset.getRdataIterator());
     for (; !rdata_iterator->isLast(); rdata_iterator->next()) {
         const Rdata& rdata(rdata_iterator->getCurrent());
         if (rrset.getType() == RRType::NS()) {
             // Need to perform the search in the "GLUE OK" mode.
             const generic::NS& ns = dynamic_cast<const generic::NS&>(rdata);
-            findAddrs(zone, ns.getNSName(), Zone::FIND_GLUE_OK);
+            findAddrs(zone, ns.getNSName(), ZoneFinder::FIND_GLUE_OK);
         } else if (rrset.getType() == RRType::MX()) {
             const generic::MX& mx(dynamic_cast<const generic::MX&>(rdata));
             findAddrs(zone, mx.getMXName());
@@ -47,8 +47,8 @@ Query::getAdditional(const Zone& zone, const RRset& rrset) const {
 }
 
 void
-Query::findAddrs(const Zone& zone, const Name& qname,
-                 const Zone::FindOptions options) const
+Query::findAddrs(ZoneFinder& zone, const Name& qname,
+                 const ZoneFinder::FindOptions options) const
 {
     // Out of zone name
     NameComparisonResult result = zone.getOrigin().compare(qname);
@@ -66,30 +66,31 @@ Query::findAddrs(const Zone& zone, const Name& qname,
 
     // Find A rrset
     if (qname_ != qname || qtype_ != RRType::A()) {
-        Zone::FindResult a_result = zone.find(qname, RRType::A(), NULL,
-                                              options);
-        if (a_result.code == Zone::SUCCESS) {
+        ZoneFinder::FindResult a_result = zone.find(qname, RRType::A(), NULL,
+                                                    options | dnssec_opt_);
+        if (a_result.code == ZoneFinder::SUCCESS) {
             response_.addRRset(Message::SECTION_ADDITIONAL,
-                    boost::const_pointer_cast<RRset>(a_result.rrset));
+                    boost::const_pointer_cast<RRset>(a_result.rrset), dnssec_);
         }
     }
 
     // Find AAAA rrset
     if (qname_ != qname || qtype_ != RRType::AAAA()) {
-        Zone::FindResult aaaa_result =
-            zone.find(qname, RRType::AAAA(), NULL, options);
-        if (aaaa_result.code == Zone::SUCCESS) {
+        ZoneFinder::FindResult aaaa_result =
+            zone.find(qname, RRType::AAAA(), NULL, options | dnssec_opt_);
+        if (aaaa_result.code == ZoneFinder::SUCCESS) {
             response_.addRRset(Message::SECTION_ADDITIONAL,
-                    boost::const_pointer_cast<RRset>(aaaa_result.rrset));
+                    boost::const_pointer_cast<RRset>(aaaa_result.rrset),
+                    dnssec_);
         }
     }
 }
 
 void
-Query::putSOA(const Zone& zone) const {
-    Zone::FindResult soa_result(zone.find(zone.getOrigin(),
-        RRType::SOA()));
-    if (soa_result.code != Zone::SUCCESS) {
+Query::putSOA(ZoneFinder& zone) const {
+    ZoneFinder::FindResult soa_result(zone.find(zone.getOrigin(),
+        RRType::SOA(), NULL, dnssec_opt_));
+    if (soa_result.code != ZoneFinder::SUCCESS) {
         isc_throw(NoSOA, "There's no SOA record in zone " <<
             zone.getOrigin().toText());
     } else {
@@ -99,21 +100,23 @@ Query::putSOA(const Zone& zone) const {
          * to insist.
          */
         response_.addRRset(Message::SECTION_AUTHORITY,
-            boost::const_pointer_cast<RRset>(soa_result.rrset));
+            boost::const_pointer_cast<RRset>(soa_result.rrset), dnssec_);
     }
 }
 
 void
-Query::getAuthAdditional(const Zone& zone) const {
+Query::getAuthAdditional(ZoneFinder& zone) const {
     // Fill in authority and addtional sections.
-    Zone::FindResult ns_result = zone.find(zone.getOrigin(), RRType::NS());
+    ZoneFinder::FindResult ns_result = zone.find(zone.getOrigin(),
+                                                 RRType::NS(), NULL,
+                                                 dnssec_opt_);
     // zone origin name should have NS records
-    if (ns_result.code != Zone::SUCCESS) {
+    if (ns_result.code != ZoneFinder::SUCCESS) {
         isc_throw(NoApexNS, "There's no apex NS records in zone " <<
                 zone.getOrigin().toText());
     } else {
         response_.addRRset(Message::SECTION_AUTHORITY,
-            boost::const_pointer_cast<RRset>(ns_result.rrset));
+            boost::const_pointer_cast<RRset>(ns_result.rrset), dnssec_);
         // Handle additional for authority section
         getAdditional(zone, *ns_result.rrset);
     }
@@ -125,8 +128,8 @@ Query::process() const {
     const bool qtype_is_any = (qtype_ == RRType::ANY());
 
     response_.setHeaderFlag(Message::HEADERFLAG_AA, false);
-    const MemoryDataSrc::FindResult result =
-        memory_datasrc_.findZone(qname_);
+    const DataSourceClient::FindResult result =
+        datasrc_client_.findZone(qname_);
 
     // If we have no matching authoritative zone for the query name, return
     // REFUSED.  In short, this is to be compatible with BIND 9, but the
@@ -145,14 +148,15 @@ Query::process() const {
     while (keep_doing) {
         keep_doing = false;
         std::auto_ptr<RRsetList> target(qtype_is_any ? new RRsetList : NULL);
-        const Zone::FindResult db_result(result.zone->find(qname_, qtype_,
-            target.get()));
-
+        const ZoneFinder::FindResult db_result(
+            result.zone_finder->find(qname_, qtype_, target.get(),
+                                     dnssec_opt_));
         switch (db_result.code) {
-            case Zone::DNAME: {
+            case ZoneFinder::DNAME: {
                 // First, put the dname into the answer
                 response_.addRRset(Message::SECTION_ANSWER,
-                    boost::const_pointer_cast<RRset>(db_result.rrset));
+                    boost::const_pointer_cast<RRset>(db_result.rrset),
+                    dnssec_);
                 /*
                  * Empty DNAME should never get in, as it is impossible to
                  * create one in master file.
@@ -188,10 +192,10 @@ Query::process() const {
                     qname_.getLabelCount() -
                     db_result.rrset->getName().getLabelCount()).
                     concatenate(dname.getDname())));
-                response_.addRRset(Message::SECTION_ANSWER, cname);
+                response_.addRRset(Message::SECTION_ANSWER, cname, dnssec_);
                 break;
             }
-            case Zone::CNAME:
+            case ZoneFinder::CNAME:
                 /*
                  * We don't do chaining yet. Therefore handling a CNAME is
                  * mostly the same as handling SUCCESS, but we didn't get
@@ -202,48 +206,59 @@ Query::process() const {
                  * So, just put it there.
                  */
                 response_.addRRset(Message::SECTION_ANSWER,
-                    boost::const_pointer_cast<RRset>(db_result.rrset));
+                    boost::const_pointer_cast<RRset>(db_result.rrset),
+                    dnssec_);
                 break;
-            case Zone::SUCCESS:
+            case ZoneFinder::SUCCESS:
                 if (qtype_is_any) {
                     // If quety type is ANY, insert all RRs under the domain
                     // into answer section.
                     BOOST_FOREACH(RRsetPtr rrset, *target) {
-                        response_.addRRset(Message::SECTION_ANSWER, rrset);
+                        response_.addRRset(Message::SECTION_ANSWER, rrset,
+                                           dnssec_);
                         // Handle additional for answer section
-                        getAdditional(*result.zone, *rrset.get());
+                        getAdditional(*result.zone_finder, *rrset.get());
                     }
                 } else {
                     response_.addRRset(Message::SECTION_ANSWER,
-                        boost::const_pointer_cast<RRset>(db_result.rrset));
+                        boost::const_pointer_cast<RRset>(db_result.rrset),
+                        dnssec_);
                     // Handle additional for answer section
-                    getAdditional(*result.zone, *db_result.rrset);
+                    getAdditional(*result.zone_finder, *db_result.rrset);
                 }
                 // If apex NS records haven't been provided in the answer
                 // section, insert apex NS records into the authority section
                 // and AAAA/A RRS of each of the NS RDATA into the additional
                 // section.
-                if (qname_ != result.zone->getOrigin() ||
-                    db_result.code != Zone::SUCCESS ||
+                if (qname_ != result.zone_finder->getOrigin() ||
+                    db_result.code != ZoneFinder::SUCCESS ||
                     (qtype_ != RRType::NS() && !qtype_is_any))
                 {
-                    getAuthAdditional(*result.zone);
+                    getAuthAdditional(*result.zone_finder);
                 }
                 break;
-            case Zone::DELEGATION:
+            case ZoneFinder::DELEGATION:
                 response_.setHeaderFlag(Message::HEADERFLAG_AA, false);
                 response_.addRRset(Message::SECTION_AUTHORITY,
-                    boost::const_pointer_cast<RRset>(db_result.rrset));
-                getAdditional(*result.zone, *db_result.rrset);
+                    boost::const_pointer_cast<RRset>(db_result.rrset),
+                    dnssec_);
+                getAdditional(*result.zone_finder, *db_result.rrset);
                 break;
-            case Zone::NXDOMAIN:
+            case ZoneFinder::NXDOMAIN:
                 // Just empty answer with SOA in authority section
                 response_.setRcode(Rcode::NXDOMAIN());
-                putSOA(*result.zone);
+                putSOA(*result.zone_finder);
                 break;
-            case Zone::NXRRSET:
+            case ZoneFinder::NXRRSET:
                 // Just empty answer with SOA in authority section
-                putSOA(*result.zone);
+                putSOA(*result.zone_finder);
+                break;
+            default:
+                // These are new result codes (WILDCARD and WILDCARD_NXRRSET)
+                // They should not happen from the in-memory and the database
+                // backend isn't used yet.
+                // TODO: Implement before letting the database backends in
+                isc_throw(isc::NotImplemented, "Unknown result code");
                 break;
         }
     }

src/bin/auth/query.h

@@ -26,7 +26,7 @@ class RRset;
 }
 
 namespace datasrc {
-class MemoryDataSrc;
+class DataSourceClient;
 }
 
 namespace auth {
@@ -36,10 +36,8 @@ namespace auth {
 ///
 /// Many of the design details for this class are still in flux.
 /// We'll revisit and update them as we add more functionality, for example:
-/// - memory_datasrc parameter of the constructor.  It is a data source that
-///   uses in memory dedicated backend.
 /// - as a related point, we may have to pass the RR class of the query.
-///   in the initial implementation the RR class is an attribute of memory
+///   in the initial implementation the RR class is an attribute of
 ///   datasource and omitted.  It's not clear if this assumption holds with
 ///   generic data sources.  On the other hand, it will help keep
 ///   implementation simpler, and we might rather want to modify the design
@@ -51,7 +49,7 @@ namespace auth {
 ///   separate attribute setter.
 /// - likewise, we'll eventually need to do per zone access control, for which
 ///   we need querier's information such as its IP address.
-/// - memory_datasrc and response may better be parameters to process() instead
+/// - datasrc_client and response may better be parameters to process() instead
 ///   of the constructor.
 ///
 /// <b>Note:</b> The class name is intentionally the same as the one used in
@@ -71,7 +69,7 @@ private:
     /// Adds a SOA of the zone into the authority zone of response_.
     /// Can throw NoSOA.
     ///
-    void putSOA(const isc::datasrc::Zone& zone) const;
+    void putSOA(isc::datasrc::ZoneFinder& zone) const;
 
     /// \brief Look up additional data (i.e., address records for the names
     /// included in NS or MX records).
@@ -83,11 +81,11 @@ private:
     /// This method may throw a exception because its underlying methods may
     /// throw exceptions.
     ///
-    /// \param zone The Zone wherein the additional data to the query is bo be
-    /// found.
+    /// \param zone The ZoneFinder through which the additional data for the
+    /// query is to be found.
     /// \param rrset The RRset (i.e., NS or MX rrset) which require additional
     /// processing.
-    void getAdditional(const isc::datasrc::Zone& zone,
+    void getAdditional(isc::datasrc::ZoneFinder& zone,
                        const isc::dns::RRset& rrset) const;
 
     /// \brief Find address records for a specified name.
@@ -102,18 +100,19 @@ private:
     /// The glue records must exactly match the name in the NS RDATA, without
     /// CNAME or wildcard processing.
     ///
-    /// \param zone The \c Zone wherein the address records is to be found.
+    /// \param zone The \c ZoneFinder through which the address records is to
+    /// be found.
     /// \param qname The name in rrset RDATA.
     /// \param options The search options.
-    void findAddrs(const isc::datasrc::Zone& zone,
+    void findAddrs(isc::datasrc::ZoneFinder& zone,
                    const isc::dns::Name& qname,
-                   const isc::datasrc::Zone::FindOptions options
-                   = isc::datasrc::Zone::FIND_DEFAULT) const;
+                   const isc::datasrc::ZoneFinder::FindOptions options
+                   = isc::datasrc::ZoneFinder::FIND_DEFAULT) const;
 
-    /// \brief Look up \c Zone's NS and address records for the NS RDATA
-    /// (domain name) for authoritative answer.
+    /// \brief Look up a zone's NS RRset and their address records for an
+    /// authoritative answer.
     ///
-    /// On returning an authoritative answer, insert the \c Zone's NS into the
+    /// On returning an authoritative answer, insert a zone's NS into the
     /// authority section and AAAA/A RRs of each of the NS RDATA into the
     /// additional section.
     ///
@@ -126,25 +125,29 @@ private:
     /// include AAAA/A RRs under a zone cut in additional section. (BIND 9
     /// excludes under-cut RRs; NSD include them.)
     ///
-    /// \param zone The \c Zone wherein the additional data to the query is to
-    /// be found.
-    void getAuthAdditional(const isc::datasrc::Zone& zone) const;
+    /// \param zone The \c ZoneFinder through which the NS and additional data
+    /// for the query are to be found.
+    void getAuthAdditional(isc::datasrc::ZoneFinder& zone) const;
 
 public:
     /// Constructor from query parameters.
     ///
     /// This constructor never throws an exception.
     ///
-    /// \param memory_datasrc The memory datasource wherein the answer to the query is
+    /// \param datasrc_client The datasource wherein the answer to the query is
     /// to be found.
     /// \param qname The query name
     /// \param qtype The RR type of the query
     /// \param response The response message to store the answer to the query.
-    Query(const isc::datasrc::MemoryDataSrc& memory_datasrc,
+    /// \param dnssec If the answer should include signatures and NSEC/NSEC3 if
+    ///     possible.
+    Query(const isc::datasrc::DataSourceClient& datasrc_client,
           const isc::dns::Name& qname, const isc::dns::RRType& qtype,
-          isc::dns::Message& response) :
-        memory_datasrc_(memory_datasrc), qname_(qname), qtype_(qtype),
-        response_(response)
+          isc::dns::Message& response, bool dnssec = false) :
+        datasrc_client_(datasrc_client), qname_(qname), qtype_(qtype),
+        response_(response), dnssec_(dnssec),
+        dnssec_opt_(dnssec ?  isc::datasrc::ZoneFinder::FIND_DNSSEC :
+                    isc::datasrc::ZoneFinder::FIND_DEFAULT)
     {}
 
     /// Process the query.
@@ -157,7 +160,7 @@ public:
     /// successful search would result in adding a corresponding RRset to
     /// the answer section of the response.
     ///
-    /// If no matching zone is found in the memory datasource, the RCODE of
+    /// If no matching zone is found in the datasource, the RCODE of
     /// SERVFAIL will be set in the response.
     /// <b>Note:</b> this is different from the error code that BIND 9 returns
     /// by default when it's configured as an authoritative-only server (and
@@ -208,10 +211,12 @@ public:
     };
 
 private:
-    const isc::datasrc::MemoryDataSrc& memory_datasrc_;
+    const isc::datasrc::DataSourceClient& datasrc_client_;
     const isc::dns::Name& qname_;
     const isc::dns::RRType& qtype_;
     isc::dns::Message& response_;
+    const bool dnssec_;
+    const isc::datasrc::ZoneFinder::FindOptions dnssec_opt_;
 };
 
 }

src/bin/auth/statistics.cc

@@ -37,11 +37,14 @@ public:
     void inc(const AuthCounters::CounterType type);
     bool submitStatistics() const;
     void setStatisticsSession(isc::cc::AbstractSession* statistics_session);
+    void registerStatisticsValidator
+    (AuthCounters::validator_type validator);
     // Currently for testing purpose only
     uint64_t getCounter(const AuthCounters::CounterType type) const;
 private:
     std::vector<uint64_t> counters_;
     isc::cc::AbstractSession* statistics_session_;
+    AuthCounters::validator_type validator_;
 };
 
 AuthCountersImpl::AuthCountersImpl() :
@@ -67,16 +70,25 @@ AuthCountersImpl::submitStatistics() const {
     }
     std::stringstream statistics_string;
     statistics_string << "{\"command\": [\"set\","
-                      <<   "{ \"stats_data\": "
-                      <<     "{ \"auth.queries.udp\": "
+                      <<   "{ \"owner\": \"Auth\","
+                      <<   "  \"data\":"
+                      <<     "{ \"queries.udp\": "
                       <<     counters_.at(AuthCounters::COUNTER_UDP_QUERY)
-                      <<     ", \"auth.queries.tcp\": "
+                      <<     ", \"queries.tcp\": "
                       <<     counters_.at(AuthCounters::COUNTER_TCP_QUERY)
                       <<   " }"
                       <<   "}"
                       << "]}";
     isc::data::ConstElementPtr statistics_element =
         isc::data::Element::fromJSON(statistics_string);
+    // validate the statistics data before send
+    if (validator_) {
+        if (!validator_(
+                statistics_element->get("command")->get(1)->get("data"))) {
+            LOG_ERROR(auth_logger, AUTH_INVALID_STATISTICS_DATA);
+            return (false);
+        }
+    }
     try {
         // group_{send,recv}msg() can throw an exception when encountering
         // an error, and group_recvmsg() will throw an exception on timeout.
@@ -105,6 +117,13 @@ AuthCountersImpl::setStatisticsSession
     statistics_session_ = statistics_session;
 }
 
+void
+AuthCountersImpl::registerStatisticsValidator
+    (AuthCounters::validator_type validator)
+{
+    validator_ = validator;
+}
+
 // Currently for testing purpose only
 uint64_t
 AuthCountersImpl::getCounter(const AuthCounters::CounterType type) const {
@@ -139,3 +158,10 @@ uint64_t
 AuthCounters::getCounter(const AuthCounters::CounterType type) const {
     return (impl_->getCounter(type));
 }
+
+void
+AuthCounters::registerStatisticsValidator
+    (AuthCounters::validator_type validator) const
+{
+    return (impl_->registerStatisticsValidator(validator));
+}

src/bin/auth/statistics.h

@@ -131,6 +131,26 @@ public:
     /// \return the value of the counter specified by \a type.
     ///
     uint64_t getCounter(const AuthCounters::CounterType type) const;
+
+    /// \brief A type of validation function for the specification in
+    /// isc::config::ModuleSpec.
+    ///
+    /// This type might be useful for not only statistics
+    /// specificatoin but also for config_data specification and for
+    /// commnad.
+    ///
+    typedef boost::function<bool(const isc::data::ConstElementPtr&)>
+    validator_type;
+
+    /// \brief Register a function type of the statistics validation
+    /// function for AuthCounters.
+    ///
+    /// This method never throws an exception.
+    ///
+    /// \param validator A function type of the validation of
+    /// statistics specification.
+    ///
+    void registerStatisticsValidator(AuthCounters::validator_type validator) const;
 };
 
 #endif // __STATISTICS_H

src/bin/auth/tests/Makefile.am

@@ -37,6 +37,13 @@ run_unittests_SOURCES += query_unittest.cc
 run_unittests_SOURCES += change_user_unittest.cc
 run_unittests_SOURCES += statistics_unittest.cc
 run_unittests_SOURCES += run_unittests.cc
+# This is a temporary workaround for #1206, where the InMemoryClient has been
+# moved to an ldopened library. We could add that library to LDADD, but that
+# is nonportable. When #1207 is done this becomes moot anyway, and the
+# specific workaround is not needed anymore, so we can then remove this
+# line again.
+run_unittests_SOURCES += ${top_srcdir}/src/lib/datasrc/memory_datasrc.cc
+
 
 nodist_run_unittests_SOURCES = ../auth_messages.h ../auth_messages.cc
 
@@ -47,6 +54,7 @@ run_unittests_LDADD += $(SQLITE_LIBS)
 run_unittests_LDADD += $(top_builddir)/src/lib/testutils/libtestutils.la
 run_unittests_LDADD +=  $(top_builddir)/src/lib/datasrc/libdatasrc.la
 run_unittests_LDADD +=  $(top_builddir)/src/lib/dns/libdns++.la
+run_unittests_LDADD += $(top_builddir)/src/lib/util/libutil.la
 run_unittests_LDADD += $(top_builddir)/src/lib/asiodns/libasiodns.la
 run_unittests_LDADD += $(top_builddir)/src/lib/asiolink/libasiolink.la
 run_unittests_LDADD += $(top_builddir)/src/lib/config/libcfgclient.la

src/bin/auth/tests/auth_srv_unittest.cc

@@ -651,17 +651,17 @@ TEST_F(AuthSrvTest, updateConfigFail) {
                 QR_FLAG | AA_FLAG, 1, 1, 1, 0);
 }
 
-TEST_F(AuthSrvTest, updateWithMemoryDataSrc) {
+TEST_F(AuthSrvTest, updateWithInMemoryClient) {
     // Test configuring memory data source.  Detailed test cases are covered
     // in the configuration tests.  We only check the AuthSrv interface here.
 
     // By default memory data source isn't enabled
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
     updateConfig(&server,
                  "{\"datasources\": [{\"type\": \"memory\"}]}", true);
     // after successful configuration, we should have one (with empty zoneset).
-    ASSERT_NE(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
-    EXPECT_EQ(0, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    ASSERT_NE(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
+    EXPECT_EQ(0, server.getInMemoryClient(rrclass)->getZoneCount());
 
     // The memory data source is empty, should return REFUSED rcode.
     createDataFromFile("examplequery_fromWire.wire");
@@ -672,7 +672,7 @@ TEST_F(AuthSrvTest, updateWithMemoryDataSrc) {
                 opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
 }
 
-TEST_F(AuthSrvTest, chQueryWithMemoryDataSrc) {
+TEST_F(AuthSrvTest, chQueryWithInMemoryClient) {
     // Configure memory data source for class IN
     updateConfig(&server, "{\"datasources\": "
                  "[{\"class\": \"IN\", \"type\": \"memory\"}]}", true);

src/bin/auth/tests/command_unittest.cc

@@ -48,9 +48,9 @@ using namespace isc::datasrc;
 using namespace isc::config;
 
 namespace {
-class AuthConmmandTest : public ::testing::Test {
+class AuthCommandTest : public ::testing::Test {
 protected:
-    AuthConmmandTest() : server(false, xfrout), rcode(-1) {
+    AuthCommandTest() : server(false, xfrout), rcode(-1) {
         server.setStatisticsSession(&statistics_session);
     }
     void checkAnswer(const int expected_code) {
@@ -60,21 +60,20 @@ protected:
     MockSession statistics_session;
     MockXfroutClient xfrout;
     AuthSrv server;
-    AuthSrv::ConstMemoryDataSrcPtr memory_datasrc;
     ConstElementPtr result;
     int rcode;
 public:
     void stopServer();          // need to be public for boost::bind
 };
 
-TEST_F(AuthConmmandTest, unknownCommand) {
+TEST_F(AuthCommandTest, unknownCommand) {
     result = execAuthServerCommand(server, "no_such_command",
                                    ConstElementPtr());
     parseAnswer(rcode, result);
     EXPECT_EQ(1, rcode);
 }
 
-TEST_F(AuthConmmandTest, DISABLED_unexpectedException) {
+TEST_F(AuthCommandTest, DISABLED_unexpectedException) {
     // execAuthServerCommand() won't catch standard exceptions.
     // Skip this test for now: ModuleCCSession doesn't seem to validate
     // commands.
@@ -83,7 +82,7 @@ TEST_F(AuthConmmandTest, DISABLED_unexpectedException) {
                  runtime_error);
 }
 
-TEST_F(AuthConmmandTest, sendStatistics) {
+TEST_F(AuthCommandTest, sendStatistics) {
     result = execAuthServerCommand(server, "sendstats", ConstElementPtr());
     // Just check some message has been sent.  Detailed tests specific to
     // statistics are done in its own tests.
@@ -92,15 +91,15 @@ TEST_F(AuthConmmandTest, sendStatistics) {
 }
 
 void
-AuthConmmandTest::stopServer() {
+AuthCommandTest::stopServer() {
     result = execAuthServerCommand(server, "shutdown", ConstElementPtr());
     parseAnswer(rcode, result);
     assert(rcode == 0); // make sure the test stops when something is wrong
 }
 
-TEST_F(AuthConmmandTest, shutdown) {
+TEST_F(AuthCommandTest, shutdown) {
     isc::asiolink::IntervalTimer itimer(server.getIOService());
-    itimer.setup(boost::bind(&AuthConmmandTest::stopServer, this), 1);
+    itimer.setup(boost::bind(&AuthCommandTest::stopServer, this), 1);
     server.getIOService().run();
     EXPECT_EQ(0, rcode);
 }
@@ -110,18 +109,18 @@ TEST_F(AuthConmmandTest, shutdown) {
 // zones, and checks the zones are correctly loaded.
 void
 zoneChecks(AuthSrv& server) {
-    EXPECT_TRUE(server.getMemoryDataSrc(RRClass::IN()));
-    EXPECT_EQ(Zone::SUCCESS, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test1.example")).zone->
+    EXPECT_TRUE(server.getInMemoryClient(RRClass::IN()));
+    EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test1.example")).zone_finder->
               find(Name("ns.test1.example"), RRType::A()).code);
-    EXPECT_EQ(Zone::NXRRSET, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test1.example")).zone->
+    EXPECT_EQ(ZoneFinder::NXRRSET, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test1.example")).zone_finder->
               find(Name("ns.test1.example"), RRType::AAAA()).code);
-    EXPECT_EQ(Zone::SUCCESS, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test2.example")).zone->
+    EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test2.example")).zone_finder->
               find(Name("ns.test2.example"), RRType::A()).code);
-    EXPECT_EQ(Zone::NXRRSET, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test2.example")).zone->
+    EXPECT_EQ(ZoneFinder::NXRRSET, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test2.example")).zone_finder->
               find(Name("ns.test2.example"), RRType::AAAA()).code);
 }
 
@@ -147,25 +146,25 @@ configureZones(AuthSrv& server) {
 
 void
 newZoneChecks(AuthSrv& server) {
-    EXPECT_TRUE(server.getMemoryDataSrc(RRClass::IN()));
-    EXPECT_EQ(Zone::SUCCESS, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test1.example")).zone->
+    EXPECT_TRUE(server.getInMemoryClient(RRClass::IN()));
+    EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test1.example")).zone_finder->
               find(Name("ns.test1.example"), RRType::A()).code);
     // now test1.example should have ns/AAAA
-    EXPECT_EQ(Zone::SUCCESS, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test1.example")).zone->
+    EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test1.example")).zone_finder->
               find(Name("ns.test1.example"), RRType::AAAA()).code);
 
     // test2.example shouldn't change
-    EXPECT_EQ(Zone::SUCCESS, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test2.example")).zone->
+    EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test2.example")).zone_finder->
               find(Name("ns.test2.example"), RRType::A()).code);
-    EXPECT_EQ(Zone::NXRRSET, server.getMemoryDataSrc(RRClass::IN())->
-              findZone(Name("ns.test2.example")).zone->
+    EXPECT_EQ(ZoneFinder::NXRRSET, server.getInMemoryClient(RRClass::IN())->
+              findZone(Name("ns.test2.example")).zone_finder->
               find(Name("ns.test2.example"), RRType::AAAA()).code);
 }
 
-TEST_F(AuthConmmandTest, loadZone) {
+TEST_F(AuthCommandTest, loadZone) {
     configureZones(server);
 
     ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR
@@ -182,7 +181,7 @@ TEST_F(AuthConmmandTest, loadZone) {
     newZoneChecks(server);
 }
 
-TEST_F(AuthConmmandTest, loadBrokenZone) {
+TEST_F(AuthCommandTest, loadBrokenZone) {
     configureZones(server);
 
     ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR
@@ -195,7 +194,7 @@ TEST_F(AuthConmmandTest, loadBrokenZone) {
     zoneChecks(server);     // zone shouldn't be replaced
 }
 
-TEST_F(AuthConmmandTest, loadUnreadableZone) {
+TEST_F(AuthCommandTest, loadUnreadableZone) {
     configureZones(server);
 
     // install the zone file as unreadable
@@ -209,7 +208,7 @@ TEST_F(AuthConmmandTest, loadUnreadableZone) {
     zoneChecks(server);     // zone shouldn't be replaced
 }
 
-TEST_F(AuthConmmandTest, loadZoneWithoutDataSrc) {
+TEST_F(AuthCommandTest, loadZoneWithoutDataSrc) {
     // try to execute load command without configuring the zone beforehand.
     // it should fail.
     result = execAuthServerCommand(server, "loadzone",
@@ -218,7 +217,7 @@ TEST_F(AuthConmmandTest, loadZoneWithoutDataSrc) {
     checkAnswer(1);
 }
 
-TEST_F(AuthConmmandTest, loadSqlite3DataSrc) {
+TEST_F(AuthCommandTest, loadSqlite3DataSrc) {
     // For sqlite3 data source we don't have to do anything (the data source
     // (re)loads itself automatically)
     result = execAuthServerCommand(server, "loadzone",
@@ -228,7 +227,7 @@ TEST_F(AuthConmmandTest, loadSqlite3DataSrc) {
     checkAnswer(0);
 }
 
-TEST_F(AuthConmmandTest, loadZoneInvalidParams) {
+TEST_F(AuthCommandTest, loadZoneInvalidParams) {
     configureZones(server);
 
     // null arg

src/bin/auth/tests/config_unittest.cc

@@ -57,12 +57,12 @@ protected:
 
 TEST_F(AuthConfigTest, datasourceConfig) {
     // By default, we don't have any in-memory data source.
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
     configureAuthServer(server, Element::fromJSON(
                             "{\"datasources\": [{\"type\": \"memory\"}]}"));
     // after successful configuration, we should have one (with empty zoneset).
-    ASSERT_NE(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
-    EXPECT_EQ(0, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    ASSERT_NE(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
+    EXPECT_EQ(0, server.getInMemoryClient(rrclass)->getZoneCount());
 }
 
 TEST_F(AuthConfigTest, databaseConfig) {
@@ -82,7 +82,7 @@ TEST_F(AuthConfigTest, versionConfig) {
 }
 
 TEST_F(AuthConfigTest, exceptionGuarantee) {
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
     // This configuration contains an invalid item, which will trigger
     // an exception.
     EXPECT_THROW(configureAuthServer(
@@ -92,7 +92,7 @@ TEST_F(AuthConfigTest, exceptionGuarantee) {
                          " \"no_such_config_var\": 1}")),
                  AuthConfigError);
     // The server state shouldn't change
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
 }
 
 TEST_F(AuthConfigTest, exceptionConversion) {
@@ -154,22 +154,22 @@ protected:
 TEST_F(MemoryDatasrcConfigTest, addZeroDataSrc) {
     parser->build(Element::fromJSON("[]"));
     parser->commit();
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
 }
 
 TEST_F(MemoryDatasrcConfigTest, addEmpty) {
     // By default, we don't have any in-memory data source.
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
     parser->build(Element::fromJSON("[{\"type\": \"memory\"}]"));
     parser->commit();
-    EXPECT_EQ(0, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(0, server.getInMemoryClient(rrclass)->getZoneCount());
 }
 
 TEST_F(MemoryDatasrcConfigTest, addZeroZone) {
     parser->build(Element::fromJSON("[{\"type\": \"memory\","
                                     "  \"zones\": []}]"));
     parser->commit();
-    EXPECT_EQ(0, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(0, server.getInMemoryClient(rrclass)->getZoneCount());
 }
 
 TEST_F(MemoryDatasrcConfigTest, addOneZone) {
@@ -179,10 +179,10 @@ TEST_F(MemoryDatasrcConfigTest, addOneZone) {
                       "               \"file\": \"" TEST_DATA_DIR
                       "/example.zone\"}]}]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(1, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
     // Check it actually loaded something
-    EXPECT_EQ(Zone::SUCCESS, server.getMemoryDataSrc(rrclass)->findZone(
-        Name("ns.example.com.")).zone->find(Name("ns.example.com."),
+    EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(rrclass)->findZone(
+        Name("ns.example.com.")).zone_finder->find(Name("ns.example.com."),
         RRType::A()).code);
 }
 
@@ -199,7 +199,7 @@ TEST_F(MemoryDatasrcConfigTest, addMultiZones) {
                       "               \"file\": \"" TEST_DATA_DIR
                       "/example.net.zone\"}]}]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(3, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(3, server.getInMemoryClient(rrclass)->getZoneCount());
 }
 
 TEST_F(MemoryDatasrcConfigTest, replace) {
@@ -209,9 +209,9 @@ TEST_F(MemoryDatasrcConfigTest, replace) {
                       "               \"file\": \"" TEST_DATA_DIR
                       "/example.zone\"}]}]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(1, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
     EXPECT_EQ(isc::datasrc::result::SUCCESS,
-              server.getMemoryDataSrc(rrclass)->findZone(
+              server.getInMemoryClient(rrclass)->findZone(
                   Name("example.com")).code);
 
     // create a new parser, and install a new set of configuration.  It
@@ -227,9 +227,9 @@ TEST_F(MemoryDatasrcConfigTest, replace) {
                       "               \"file\": \"" TEST_DATA_DIR
                       "/example.net.zone\"}]}]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(2, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(2, server.getInMemoryClient(rrclass)->getZoneCount());
     EXPECT_EQ(isc::datasrc::result::NOTFOUND,
-              server.getMemoryDataSrc(rrclass)->findZone(
+              server.getInMemoryClient(rrclass)->findZone(
                   Name("example.com")).code);
 }
 
@@ -241,9 +241,9 @@ TEST_F(MemoryDatasrcConfigTest, exception) {
                       "               \"file\": \"" TEST_DATA_DIR
                       "/example.zone\"}]}]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(1, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
     EXPECT_EQ(isc::datasrc::result::SUCCESS,
-              server.getMemoryDataSrc(rrclass)->findZone(
+              server.getInMemoryClient(rrclass)->findZone(
                   Name("example.com")).code);
 
     // create a new parser, and try to load something. It will throw,
@@ -262,9 +262,9 @@ TEST_F(MemoryDatasrcConfigTest, exception) {
     // commit it
 
     // The original should be untouched
-    EXPECT_EQ(1, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
     EXPECT_EQ(isc::datasrc::result::SUCCESS,
-              server.getMemoryDataSrc(rrclass)->findZone(
+              server.getInMemoryClient(rrclass)->findZone(
                   Name("example.com")).code);
 }
 
@@ -275,13 +275,13 @@ TEST_F(MemoryDatasrcConfigTest, remove) {
                       "               \"file\": \"" TEST_DATA_DIR
                       "/example.zone\"}]}]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(1, server.getMemoryDataSrc(rrclass)->getZoneCount());
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
 
     delete parser;
     parser = createAuthConfigParser(server, "datasources"); 
     EXPECT_NO_THROW(parser->build(Element::fromJSON("[]")));
     EXPECT_NO_THROW(parser->commit());
-    EXPECT_EQ(AuthSrv::MemoryDataSrcPtr(), server.getMemoryDataSrc(rrclass));
+    EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
 }
 
 TEST_F(MemoryDatasrcConfigTest, adDuplicateZones) {

src/bin/auth/tests/query_unittest.cc

@@ -93,9 +93,9 @@ const char* const other_zone_rrs =
     "mx.delegation.example.com. 3600 IN A 192.0.2.100\n";
 
 // This is a mock Zone class for testing.
-// It is a derived class of Zone for the convenient of tests.
+// It is a derived class of ZoneFinder for the convenient of tests.
 // Its find() method emulates the common behavior of protocol compliant
-// zone classes, but simplifies some minor cases and also supports broken
+// ZoneFinder classes, but simplifies some minor cases and also supports broken
 // behavior.
 // For simplicity, most names are assumed to be "in zone"; there's only
 // one zone cut at the point of name "delegation.example.com".
@@ -103,15 +103,16 @@ const char* const other_zone_rrs =
 // will result in DNAME.
 // This mock zone doesn't handle empty non terminal nodes (if we need to test
 // such cases find() should have specialized code for it).
-class MockZone : public Zone {
+class MockZoneFinder : public ZoneFinder {
 public:
-    MockZone() :
+    MockZoneFinder() :
         origin_(Name("example.com")),
         delegation_name_("delegation.example.com"),
         dname_name_("dname.example.com"),
         has_SOA_(true),
         has_apex_NS_(true),
-        rrclass_(RRClass::IN())
+        rrclass_(RRClass::IN()),
+        include_rrsig_anyway_(false)
     {
         stringstream zone_stream;
         zone_stream << soa_txt << zone_ns_txt << ns_addrs_txt <<
@@ -120,14 +121,14 @@ public:
             other_zone_rrs;
 
         masterLoad(zone_stream, origin_, rrclass_,
-                   boost::bind(&MockZone::loadRRset, this, _1));
+                   boost::bind(&MockZoneFinder::loadRRset, this, _1));
     }
-    virtual const isc::dns::Name& getOrigin() const { return (origin_); }
-    virtual const isc::dns::RRClass& getClass() const { return (rrclass_); }
+    virtual isc::dns::Name getOrigin() const { return (origin_); }
+    virtual isc::dns::RRClass getClass() const { return (rrclass_); }
     virtual FindResult find(const isc::dns::Name& name,
                             const isc::dns::RRType& type,
                             RRsetList* target = NULL,
-                            const FindOptions options = FIND_DEFAULT) const;
+                            const FindOptions options = FIND_DEFAULT);
 
     // If false is passed, it makes the zone broken as if it didn't have the
     // SOA.
@@ -137,11 +138,18 @@ public:
     // the apex NS.
     void setApexNSFlag(bool on) { has_apex_NS_ = on; }
 
+    // Turn this on if you want it to return RRSIGs regardless of FIND_GLUE_OK
+    void setIncludeRRSIGAnyway(bool on) { include_rrsig_anyway_ = on; }
+
+    Name findPreviousName(const Name&) const {
+        isc_throw(isc::NotImplemented, "Mock doesn't support previous name");
+    }
+
 private:
     typedef map<RRType, ConstRRsetPtr> RRsetStore;
     typedef map<Name, RRsetStore> Domains;
     Domains domains_;
-    void loadRRset(ConstRRsetPtr rrset) {
+    void loadRRset(RRsetPtr rrset) {
         domains_[rrset->getName()][rrset->getType()] = rrset;
         if (rrset->getName() == delegation_name_ &&
             rrset->getType() == RRType::NS()) {
@@ -149,6 +157,26 @@ private:
         } else if (rrset->getName() == dname_name_ &&
             rrset->getType() == RRType::DNAME()) {
             dname_rrset_ = rrset;
+        // Add some signatures
+        } else if (rrset->getName() == Name("example.com.") &&
+                   rrset->getType() == RRType::NS()) {
+            rrset->addRRsig(RdataPtr(new generic::RRSIG("NS 5 3 3600 "
+                                                        "20000101000000 "
+                                                        "20000201000000 "
+                                                        "12345 example.com. "
+                                                        "FAKEFAKEFAKE")));
+        } else if (rrset->getType() == RRType::A()) {
+            rrset->addRRsig(RdataPtr(new generic::RRSIG("A 5 3 3600 "
+                                                        "20000101000000 "
+                                                        "20000201000000 "
+                                                        "12345 example.com. "
+                                                        "FAKEFAKEFAKE")));
+        } else if (rrset->getType() == RRType::AAAA()) {
+            rrset->addRRsig(RdataPtr(new generic::RRSIG("AAAA 5 3 3600 "
+                                                        "20000101000000 "
+                                                        "20000201000000 "
+                                                        "12345 example.com. "
+                                                        "FAKEFAKEFAKE")));
         }
     }
 
@@ -161,11 +189,12 @@ private:
     ConstRRsetPtr delegation_rrset_;
     ConstRRsetPtr dname_rrset_;
     const RRClass rrclass_;
+    bool include_rrsig_anyway_;
 };
 
-Zone::FindResult
-MockZone::find(const Name& name, const RRType& type,
-               RRsetList* target, const FindOptions options) const
+ZoneFinder::FindResult
+MockZoneFinder::find(const Name& name, const RRType& type,
+                     RRsetList* target, const FindOptions options)
 {
     // Emulating a broken zone: mandatory apex RRs are missing if specifically
     // configured so (which are rare cases).
@@ -195,7 +224,26 @@ MockZone::find(const Name& name, const RRType& type,
         RRsetStore::const_iterator found_rrset =
             found_domain->second.find(type);
         if (found_rrset != found_domain->second.end()) {
-            return (FindResult(SUCCESS, found_rrset->second));
+            ConstRRsetPtr rrset;
+            // Strip whatever signature there is in case DNSSEC is not required
+            // Just to make sure the Query asks for it when it is needed
+            if (options & ZoneFinder::FIND_DNSSEC ||
+                include_rrsig_anyway_ ||
+                !found_rrset->second->getRRsig()) {
+                rrset = found_rrset->second;
+            } else {
+                RRsetPtr noconst(new RRset(found_rrset->second->getName(),
+                                           found_rrset->second->getClass(),
+                                           found_rrset->second->getType(),
+                                           found_rrset->second->getTTL()));
+                for (RdataIteratorPtr
+                     i(found_rrset->second->getRdataIterator());
+                     !i->isLast(); i->next()) {
+                    noconst->addRdata(i->getCurrent());
+                }
+                rrset = noconst;
+            }
+            return (FindResult(SUCCESS, rrset));
         }
 
         // If not found but we have a target, fill it with all RRsets here
@@ -233,11 +281,15 @@ protected:
         response.setRcode(Rcode::NOERROR());
         response.setOpcode(Opcode::QUERY());
         // create and add a matching zone.
-        mock_zone = new MockZone();
-        memory_datasrc.addZone(ZonePtr(mock_zone));
+        mock_finder = new MockZoneFinder();
+        memory_client.addZone(ZoneFinderPtr(mock_finder));
     }
-    MockZone* mock_zone;
-    MemoryDataSrc memory_datasrc;
+    MockZoneFinder* mock_finder;
+    // We use InMemoryClient here. We could have some kind of mock client
+    // here, but historically, the Query supported only InMemoryClient
+    // (originally named MemoryDataSrc) and was tested with it, so we keep
+    // it like this for now.
+    InMemoryClient memory_client;
     const Name qname;
     const RRClass qclass;
     const RRType qtype;
@@ -286,24 +338,76 @@ responseCheck(Message& response, const isc::dns::Rcode& rcode,
 TEST_F(QueryTest, noZone) {
     // There's no zone in the memory datasource.  So the response should have
     // REFUSED.
-    MemoryDataSrc empty_memory_datasrc;
-    Query nozone_query(empty_memory_datasrc, qname, qtype, response);
+    InMemoryClient empty_memory_client;
+    Query nozone_query(empty_memory_client, qname, qtype, response);
     EXPECT_NO_THROW(nozone_query.process());
     EXPECT_EQ(Rcode::REFUSED(), response.getRcode());
 }
 
 TEST_F(QueryTest, exactMatch) {
-    Query query(memory_datasrc, qname, qtype, response);
+    Query query(memory_client, qname, qtype, response);
     EXPECT_NO_THROW(query.process());
     // find match rrset
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
                   www_a_txt, zone_ns_txt, ns_addrs_txt);
 }
 
+TEST_F(QueryTest, exactMatchIgnoreSIG) {
+    // Check that we do not include the RRSIG when not requested even when
+    // we receive it from the data source.
+    mock_finder->setIncludeRRSIGAnyway(true);
+    Query query(memory_client, qname, qtype, response);
+    EXPECT_NO_THROW(query.process());
+    // find match rrset
+    responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
+                  www_a_txt, zone_ns_txt, ns_addrs_txt);
+}
+
+TEST_F(QueryTest, dnssecPositive) {
+    // Just like exactMatch, but the signatures should be included as well
+    Query query(memory_client, qname, qtype, response, true);
+    EXPECT_NO_THROW(query.process());
+    // find match rrset
+    // We can't let responseCheck to check the additional section as well,
+    // it gets confused by the two RRs for glue.delegation.../RRSIG due
+    // to it's design and fixing it would be hard. Therefore we simply
+    // check manually this one time.
+    responseCheck(response, Rcode::NOERROR(), AA_FLAG, 2, 4, 6,
+                  (www_a_txt + std::string("www.example.com. 3600 IN RRSIG "
+                                           "A 5 3 3600 20000101000000 "
+                                           "20000201000000 12345 example.com. "
+                                           "FAKEFAKEFAKE\n")).c_str(),
+                  (zone_ns_txt + std::string("example.com. 3600 IN RRSIG NS 5 "
+                                             "3 3600 20000101000000 "
+                                             "20000201000000 12345 "
+                                             "example.com. FAKEFAKEFAKE\n")).
+                  c_str(), NULL);
+    RRsetIterator iterator(response.beginSection(Message::SECTION_ADDITIONAL));
+    const char* additional[] = {
+        "glue.delegation.example.com. 3600 IN A 192.0.2.153\n",
+        "glue.delegation.example.com. 3600 IN RRSIG A 5 3 3600 20000101000000 "
+            "20000201000000 12345 example.com. FAKEFAKEFAKE\n",
+        "glue.delegation.example.com. 3600 IN AAAA 2001:db8::53\n",
+        "glue.delegation.example.com. 3600 IN RRSIG AAAA 5 3 3600 "
+            "20000101000000 20000201000000 12345 example.com. FAKEFAKEFAKE\n",
+        "noglue.example.com. 3600 IN A 192.0.2.53\n",
+        "noglue.example.com. 3600 IN RRSIG A 5 3 3600 20000101000000 "
+            "20000201000000 12345 example.com. FAKEFAKEFAKE\n",
+        NULL
+    };
+    for (const char** rr(additional); *rr != NULL; ++ rr) {
+        ASSERT_FALSE(iterator ==
+                     response.endSection(Message::SECTION_ADDITIONAL));
+        EXPECT_EQ(*rr, (*iterator)->toText());
+        iterator ++;
+    }
+    EXPECT_TRUE(iterator == response.endSection(Message::SECTION_ADDITIONAL));
+}
+
 TEST_F(QueryTest, exactAddrMatch) {
     // find match rrset, omit additional data which has already been provided
     // in the answer section from the additional.
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("noglue.example.com"), qtype,
+    EXPECT_NO_THROW(Query(memory_client, Name("noglue.example.com"), qtype,
                           response).process());
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 2,
@@ -315,7 +419,7 @@ TEST_F(QueryTest, exactAddrMatch) {
 TEST_F(QueryTest, apexNSMatch) {
     // find match rrset, omit authority data which has already been provided
     // in the answer section from the authority section.
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("example.com"), RRType::NS(),
+    EXPECT_NO_THROW(Query(memory_client, Name("example.com"), RRType::NS(),
                           response).process());
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 3, 0, 3,
@@ -326,7 +430,7 @@ TEST_F(QueryTest, apexNSMatch) {
 TEST_F(QueryTest, exactAnyMatch) {
     // find match rrset, omit additional data which has already been provided
     // in the answer section from the additional.
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("noglue.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("noglue.example.com"),
                           RRType::ANY(), response).process());
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 2,
@@ -339,18 +443,18 @@ TEST_F(QueryTest, exactAnyMatch) {
 TEST_F(QueryTest, apexAnyMatch) {
     // find match rrset, omit additional data which has already been provided
     // in the answer section from the additional.
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("example.com"),
                           RRType::ANY(), response).process());
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 4, 0, 3,
                   "example.com. 3600 IN SOA . . 0 0 0 0 0\n"
                   "example.com. 3600 IN NS glue.delegation.example.com.\n"
                   "example.com. 3600 IN NS noglue.example.com.\n"
                   "example.com. 3600 IN NS example.net.\n",
-                  NULL, ns_addrs_txt, mock_zone->getOrigin());
+                  NULL, ns_addrs_txt, mock_finder->getOrigin());
 }
 
 TEST_F(QueryTest, mxANYMatch) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("mx.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("mx.example.com"),
                           RRType::ANY(), response).process());
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 3, 3, 4,
                   mx_txt, zone_ns_txt,
@@ -358,17 +462,17 @@ TEST_F(QueryTest, mxANYMatch) {
 }
 
 TEST_F(QueryTest, glueANYMatch) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("delegation.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("delegation.example.com"),
                           RRType::ANY(), response).process());
     responseCheck(response, Rcode::NOERROR(), 0, 0, 4, 3,
                   NULL, delegation_txt, ns_addrs_txt);
 }
 
 TEST_F(QueryTest, nodomainANY) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("nxdomain.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("nxdomain.example.com"),
                           RRType::ANY(), response).process());
     responseCheck(response, Rcode::NXDOMAIN(), AA_FLAG, 0, 1, 0,
-                  NULL, soa_txt, NULL, mock_zone->getOrigin());
+                  NULL, soa_txt, NULL, mock_finder->getOrigin());
 }
 
 // This tests that when we need to look up Zone's apex NS records for
@@ -376,15 +480,15 @@ TEST_F(QueryTest, nodomainANY) {
 // throw in that case.
 TEST_F(QueryTest, noApexNS) {
     // Disable apex NS record
-    mock_zone->setApexNSFlag(false);
+    mock_finder->setApexNSFlag(false);
 
-    EXPECT_THROW(Query(memory_datasrc, Name("noglue.example.com"), qtype,
+    EXPECT_THROW(Query(memory_client, Name("noglue.example.com"), qtype,
                        response).process(), Query::NoApexNS);
     // We don't look into the response, as it threw
 }
 
 TEST_F(QueryTest, delegation) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("delegation.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("delegation.example.com"),
                           qtype, response).process());
 
     responseCheck(response, Rcode::NOERROR(), 0, 0, 4, 3,
@@ -392,18 +496,18 @@ TEST_F(QueryTest, delegation) {
 }
 
 TEST_F(QueryTest, nxdomain) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("nxdomain.example.com"), qtype,
+    EXPECT_NO_THROW(Query(memory_client, Name("nxdomain.example.com"), qtype,
                           response).process());
     responseCheck(response, Rcode::NXDOMAIN(), AA_FLAG, 0, 1, 0,
-                  NULL, soa_txt, NULL, mock_zone->getOrigin());
+                  NULL, soa_txt, NULL, mock_finder->getOrigin());
 }
 
 TEST_F(QueryTest, nxrrset) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("www.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("www.example.com"),
                           RRType::TXT(), response).process());
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 0, 1, 0,
-                  NULL, soa_txt, NULL, mock_zone->getOrigin());
+                  NULL, soa_txt, NULL, mock_finder->getOrigin());
 }
 
 /*
@@ -412,22 +516,22 @@ TEST_F(QueryTest, nxrrset) {
  */
 TEST_F(QueryTest, noSOA) {
     // disable zone's SOA RR.
-    mock_zone->setSOAFlag(false);
+    mock_finder->setSOAFlag(false);
 
     // The NX Domain
-    EXPECT_THROW(Query(memory_datasrc, Name("nxdomain.example.com"),
+    EXPECT_THROW(Query(memory_client, Name("nxdomain.example.com"),
                        qtype, response).process(), Query::NoSOA);
     // Of course, we don't look into the response, as it throwed
 
     // NXRRSET
-    EXPECT_THROW(Query(memory_datasrc, Name("nxrrset.example.com"),
+    EXPECT_THROW(Query(memory_client, Name("nxrrset.example.com"),
                        qtype, response).process(), Query::NoSOA);
 }
 
 TEST_F(QueryTest, noMatchZone) {
     // there's a zone in the memory datasource but it doesn't match the qname.
     // should result in REFUSED.
-    Query(memory_datasrc, Name("example.org"), qtype, response).process();
+    Query(memory_client, Name("example.org"), qtype, response).process();
     EXPECT_EQ(Rcode::REFUSED(), response.getRcode());
 }
 
@@ -438,7 +542,7 @@ TEST_F(QueryTest, noMatchZone) {
  * A record, other to unknown out of zone one.
  */
 TEST_F(QueryTest, MX) {
-    Query(memory_datasrc, Name("mx.example.com"), RRType::MX(),
+    Query(memory_client, Name("mx.example.com"), RRType::MX(),
           response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 3, 3, 4,
@@ -452,7 +556,7 @@ TEST_F(QueryTest, MX) {
  * This should not trigger the additional processing for the exchange.
  */
 TEST_F(QueryTest, MXAlias) {
-    Query(memory_datasrc, Name("cnamemx.example.com"), RRType::MX(),
+    Query(memory_client, Name("cnamemx.example.com"), RRType::MX(),
           response).process();
 
     // there shouldn't be no additional RRs for the exchanges (we have 3
@@ -472,7 +576,7 @@ TEST_F(QueryTest, MXAlias) {
  * returned.
  */
 TEST_F(QueryTest, CNAME) {
-    Query(memory_datasrc, Name("cname.example.com"), RRType::A(),
+    Query(memory_client, Name("cname.example.com"), RRType::A(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 0, 0,
@@ -482,7 +586,7 @@ TEST_F(QueryTest, CNAME) {
 TEST_F(QueryTest, explicitCNAME) {
     // same owner name as the CNAME test but explicitly query for CNAME RR.
     // expect the same response as we don't provide a full chain yet.
-    Query(memory_datasrc, Name("cname.example.com"), RRType::CNAME(),
+    Query(memory_client, Name("cname.example.com"), RRType::CNAME(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
@@ -494,7 +598,7 @@ TEST_F(QueryTest, CNAME_NX_RRSET) {
     // note: with chaining, what should be expected is not trivial:
     // BIND 9 returns the CNAME in answer and SOA in authority, no additional.
     // NSD returns the CNAME, NS in authority, A/AAAA for NS in additional.
-    Query(memory_datasrc, Name("cname.example.com"), RRType::TXT(),
+    Query(memory_client, Name("cname.example.com"), RRType::TXT(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 0, 0,
@@ -503,7 +607,7 @@ TEST_F(QueryTest, CNAME_NX_RRSET) {
 
 TEST_F(QueryTest, explicitCNAME_NX_RRSET) {
     // same owner name as the NXRRSET test but explicitly query for CNAME RR.
-    Query(memory_datasrc, Name("cname.example.com"), RRType::CNAME(),
+    Query(memory_client, Name("cname.example.com"), RRType::CNAME(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
@@ -517,7 +621,7 @@ TEST_F(QueryTest, CNAME_NX_DOMAIN) {
     // RCODE being NXDOMAIN.
     // NSD returns the CNAME, NS in authority, A/AAAA for NS in additional,
     // RCODE being NOERROR.
-    Query(memory_datasrc, Name("cnamenxdom.example.com"), RRType::A(),
+    Query(memory_client, Name("cnamenxdom.example.com"), RRType::A(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 0, 0,
@@ -526,7 +630,7 @@ TEST_F(QueryTest, CNAME_NX_DOMAIN) {
 
 TEST_F(QueryTest, explicitCNAME_NX_DOMAIN) {
     // same owner name as the NXDOMAIN test but explicitly query for CNAME RR.
-    Query(memory_datasrc, Name("cnamenxdom.example.com"), RRType::CNAME(),
+    Query(memory_client, Name("cnamenxdom.example.com"), RRType::CNAME(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
@@ -542,7 +646,7 @@ TEST_F(QueryTest, CNAME_OUT) {
      * Then the same test should be done with .org included there and
      * see what it does (depends on what we want to do)
      */
-    Query(memory_datasrc, Name("cnameout.example.com"), RRType::A(),
+    Query(memory_client, Name("cnameout.example.com"), RRType::A(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 0, 0,
@@ -551,7 +655,7 @@ TEST_F(QueryTest, CNAME_OUT) {
 
 TEST_F(QueryTest, explicitCNAME_OUT) {
     // same owner name as the OUT test but explicitly query for CNAME RR.
-    Query(memory_datasrc, Name("cnameout.example.com"), RRType::CNAME(),
+    Query(memory_client, Name("cnameout.example.com"), RRType::CNAME(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
@@ -567,7 +671,7 @@ TEST_F(QueryTest, explicitCNAME_OUT) {
  * pointing to NXRRSET and NXDOMAIN cases (similarly as with CNAME).
  */
 TEST_F(QueryTest, DNAME) {
-    Query(memory_datasrc, Name("www.dname.example.com"), RRType::A(),
+    Query(memory_client, Name("www.dname.example.com"), RRType::A(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 2, 0, 0,
@@ -583,7 +687,7 @@ TEST_F(QueryTest, DNAME) {
  * DNAME.
  */
 TEST_F(QueryTest, DNAME_ANY) {
-    Query(memory_datasrc, Name("www.dname.example.com"), RRType::ANY(),
+    Query(memory_client, Name("www.dname.example.com"), RRType::ANY(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 2, 0, 0,
@@ -592,7 +696,7 @@ TEST_F(QueryTest, DNAME_ANY) {
 
 // Test when we ask for DNAME explicitly, it does no synthetizing.
 TEST_F(QueryTest, explicitDNAME) {
-    Query(memory_datasrc, Name("dname.example.com"), RRType::DNAME(),
+    Query(memory_client, Name("dname.example.com"), RRType::DNAME(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
@@ -604,7 +708,7 @@ TEST_F(QueryTest, explicitDNAME) {
  * the CNAME, it should return the RRset.
  */
 TEST_F(QueryTest, DNAME_A) {
-    Query(memory_datasrc, Name("dname.example.com"), RRType::A(),
+    Query(memory_client, Name("dname.example.com"), RRType::A(),
         response).process();
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 1, 3, 3,
@@ -616,11 +720,11 @@ TEST_F(QueryTest, DNAME_A) {
  * It should not synthetize the CNAME.
  */
 TEST_F(QueryTest, DNAME_NX_RRSET) {
-    EXPECT_NO_THROW(Query(memory_datasrc, Name("dname.example.com"),
+    EXPECT_NO_THROW(Query(memory_client, Name("dname.example.com"),
         RRType::TXT(), response).process());
 
     responseCheck(response, Rcode::NOERROR(), AA_FLAG, 0, 1, 0,
-        NULL, soa_txt, NULL, mock_zone->getOrigin());
+        NULL, soa_txt, NULL, mock_finder->getOrigin());
 }
 
 /*
@@ -636,7 +740,7 @@ TEST_F(QueryTest, LongDNAME) {
         "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa."
         "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa."
         "dname.example.com.");
-    EXPECT_NO_THROW(Query(memory_datasrc, longname, RRType::A(),
+    EXPECT_NO_THROW(Query(memory_client, longname, RRType::A(),
         response).process());
 
     responseCheck(response, Rcode::YXDOMAIN(), AA_FLAG, 1, 0, 0,
@@ -655,7 +759,7 @@ TEST_F(QueryTest, MaxLenDNAME) {
         "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa."
         "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa."
         "dname.example.com.");
-    EXPECT_NO_THROW(Query(memory_datasrc, longname, RRType::A(),
+    EXPECT_NO_THROW(Query(memory_client, longname, RRType::A(),
         response).process());
 
     // Check the answer is OK

src/bin/auth/tests/statistics_unittest.cc

@@ -16,6 +16,8 @@
 
 #include <gtest/gtest.h>
 
+#include <boost/bind.hpp>
+
 #include <cc/data.h>
 #include <cc/session.h>
 
@@ -76,6 +78,13 @@ protected:
     }
     MockSession statistics_session_;
     AuthCounters counters;
+    // no need to be inherited from the original class here.
+    class MockModuleSpec {
+    public:
+        bool validateStatistics(ConstElementPtr, const bool valid) const
+            { return (valid); }
+    };
+    MockModuleSpec module_spec_;
 };
 
 void
@@ -181,7 +190,7 @@ TEST_F(AuthCountersTest, submitStatisticsWithException) {
     statistics_session_.setThrowSessionTimeout(false);
 }
 
-TEST_F(AuthCountersTest, submitStatistics) {
+TEST_F(AuthCountersTest, submitStatisticsWithoutValidator) {
     // Submit statistics data.
     // Validate if it submits correct data.
 
@@ -201,12 +210,69 @@ TEST_F(AuthCountersTest, submitStatistics) {
     // Command is "set".
     EXPECT_EQ("set", statistics_session_.sent_msg->get("command")
                          ->get(0)->stringValue());
+    EXPECT_EQ("Auth", statistics_session_.sent_msg->get("command")
+                         ->get(1)->get("owner")->stringValue());
     ConstElementPtr statistics_data = statistics_session_.sent_msg
                                           ->get("command")->get(1)
-                                          ->get("stats_data");
+                                          ->get("data");
     // UDP query counter is 2 and TCP query counter is 1.
-    EXPECT_EQ(2, statistics_data->get("auth.queries.udp")->intValue());
-    EXPECT_EQ(1, statistics_data->get("auth.queries.tcp")->intValue());
+    EXPECT_EQ(2, statistics_data->get("queries.udp")->intValue());
+    EXPECT_EQ(1, statistics_data->get("queries.tcp")->intValue());
 }
 
+TEST_F(AuthCountersTest, submitStatisticsWithValidator) {
+
+    //a validator for the unittest
+    AuthCounters::validator_type validator;
+    ConstElementPtr el;
+
+    // Submit statistics data with correct statistics validator.
+    validator = boost::bind(
+        &AuthCountersTest::MockModuleSpec::validateStatistics,
+        &module_spec_, _1, true);
+
+    EXPECT_TRUE(validator(el));
+
+    // register validator to AuthCounters
+    counters.registerStatisticsValidator(validator);
+
+    // Counters should be initialized to 0.
+    EXPECT_EQ(0, counters.getCounter(AuthCounters::COUNTER_UDP_QUERY));
+    EXPECT_EQ(0, counters.getCounter(AuthCounters::COUNTER_TCP_QUERY));
+
+    // UDP query counter is set to 2.
+    counters.inc(AuthCounters::COUNTER_UDP_QUERY);
+    counters.inc(AuthCounters::COUNTER_UDP_QUERY);
+    // TCP query counter is set to 1.
+    counters.inc(AuthCounters::COUNTER_TCP_QUERY);
+
+    // checks the value returned by submitStatistics
+    EXPECT_TRUE(counters.submitStatistics());
+
+    // Destination is "Stats".
+    EXPECT_EQ("Stats", statistics_session_.msg_destination);
+    // Command is "set".
+    EXPECT_EQ("set", statistics_session_.sent_msg->get("command")
+                         ->get(0)->stringValue());
+    EXPECT_EQ("Auth", statistics_session_.sent_msg->get("command")
+                         ->get(1)->get("owner")->stringValue());
+    ConstElementPtr statistics_data = statistics_session_.sent_msg
+                                          ->get("command")->get(1)
+                                          ->get("data");
+    // UDP query counter is 2 and TCP query counter is 1.
+    EXPECT_EQ(2, statistics_data->get("queries.udp")->intValue());
+    EXPECT_EQ(1, statistics_data->get("queries.tcp")->intValue());
+
+    // Submit statistics data with incorrect statistics validator.
+    validator = boost::bind(
+        &AuthCountersTest::MockModuleSpec::validateStatistics,
+        &module_spec_, _1, false);
+
+    EXPECT_FALSE(validator(el));
+
+    counters.registerStatisticsValidator(validator);
+
+    // checks the value returned by submitStatistics
+    EXPECT_FALSE(counters.submitStatistics());
+}
 }

src/bin/auth/tests/testdata/Makefile.am

@@ -23,4 +23,4 @@ EXTRA_DIST += example.com
 EXTRA_DIST += example.sqlite3
 
 .spec.wire:
-	$(abs_top_builddir)/src/lib/dns/tests/testdata/gen-wiredata.py -o $@ $<
+	$(PYTHON) $(top_builddir)/src/lib/util/python/gen_wiredata.py -o $@ $<

src/bin/bind10/Makefile.am

@@ -1,16 +1,23 @@
 SUBDIRS = . tests
 
 sbin_SCRIPTS = bind10
-CLEANFILES = bind10 bind10.pyc
+CLEANFILES = bind10 bind10_src.pyc
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/bind10_messages.py
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/bind10_messages.pyc
 
 pkglibexecdir = $(libexecdir)/@PACKAGE@
 
+nodist_pylogmessage_PYTHON = $(PYTHON_LOGMSGPKG_DIR)/work/bind10_messages.py
+pylogmessagedir = $(pyexecdir)/isc/log_messages/
+
+noinst_SCRIPTS = run_bind10.sh
+
 bind10dir = $(pkgdatadir)
 bind10_DATA = bob.spec
 EXTRA_DIST = bob.spec
 
 man_MANS = bind10.8
-EXTRA_DIST += $(man_MANS) bind10.xml
+EXTRA_DIST += $(man_MANS) bind10.xml bind10_messages.mes
 
 if ENABLE_MAN
 
@@ -19,10 +26,14 @@ bind10.8: bind10.xml
 
 endif
 
+$(PYTHON_LOGMSGPKG_DIR)/work/bind10_messages.py : bind10_messages.mes
+	$(top_builddir)/src/lib/log/compiler/message \
+	-d $(PYTHON_LOGMSGPKG_DIR)/work -p $(srcdir)/bind10_messages.mes
+
 # this is done here since configure.ac AC_OUTPUT doesn't expand exec_prefix
-bind10: bind10.py
+bind10: bind10_src.py $(PYTHON_LOGMSGPKG_DIR)/work/bind10_messages.py
 	$(SED) -e "s|@@PYTHONPATH@@|@pyexecdir@|" \
-	       -e "s|@@LIBEXECDIR@@|$(pkglibexecdir)|" bind10.py >$@
+	       -e "s|@@LIBEXECDIR@@|$(pkglibexecdir)|" bind10_src.py >$@
 	chmod a+x $@
 
 pytest:

src/bin/bind10/bind10.8

@@ -2,12 +2,12 @@
 .\"     Title: bind10
 .\"    Author: [see the "AUTHORS" section]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: March 31, 2011
+.\"      Date: August 11, 2011
 .\"    Manual: BIND10
 .\"    Source: BIND10
 .\"  Language: English
 .\"
-.TH "BIND10" "8" "March 31, 2011" "BIND10" "BIND10"
+.TH "BIND10" "8" "August 11, 2011" "BIND10" "BIND10"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -107,6 +107,18 @@ Display more about what is going on for
 \fBbind10\fR
 and its child processes\&.
 .RE
+.SH "STATISTICS DATA"
+.PP
+The statistics data collected by the
+\fBb10\-stats\fR
+daemon include:
+.PP
+bind10\&.boot_time
+.RS 4
+The date and time that the
+\fBbind10\fR
+process started\&. This is represented in ISO 8601 format\&.
+.RE
 .SH "SEE ALSO"
 .PP
 

src/bin/bind10/bind10.xml

@@ -2,7 +2,7 @@
                "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
 	       [<!ENTITY mdash "&#8212;">]>
 <!--
- - Copyright (C) 2010  Internet Systems Consortium, Inc. ("ISC")
+ - Copyright (C) 2010-2011  Internet Systems Consortium, Inc. ("ISC")
  -
  - Permission to use, copy, modify, and/or distribute this software for any
  - purpose with or without fee is hereby granted, provided that the above
@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>March 31, 2011</date>
+    <date>August 11, 2011</date>
   </refentryinfo>
 
   <refmeta>
@@ -217,6 +217,30 @@ The default is the basename of ARG 0.
 <!--
 TODO: configuration section
 -->
+
+  <refsect1>
+    <title>STATISTICS DATA</title>
+
+    <para>
+      The statistics data collected by the <command>b10-stats</command>
+      daemon include:
+    </para>
+
+    <variablelist>
+
+      <varlistentry>
+        <term>bind10.boot_time</term>
+        <listitem><para>
+          The date and time that the <command>bind10</command>
+          process started.
+          This is represented in ISO 8601 format.
+        </para></listitem>
+      </varlistentry>
+
+    </variablelist>
+
+  </refsect1>
+
 <!--
   <refsect1>
     <title>FILES</title>

src/bin/bind10/bind10_messages.mes

@@ -0,0 +1,204 @@
+# Copyright (C) 2011  Internet Systems Consortium, Inc. ("ISC")
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
+# No namespace declaration - these constants go in the global namespace
+# of the xfrin messages python module.
+
+% BIND10_CHECK_MSGQ_ALREADY_RUNNING checking if msgq is already running
+The boss process is starting up and will now check if the message bus
+daemon is already running. If so, it will not be able to start, as it
+needs a dedicated message bus.
+
+% BIND10_CONFIGURATION_START_AUTH start authoritative server: %1
+This message shows whether or not the authoritative server should be
+started according to the configuration.
+
+% BIND10_CONFIGURATION_START_RESOLVER start resolver: %1
+This message shows whether or not the resolver should be
+started according to the configuration.
+
+% BIND10_INVALID_USER invalid user: %1
+The boss process was started with the -u option, to drop root privileges
+and continue running as the specified user, but the user is unknown.
+
+% BIND10_KILLING_ALL_PROCESSES killing all started processes
+The boss module was not able to start every process it needed to start
+during startup, and will now kill the processes that did get started.
+
+% BIND10_KILL_PROCESS killing process %1
+The boss module is sending a kill signal to process with the given name,
+as part of the process of killing all started processes during a failed
+startup, as described for BIND10_KILLING_ALL_PROCESSES
+
+% BIND10_MSGQ_ALREADY_RUNNING msgq daemon already running, cannot start
+There already appears to be a message bus daemon running. Either an
+old process was not shut down correctly, and needs to be killed, or
+another instance of BIND10, with the same msgq domain socket, is
+running, which needs to be stopped.
+
+% BIND10_MSGQ_DAEMON_ENDED b10-msgq process died, shutting down
+The message bus daemon has died. This is a fatal error, since it may
+leave the system in an inconsistent state. BIND10 will now shut down.
+
+% BIND10_MSGQ_DISAPPEARED msgq channel disappeared
+While listening on the message bus channel for messages, it suddenly
+disappeared. The msgq daemon may have died. This might lead to an
+inconsistent state of the system, and BIND 10 will now shut down.
+
+% BIND10_PROCESS_ENDED_NO_EXIT_STATUS process %1 (PID %2) died: exit status not available
+The given process ended unexpectedly, but no exit status is
+available. See BIND10_PROCESS_ENDED_WITH_EXIT_STATUS for a longer
+description.
+
+% BIND10_PROCESS_ENDED_WITH_EXIT_STATUS process %1 (PID %2) terminated, exit status = %3
+The given process ended unexpectedly with the given exit status.
+Depending on which module it was, it may simply be restarted, or it
+may be a problem that will cause the boss module to shut down too.
+The latter happens if it was the message bus daemon, which, if it has
+died suddenly, may leave the system in an inconsistent state. BIND10
+will also shut down now if it has been run with --brittle.
+
+% BIND10_READING_BOSS_CONFIGURATION reading boss configuration
+The boss process is starting up, and will now process the initial
+configuration, as received from the configuration manager.
+
+% BIND10_RECEIVED_COMMAND received command: %1
+The boss module received a command and shall now process it. The command
+is printed.
+
+% BIND10_RECEIVED_NEW_CONFIGURATION received new configuration: %1
+The boss module received a configuration update and is going to apply
+it now. The new configuration is printed.
+
+% BIND10_RECEIVED_SIGNAL received signal %1
+The boss module received the given signal.
+
+% BIND10_RESURRECTED_PROCESS resurrected %1 (PID %2)
+The given process has been restarted successfully, and is now running
+with the given process id.
+
+% BIND10_RESURRECTING_PROCESS resurrecting dead %1 process...
+The given process has ended unexpectedly, and is now restarted.
+
+% BIND10_SELECT_ERROR error in select() call: %1
+There was a fatal error in the call to select(), used to see if a child
+process has ended or if there is a message on the message bus. This
+should not happen under normal circumstances and is considered fatal,
+so BIND 10 will now shut down. The specific error is printed.
+
+% BIND10_SEND_SIGKILL sending SIGKILL to %1 (PID %2)
+The boss module is sending a SIGKILL signal to the given process.
+
+% BIND10_SEND_SIGTERM sending SIGTERM to %1 (PID %2)
+The boss module is sending a SIGTERM signal to the given process.
+
+% BIND10_SHUTDOWN stopping the server
+The boss process received a command or signal telling it to shut down.
+It will send a shutdown command to each process. The processes that do
+not shut down will then receive a SIGTERM signal. If that doesn't work,
+it shall send SIGKILL signals to the processes still alive.
+
+% BIND10_SHUTDOWN_COMPLETE all processes ended, shutdown complete
+All child processes have been stopped, and the boss process will now
+stop itself.
+
+% BIND10_SOCKCREATOR_BAD_CAUSE unknown error cause from socket creator: %1
+The socket creator reported an error when creating a socket. But the function
+which failed is unknown (not one of 'S' for socket or 'B' for bind).
+
+% BIND10_SOCKCREATOR_BAD_RESPONSE unknown response for socket request: %1
+The boss requested a socket from the creator, but the answer is unknown. This
+looks like a programmer error.
+
+% BIND10_SOCKCREATOR_CRASHED the socket creator crashed
+The socket creator terminated unexpectedly. It is not possible to restart it
+(because the boss already gave up root privileges), so the system is going
+to terminate.
+
+% BIND10_SOCKCREATOR_EOF eof while expecting data from socket creator
+There should be more data from the socket creator, but it closed the socket.
+It probably crashed.
+
+% BIND10_SOCKCREATOR_INIT initializing socket creator parser
+The boss module initializes routines for parsing the socket creator
+protocol.
+
+% BIND10_SOCKCREATOR_KILL killing the socket creator
+The socket creator is being terminated the aggressive way, by sending it
+sigkill. This should not happen usually.
+
+% BIND10_SOCKCREATOR_TERMINATE terminating socket creator
+The boss module sends a request to terminate to the socket creator.
+
+% BIND10_SOCKCREATOR_TRANSPORT_ERROR transport error when talking to the socket creator: %1
+Either sending or receiving data from the socket creator failed with the given
+error. The creator probably crashed or some serious OS-level problem happened,
+as the communication happens only on local host.
+
+% BIND10_SOCKET_CREATED successfully created socket %1
+The socket creator successfully created and sent a requested socket, it has
+the given file number.
+
+% BIND10_SOCKET_ERROR error on %1 call in the creator: %2/%3
+The socket creator failed to create the requested socket. It failed on the
+indicated OS API function with given error.
+
+% BIND10_SOCKET_GET requesting socket [%1]:%2 of type %3 from the creator
+The boss forwards a request for a socket to the socket creator.
+
+% BIND10_STARTED_PROCESS started %1
+The given process has successfully been started.
+
+% BIND10_STARTED_PROCESS_PID started %1 (PID %2)
+The given process has successfully been started, and has the given PID.
+
+% BIND10_STARTING starting BIND10: %1
+Informational message on startup that shows the full version.
+
+% BIND10_STARTING_PROCESS starting process %1
+The boss module is starting the given process.
+
+% BIND10_STARTING_PROCESS_PORT starting process %1 (to listen on port %2)
+The boss module is starting the given process, which will listen on the
+given port number.
+
+% BIND10_STARTING_PROCESS_PORT_ADDRESS starting process %1 (to listen on %2#%3)
+The boss module is starting the given process, which will listen on the
+given address and port number (written as <address>#<port>).
+
+% BIND10_STARTUP_COMPLETE BIND 10 started
+All modules have been successfully started, and BIND 10 is now running.
+
+% BIND10_STARTUP_ERROR error during startup: %1
+There was a fatal error when BIND10 was trying to start. The error is
+shown, and BIND10 will now shut down.
+
+% BIND10_START_AS_NON_ROOT starting %1 as a user, not root. This might fail.
+The given module is being started or restarted without root privileges.
+If the module needs these privileges, it may have problems starting.
+Note that this issue should be resolved by the pending 'socket-creator'
+process; once that has been implemented, modules should not need root
+privileges anymore. See tickets #800 and #801 for more information.
+
+% BIND10_STOP_PROCESS asking %1 to shut down
+The boss module is sending a shutdown command to the given module over
+the message channel.
+
+% BIND10_UNKNOWN_CHILD_PROCESS_ENDED unknown child pid %1 exited
+An unknown child process has exited. The PID is printed, but no further
+action will be taken by the boss process.
+
+% BIND10_INVALID_STATISTICS_DATA invalid specification of statistics data specified
+An error was encountered when the boss module specified
+statistics data which is invalid for the boss specification file.

src/bin/bind10/bind10.py.in

@@ -65,6 +65,17 @@ import posix
 import isc.cc
 import isc.util.process
 import isc.net.parse
+import isc.log
+from isc.log_messages.bind10_messages import *
+import isc.bind10.sockcreator
+
+isc.log.init("b10-boss")
+logger = isc.log.Logger("boss")
+
+# Pending system-wide debug level definitions, the ones we
+# use here are hardcoded for now
+DBG_PROCESS = 10
+DBG_COMMANDS = 30
 
 # Assign this process some longer name
 isc.util.process.rename(sys.argv[0])
@@ -74,7 +85,7 @@ isc.util.process.rename(sys.argv[0])
 # number, and the overall BIND 10 version number (set in configure.ac).
 VERSION = "bind10 20110223 (BIND 10 @PACKAGE_VERSION@)"
 
-# This is for bind10.boottime of stats module
+# This is for boot_time of Boss
 _BASETIME = time.gmtime()
 
 class RestartSchedule:
@@ -238,6 +249,7 @@ class BoB:
         self.config_filename = config_filename
         self.cmdctl_port = cmdctl_port
         self.brittle = brittle
+        self.sockcreator = None
 
     def config_handler(self, new_config):
         # If this is initial update, don't do anything now, leave it to startup
@@ -252,8 +264,7 @@ class BoB:
             if new_config['start_' + name]:
                 if not started:
                     if self.uid is not None:
-                        sys.stderr.write("[bind10] Starting " + name + " as " +
-                            "a user, not root. This might fail.\n")
+                        logger.info(BIND10_START_AS_NON_ROOT, name)
                     start()
             else:
                 stop()
@@ -279,9 +290,8 @@ class BoB:
             self.started_auth_family = False
 
         # The real code of the config handler function follows here
-        if self.verbose:
-            sys.stdout.write("[bind10] Handling new configuration: " +
-                str(new_config) + "\n")
+        logger.debug(DBG_COMMANDS, BIND10_RECEIVED_NEW_CONFIGURATION,
+                     new_config)
         start_stop('resolver', self.started_resolver_family, resolver_on,
             resolver_off)
         start_stop('auth', self.started_auth_family, auth_on, auth_off)
@@ -297,9 +307,15 @@ class BoB:
             process_list.append([pid, self.processes[pid].name])
         return process_list
 
+    def _get_stats_data(self):
+        return { "owner": "Boss",
+                 "data": { 'boot_time':
+                               time.strftime('%Y-%m-%dT%H:%M:%SZ', _BASETIME)
+                           }
+                 }
+
     def command_handler(self, command, args):
-        if self.verbose:
-            sys.stdout.write("[bind10] Boss got command: " + str(command) + "\n")
+        logger.debug(DBG_COMMANDS, BIND10_RECEIVED_COMMAND, command)
         answer = isc.config.ccsession.create_answer(1, "command not implemented")
         if type(command) != str:
             answer = isc.config.ccsession.create_answer(1, "bad command")
@@ -307,15 +323,26 @@ class BoB:
             if command == "shutdown":
                 self.runnable = False
                 answer = isc.config.ccsession.create_answer(0)
+            elif command == "getstats":
+                answer = isc.config.ccsession.create_answer(0, self._get_stats_data())
             elif command == "sendstats":
                 # send statistics data to the stats daemon immediately
-                cmd = isc.config.ccsession.create_command(
-                    'set', { "stats_data": {
-                            'bind10.boot_time': time.strftime('%Y-%m-%dT%H:%M:%SZ', _BASETIME)
-                            }})
-                seq = self.cc_session.group_sendmsg(cmd, 'Stats')
-                self.cc_session.group_recvmsg(True, seq)
-                answer = isc.config.ccsession.create_answer(0)
+                stats_data = self._get_stats_data()
+                valid = self.ccs.get_module_spec().validate_statistics(
+                    True, stats_data["data"])
+                if valid:
+                    cmd = isc.config.ccsession.create_command('set', stats_data)
+                    seq = self.cc_session.group_sendmsg(cmd, 'Stats')
+                    # Consume the answer, in case it becomes a orphan message.
+                    try:
+                        self.cc_session.group_recvmsg(False, seq)
+                    except isc.cc.session.SessionTimeout:
+                        pass
+                    answer = isc.config.ccsession.create_answer(0)
+                else:
+                    logger.fatal(BIND10_INVALID_STATISTICS_DATA);
+                    answer = isc.config.ccsession.create_answer(
+                        1, "specified statistics data is invalid")
             elif command == "ping":
                 answer = isc.config.ccsession.create_answer(0, "pong")
             elif command == "show_processes":
@@ -326,18 +353,32 @@ class BoB:
                                                             "Unknown command")
         return answer
 
+    def start_creator(self):
+        self.curproc = 'b10-sockcreator'
+        self.sockcreator = isc.bind10.sockcreator.Creator("@@LIBEXECDIR@@:" +
+                                                          os.environ['PATH'])
+
+    def stop_creator(self, kill=False):
+        if self.sockcreator is None:
+            return
+        if kill:
+            self.sockcreator.kill()
+        else:
+            self.sockcreator.terminate()
+        self.sockcreator = None
+
     def kill_started_processes(self):
         """
             Called as part of the exception handling when a process fails to
             start, this runs through the list of started processes, killing
             each one.  It then clears that list.
         """
-        if self.verbose:
-            sys.stdout.write("[bind10] killing started processes:\n")
+        logger.info(BIND10_KILLING_ALL_PROCESSES)
+
+        self.stop_creator(True)
 
         for pid in self.processes:
-            if self.verbose:
-                sys.stdout.write("[bind10] - %s\n" % self.processes[pid].name)
+            logger.info(BIND10_KILL_PROCESS, self.processes[pid].name)
             self.processes[pid].process.kill()
         self.processes = {}
 
@@ -351,23 +392,20 @@ class BoB:
             xfrin/xfrout and zone manager as we don't need to start those if we
             are not running the authoritative server.)
         """
-        if self.verbose:
-            sys.stdout.write("[bind10] Reading Boss configuration:\n")
+        logger.info(BIND10_READING_BOSS_CONFIGURATION)
 
         config_data = self.ccs.get_full_config()
         self.cfg_start_auth = config_data.get("start_auth")
         self.cfg_start_resolver = config_data.get("start_resolver")
 
-        if self.verbose:
-            sys.stdout.write("[bind10] - start_auth: %s\n" %
-                str(self.cfg_start_auth))
-            sys.stdout.write("[bind10] - start_resolver: %s\n" %
-                str(self.cfg_start_resolver))
+        logger.info(BIND10_CONFIGURATION_START_AUTH, self.cfg_start_auth)
+        logger.info(BIND10_CONFIGURATION_START_RESOLVER, self.cfg_start_resolver)
 
     def log_starting(self, process, port = None, address = None):
         """
             A convenience function to output a "Starting xxx" message if the
-            verbose option is set.  Putting this into a separate method ensures
+            logging is set to DEBUG with debuglevel DBG_PROCESS or higher.
+            Putting this into a separate method ensures
             that the output form is consistent across all processes.
 
             The process name (passed as the first argument) is put into
@@ -377,13 +415,14 @@ class BoB:
             appended to the message (if present).
         """
         self.curproc = process
-        if self.verbose:
-            sys.stdout.write("[bind10] Starting %s" % self.curproc)
-            if port is not None:
-                sys.stdout.write(" on port %d" % port)
-                if address is not None:
-                    sys.stdout.write(" (address %s)" % str(address))
-            sys.stdout.write("\n")
+        if port is None and address is None:
+            logger.info(BIND10_STARTING_PROCESS, self.curproc)
+        elif address is None:
+            logger.info(BIND10_STARTING_PROCESS_PORT, self.curproc,
+                        port)
+        else:
+            logger.info(BIND10_STARTING_PROCESS_PORT_ADDRESS,
+                        self.curproc, address, port)
 
     def log_started(self, pid = None):
         """
@@ -391,11 +430,10 @@ class BoB:
             message.  As with starting_message(), this ensures a consistent
             format.
         """
-        if self.verbose:
-            sys.stdout.write("[bind10] Started %s" % self.curproc)
-            if pid is not None:
-                sys.stdout.write(" (PID %d)" % pid)
-            sys.stdout.write("\n")
+        if pid is None:
+            logger.debug(DBG_PROCESS, BIND10_STARTED_PROCESS, self.curproc)
+        else:
+            logger.debug(DBG_PROCESS, BIND10_STARTED_PROCESS_PID, self.curproc, pid)
 
     # The next few methods start the individual processes of BIND-10.  They
     # are called via start_all_processes().  If any fail, an exception is
@@ -459,7 +497,8 @@ class BoB:
         """
         self.log_starting("ccsession")
         self.ccs = isc.config.ModuleCCSession(SPECFILE_LOCATION, 
-                                      self.config_handler, self.command_handler)
+                                      self.config_handler,
+                                      self.command_handler)
         self.ccs.start()
         self.log_started()
 
@@ -568,6 +607,11 @@ class BoB:
             Starts up all the processes.  Any exception generated during the
             starting of the processes is handled by the caller.
         """
+        # The socket creator first, as it is the only thing that needs root
+        self.start_creator()
+        # TODO: Once everything uses the socket creator, we can drop root
+        # privileges right now
+
         c_channel_env = self.c_channel_env
         self.start_msgq(c_channel_env)
         self.start_cfgmgr(c_channel_env)
@@ -620,12 +664,12 @@ class BoB:
         # running
         c_channel_env = {}
         if self.msgq_socket_file is not None:
-             c_channel_env["BIND10_MSGQ_SOCKET_FILE"] = self.msgq_socket_file 
-        if self.verbose:
-           sys.stdout.write("[bind10] Checking for already running b10-msgq\n")
+             c_channel_env["BIND10_MSGQ_SOCKET_FILE"] = self.msgq_socket_file
+        logger.debug(DBG_PROCESS, BIND10_CHECK_MSGQ_ALREADY_RUNNING)
         # try to connect, and if we can't wait a short while
         try:
             self.cc_session = isc.cc.Session(self.msgq_socket_file)
+            logger.fatal(BIND10_MSGQ_ALREADY_RUNNING)
             return "b10-msgq already running, or socket file not cleaned , cannot start"
         except isc.cc.session.SessionError:
             # this is the case we want, where the msgq is not running
@@ -657,14 +701,15 @@ class BoB:
         self.cc_session.group_sendmsg(cmd, "Zonemgr", "Zonemgr")
         self.cc_session.group_sendmsg(cmd, "Stats", "Stats")
         self.cc_session.group_sendmsg(cmd, "StatsHttpd", "StatsHttpd")
+        # Terminate the creator last
+        self.stop_creator()
 
     def stop_process(self, process, recipient):
         """
         Stop the given process, friendly-like. The process is the name it has
         (in logs, etc), the recipient is the address on msgq.
         """
-        if self.verbose:
-            sys.stdout.write("[bind10] Asking %s to terminate\n" % process)
+        logger.info(BIND10_STOP_PROCESS, process)
         # TODO: Some timeout to solve processes that don't want to die would
         # help. We can even store it in the dict, it is used only as a set
         self.expected_shutdowns[process] = 1
@@ -690,8 +735,7 @@ class BoB:
 
     def shutdown(self):
         """Stop the BoB instance."""
-        if self.verbose:
-            sys.stdout.write("[bind10] Stopping the server.\n")
+        logger.info(BIND10_SHUTDOWN)
         # first try using the BIND 10 request to stop
         try:
             self.stop_all_processes()
@@ -705,9 +749,8 @@ class BoB:
         # next try sending a SIGTERM
         processes_to_stop = list(self.processes.values())
         for proc_info in processes_to_stop:
-            if self.verbose:
-                sys.stdout.write("[bind10] Sending SIGTERM to %s (PID %d).\n" % 
-                                 (proc_info.name, proc_info.pid))
+            logger.info(BIND10_SEND_SIGTERM, proc_info.name,
+                        proc_info.pid)
             try:
                 proc_info.process.terminate()
             except OSError:
@@ -721,17 +764,15 @@ class BoB:
             self.reap_children()
             processes_to_stop = list(self.processes.values())
             for proc_info in processes_to_stop:
-                if self.verbose:
-                    sys.stdout.write("[bind10] Sending SIGKILL to %s (PID %d).\n" % 
-                                     (proc_info.name, proc_info.pid))
+                logger.info(BIND10_SEND_SIGKILL, proc_info.name,
+                            proc_info.pid)
                 try:
                     proc_info.process.kill()
                 except OSError:
                     # ignore these (usually ESRCH because the child
                     # finally exited)
                     pass
-        if self.verbose:
-            sys.stdout.write("[bind10] All processes ended, server done.\n")
+        logger.info(BIND10_SHUTDOWN_COMPLETE)
 
     def _get_process_exit_status(self):
         return os.waitpid(-1, os.WNOHANG)
@@ -748,7 +789,14 @@ class BoB:
                 # XXX: should be impossible to get any other error here
                 raise
             if pid == 0: break
-            if pid in self.processes:
+            if self.sockcreator is not None and self.sockcreator.pid() == pid:
+                # This is the socket creator, started and terminated
+                # differently. This can't be restarted.
+                if self.runnable:
+                    logger.fatal(BIND10_SOCKCREATOR_CRASHED)
+                    self.sockcreator = None
+                    self.runnable = False
+            elif pid in self.processes:
                 # One of the processes we know about.  Get information on it.
                 proc_info = self.processes.pop(pid)
                 proc_info.restart_schedule.set_run_stop_time()
@@ -759,18 +807,16 @@ class BoB:
                 # elsewhere.
                 if self.runnable:
                     if exit_status is None:
-                        sys.stdout.write(
-                            "[bind10] Process %s (PID %d) died: exit status not available" % 
-                            (proc_info.name, proc_info.pid))
+                        logger.warn(BIND10_PROCESS_ENDED_NO_EXIT_STATUS,
+                                    proc_info.name, proc_info.pid)
                     else:
-                        sys.stdout.write(
-                            "[bind10] Process %s (PID %d) terminated, exit status = %d\n" % 
-                            (proc_info.name, proc_info.pid, exit_status))
+                        logger.warn(BIND10_PROCESS_ENDED_WITH_EXIT_STATUS,
+                                    proc_info.name, proc_info.pid,
+                                    exit_status)
 
                     # Was it a special process?
                     if proc_info.name == "b10-msgq":
-                        sys.stdout.write(
-                                 "[bind10] The b10-msgq process died, shutting down.\n")
+                        logger.fatal(BIND10_MSGQ_DAEMON_ENDED)
                         self.runnable = False
 
                 # If we're in 'brittle' mode, we want to shutdown after
@@ -778,7 +824,7 @@ class BoB:
                 if self.brittle:
                     self.runnable = False
             else:
-                sys.stdout.write("[bind10] Unknown child pid %d exited.\n" % pid)
+                logger.info(BIND10_UNKNOWN_CHILD_PROCESS_ENDED, pid)
 
     def restart_processes(self):
         """
@@ -809,14 +855,11 @@ class BoB:
                     next_restart = restart_time
                 still_dead[proc_info.pid] = proc_info
             else:
-                if self.verbose:
-                    sys.stdout.write("[bind10] Resurrecting dead %s process...\n" % 
-                        proc_info.name)
+                logger.info(BIND10_RESURRECTING_PROCESS, proc_info.name)
                 try:
                     proc_info.respawn()
                     self.processes[proc_info.pid] = proc_info
-                    sys.stdout.write("[bind10] Resurrected %s (PID %d)\n" %
-                                     (proc_info.name, proc_info.pid))
+                    logger.info(BIND10_RESURRECTED_PROCESS, proc_info.name, proc_info.pid)
                 except:
                     still_dead[proc_info.pid] = proc_info
         # remember any processes that refuse to be resurrected
@@ -848,8 +891,7 @@ def fatal_signal(signal_number, stack_frame):
     """We need to exit (SIGINT or SIGTERM received)."""
     global options
     global boss_of_bind
-    if options.verbose:
-        sys.stdout.write("[bind10] Received %s.\n" % get_signame(signal_number))
+    logger.info(BIND10_RECEIVED_SIGNAL, get_signame(signal_number))
     signal.signal(signal.SIGCHLD, signal.SIG_DFL)
     boss_of_bind.runnable = False
 
@@ -967,12 +1009,11 @@ def main():
             pass
 
         if setuid is None:
-            sys.stderr.write("bind10: invalid user: '%s'\n" % options.user)
+            logger.fatal(BIND10_INVALID_USER, options.user)
             sys.exit(1)
 
     # Announce startup.
-    if options.verbose:
-        sys.stdout.write("%s\n" % VERSION)
+    logger.info(BIND10_STARTING, VERSION)
 
     # Create wakeup pipe for signal handlers
     wakeup_pipe = os.pipe()
@@ -994,9 +1035,9 @@ def main():
                        setuid, username, options.cmdctl_port, options.brittle)
     startup_result = boss_of_bind.startup()
     if startup_result:
-        sys.stderr.write("[bind10] Error on startup: %s\n" % startup_result)
+        logger.fatal(BIND10_STARTUP_ERROR, startup_result)
         sys.exit(1)
-    sys.stdout.write("[bind10] BIND 10 started\n")
+    logger.info(BIND10_STARTUP_COMPLETE)
     dump_pid(options.pid_file)
 
     # In our main loop, we check for dead processes or messages 
@@ -1022,7 +1063,7 @@ def main():
             if err.args[0] == errno.EINTR:
                 (rlist, wlist, xlist) = ([], [], [])
             else:
-                sys.stderr.write("[bind10] Error with select(); %s\n" % err)
+                logger.fatal(BIND10_SELECT_ERROR, err)
                 break
 
         for fd in rlist + xlist:
@@ -1030,8 +1071,8 @@ def main():
                 try:
                     boss_of_bind.ccs.check_command()
                 except isc.cc.session.ProtocolError:
-                    if options.verbose:
-                        sys.stderr.write("[bind10] msgq channel disappeared.\n")
+                    logger.fatal(BIND10_MSGQ_DISAPPEARED)
+                    self.runnable = False
                     break
             elif fd == wakeup_fd:
                 os.read(wakeup_fd, 32)
@@ -1039,7 +1080,6 @@ def main():
     # shutdown
     signal.signal(signal.SIGCHLD, signal.SIG_DFL)
     boss_of_bind.shutdown()
-    sys.stdout.write("[bind10] BIND 10 exiting\n");
     unlink_pid_file(options.pid_file)
     sys.exit(0)
 

src/bin/bind10/bob.spec

@@ -37,6 +37,17 @@
         "command_description": "List the running BIND 10 processes",
         "command_args": []
       }
+    ],
+    "statistics": [
+      {
+        "item_name": "boot_time",
+        "item_type": "string",
+        "item_optional": false,
+        "item_default": "1970-01-01T00:00:00Z",
+        "item_title": "Boot time",
+        "item_description": "A date time when bind10 process starts initially",
+        "item_format": "date-time"
+      }
     ]
   }
 }

src/bin/bind10/creatorapi.txt

@@ -0,0 +1,123 @@
+Socket creator API
+==================
+
+This API is between Boss and other modules to allow them requesting of sockets.
+For simplicity, we will use the socket creator for all (even non-privileged)
+ports for now, but we should have some function where we can abstract it later.
+
+Goals
+-----
+* Be able to request a socket of any combination IPv4/IPv6 UDP/TCP bound to given
+  port and address (sockets that are not bound to anything can be created
+  without privileges, therefore are not requested from the socket creator).
+* Allow to provide the same socket to multiple modules (eg. multiple running
+  auth servers).
+* Allow releasing the sockets (in case all modules using it give it up,
+  terminate or crash).
+* Allow restricting of the sharing (don't allow shared socket between auth
+  and recursive, as the packets would often get to the wrong application,
+  show error instead).
+* Get the socket to the application.
+
+Transport of sockets
+--------------------
+It seems we are stuck with current msgq for a while and there's a chance the
+new replacement will not be able to send sockets inbound. So, we need another
+channel.
+
+The boss will create a unix-domain socket and listen on it. When something
+requests a socket over the command channel and the socket is created, some kind
+of token is returned to the application (which will represent the future
+socket). The application then connects to the unix-domain socket, sends the
+token over the connection (so Boss will know which socket to send there, in case
+multiple applications ask for sockets simultaneously) and Boss sends the socket
+in return.
+
+In theory, we could send the requests directly over the unix-domain
+socket, but it has two disadvantages:
+* The msgq handles serializing/deserializing of structured
+  information (like the parameters to be used), we would have to do it
+  manually on the socket.
+* We could place some kind of security in front of msgq (in case file
+  permissions are not enough, for example if they are not honored on
+  socket files, as indicated in the first paragraph of:
+  http://lkml.indiana.edu/hypermail/linux/kernel/0505.2/0008.html).
+  The socket would have to be secured separately. With the tokens,
+  there's some level of security already - someone not having the
+  token can't request a priviledged socket.
+
+Caching of sockets
+------------------
+To allow sending the same socket to multiple application, the Boss process will
+hold a cache. Each socket that is created and sent is kept open in Boss and
+preserved there as well. A reference count is kept with each of them.
+
+When another application asks for the same socket, it is simply sent from the
+cache instead of creating it again by the creator.
+
+When application gives the socket willingly (by sending a message over the
+command channel), the reference count can be decreased without problems. But
+when the application terminates or crashes, we need to decrease it as well.
+There's a problem, since we don't know which command channel connection (eg.
+lname) belongs to which PID. Furthermore, the applications don't need to be
+started by boss.
+
+There are two possibilities:
+* Let the msgq send messages about disconnected clients (eg. group message to
+  some name). This one is better if we want to migrate to dbus, since dbus
+  already has this capability as well as sending the sockets inbound (at least it
+  seems so on unix) and we could get rid of the unix-domain socket completely.
+* Keep the unix-domain connections open forever. Boss can remember which socket
+  was sent to which connection and when the connection closes (because the
+  application crashed), it can drop all the references on the sockets. This
+  seems easier to implement.
+
+The commands
+------------
+* Command to release a socket. This one would have single parameter, the token
+  used to get the socket. After this, boss would decrease its reference count
+  and if it drops to zero, close its own copy of the socket. This should be used
+  when the module stops using the socket (and after closes it). The
+  library could remember the file-descriptor to token mapping (for
+  common applications that don't request the same socket multiple
+  times in parallel).
+* Command to request a socket. It would have parameters to specify which socket
+  (IP address, address family, port) and how to allow sharing. Sharing would be
+  one of:
+  - None
+  - Same kind of application (however, it is not entirely clear what
+    this means, in case it won't work out intuitively, we'll need to
+    define it somehow)
+  - Any kind of application
+  And a kind of application would be provided, to decide if the sharing is
+  possible (eg. if auth allows sharing with the same kind and something else
+  allows sharing with anything, the sharing is not possible, two auths can).
+
+  It would return either error (the socket can't be created or sharing is not
+  possible) or the token. Then there would be some time for the application to
+  pick up the requested socket.
+
+Examples
+--------
+We probably would have a library with blocking calls to request the
+sockets, so a code could look like:
+
+(socket_fd, token) = request_socket(address, port, 'UDP', SHARE_SAMENAME, 'test-application')
+sock = socket.fromfd(socket_fd)
+
+# Some sock.send and sock.recv stuff here
+
+sock.close()
+release_socket(socket_fd) # or release_socket(token)
+
+Known limitations
+-----------------
+Currently the socket creator doesn't support specifying any socket
+options. If it turns out there are any options that need to be set
+before bind(), we'll need to extend it (and extend the protocol as
+well). If we want to support them, we'll have to solve a possible
+conflict (what to do when two applications request the same socket and
+want to share it, but want different options).
+
+The current socket creator doesn't know raw sockets, but if they are
+needed, it should be easy to add.

src/bin/bind10/run_bind10.sh.in

@@ -20,17 +20,17 @@ export PYTHON_EXEC
 
 BIND10_PATH=@abs_top_builddir@/src/bin/bind10
 
-PATH=@abs_top_builddir@/src/bin/msgq:@abs_top_builddir@/src/bin/auth:@abs_top_builddir@/src/bin/resolver:@abs_top_builddir@/src/bin/cfgmgr:@abs_top_builddir@/src/bin/cmdctl:@abs_top_builddir@/src/bin/stats:@abs_top_builddir@/src/bin/xfrin:@abs_top_builddir@/src/bin/xfrout:@abs_top_builddir@/src/bin/zonemgr:@abs_top_builddir@/src/bin/dhcp6:$PATH
+PATH=@abs_top_builddir@/src/bin/msgq:@abs_top_builddir@/src/bin/auth:@abs_top_builddir@/src/bin/resolver:@abs_top_builddir@/src/bin/cfgmgr:@abs_top_builddir@/src/bin/cmdctl:@abs_top_builddir@/src/bin/stats:@abs_top_builddir@/src/bin/xfrin:@abs_top_builddir@/src/bin/xfrout:@abs_top_builddir@/src/bin/zonemgr:@abs_top_builddir@/src/bin/dhcp6:@abs_top_builddir@/src/bin/sockcreator:$PATH
 export PATH
 
-PYTHONPATH=@abs_top_builddir@/src/lib/python:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/xfr/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/python/isc/config
+PYTHONPATH=@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_builddir@/src/lib/python:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/xfr/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/python/isc/config:@abs_top_builddir@/src/lib/python/isc/acl/.libs:@abs_top_builddir@/src/lib/python/isc/datasrc/.libs
 export PYTHONPATH
 
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 SET_ENV_LIBRARY_PATH=@SET_ENV_LIBRARY_PATH@
 if test $SET_ENV_LIBRARY_PATH = yes; then
-	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:$@ENV_LIBRARY_PATH@
+	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/acl/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:@abs_top_builddir@/src/lib/datasrc/.libs:$@ENV_LIBRARY_PATH@
 	export @ENV_LIBRARY_PATH@
 fi
 

src/bin/bind10/tests/Makefile.am

@@ -2,13 +2,13 @@ PYCOVERAGE_RUN = @PYCOVERAGE_RUN@
 #PYTESTS = args_test.py bind10_test.py
 # NOTE: this has a generated test found in the builddir
 PYTESTS = bind10_test.py
-EXTRA_DIST = $(PYTESTS)
+noinst_SCRIPTS = $(PYTESTS)
 
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
@@ -20,8 +20,9 @@ if ENABLE_PYTHON_COVERAGE
 endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
+	chmod +x $(abs_builddir)/$$pytest ; \
 	$(LIBRARY_PATH_PLACEHOLDER) \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/bind10 \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_srcdir)/src/bin:$(abs_top_builddir)/src/bin/bind10:$(abs_top_builddir)/src/lib/util/io/.libs \
 	BIND10_MSGQ_SOCKET_FILE=$(abs_top_builddir)/msgq_socket \
 		$(PYCOVERAGE_RUN) $(abs_builddir)/$$pytest || exit ; \
 	done

src/bin/bind10/tests/bind10_test.py.in

@@ -13,7 +13,7 @@
 # NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
 # WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
-from bind10 import ProcessInfo, BoB, parse_args, dump_pid, unlink_pid_file, _BASETIME
+from bind10_src import ProcessInfo, BoB, parse_args, dump_pid, unlink_pid_file, _BASETIME
 
 # XXX: environment tests are currently disabled, due to the preprocessor
 #      setup that we have now complicating the environment
@@ -26,6 +26,7 @@ import socket
 from isc.net.addr import IPAddr
 import time
 import isc
+import isc.log
 
 from isc.testutils.parse_args import TestOptParser, OptsError
 
@@ -136,9 +137,27 @@ class TestBoB(unittest.TestCase):
             def group_sendmsg(self, msg, group):
                 (self.msg, self.group) = (msg, group)
             def group_recvmsg(self, nonblock, seq): pass
+        class DummyModuleCCSession():
+            module_spec = isc.config.module_spec.ModuleSpec({
+                    "module_name": "Boss",
+                    "statistics": [
+                        {
+                            "item_name": "boot_time",
+                            "item_type": "string",
+                            "item_optional": False,
+                            "item_default": "1970-01-01T00:00:00Z",
+                            "item_title": "Boot time",
+                            "item_description": "A date time when bind10 process starts initially",
+                            "item_format": "date-time"
+                            }
+                        ]
+                    })
+            def get_module_spec(self):
+                return self.module_spec
         bob = BoB()
         bob.verbose = True
         bob.cc_session = DummySession()
+        bob.ccs = DummyModuleCCSession()
         # a bad command
         self.assertEqual(bob.command_handler(-1, None),
                          isc.config.ccsession.create_answer(1, "bad command"))
@@ -146,14 +165,22 @@ class TestBoB(unittest.TestCase):
         self.assertEqual(bob.command_handler("shutdown", None),
                          isc.config.ccsession.create_answer(0))
         self.assertFalse(bob.runnable)
+        # "getstats" command
+        self.assertEqual(bob.command_handler("getstats", None),
+                         isc.config.ccsession.create_answer(0,
+                            { "owner": "Boss",
+                              "data": {
+                                'boot_time': time.strftime('%Y-%m-%dT%H:%M:%SZ', _BASETIME)
+                            }}))
         # "sendstats" command
         self.assertEqual(bob.command_handler("sendstats", None),
                          isc.config.ccsession.create_answer(0))
         self.assertEqual(bob.cc_session.group, "Stats")
         self.assertEqual(bob.cc_session.msg,
                          isc.config.ccsession.create_command(
-                'set', { "stats_data": {
-                        'bind10.boot_time': time.strftime('%Y-%m-%dT%H:%M:%SZ', _BASETIME)
+                "set", { "owner": "Boss",
+                         "data": {
+                        "boot_time": time.strftime("%Y-%m-%dT%H:%M:%SZ", _BASETIME)
                         }}))
         # "ping" command
         self.assertEqual(bob.command_handler("ping", None),
@@ -192,6 +219,13 @@ class MockBob(BoB):
         self.cmdctl = False
         self.c_channel_env = {}
         self.processes = { }
+        self.creator = False
+
+    def start_creator(self):
+        self.creator = True
+
+    def stop_creator(self, kill=False):
+        self.creator = False
 
     def read_bind10_config(self):
         # Configuration options are set directly
@@ -336,6 +370,7 @@ class TestStartStopProcessesBob(unittest.TestCase):
         self.assertEqual(bob.msgq, core)
         self.assertEqual(bob.cfgmgr, core)
         self.assertEqual(bob.ccsession, core)
+        self.assertEqual(bob.creator, core)
         self.assertEqual(bob.auth, auth)
         self.assertEqual(bob.resolver, resolver)
         self.assertEqual(bob.xfrout, auth)
@@ -764,4 +799,5 @@ class TestBrittle(unittest.TestCase):
         self.assertFalse(bob.runnable)
 
 if __name__ == '__main__':
+    isc.log.resetUnitTestRootLogger()
     unittest.main()

src/bin/bindctl/Makefile.am

@@ -5,6 +5,8 @@ man_MANS = bindctl.1
 
 EXTRA_DIST = $(man_MANS) bindctl.xml
 
+noinst_SCRIPTS = run_bindctl.sh
+
 python_PYTHON = __init__.py bindcmd.py cmdparse.py exception.py moduleinfo.py \
 		mycollections.py
 pythondir = $(pyexecdir)/bindctl

src/bin/bindctl/bindcmd.py

@@ -398,6 +398,8 @@ class BindCmdInterpreter(Cmd):
                 print("Error: " + str(dte))
             except isc.cc.data.DataNotFoundError as dnfe:
                 print("Error: " + str(dnfe))
+            except isc.cc.data.DataAlreadyPresentError as dape:
+                print("Error: " + str(dape))
             except KeyError as ke:
                 print("Error: missing " + str(ke))
         else:
@@ -634,7 +636,15 @@ class BindCmdInterpreter(Cmd):
                     # we have more data to show
                     line += "/"
                 else:
-                    line += "\t" + json.dumps(value_map['value'])
+                    # if type is named_set, don't print value if None
+                    # (it is either {} meaning empty, or None, meaning
+                    # there actually is data, but not to be shown with
+                    # the current command
+                    if value_map['type'] == 'named_set' and\
+                       value_map['value'] is None:
+                        line += "/\t"
+                    else:
+                        line += "\t" + json.dumps(value_map['value'])
                 line += "\t" + value_map['type']
                 line += "\t"
                 if value_map['default']:
@@ -649,10 +659,9 @@ class BindCmdInterpreter(Cmd):
                 data, default = self.config_data.get_value(identifier)
                 print(json.dumps(data))
         elif cmd.command == "add":
-            if 'value' in cmd.params:
-                self.config_data.add_value(identifier, cmd.params['value'])
-            else:
-                self.config_data.add_value(identifier)
+            self.config_data.add_value(identifier,
+                                       cmd.params.get('value_or_name'),
+                                       cmd.params.get('value_for_set'))
         elif cmd.command == "remove":
             if 'value' in cmd.params:
                 self.config_data.remove_value(identifier, cmd.params['value'])
@@ -674,9 +683,12 @@ class BindCmdInterpreter(Cmd):
         elif cmd.command == "revert":
             self.config_data.clear_local_changes()
         elif cmd.command == "commit":
-            self.config_data.commit()
+            try:
+                self.config_data.commit()
+            except isc.config.ModuleCCSessionError as mcse:
+                print(str(mcse))
         elif cmd.command == "diff":
-            print(self.config_data.get_local_changes());
+            print(self.config_data.get_local_changes())
         elif cmd.command == "go":
             self.go(identifier)
 

src/bin/bindctl/bindctl_main.py.in

@@ -50,17 +50,28 @@ def prepare_config_commands(tool):
     cmd.add_param(param)
     module.add_command(cmd)
 
-    cmd = CommandInfo(name = "add", desc = "Add an entry to configuration list. If no value is given, a default value is added.")
+    cmd = CommandInfo(name = "add", desc =
+        "Add an entry to configuration list or a named set. "
+        "When adding to a list, the command has one optional argument, "
+        "a value to add to the list. The value must be in correct JSON "
+        "and complete. When adding to a named set, it has one "
+        "mandatory parameter (the name to add), and an optional "
+        "parameter value, similar to when adding to a list. "
+        "In either case, when no value is given, an entry will be "
+        "constructed with default values.")
     param = ParamInfo(name = "identifier", type = "string", optional=True, desc = DEFAULT_IDENTIFIER_DESC)
     cmd.add_param(param)
-    param = ParamInfo(name = "value", type = "string", optional=True, desc = "Specifies a value to add to the list. It must be in correct JSON format and complete.")
+    param = ParamInfo(name = "value_or_name", type = "string", optional=True, desc = "Specifies a value to add to the list, or the name when adding to a named set. It must be in correct JSON format and complete.")
+    cmd.add_param(param)
+    module.add_command(cmd)
+    param = ParamInfo(name = "value_for_set", type = "string", optional=True, desc = "Specifies an optional value to add to the named map. It must be in correct JSON format and complete.")
     cmd.add_param(param)
     module.add_command(cmd)
 
-    cmd = CommandInfo(name = "remove", desc = "Remove entry from configuration list.")
+    cmd = CommandInfo(name = "remove", desc = "Remove entry from configuration list or named set.")
     param = ParamInfo(name = "identifier", type = "string", optional=True, desc = DEFAULT_IDENTIFIER_DESC)
     cmd.add_param(param)
-    param = ParamInfo(name = "value", type = "string", optional=True, desc = "Specifies a value to remove from the list. It must be in correct JSON format and complete.")
+    param = ParamInfo(name = "value", type = "string", optional=True, desc = "When identifier is a list, specifies a value to remove from the list. It must be in correct JSON format and complete. When it is a named set, specifies the name to remove.")
     cmd.add_param(param)
     module.add_command(cmd)
 

src/bin/bindctl/run_bindctl.sh.in

@@ -20,14 +20,14 @@ export PYTHON_EXEC
 
 BINDCTL_PATH=@abs_top_builddir@/src/bin/bindctl
 
-PYTHONPATH=@abs_top_srcdir@/src/bin:@abs_top_builddir@/src/lib/python:@abs_top_builddir@/src/bin:@abs_top_srcdir@/src/lib/python
+PYTHONPATH=@abs_top_srcdir@/src/bin:@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_builddir@/src/lib/python:@abs_top_builddir@/src/bin:@abs_top_srcdir@/src/lib/python
 export PYTHONPATH
 
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 SET_ENV_LIBRARY_PATH=@SET_ENV_LIBRARY_PATH@
 if test $SET_ENV_LIBRARY_PATH = yes; then
-	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:$@ENV_LIBRARY_PATH@
+	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:@abs_top_builddir@/src/lib/datasrc/.libs:$@ENV_LIBRARY_PATH@
 	export @ENV_LIBRARY_PATH@
 fi
 

src/bin/bindctl/tests/Makefile.am

@@ -6,7 +6,7 @@ EXTRA_DIST = $(PYTESTS)
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
@@ -19,6 +19,6 @@ endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
 	$(LIBRARY_PATH_PLACEHOLDER) \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/bindctl:$(abs_top_srcdir)/src/bin  \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_builddir)/src/bin/bindctl:$(abs_top_srcdir)/src/bin  \
 	$(PYCOVERAGE_RUN) $(abs_srcdir)/$$pytest || exit ; \
 	done

src/bin/cfgmgr/b10-cfgmgr.py.in

@@ -28,7 +28,7 @@ import os.path
 import isc.log
 isc.log.init("b10-cfgmgr")
 from isc.config.cfgmgr import ConfigManager, ConfigManagerDataReadError, logger
-from cfgmgr_messages import *
+from isc.log_messages.cfgmgr_messages import *
 
 isc.util.process.rename()
 

src/bin/cfgmgr/plugins/tests/Makefile.am

@@ -7,7 +7,7 @@ EXTRA_DIST = $(PYTESTS)
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
@@ -19,8 +19,8 @@ if ENABLE_PYTHON_COVERAGE
 endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
-	env B10_TEST_PLUGIN_DIR=$(abs_srcdir)/..:$(abs_builddir)/.. \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/cfgmgr:$(abs_top_builddir)/src/lib/dns/python/.libs \
+	B10_TEST_PLUGIN_DIR=$(abs_srcdir)/..:$(abs_builddir)/.. \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_builddir)/src/bin/cfgmgr:$(abs_top_builddir)/src/lib/dns/python/.libs \
 	$(LIBRARY_PATH_PLACEHOLDER) \
 	$(PYCOVERAGE_RUN) $(abs_srcdir)/$$pytest || exit ; \
 	done

src/bin/cfgmgr/tests/Makefile.am

@@ -1,13 +1,14 @@
 PYCOVERAGE_RUN = @PYCOVERAGE_RUN@
 PYTESTS = b10-cfgmgr_test.py
 
-EXTRA_DIST = $(PYTESTS) testdata/plugins/testplugin.py
+noinst_SCRIPTS = $(PYTESTS)
+EXTRA_DIST = testdata/plugins/testplugin.py
 
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
@@ -19,9 +20,10 @@ if ENABLE_PYTHON_COVERAGE
 endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
-	env TESTDATA_PATH=$(abs_srcdir)/testdata \
+	chmod +x $(abs_builddir)/$$pytest ; \
+	TESTDATA_PATH=$(abs_srcdir)/testdata \
 	$(LIBRARY_PATH_PLACEHOLDER) \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/cfgmgr:$(abs_top_builddir)/src/lib/python/isc/config \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_builddir)/src/bin/cfgmgr:$(abs_top_builddir)/src/lib/python/isc/config \
 	$(PYCOVERAGE_RUN) $(abs_builddir)/$$pytest || exit ; \
 	done
 

src/bin/cmdctl/Makefile.am

@@ -4,6 +4,9 @@ pkglibexecdir = $(libexecdir)/@PACKAGE@
 
 pkglibexec_SCRIPTS = b10-cmdctl
 
+nodist_pylogmessage_PYTHON = $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py
+pylogmessagedir = $(pyexecdir)/isc/log_messages/
+
 b10_cmdctldir = $(pkgdatadir)
 
 # NOTE: this will overwrite on install
@@ -18,10 +21,12 @@ b10_cmdctl_DATA += cmdctl.spec
 
 EXTRA_DIST = $(CMDCTL_CONFIGURATIONS)
 
-CLEANFILES=	b10-cmdctl cmdctl.pyc cmdctl.spec
+CLEANFILES= b10-cmdctl cmdctl.pyc cmdctl.spec
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.pyc
 
 man_MANS = b10-cmdctl.8
-EXTRA_DIST += $(man_MANS) b10-cmdctl.xml
+EXTRA_DIST += $(man_MANS) b10-cmdctl.xml cmdctl_messages.mes
 
 if ENABLE_MAN
 
@@ -33,8 +38,12 @@ endif
 cmdctl.spec: cmdctl.spec.pre
 	$(SED) -e "s|@@SYSCONFDIR@@|$(sysconfdir)|" cmdctl.spec.pre >$@
 
+$(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py : cmdctl_messages.mes
+	$(top_builddir)/src/lib/log/compiler/message \
+	-d $(PYTHON_LOGMSGPKG_DIR)/work -p $(srcdir)/cmdctl_messages.mes
+
 # this is done here since configure.ac AC_OUTPUT doesn't expand exec_prefix
-b10-cmdctl: cmdctl.py
+b10-cmdctl: cmdctl.py $(PYTHON_LOGMSGPKG_DIR)/work/cmdctl_messages.py
 	$(SED) "s|@@PYTHONPATH@@|@pyexecdir@|" cmdctl.py >$@
 	chmod a+x $@
 

src/bin/cmdctl/cmdctl.py.in

@@ -47,6 +47,18 @@ import isc.net.parse
 from optparse import OptionParser, OptionValueError
 from hashlib import sha1
 from isc.util import socketserver_mixin
+from isc.log_messages.cmdctl_messages import *
+
+# TODO: these debug-levels are hard-coded here; we are planning on
+# creating a general set of debug levels, see ticket #1074. When done,
+# we should remove these values and use the general ones in the
+# logger.debug calls
+
+# Debug level for communication with BIND10
+DBG_CMDCTL_MESSAGING = 30
+
+isc.log.init("b10-cmdctl")
+logger = isc.log.Logger("cmdctl")
 
 try:
     import threading
@@ -173,7 +185,8 @@ class SecureHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
         if not user_name:
             return False, ["need user name"]
         if not self.server.get_user_info(user_name):
-            return False, ["user doesn't exist"]
+            logger.info(CMDCTL_NO_SUCH_USER, user_name)
+            return False, ["username or password error"]
 
         user_pwd = user_info.get('password')
         if not user_pwd:
@@ -181,7 +194,8 @@ class SecureHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
         local_info = self.server.get_user_info(user_name)
         pwd_hashval = sha1((user_pwd + local_info[1]).encode())
         if pwd_hashval.hexdigest() != local_info[0]:
-            return False, ["password doesn't match"] 
+            logger.info(CMDCTL_BAD_PASSWORD, user_name)
+            return False, ["username or password error"]
 
         return True, None
    
@@ -281,7 +295,7 @@ class CommandControl():
                 errstr = 'unknown config item: ' + key
             
             if errstr != None:
-                self.log_info('Fail to apply config data, ' + errstr) 
+                logger.error(CMDCTL_BAD_CONFIG_DATA, errstr);
                 return ccsession.create_answer(1, errstr)
 
         return ccsession.create_answer(0)
@@ -387,8 +401,8 @@ class CommandControl():
         '''Send the command from bindctl to proper module. '''
         errstr = 'unknown error'
         answer = None
-        if self._verbose:
-            self.log_info("Begin send command '%s' to module '%s'" %(command_name, module_name))
+        logger.debug(DBG_CMDCTL_MESSAGING, CMDCTL_SEND_COMMAND,
+                     command_name, module_name)
 
         if module_name == self._module_name:
             # Process the command sent to cmdctl directly. 
@@ -396,15 +410,14 @@ class CommandControl():
         else:
             msg = ccsession.create_command(command_name, params)
             seq = self._cc.group_sendmsg(msg, module_name)
+            logger.debug(DBG_CMDCTL_MESSAGING, CMDCTL_COMMAND_SENT,
+                         command_name, module_name)
             #TODO, it may be blocked, msqg need to add a new interface waiting in timeout.
             try:
                 answer, env = self._cc.group_recvmsg(False, seq)
             except isc.cc.session.SessionTimeout:
                 errstr = "Module '%s' not responding" % module_name
 
-        if self._verbose:
-            self.log_info("Finish send command '%s' to module '%s'" % (command_name, module_name))
-
         if answer:
             try:
                 rcode, arg = ccsession.parse_answer(answer)
@@ -415,16 +428,13 @@ class CommandControl():
                     else:
                         return rcode, {}
                 else:
-                    # TODO: exception
                     errstr = str(answer['result'][1])
             except ccsession.ModuleCCSessionError as mcse:
                 errstr = str("Error in ccsession answer:") + str(mcse)
-                self.log_info(errstr)
+
+        logger.error(CMDCTL_COMMAND_ERROR, command_name, module_name, errstr)
         return 1, {'error': errstr}
     
-    def log_info(self, msg):
-        sys.stdout.write("[b10-cmdctl] %s\n" % str(msg))
-
     def get_cmdctl_config_data(self):
         ''' If running in source code tree, use keyfile, certificate
         and user accounts file in source code. '''
@@ -481,14 +491,15 @@ class SecureHTTPServer(socketserver_mixin.NoPollMixIn,
                 for row in reader:
                     self._user_infos[row[0]] = [row[1], row[2]]
             except (IOError, IndexError) as e:
-                self.log_info("Fail to read user database, %s" % e)                
+                logger.error(CMDCTL_USER_DATABASE_READ_ERROR,
+                             accounts_file, e)
             finally:
                 if csvfile:
                     csvfile.close()
 
         self._accounts_file = accounts_file
         if len(self._user_infos) == 0:
-            self.log_info("Fail to get user information, will deny any user")                
+            logger.error(CMDCTL_NO_USER_ENTRIES_READ)
          
     def get_user_info(self, username):
         '''Get user's salt and hashed string. If the user
@@ -520,7 +531,7 @@ class SecureHTTPServer(socketserver_mixin.NoPollMixIn,
                                       ssl_version = ssl.PROTOCOL_SSLv23)
             return ssl_sock 
         except (ssl.SSLError, CmdctlException) as err :
-            self.log_info("Deny client's connection because %s" % str(err))
+            logger.info(CMDCTL_SSL_SETUP_FAILURE_USER_DENIED, err)
             self.close_request(sock)
             # raise socket error to finish the request
             raise socket.error
@@ -547,9 +558,6 @@ class SecureHTTPServer(socketserver_mixin.NoPollMixIn,
     def send_command_to_module(self, module_name, command_name, params):
         return self.cmdctl.send_command_with_check(module_name, command_name, params)
    
-    def log_info(self, msg):
-        sys.stdout.write("[b10-cmdctl] %s\n" % str(msg))
-
 httpd = None
 
 def signal_handler(signal, frame):
@@ -607,15 +615,13 @@ if __name__ == '__main__':
         run(options.addr, options.port, options.idle_timeout, options.verbose)
         result = 0
     except isc.cc.SessionError as err:
-        sys.stderr.write("[b10-cmdctl] Error creating b10-cmdctl, "
-                         "is the command channel daemon running?\n")        
+        logger.fatal(CMDCTL_CC_SESSION_ERROR, err)
     except isc.cc.SessionTimeout:
-        sys.stderr.write("[b10-cmdctl] Error creating b10-cmdctl, "
-                         "is the configuration manager running?\n")        
+        logger.fatal(CMDCTL_CC_SESSION_TIMEOUT)
     except KeyboardInterrupt:
-        sys.stderr.write("[b10-cmdctl] exit from Cmdctl\n")
+        logger.info(CMDCTL_STOPPED_BY_KEYBOARD)
     except CmdctlException as err:
-        sys.stderr.write("[b10-cmdctl] " + str(err) + "\n")
+        logger.fatal(CMDCTL_UNCAUGHT_EXCEPTION, err);
 
     if httpd:
         httpd.shutdown()

src/bin/cmdctl/cmdctl_messages.mes

@@ -0,0 +1,81 @@
+# Copyright (C) 2011  Internet Systems Consortium, Inc. ("ISC")
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
+# No namespace declaration - these constants go in the global namespace
+# of the cmdctl_messages python module.
+
+% CMDCTL_BAD_CONFIG_DATA error in config data: %1
+There was an error reading the updated configuration data. The specific
+error is printed.
+
+% CMDCTL_BAD_PASSWORD bad password for user: %1
+A login attempt was made to b10-cmdctl, but the password was wrong.
+Users can be managed with the tool b10-cmdctl-usermgr.
+
+% CMDCTL_CC_SESSION_ERROR error reading from cc channel: %1
+There was a problem reading from the command and control channel. The
+most likely cause is that the message bus daemon is not running.
+
+% CMDCTL_CC_SESSION_TIMEOUT timeout on cc channel
+A timeout occurred when waiting for essential data from the cc session.
+This usually occurs when b10-cfgmgr is not running or not responding.
+Since we are waiting for essential information, this is a fatal error,
+and the cmdctl daemon will now shut down.
+
+% CMDCTL_COMMAND_ERROR error in command %1 to module %2: %3
+An error was encountered sending the given command to the given module.
+Either there was a communication problem with the module, or the module
+was not able to process the command, and sent back an error. The
+specific error is printed in the message.
+
+% CMDCTL_COMMAND_SENT command '%1' to module '%2' was sent
+This debug message indicates that the given command has been sent to
+the given module.
+
+% CMDCTL_NO_SUCH_USER username not found in user database: %1
+A login attempt was made to b10-cmdctl, but the username was not known.
+Users can be added with the tool b10-cmdctl-usermgr.
+
+% CMDCTL_NO_USER_ENTRIES_READ failed to read user information, all users will be denied
+The b10-cmdctl daemon was unable to find any user data in the user
+database file. Either it was unable to read the file (in which case
+this message follows a message CMDCTL_USER_DATABASE_READ_ERROR
+containing a specific error), or the file was empty. Users can be added
+with the tool b10-cmdctl-usermgr.
+
+% CMDCTL_SEND_COMMAND sending command %1 to module %2
+This debug message indicates that the given command is being sent to
+the given module.
+
+% CMDCTL_SSL_SETUP_FAILURE_USER_DENIED failed to create an SSL connection (user denied): %1
+The user was denied because the SSL connection could not successfully
+be set up. The specific error is given in the log message. Possible
+causes may be that the ssl request itself was bad, or the local key or
+certificate file could not be read.
+
+% CMDCTL_STOPPED_BY_KEYBOARD keyboard interrupt, shutting down
+There was a keyboard interrupt signal to stop the cmdctl daemon. The
+daemon will now shut down.
+
+% CMDCTL_UNCAUGHT_EXCEPTION uncaught exception: %1
+The b10-cmdctl daemon encountered an uncaught exception and
+will now shut down. This is indicative of a programming error and
+should not happen under normal circumstances. The exception message
+is printed.
+
+% CMDCTL_USER_DATABASE_READ_ERROR failed to read user database file %1: %2
+The b10-cmdctl daemon was unable to read the user database file. The
+file may be unreadable for the daemon, or it may be corrupted. In the
+latter case, it can be recreated with b10-cmdctl-usermgr. The specific
+error is printed in the log message.

src/bin/cmdctl/run_b10-cmdctl.sh.in

@@ -19,9 +19,17 @@ PYTHON_EXEC=${PYTHON_EXEC:-@PYTHON@}
 export PYTHON_EXEC
 
 CMD_CTRLD_PATH=@abs_top_builddir@/src/bin/cmdctl
-PYTHONPATH=@abs_top_srcdir@/src/lib/python
+PYTHONPATH=@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_builddir@/src/lib/python:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/xfr/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/python/isc/config:@abs_top_builddir@/src/lib/python/isc/acl/.libs:@abs_top_builddir@/src/lib/python/isc/datasrc/.libs
 export PYTHONPATH
 
+# If necessary (rare cases), explicitly specify paths to dynamic libraries
+# required by loadable python modules.
+SET_ENV_LIBRARY_PATH=@SET_ENV_LIBRARY_PATH@
+if test $SET_ENV_LIBRARY_PATH = yes; then
+        @ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:@abs_top_builddir@/src/lib/datasrc/.libs:$@ENV_LIBRARY_PATH@
+        export @ENV_LIBRARY_PATH@
+fi
+
 BIND10_MSGQ_SOCKET_FILE=@abs_top_builddir@/msgq_socket
 export BIND10_MSGQ_SOCKET_FILE
 

src/bin/cmdctl/tests/Makefile.am

@@ -6,7 +6,7 @@ EXTRA_DIST = $(PYTESTS)
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
@@ -19,7 +19,7 @@ endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
 	$(LIBRARY_PATH_PLACEHOLDER) \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/cmdctl \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_builddir)/src/bin/cmdctl \
 	CMDCTL_SPEC_PATH=$(abs_top_builddir)/src/bin/cmdctl \
 	CMDCTL_SRC_PATH=$(abs_top_srcdir)/src/bin/cmdctl \
 	$(PYCOVERAGE_RUN) $(abs_srcdir)/$$pytest || exit ; \

src/bin/cmdctl/tests/cmdctl_test.py

@@ -19,6 +19,7 @@ import socket
 import tempfile
 import sys
 from cmdctl import *
+import isc.log
 
 SPEC_FILE_PATH = '..' + os.sep
 if 'CMDCTL_SPEC_PATH' in os.environ:
@@ -173,7 +174,7 @@ class TestSecureHTTPRequestHandler(unittest.TestCase):
         self.handler.server._user_infos['root'] = ['aa', 'aaa']
         ret, msg = self.handler._check_user_name_and_pwd()
         self.assertFalse(ret)
-        self.assertEqual(msg, ['password doesn\'t match'])
+        self.assertEqual(msg, ['username or password error'])
 
     def test_check_user_name_and_pwd_2(self):
         user_info = {'username':'root', 'password':'abc123'}
@@ -214,7 +215,7 @@ class TestSecureHTTPRequestHandler(unittest.TestCase):
 
         ret, msg = self.handler._check_user_name_and_pwd()
         self.assertFalse(ret)
-        self.assertEqual(msg, ['user doesn\'t exist'])
+        self.assertEqual(msg, ['username or password error'])
 
     def test_do_POST(self):
         self.handler.headers = {}
@@ -447,6 +448,7 @@ class TestFuncNotInClass(unittest.TestCase):
 
 
 if __name__== "__main__":
+    isc.log.resetUnitTestRootLogger()
     unittest.main()
 
 

src/bin/dhcp6/Makefile.am

@@ -35,6 +35,7 @@ b10_dhcp6_SOURCES = main.cc iface_mgr.cc pkt6.cc dhcp6_srv.cc
 b10_dhcp6_SOURCES += iface_mgr.h pkt6.h dhcp6_srv.h dhcp6.h
 b10_dhcp6_LDADD =  $(top_builddir)/src/lib/datasrc/libdatasrc.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/dns/libdns++.la
+b10_dhcp6_LDADD += $(top_builddir)/src/lib/util/libutil.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/config/libcfgclient.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/cc/libcc.la
 b10_dhcp6_LDADD += $(top_builddir)/src/lib/exceptions/libexceptions.la

src/bin/dhcp6/tests/Makefile.am

@@ -8,14 +8,14 @@ EXTRA_DIST = $(PYTESTS)
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
 check-local:
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/bind10 \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_srcdir)/src/bin:$(abs_top_builddir)/src/bin/bind10:$(abs_top_builddir)/src/lib/util/io/.libs \
 	$(LIBRARY_PATH_PLACEHOLDER) \
 	BIND10_MSGQ_SOCKET_FILE=$(abs_top_builddir)/msgq_socket \
 		$(PYCOVERAGE_RUN) $(abs_srcdir)/$$pytest || exit ; \

src/bin/dhcp6/tests/dhcp6_test.py

@@ -13,7 +13,7 @@
 # NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
 # WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
-from bind10 import ProcessInfo, parse_args, dump_pid, unlink_pid_file, _BASETIME
+from bind10_src import ProcessInfo, parse_args, dump_pid, unlink_pid_file, _BASETIME
 
 import unittest
 import sys

src/bin/host/Makefile.am

@@ -13,6 +13,7 @@ CLEANFILES = *.gcno *.gcda
 bin_PROGRAMS = b10-host
 b10_host_SOURCES = host.cc
 b10_host_LDADD = $(top_builddir)/src/lib/dns/libdns++.la
+b10_host_LDADD += $(top_builddir)/src/lib/util/libutil.la
 b10_host_LDADD += $(top_builddir)/src/lib/exceptions/libexceptions.la
 
 man_MANS = b10-host.1

src/bin/host/b10-host.1

@@ -103,10 +103,6 @@ It doesn\'t use
 at this time\&. The default name server used is 127\&.0\&.0\&.1\&.
 .PP
 
-\fBb10\-host\fR
-does not do reverse lookups by default yet (by detecting if name is a IPv4 or IPv6 address)\&.
-.PP
-
 \fB\-p\fR
 is not a standard feature\&.
 .SH "HISTORY"

src/bin/host/b10-host.xml

@@ -176,11 +176,6 @@
     </para>
 
     <para>
-      <command>b10-host</command> does not do reverse lookups by
-      default yet (by detecting if name is a IPv4 or IPv6 address).
-    </para>
-
-    <para>
       <option>-p</option> is not a standard feature.
     </para>
   </refsect1>

src/bin/loadzone/Makefile.am

@@ -1,5 +1,6 @@
 SUBDIRS = . tests/correct tests/error
 bin_SCRIPTS = b10-loadzone
+noinst_SCRIPTS = run_loadzone.sh
 
 CLEANFILES = b10-loadzone
 

src/bin/loadzone/run_loadzone.sh.in

@@ -18,14 +18,14 @@
 PYTHON_EXEC=${PYTHON_EXEC:-@PYTHON@}
 export PYTHON_EXEC
 
-PYTHONPATH=@abs_top_builddir@/src/lib/python
+PYTHONPATH=@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_builddir@/src/lib/python
 export PYTHONPATH
 
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 SET_ENV_LIBRARY_PATH=@SET_ENV_LIBRARY_PATH@
 if test $SET_ENV_LIBRARY_PATH = yes; then
-	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:$@ENV_LIBRARY_PATH@
+	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:@abs_top_builddir@/src/lib/datasrc/.libs:$@ENV_LIBRARY_PATH@
 	export @ENV_LIBRARY_PATH@
 fi
 

src/bin/loadzone/tests/correct/Makefile.am

@@ -13,11 +13,13 @@ EXTRA_DIST += ttl2.db
 EXTRA_DIST += ttlext.db
 EXTRA_DIST += example.db
 
+noinst_SCRIPTS = correct_test.sh
+
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # TODO: maybe use TESTS?

src/bin/loadzone/tests/correct/correct_test.sh.in

@@ -18,7 +18,7 @@
 PYTHON_EXEC=${PYTHON_EXEC:-@PYTHON@}
 export PYTHON_EXEC
 
-PYTHONPATH=@abs_top_srcdir@/src/lib/python:@abs_top_builddir@/src/lib/python
+PYTHONPATH=@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_srcdir@/src/lib/python:@abs_top_builddir@/src/lib/python
 export PYTHONPATH
 
 LOADZONE_PATH=@abs_top_builddir@/src/bin/loadzone

src/bin/loadzone/tests/error/Makefile.am

@@ -12,11 +12,13 @@ EXTRA_DIST += keyerror3.db
 EXTRA_DIST += originerr1.db
 EXTRA_DIST += originerr2.db
 
+noinst_SCRIPTS = error_test.sh
+
 # If necessary (rare cases), explicitly specify paths to dynamic libraries
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # TODO: use TESTS ?

src/bin/loadzone/tests/error/error_test.sh.in

@@ -18,7 +18,7 @@
 PYTHON_EXEC=${PYTHON_EXEC:-@PYTHON@}
 export PYTHON_EXEC
 
-PYTHONPATH=@abs_top_srcdir@/src/lib/python:@abs_top_builddir@/src/lib/python
+PYTHONPATH=@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_srcdir@/src/lib/python:@abs_top_builddir@/src/lib/python
 export PYTHONPATH
 
 LOADZONE_PATH=@abs_top_builddir@/src/bin/loadzone

src/bin/msgq/tests/Makefile.am

@@ -6,7 +6,7 @@ EXTRA_DIST = $(PYTESTS)
 # required by loadable python modules.
 LIBRARY_PATH_PLACEHOLDER =
 if SET_ENV_LIBRARY_PATH
-LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$$$(ENV_LIBRARY_PATH)
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/util/io/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
 endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
@@ -19,7 +19,7 @@ endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
 	$(LIBRARY_PATH_PLACEHOLDER) \
-	env PYTHONPATH=$(abs_top_builddir)/src/bin/msgq:$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_builddir)/src/bin/msgq \
 	BIND10_TEST_SOCKET_FILE=$(builddir)/test_msgq_socket.sock \
 	$(PYCOVERAGE_RUN) $(abs_srcdir)/$$pytest || exit ; \
 	done

src/bin/resolver/Makefile.am

@@ -59,6 +59,8 @@ nodist_b10_resolver_SOURCES = resolver_messages.cc resolver_messages.h
 b10_resolver_LDADD =  $(top_builddir)/src/lib/dns/libdns++.la
 b10_resolver_LDADD += $(top_builddir)/src/lib/config/libcfgclient.la
 b10_resolver_LDADD += $(top_builddir)/src/lib/cc/libcc.la
+b10_resolver_LDADD += $(top_builddir)/src/lib/util/libutil.la
+b10_resolver_LDADD += $(top_builddir)/src/lib/acl/libdnsacl.la
 b10_resolver_LDADD += $(top_builddir)/src/lib/exceptions/libexceptions.la
 b10_resolver_LDADD += $(top_builddir)/src/lib/asiodns/libasiodns.la
 b10_resolver_LDADD += $(top_builddir)/src/lib/asiolink/libasiolink.la

src/bin/resolver/b10-resolver.8

@@ -2,12 +2,12 @@
 .\"     Title: b10-resolver
 .\"    Author: [FIXME: author] [see http://docbook.sf.net/el/author]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: February 17, 2011
+.\"      Date: August 17, 2011
 .\"    Manual: BIND10
 .\"    Source: BIND10
 .\"  Language: English
 .\"
-.TH "B10\-RESOLVER" "8" "February 17, 2011" "BIND10" "BIND10"
+.TH "B10\-RESOLVER" "8" "August 17, 2011" "BIND10" "BIND10"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -54,7 +54,7 @@ must be either a valid numeric user ID or a valid user name\&. By default the da
 .PP
 \fB\-v\fR
 .RS 4
-Enabled verbose mode\&. This enables diagnostic messages to STDERR\&.
+Enable verbose mode\&. This sets logging to the maximum debugging level\&.
 .RE
 .SH "CONFIGURATION AND COMMANDS"
 .PP
@@ -77,6 +77,25 @@ string and
 number\&. The defaults are address ::1 port 53 and address 127\&.0\&.0\&.1 port 53\&.
 .PP
 
+
+
+
+
+
+\fIquery_acl\fR
+is a list of query access control rules\&. The list items are the
+\fIaction\fR
+string and the
+\fIfrom\fR
+or
+\fIkey\fR
+strings\&. The possible actions are ACCEPT, REJECT and DROP\&. The
+\fIfrom\fR
+is a remote (source) IPv4 or IPv6 address or special keyword\&. The
+\fIkey\fR
+is a TSIG key name\&. The default configuration accepts queries from 127\&.0\&.0\&.1 and ::1\&.
+.PP
+
 \fIretries\fR
 is the number of times to retry (resend query) after a query timeout (\fItimeout_query\fR)\&. The default is 3\&.
 .PP
@@ -88,7 +107,7 @@ to use directly as root servers to start resolving\&. The list items are the
 \fIaddress\fR
 string and
 \fIport\fR
-number\&. If empty, a hardcoded address for F\-root (192\&.5\&.5\&.241) is used\&.
+number\&. By default, a hardcoded address for l\&.root\-servers\&.net (199\&.7\&.83\&.42 or 2001:500:3::42) is used\&.
 .PP
 
 \fItimeout_client\fR
@@ -121,8 +140,7 @@ BIND 10 Guide\&.
 .PP
 The
 \fBb10\-resolver\fR
-daemon was first coded in September 2010\&. The initial implementation only provided forwarding\&. Iteration was introduced in January 2011\&.
-
+daemon was first coded in September 2010\&. The initial implementation only provided forwarding\&. Iteration was introduced in January 2011\&. Caching was implemented in February 2011\&. Access control was introduced in June 2011\&.
 .SH "COPYRIGHT"
 .br
 Copyright \(co 2010 Internet Systems Consortium, Inc. ("ISC")

src/bin/resolver/b10-resolver.xml

@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>February 17, 2011</date>
+    <date>August 17, 2011</date>
   </refentryinfo>
 
   <refmeta>
@@ -99,11 +99,14 @@
         </listitem>
       </varlistentry>
 
+<!-- TODO: this needs to be fixed as -v on command line
+should imply stdout or stderr output also -->
+<!-- TODO: can this -v be overidden by configuration or bindctl? -->
       <varlistentry>
         <term><option>-v</option></term>
         <listitem><para>
-          Enabled verbose mode. This enables diagnostic messages to
-          STDERR.
+          Enable verbose mode.
+          This sets logging to the maximum debugging level.
         </para></listitem>
       </varlistentry>
 
@@ -147,6 +150,22 @@ once that is merged you can for instance do 'config add Resolver/forward_address
     </para>
 
     <para>
+<!-- TODO: need more explanation or point to guide. -->
+<!-- TODO: what about a netmask or cidr? -->
+<!-- TODO: document "key" -->
+<!-- TODO: where are the TSIG keys defined? -->
+<!-- TODO: key and from are mutually exclusive? what if both defined? -->
+      <varname>query_acl</varname> is a list of query access control
+      rules. The list items are the <varname>action</varname> string
+      and the <varname>from</varname> or <varname>key</varname> strings.
+      The possible actions are ACCEPT, REJECT and DROP.
+      The <varname>from</varname> is a remote (source) IPv4 or IPv6
+      address or special keyword.
+      The <varname>key</varname> is a TSIG key name.
+      The default configuration accepts queries from 127.0.0.1 and ::1.
+    </para>
+
+    <para>
       <varname>retries</varname> is the number of times to retry
       (resend query) after a query timeout
       (<varname>timeout_query</varname>).
@@ -159,8 +178,10 @@ once that is merged you can for instance do 'config add Resolver/forward_address
       root servers to start resolving.
       The list items are the <varname>address</varname> string
       and <varname>port</varname> number.
-      If empty, a hardcoded address for F-root (192.5.5.241) is used.
+      By default, a hardcoded address for l.root-servers.net
+      (199.7.83.42 or 2001:500:3::42) is used.
     </para>
+<!-- TODO: this is broken, see ticket #1184 -->
 
     <para>
       <varname>timeout_client</varname> is the number of milliseconds
@@ -234,7 +255,8 @@ once that is merged you can for instance do 'config add Resolver/forward_address
       The <command>b10-resolver</command> daemon was first coded in
       September 2010. The initial implementation only provided
       forwarding. Iteration was introduced in January 2011.
-<!-- TODO: document when caching was added -->
+      Caching was implemented in February 2011.
+      Access control was introduced in June 2011.
 <!-- TODO: document when validation was added -->
     </para>
   </refsect1>

src/bin/resolver/main.cc

@@ -208,8 +208,7 @@ main(int argc, char* argv[]) {
         cc_session = new Session(io_service.get_io_service());
         config_session = new ModuleCCSession(specfile, *cc_session,
                                              my_config_handler,
-                                             my_command_handler,
-                                             true, true);
+                                             my_command_handler);
         LOG_DEBUG(resolver_logger, RESOLVER_DBG_INIT, RESOLVER_CONFIG_CHANNEL);
 
         // FIXME: This does not belong here, but inside Boss

src/bin/resolver/resolver.cc

@@ -26,7 +26,7 @@
 
 #include <exceptions/exceptions.h>
 
-#include <acl/acl.h>
+#include <acl/dns.h>
 #include <acl/loader.h>
 
 #include <asiodns/asiodns.h>
@@ -62,6 +62,7 @@ using boost::shared_ptr;
 using namespace isc;
 using namespace isc::util;
 using namespace isc::acl;
+using isc::acl::dns::RequestACL;
 using namespace isc::dns;
 using namespace isc::data;
 using namespace isc::config;
@@ -82,7 +83,9 @@ public:
         client_timeout_(4000),
         lookup_timeout_(30000),
         retries_(3),
-        query_acl_(new Resolver::ClientACL(REJECT)),
+        // we apply "reject all" (implicit default of the loader) ACL by
+        // default:
+        query_acl_(acl::dns::getRequestLoader().load(Element::fromJSON("[]"))),
         rec_query_(NULL)
     {}
 
@@ -160,11 +163,11 @@ public:
                                          OutputBufferPtr buffer,
                                          DNSServer* server);
 
-    const Resolver::ClientACL& getQueryACL() const {
+    const RequestACL& getQueryACL() const {
         return (*query_acl_);
     }
 
-    void setQueryACL(shared_ptr<const Resolver::ClientACL> new_acl) {
+    void setQueryACL(shared_ptr<const RequestACL> new_acl) {
         query_acl_ = new_acl;
     }
 
@@ -192,7 +195,7 @@ public:
 
 private:
     /// ACL on incoming queries
-    shared_ptr<const Resolver::ClientACL> query_acl_;
+    shared_ptr<const RequestACL> query_acl_;
 
     /// Object to handle upstream queries
     RecursiveQuery* rec_query_;
@@ -514,8 +517,11 @@ ResolverImpl::processNormalQuery(const IOMessage& io_message,
     const RRClass qclass = question->getClass();
 
     // Apply query ACL
-    Client client(io_message);
-    const BasicAction query_action(getQueryACL().execute(client));
+    const Client client(io_message);
+    const BasicAction query_action(
+        getQueryACL().execute(acl::dns::RequestContext(
+                                  client.getRequestSourceIPAddress(),
+                                  query_message->getTSIGRecord())));
     if (query_action == isc::acl::REJECT) {
         LOG_INFO(resolver_logger, RESOLVER_QUERY_REJECTED)
             .arg(question->getName()).arg(qtype).arg(qclass).arg(client);
@@ -574,32 +580,6 @@ ResolverImpl::processNormalQuery(const IOMessage& io_message,
     return (RECURSION);
 }
 
-namespace {
-// This is a simplified ACL parser for the initial implementation with minimal
-// external dependency.  For a longer term we'll switch to a more generic
-// loader with allowing more complicated ACL syntax.
-shared_ptr<const Resolver::ClientACL>
-createQueryACL(isc::data::ConstElementPtr acl_config) {
-    if (!acl_config) {
-        return (shared_ptr<const Resolver::ClientACL>());
-    }
-
-    shared_ptr<Resolver::ClientACL> new_acl(
-        new Resolver::ClientACL(REJECT));
-    BOOST_FOREACH(ConstElementPtr rule, acl_config->listValue()) {
-        ConstElementPtr action = rule->get("action");
-        ConstElementPtr from = rule->get("from");
-        if (!action || !from) {
-            isc_throw(BadValue, "query ACL misses mandatory parameter");
-        }
-        new_acl->append(shared_ptr<IPCheck<Client> >(
-                            new IPCheck<Client>(from->stringValue())),
-                        defaultActionLoader(action));
-    }
-    return (new_acl);
-}
-}
-
 ConstElementPtr
 Resolver::updateConfig(ConstElementPtr config) {
     LOG_DEBUG(resolver_logger, RESOLVER_DBG_CONFIG, RESOLVER_CONFIG_UPDATED)
@@ -616,8 +596,10 @@ Resolver::updateConfig(ConstElementPtr config) {
         ConstElementPtr listenAddressesE(config->get("listen_on"));
         AddressList listenAddresses(parseAddresses(listenAddressesE,
                                                       "listen_on"));
-        shared_ptr<const ClientACL> query_acl(createQueryACL(
-                                                  config->get("query_acl")));
+        const ConstElementPtr query_acl_cfg(config->get("query_acl"));
+        const shared_ptr<const RequestACL> query_acl =
+            query_acl_cfg ? acl::dns::getRequestLoader().load(query_acl_cfg) :
+            shared_ptr<RequestACL>();
         bool set_timeouts(false);
         int qtimeout = impl_->query_timeout_;
         int ctimeout = impl_->client_timeout_;
@@ -777,13 +759,13 @@ Resolver::getListenAddresses() const {
     return (impl_->listen_);
 }
 
-const Resolver::ClientACL&
+const RequestACL&
 Resolver::getQueryACL() const {
     return (impl_->getQueryACL());
 }
 
 void
-Resolver::setQueryACL(shared_ptr<const ClientACL> new_acl) {
+Resolver::setQueryACL(shared_ptr<const RequestACL> new_acl) {
     if (!new_acl) {
         isc_throw(InvalidParameter, "NULL pointer is passed to setQueryACL");
     }

src/bin/resolver/resolver.h

@@ -21,10 +21,9 @@
 
 #include <boost/shared_ptr.hpp>
 
-#include <acl/acl.h>
-
 #include <cc/data.h>
 #include <config/ccsession.h>
+#include <acl/dns.h>
 #include <dns/message.h>
 #include <util/buffer.h>
 
@@ -41,12 +40,6 @@
 
 #include <resolve/resolver_interface.h>
 
-namespace isc {
-namespace server_common {
-class Client;
-}
-}
-
 class ResolverImpl;
 
 /**
@@ -246,13 +239,10 @@ public:
      */
     int getRetries() const;
 
-    // Shortcut typedef used for query ACL.
-    typedef isc::acl::ACL<isc::server_common::Client> ClientACL;
-
     /// Get the query ACL.
     ///
     /// \exception None
-    const ClientACL& getQueryACL() const;
+    const isc::acl::dns::RequestACL& getQueryACL() const;
 
     /// Set the new query ACL.
     ///
@@ -265,7 +255,8 @@ public:
     /// \exception InvalidParameter The given pointer is NULL
     ///
     /// \param new_acl The new ACL to replace the existing one.
-    void setQueryACL(boost::shared_ptr<const ClientACL> new_acl);
+    void setQueryACL(boost::shared_ptr<const isc::acl::dns::RequestACL>
+                     new_acl);
 
 private:
     ResolverImpl* impl_;

src/bin/resolver/resolver_messages.mes

@@ -16,151 +16,174 @@
 # along with the resolver methods.
 
 % RESOLVER_AXFR_TCP AXFR request received over TCP
-A debug message, the resolver received a NOTIFY message over TCP.  The server
-cannot process it and will return an error message to the sender with the
-RCODE set to NOTIMP.
+This is a debug message output when the resolver received a request for
+an AXFR (full transfer of a zone) over TCP.  Only authoritative servers
+are able to handle AXFR requests, so the resolver will return an error
+message to the sender with the RCODE set to NOTIMP.
 
 % RESOLVER_AXFR_UDP AXFR request received over UDP
-A debug message, the resolver received a NOTIFY message over UDP.  The server
-cannot process it (and in any case, an AXFR request should be sent over TCP)
-and will return an error message to the sender with the RCODE set to FORMERR.
+This is a debug message output when the resolver received a request for
+an AXFR (full transfer of a zone) over UDP.  Only authoritative servers
+are able to handle AXFR requests (and in any case, an AXFR request should
+be sent over TCP), so the resolver will return an error message to the
+sender with the RCODE set to NOTIMP.
 
 % RESOLVER_CLIENT_TIME_SMALL client timeout of %1 is too small
-An error indicating that the configuration value specified for the query
-timeout is too small.
+During the update of the resolver's configuration parameters, the value
+of the client timeout was found to be too small.  The configuration
+update was abandoned and the parameters were not changed.
 
 % RESOLVER_CONFIG_CHANNEL configuration channel created
-A debug message, output when the resolver has successfully established a
-connection to the configuration channel.
+This is a debug message output when the resolver has successfully
+established a connection to the configuration channel.
 
 % RESOLVER_CONFIG_ERROR error in configuration: %1
-An error was detected in a configuration update received by the resolver. This
-may be in the format of the configuration message (in which case this is a
-programming error) or it may be in the data supplied (in which case it is
-a user error).  The reason for the error, given as a parameter in the message,
-will give more details.
+An error was detected in a configuration update received by the
+resolver. This may be in the format of the configuration message (in
+which case this is a programming error) or it may be in the data supplied
+(in which case it is a user error).  The reason for the error, included
+in the message, will give more details.  The configuration update is
+not applied and the resolver parameters were not changed.
 
 % RESOLVER_CONFIG_LOADED configuration loaded
-A debug message, output when the resolver configuration has been successfully
-loaded.
+This is a debug message output when the resolver configuration has been
+successfully loaded.
 
 % RESOLVER_CONFIG_UPDATED configuration updated: %1
-A debug message, the configuration has been updated with the specified
-information.
+This is a debug message output when the resolver configuration is being
+updated with the specified information.
 
 % RESOLVER_CREATED main resolver object created
-A debug message, output when the Resolver() object has been created.
+This is a debug message indicating that the main resolver object has
+been created.
 
 % RESOLVER_DNS_MESSAGE_RECEIVED DNS message received: %1
-A debug message, this always precedes some other logging message and is the
-formatted contents of the DNS packet that the other message refers to.
+This is a debug message from the resolver listing the contents of a
+received DNS message.
 
 % RESOLVER_DNS_MESSAGE_SENT DNS message of %1 bytes sent: %2
-A debug message, this contains details of the response sent back to the querying
-system.
+This is a debug message containing details of the response returned by
+the resolver to the querying system.
 
 % RESOLVER_FAILED resolver failed, reason: %1
-This is an error message output when an unhandled exception is caught by the
-resolver.  All it can do is to shut down.
+This is an error message output when an unhandled exception is caught
+by the resolver.  After this, the resolver will shut itself down.
+Please submit a bug report.
 
 % RESOLVER_FORWARD_ADDRESS setting forward address %1(%2)
-This message may appear multiple times during startup, and it lists the
-forward addresses used by the resolver when running in forwarding mode.
+If the resolver is running in forward mode, this message will appear
+during startup to list the forward address.  If multiple addresses are
+specified, it will appear once for each address.
 
 % RESOLVER_FORWARD_QUERY processing forward query
-The received query has passed all checks and is being forwarded to upstream
+This is a debug message indicating that a query received by the resolver
+has passed a set of checks (message is well-formed, it is allowed by the
+ACL, it is a supported opcode, etc.) and is being forwarded to upstream
 servers.
 
 % RESOLVER_HEADER_ERROR message received, exception when processing header: %1
-A debug message noting that an exception occurred during the processing of
-a received packet.  The packet has been dropped.
+This is a debug message from the resolver noting that an exception
+occurred during the processing of a received packet.  The packet has
+been dropped.
 
 % RESOLVER_IXFR IXFR request received
-The resolver received a NOTIFY message over TCP.  The server cannot process it
-and will return an error message to the sender with the RCODE set to NOTIMP.
+This is a debug message indicating that the resolver received a request
+for an IXFR (incremental transfer of a zone).  Only authoritative servers
+are able to handle IXFR requests, so the resolver will return an error
+message to the sender with the RCODE set to NOTIMP.
 
 % RESOLVER_LOOKUP_TIME_SMALL lookup timeout of %1 is too small
-An error indicating that the configuration value specified for the lookup
-timeout is too small.
+During the update of the resolver's configuration parameters, the value
+of the lookup timeout was found to be too small.  The configuration
+update will not be applied.
 
 % RESOLVER_MESSAGE_ERROR error parsing received message: %1 - returning %2
-A debug message noting that the resolver received a message and the
-parsing of the body of the message failed due to some error (although
-the parsing of the header succeeded).  The message parameters give a
-textual description of the problem and the RCODE returned.
+This is a debug message noting that parsing of the body of a received
+message by the resolver failed due to some error (although the parsing of
+the header succeeded).  The message parameters give a textual description
+of the problem and the RCODE returned.
 
 % RESOLVER_NEGATIVE_RETRIES negative number of retries (%1) specified in the configuration
-An error message indicating that the resolver configuration has specified a
-negative retry count.  Only zero or positive values are valid.
+This error is issued when a resolver configuration update has specified
+a negative retry count: only zero or positive values are valid.  The
+configuration update was abandoned and the parameters were not changed.
 
 % RESOLVER_NON_IN_PACKET non-IN class request received, returning REFUSED message
-A debug message, the resolver has received a DNS packet that was not IN class.
-The resolver cannot handle such packets, so is returning a REFUSED response to
-the sender.
+This debug message is issued when resolver has received a DNS packet that
+was not IN (Internet) class.  The resolver cannot handle such packets,
+so is returning a REFUSED response to the sender.
 
 % RESOLVER_NORMAL_QUERY processing normal query
-The received query has passed all checks and is being processed by the resolver.
+This is a debug message indicating that the query received by the resolver
+has passed a set of checks (message is well-formed, it is allowed by the
+ACL, it is a supported opcode, etc.) and is being processed by the resolver.
 
 % RESOLVER_NOTIFY_RECEIVED NOTIFY arrived but server is not authoritative
-The resolver received a NOTIFY message.  As the server is not authoritative it
-cannot process it, so it returns an error message to the sender with the RCODE
-set to NOTAUTH.
+The resolver has received a NOTIFY message.  As the server is not
+authoritative it cannot process it, so it returns an error message to
+the sender with the RCODE set to NOTAUTH.
 
 % RESOLVER_NOT_ONE_QUESTION query contained %1 questions, exactly one question was expected
-A debug message, the resolver received a query that contained the number of
-entires in the question section detailed in the message.  This is a malformed
-message, as a DNS query must contain only one question.  The resolver will
-return a message to the sender with the RCODE set to FORMERR.
+This debug message indicates that the resolver received a query that
+contained the number of entries in the question section detailed in
+the message.  This is a malformed message, as a DNS query must contain
+only one question.  The resolver will return a message to the sender
+with the RCODE set to FORMERR.
 
 % RESOLVER_NO_ROOT_ADDRESS no root addresses available
-A warning message during startup, indicates that no root addresses have been
-set.  This may be because the resolver will get them from a priming query.
+A warning message issued during resolver startup, this indicates that
+no root addresses have been set.  This may be because the resolver will
+get them from a priming query.
 
 % RESOLVER_PARSE_ERROR error parsing received message: %1 - returning %2
-A debug message noting that the resolver received a message and the parsing
-of the body of the message failed due to some non-protocol related reason
-(although the parsing of the header succeeded).  The message parameters give
-a textual description of the problem and the RCODE returned.
+This is a debug message noting that the resolver received a message and
+the parsing of the body of the message failed due to some non-protocol
+related reason (although the parsing of the header succeeded).
+The message parameters give a textual description of the problem and
+the RCODE returned.
 
 % RESOLVER_PRINT_COMMAND print message command, arguments are: %1
-This message is logged when a "print_message" command is received over the
-command channel.
+This debug message is logged when a "print_message" command is received
+by the resolver over the command channel.
 
 % RESOLVER_PROTOCOL_ERROR protocol error parsing received message: %1 - returning %2
-A debug message noting that the resolver received a message and the parsing
-of the body of the message failed due to some protocol error (although the
-parsing of the header succeeded).  The message parameters give a textual
-description of the problem and the RCODE returned.
+This is a debug message noting that the resolver received a message and
+the parsing of the body of the message failed due to some protocol error
+(although the parsing of the header succeeded).  The message parameters
+give a textual description of the problem and the RCODE returned.
 
 % RESOLVER_QUERY_SETUP query setup
-A debug message noting that the resolver is creating a RecursiveQuery object.
+This is a debug message noting that the resolver is creating a
+RecursiveQuery object.
 
 % RESOLVER_QUERY_SHUTDOWN query shutdown
-A debug message noting that the resolver is destroying a RecursiveQuery object.
+This is a debug message noting that the resolver is destroying a
+RecursiveQuery object.
 
 % RESOLVER_QUERY_TIME_SMALL query timeout of %1 is too small
-An error indicating that the configuration value specified for the query
-timeout is too small.
+During the update of the resolver's configuration parameters, the value
+of the query timeout was found to be too small.  The configuration
+parameters were not changed.
 
 % RESOLVER_RECEIVED_MESSAGE resolver has received a DNS message
-A debug message indicating that the resolver has received a message.  Depending
-on the debug settings, subsequent log output will indicate the nature of the
-message.
+This is a debug message indicating that the resolver has received a
+DNS message.  Depending on the debug settings, subsequent log output
+will indicate the nature of the message.
 
 % RESOLVER_RECURSIVE running in recursive mode
-This is an informational message that appears at startup noting that the
-resolver is running in recursive mode.
+This is an informational message that appears at startup noting that
+the resolver is running in recursive mode.
 
 % RESOLVER_SERVICE_CREATED service object created
-A debug message, output when the main service object (which handles the
-received queries) is created.
+This debug message is output when resolver creates the main service object
+(which handles the received queries).
 
 % RESOLVER_SET_PARAMS query timeout: %1, client timeout: %2, lookup timeout: %3, retry count: %4
-A debug message, lists the parameters being set for the resolver.  These are:
+This debug message lists the parameters being set for the resolver.  These are:
 query timeout: the timeout (in ms) used for queries originated by the resolver
-to upstream servers.  Client timeout: the interval to resolver a query by
+to upstream servers.  Client timeout: the interval to resolve a query by
 a client: after this time, the resolver sends back a SERVFAIL to the client
-whilst continuing to resolver the query. Lookup timeout: the time at which the
+whilst continuing to resolve the query. Lookup timeout: the time at which the
 resolver gives up trying to resolve a query.  Retry count: the number of times
 the resolver will retry a query to an upstream server if it gets a timeout.
 
@@ -169,17 +192,18 @@ resolution of the client query might require a large number of queries to
 upstream nameservers.  Even if none of these queries timeout, the total time
 taken to perform all the queries may exceed the client timeout.  When this
 happens, a SERVFAIL is returned to the client, but the resolver continues
-with the resolution process. Data received is added to the cache.  However,
+with the resolution process; data received is added to the cache.  However,
 there comes a time - the lookup timeout - when even the resolver gives up.
 At this point it will wait for pending upstream queries to complete or
 timeout and drop the query.
 
 % RESOLVER_SET_ROOT_ADDRESS setting root address %1(%2)
-This message may appear multiple times during startup; it lists the root
-addresses used by the resolver.
+This message gives the address of one of the root servers used by the
+resolver.  It is output during startup and may appear multiple times,
+once for each root server address.
 
 % RESOLVER_SHUTDOWN resolver shutdown complete
-This information message is output when the resolver has shut down.
+This informational message is output when the resolver has shut down.
 
 % RESOLVER_STARTED resolver started
 This informational message is output by the resolver when all initialization
@@ -189,31 +213,36 @@ has been completed and it is entering its main loop.
 An informational message, this is output when the resolver starts up.
 
 % RESOLVER_UNEXPECTED_RESPONSE received unexpected response, ignoring
-A debug message noting that the server has received a response instead of a
-query and is ignoring it.
+This is a debug message noting that the resolver received a DNS response
+packet on the port on which is it listening for queries.  The packet
+has been ignored.
 
 % RESOLVER_UNSUPPORTED_OPCODE opcode %1 not supported by the resolver
-A debug message, the resolver received a message with an unsupported opcode
-(it can only process QUERY opcodes).  It will return a message to the sender
-with the RCODE set to NOTIMP.
-
-% RESOLVER_SET_QUERY_ACL   query ACL is configured
-A debug message that appears when a new query ACL is configured for the
-resolver.
-
-% RESOLVER_QUERY_ACCEPTED   query accepted: '%1/%2/%3' from %4
-A debug message that indicates an incoming query is accepted in terms of
-the query ACL.  The log message shows the query in the form of
-<query name>/<query type>/<query class>, and the client that sends the
-query in the form of <Source IP address>#<source port>.
-
-% RESOLVER_QUERY_REJECTED   query rejected: '%1/%2/%3' from %4
-An informational message that indicates an incoming query is rejected
-in terms of the query ACL.  This results in a response with an RCODE of
-REFUSED.  See QUERYACCEPTED for the information given in the message.
-
-% RESOLVER_QUERY_DROPPED    query dropped: '%1/%2/%3' from %4
-An informational message that indicates an incoming query is dropped
-in terms of the query ACL.  Unlike the QUERYREJECTED case, the server does
-not return any response.  See QUERYACCEPTED for the information given in
-the message.
+This is debug message output when the resolver received a message with an
+unsupported opcode (it can only process QUERY opcodes).  It will return
+a message to the sender with the RCODE set to NOTIMP.
+
+% RESOLVER_SET_QUERY_ACL query ACL is configured
+This debug message is generated when a new query ACL is configured for
+the resolver.
+
+% RESOLVER_QUERY_ACCEPTED query accepted: '%1/%2/%3' from %4
+This debug message is produced by the resolver when an incoming query
+is accepted in terms of the query ACL.  The log message shows the query
+in the form of <query name>/<query type>/<query class>, and the client
+that sends the query in the form of <Source IP address>#<source port>.
+
+% RESOLVER_QUERY_REJECTED query rejected: '%1/%2/%3' from %4
+This is an informational message that indicates an incoming query has
+been rejected by the resolver because of the query ACL.  This results
+in a response with an RCODE of REFUSED. The log message shows the query
+in the form of <query name>/<query type>/<query class>, and the client
+that sends the query in the form of <Source IP address>#<source port>.
+
+% RESOLVER_QUERY_DROPPED query dropped: '%1/%2/%3' from %4
+This is an informational message that indicates an incoming query has
+been dropped by the resolver because of the query ACL.  Unlike the
+RESOLVER_QUERY_REJECTED case, the server does not return any response.
+The log message shows the query in the form of <query name>/<query
+type>/<query class>, and the client that sends the query in the form of
+<Source IP address>#<source port>.

src/bin/resolver/tests/Makefile.am

@@ -39,6 +39,7 @@ run_unittests_LDADD += $(top_builddir)/src/lib/dns/libdns++.la
 run_unittests_LDADD += $(top_builddir)/src/lib/asiodns/libasiodns.la
 run_unittests_LDADD += $(top_builddir)/src/lib/asiolink/libasiolink.la
 run_unittests_LDADD += $(top_builddir)/src/lib/config/libcfgclient.la
+run_unittests_LDADD += $(top_builddir)/src/lib/acl/libdnsacl.la
 run_unittests_LDADD += $(top_builddir)/src/lib/cc/libcc.la
 run_unittests_LDADD += $(top_builddir)/src/lib/exceptions/libexceptions.la
 run_unittests_LDADD += $(top_builddir)/src/lib/xfr/libxfr.la

src/bin/resolver/tests/resolver_config_unittest.cc

@@ -43,6 +43,7 @@
 using namespace std;
 using boost::scoped_ptr;
 using namespace isc::acl;
+using isc::acl::dns::RequestContext;
 using namespace isc::data;
 using namespace isc::testutils;
 using namespace isc::asiodns;
@@ -57,19 +58,23 @@ protected:
     DNSService dnss;
     Resolver server;
     scoped_ptr<const IOEndpoint> endpoint;
-    scoped_ptr<const IOMessage> request;
+    scoped_ptr<const IOMessage> query_message;
     scoped_ptr<const Client> client;
+    scoped_ptr<const RequestContext> request;
     ResolverConfig() : dnss(ios, NULL, NULL, NULL) {
         server.setDNSService(dnss);
         server.setConfigured();
     }
-    const Client& createClient(const string& source_addr) {
+    const RequestContext& createRequest(const string& source_addr) {
         endpoint.reset(IOEndpoint::create(IPPROTO_UDP, IOAddress(source_addr),
                                           53210));
-        request.reset(new IOMessage(NULL, 0, IOSocket::getDummyUDPSocket(),
-                                    *endpoint));
-        client.reset(new Client(*request));
-        return (*client);
+        query_message.reset(new IOMessage(NULL, 0,
+                                          IOSocket::getDummyUDPSocket(),
+                                          *endpoint));
+        client.reset(new Client(*query_message));
+        request.reset(new RequestContext(client->getRequestSourceIPAddress(),
+                                         NULL));
+        return (*request);
     }
     void invalidTest(const string &JSON, const string& name);
 };
@@ -100,14 +105,14 @@ TEST_F(ResolverConfig, forwardAddresses) {
 
 TEST_F(ResolverConfig, forwardAddressConfig) {
     // Try putting there some address
-    ElementPtr config(Element::fromJSON("{"
-        "\"forward_addresses\": ["
-        "   {"
-        "       \"address\": \"192.0.2.1\","
-        "       \"port\": 53"
-        "   }"
-        "]"
-        "}"));
+    ConstElementPtr config(Element::fromJSON("{"
+                                             "\"forward_addresses\": ["
+                                             " {"
+                                             "   \"address\": \"192.0.2.1\","
+                                             "   \"port\": 53"
+                                             " }"
+                                             "]"
+                                             "}"));
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
     EXPECT_TRUE(server.isForwarding());
@@ -127,14 +132,14 @@ TEST_F(ResolverConfig, forwardAddressConfig) {
 
 TEST_F(ResolverConfig, rootAddressConfig) {
     // Try putting there some address
-    ElementPtr config(Element::fromJSON("{"
-        "\"root_addresses\": ["
-        "   {"
-        "       \"address\": \"192.0.2.1\","
-        "       \"port\": 53"
-        "   }"
-        "]"
-        "}"));
+    ConstElementPtr config(Element::fromJSON("{"
+                                             "\"root_addresses\": ["
+                                             " {"
+                                             "    \"address\": \"192.0.2.1\","
+                                             "    \"port\": 53"
+                                             " }"
+                                             "]"
+                                             "}"));
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
     ASSERT_EQ(1, server.getRootAddresses().size());
@@ -210,12 +215,12 @@ TEST_F(ResolverConfig, timeouts) {
 }
 
 TEST_F(ResolverConfig, timeoutsConfig) {
-    ElementPtr config = Element::fromJSON("{"
-            "\"timeout_query\": 1000,"
-            "\"timeout_client\": 2000,"
-            "\"timeout_lookup\": 3000,"
-            "\"retries\": 4"
-            "}");
+    ConstElementPtr config = Element::fromJSON("{"
+                                               "\"timeout_query\": 1000,"
+                                               "\"timeout_client\": 2000,"
+                                               "\"timeout_lookup\": 3000,"
+                                               "\"retries\": 4"
+                                               "}");
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
     EXPECT_EQ(1000, server.getQueryTimeout());
@@ -253,51 +258,51 @@ TEST_F(ResolverConfig, invalidTimeoutsConfig) {
 
 TEST_F(ResolverConfig, defaultQueryACL) {
     // If no configuration is loaded, the default ACL should reject everything.
-    EXPECT_EQ(REJECT, server.getQueryACL().execute(createClient("192.0.2.1")));
+    EXPECT_EQ(REJECT, server.getQueryACL().execute(createRequest("192.0.2.1")));
     EXPECT_EQ(REJECT, server.getQueryACL().execute(
-                  createClient("2001:db8::1")));
+                  createRequest("2001:db8::1")));
 
     // The following would be allowed if the server had loaded the default
     // configuration from the spec file.  In this context it should not have
     // happened, and they should be rejected just like the above cases.
-    EXPECT_EQ(REJECT, server.getQueryACL().execute(createClient("127.0.0.1")));
-    EXPECT_EQ(REJECT, server.getQueryACL().execute(createClient("::1")));
+    EXPECT_EQ(REJECT, server.getQueryACL().execute(createRequest("127.0.0.1")));
+    EXPECT_EQ(REJECT, server.getQueryACL().execute(createRequest("::1")));
 }
 
 TEST_F(ResolverConfig, emptyQueryACL) {
     // Explicitly configured empty ACL should have the same effect.
-    ElementPtr config(Element::fromJSON("{ \"query_acl\": [] }"));
+    ConstElementPtr config(Element::fromJSON("{ \"query_acl\": [] }"));
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
-    EXPECT_EQ(REJECT, server.getQueryACL().execute(createClient("192.0.2.1")));
+    EXPECT_EQ(REJECT, server.getQueryACL().execute(createRequest("192.0.2.1")));
     EXPECT_EQ(REJECT, server.getQueryACL().execute(
-                  createClient("2001:db8::1")));
+                  createRequest("2001:db8::1")));
 }
 
 TEST_F(ResolverConfig, queryACLIPv4) {
     // A simple "accept" query for a specific IPv4 address
-    ElementPtr config(Element::fromJSON(
-                          "{ \"query_acl\": "
-                          "  [ {\"action\": \"ACCEPT\","
-                          "     \"from\": \"192.0.2.1\"} ] }"));
+    ConstElementPtr config(Element::fromJSON(
+                               "{ \"query_acl\": "
+                               "  [ {\"action\": \"ACCEPT\","
+                               "     \"from\": \"192.0.2.1\"} ] }"));
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
-    EXPECT_EQ(ACCEPT, server.getQueryACL().execute(createClient("192.0.2.1")));
+    EXPECT_EQ(ACCEPT, server.getQueryACL().execute(createRequest("192.0.2.1")));
     EXPECT_EQ(REJECT, server.getQueryACL().execute(
-                  createClient("2001:db8::1")));
+                  createRequest("2001:db8::1")));
 }
 
 TEST_F(ResolverConfig, queryACLIPv6) {
     // same for IPv6
-    ElementPtr config(Element::fromJSON(
-                          "{ \"query_acl\": "
-                          "  [ {\"action\": \"ACCEPT\","
-                          "     \"from\": \"2001:db8::1\"} ] }"));
+    ConstElementPtr config(Element::fromJSON(
+                               "{ \"query_acl\": "
+                               "  [ {\"action\": \"ACCEPT\","
+                               "     \"from\": \"2001:db8::1\"} ] }"));
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
-    EXPECT_EQ(REJECT, server.getQueryACL().execute(createClient("192.0.2.1")));
+    EXPECT_EQ(REJECT, server.getQueryACL().execute(createRequest("192.0.2.1")));
     EXPECT_EQ(ACCEPT, server.getQueryACL().execute(
-                  createClient("2001:db8::1")));
+                  createRequest("2001:db8::1")));
 }
 
 TEST_F(ResolverConfig, multiEntryACL) {
@@ -306,25 +311,26 @@ TEST_F(ResolverConfig, multiEntryACL) {
     // as it should have been tested in the underlying ACL module.  All we
     // have to do to check is a reasonably complicated ACL configuration is
     // loaded as expected.
-    ElementPtr config(Element::fromJSON(
-                          "{ \"query_acl\": "
-                          "  [ {\"action\": \"ACCEPT\","
-                          "     \"from\": \"192.0.2.1\"},"
-                          "    {\"action\": \"REJECT\","
-                          "     \"from\": \"192.0.2.0/24\"},"
-                          "    {\"action\": \"DROP\","
-                          "     \"from\": \"2001:db8::1\"},"
-                          "] }"));
+    ConstElementPtr config(Element::fromJSON(
+                               "{ \"query_acl\": "
+                               "  [ {\"action\": \"ACCEPT\","
+                               "     \"from\": \"192.0.2.1\"},"
+                               "    {\"action\": \"REJECT\","
+                               "     \"from\": \"192.0.2.0/24\"},"
+                               "    {\"action\": \"DROP\","
+                               "     \"from\": \"2001:db8::1\"},"
+                               "] }"));
     ConstElementPtr result(server.updateConfig(config));
     EXPECT_EQ(result->toWire(), isc::config::createAnswer()->toWire());
-    EXPECT_EQ(ACCEPT, server.getQueryACL().execute(createClient("192.0.2.1")));
-    EXPECT_EQ(REJECT, server.getQueryACL().execute(createClient("192.0.2.2")));
+    EXPECT_EQ(ACCEPT, server.getQueryACL().execute(createRequest("192.0.2.1")));
+    EXPECT_EQ(REJECT, server.getQueryACL().execute(createRequest("192.0.2.2")));
     EXPECT_EQ(DROP, server.getQueryACL().execute(
-                  createClient("2001:db8::1")));
+                  createRequest("2001:db8::1")));
     EXPECT_EQ(REJECT, server.getQueryACL().execute(
-                  createClient("2001:db8::2"))); // match the default rule
+                  createRequest("2001:db8::2"))); // match the default rule
 }
 
+
 int
 getResultCode(ConstElementPtr result) {
     int rcode;
@@ -332,6 +338,22 @@ getResultCode(ConstElementPtr result) {
     return (rcode);
 }
 
+TEST_F(ResolverConfig, queryACLActionOnly) {
+    // "action only" rule will be accepted by the loader, which can
+    // effectively change the default action.
+    ConstElementPtr config(Element::fromJSON(
+                               "{ \"query_acl\": "
+                               "  [ {\"action\": \"ACCEPT\","
+                               "     \"from\": \"192.0.2.1\"},"
+                               "    {\"action\": \"DROP\"} ] }"));
+    EXPECT_EQ(0, getResultCode(server.updateConfig(config)));
+    EXPECT_EQ(ACCEPT, server.getQueryACL().execute(createRequest("192.0.2.1")));
+
+    // We reject non matching queries by default, but the last resort
+    // rule should have changed the action in that case to "DROP".
+    EXPECT_EQ(DROP, server.getQueryACL().execute(createRequest("192.0.2.2")));
+}
+
 TEST_F(ResolverConfig, badQueryACL) {
     // Most of these cases shouldn't happen in practice because the syntax
     // check should be performed before updateConfig().  But we check at
@@ -346,10 +368,6 @@ TEST_F(ResolverConfig, badQueryACL) {
                   server.updateConfig(
                       Element::fromJSON("{ \"query_acl\":"
                                         " [ {\"from\": \"192.0.2.1\"} ] }"))));
-    EXPECT_EQ(1, getResultCode(
-                  server.updateConfig(
-                      Element::fromJSON("{ \"query_acl\":"
-                                        " [ {\"action\": \"DROP\"} ] }"))));
     // invalid "action"
     EXPECT_EQ(1, getResultCode(
                   server.updateConfig(
@@ -361,7 +379,6 @@ TEST_F(ResolverConfig, badQueryACL) {
                       Element::fromJSON("{ \"query_acl\":"
                                         " [ {\"action\": \"BADACTION\","
                                         "    \"from\": \"192.0.2.1\"}]}"))));
-
     // invalid "from"
     EXPECT_EQ(1, getResultCode(
                   server.updateConfig(

src/bin/resolver/tests/resolver_unittest.cc

@@ -27,6 +27,7 @@
 using namespace std;
 using namespace isc::dns;
 using namespace isc::data;
+using isc::acl::dns::RequestACL;
 using namespace isc::testutils;
 using isc::UnitTestUtil;
 
@@ -156,8 +157,7 @@ TEST_F(ResolverTest, notifyFail) {
 TEST_F(ResolverTest, setQueryACL) {
     // valid cases are tested through other tests.  We only explicitly check
     // an invalid case: passing a NULL shared pointer.
-    EXPECT_THROW(server.setQueryACL(
-                     boost::shared_ptr<const Resolver::ClientACL>()),
+    EXPECT_THROW(server.setQueryACL(boost::shared_ptr<const RequestACL>()),
                  isc::InvalidParameter);
 }
 

src/bin/sockcreator/README

@@ -3,7 +3,7 @@ The socket creator
 
 The only thing we need higher rights than standard user is binding sockets to
 ports lower than 1024. So we will have a separate process that keeps the
-rights, while the rests drop them for security reasons.
+rights, while the rest drops them for security reasons.
 
 This process is the socket creator. Its goal is to be as simple as possible
 and to contain as little code as possible to minimise the amount of code

src/bin/stats/Makefile.am

@@ -5,16 +5,25 @@ pkglibexecdir = $(libexecdir)/@PACKAGE@
 pkglibexec_SCRIPTS = b10-stats b10-stats-httpd
 
 b10_statsdir = $(pkgdatadir)
-b10_stats_DATA = stats.spec stats-httpd.spec stats-schema.spec
+b10_stats_DATA = stats.spec stats-httpd.spec
 b10_stats_DATA += stats-httpd-xml.tpl stats-httpd-xsd.tpl stats-httpd-xsl.tpl
 
+nodist_pylogmessage_PYTHON = $(PYTHON_LOGMSGPKG_DIR)/work/stats_messages.py
+nodist_pylogmessage_PYTHON += $(PYTHON_LOGMSGPKG_DIR)/work/stats_httpd_messages.py
+pylogmessagedir = $(pyexecdir)/isc/log_messages/
+
 CLEANFILES = b10-stats stats.pyc
 CLEANFILES += b10-stats-httpd stats_httpd.pyc
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/stats_messages.py
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/stats_messages.pyc
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/stats_httpd_messages.py
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/stats_httpd_messages.pyc
 
 man_MANS = b10-stats.8 b10-stats-httpd.8
 EXTRA_DIST = $(man_MANS) b10-stats.xml b10-stats-httpd.xml
-EXTRA_DIST += stats.spec stats-httpd.spec stats-schema.spec
+EXTRA_DIST += stats.spec stats-httpd.spec
 EXTRA_DIST += stats-httpd-xml.tpl stats-httpd-xsd.tpl stats-httpd-xsl.tpl
+EXTRA_DIST += stats_messages.mes stats_httpd_messages.mes
 
 if ENABLE_MAN
 
@@ -26,12 +35,20 @@ b10-stats-httpd.8: b10-stats-httpd.xml
 
 endif
 
+$(PYTHON_LOGMSGPKG_DIR)/work/stats_messages.py : stats_messages.mes
+	$(top_builddir)/src/lib/log/compiler/message \
+	-d $(PYTHON_LOGMSGPKG_DIR)/work -p $(srcdir)/stats_messages.mes
+
+$(PYTHON_LOGMSGPKG_DIR)/work/stats_httpd_messages.py : stats_httpd_messages.mes
+	$(top_builddir)/src/lib/log/compiler/message \
+	-d $(PYTHON_LOGMSGPKG_DIR)/work -p $(srcdir)/stats_httpd_messages.mes
+
 # this is done here since configure.ac AC_OUTPUT doesn't expand exec_prefix
-b10-stats: stats.py
+b10-stats: stats.py $(PYTHON_LOGMSGPKG_DIR)/work/stats_messages.py
 	$(SED) -e "s|@@PYTHONPATH@@|@pyexecdir@|"  stats.py >$@
 	chmod a+x $@
 
-b10-stats-httpd: stats_httpd.py
+b10-stats-httpd: stats_httpd.py $(PYTHON_LOGMSGPKG_DIR)/work/stats_httpd_messages.py
 	$(SED) -e "s|@@PYTHONPATH@@|@pyexecdir@|" stats_httpd.py >$@
 	chmod a+x $@
 

src/bin/stats/b10-stats-httpd.xml

@@ -57,7 +57,7 @@
       by the BIND 10 boss process (<command>bind10</command>) and eventually
       exited by it.  The server is intended to be server requests by HTTP
       clients like web browsers and third-party modules. When the server is
-      asked, it requests BIND 10 statistics data from
+      asked, it requests BIND 10 statistics data or its schema from
       <command>b10-stats</command>, and it sends the data back in Python
       dictionary format and the server converts it into XML format. The server
       sends it to the HTTP client. The server can send three types of document,
@@ -112,12 +112,6 @@
       of <refentrytitle>bindctl</refentrytitle><manvolnum>1</manvolnum> about
       how to configure the settings.
     </para>
-    <para><filename>/usr/local/share/bind10-devel/stats-schema.spec</filename>
-      <!--TODO: The filename should be computed from prefix-->
-      &mdash; This is a spec file for data schema of
-      of BIND 10 statistics. This schema cannot be configured 
-      via <refentrytitle>bindctl</refentrytitle><manvolnum>1</manvolnum>.
-    </para>
     <para>
       <filename>/usr/local/share/bind10-devel/stats-httpd-xml.tpl</filename>
       <!--TODO: The filename should be computed from prefix-->
@@ -138,7 +132,7 @@
   <refsect1>
     <title>CONFIGURATION AND COMMANDS</title>
     <para>
-      The configurable setting in 
+      The configurable setting in
       <filename>stats-httpd.spec</filename> is:
     </para>
     <variablelist>

src/bin/stats/b10-stats.8

@@ -1,22 +1,13 @@
 '\" t
 .\"     Title: b10-stats
 .\"    Author: [FIXME: author] [see http://docbook.sf.net/el/author]
-.\" Generator: DocBook XSL Stylesheets v1.76.1 <http://docbook.sf.net/>
-.\"      Date: Oct 15, 2010
+.\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
+.\"      Date: August 11, 2011
 .\"    Manual: BIND10
 .\"    Source: BIND10
 .\"  Language: English
 .\"
-.TH "B10\-STATS" "8" "Oct 15, 2010" "BIND10" "BIND10"
-.\" -----------------------------------------------------------------
-.\" * Define some portability stuff
-.\" -----------------------------------------------------------------
-.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.\" http://bugs.debian.org/507673
-.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html
-.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.ie \n(.g .ds Aq \(aq
-.el       .ds Aq '
+.TH "B10\-STATS" "8" "August 11, 2011" "BIND10" "BIND10"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -45,9 +36,9 @@ with other modules like
 \fBb10\-auth\fR
 and so on\&. It waits for coming data from other modules, then other modules send data to stats module periodically\&. Other modules send stats data to stats module independently from implementation of stats module, so the frequency of sending data may not be constant\&. Stats module collects data and aggregates it\&.
 \fBb10\-stats\fR
-invokes "sendstats" command for
+invokes an internal command for
 \fBbind10\fR
-after its initial starting because it\*(Aqs sure to collect statistics data from
+after its initial starting because it\'s sure to collect statistics data from
 \fBbind10\fR\&.
 .SH "OPTIONS"
 .PP
@@ -59,6 +50,84 @@ This
 \fBb10\-stats\fR
 switches to verbose mode\&. It sends verbose messages to STDOUT\&.
 .RE
+.SH "CONFIGURATION AND COMMANDS"
+.PP
+The
+\fBb10\-stats\fR
+command does not have any configurable settings\&.
+.PP
+The configuration commands are:
+.PP
+
+
+\fBremove\fR
+removes the named statistics name and data\&.
+.PP
+
+
+\fBreset\fR
+will reset all statistics data to default values except for constant names\&. This may re\-add previously removed statistics names\&.
+.PP
+
+\fBset\fR
+.PP
+
+\fBshow\fR
+will send the statistics data in JSON format\&. By default, it outputs all the statistics data it has collected\&. An optional item name may be specified to receive individual output\&.
+.PP
+
+\fBshutdown\fR
+will shutdown the
+\fBb10\-stats\fR
+process\&. (Note that the
+\fBbind10\fR
+parent may restart it\&.)
+.PP
+
+\fBstatus\fR
+simply indicates that the daemon is running\&.
+.SH "STATISTICS DATA"
+.PP
+The
+\fBb10\-stats\fR
+daemon contains these statistics:
+.PP
+report_time
+.RS 4
+The latest report date and time in ISO 8601 format\&.
+.RE
+.PP
+stats\&.boot_time
+.RS 4
+The date and time when this daemon was started in ISO 8601 format\&. This is a constant which can\'t be reset except by restarting
+\fBb10\-stats\fR\&.
+.RE
+.PP
+stats\&.last_update_time
+.RS 4
+The date and time (in ISO 8601 format) when this daemon last received data from another component\&.
+.RE
+.PP
+stats\&.lname
+.RS 4
+This is the name used for the
+\fBb10\-msgq\fR
+command\-control channel\&. (This is a constant which can\'t be reset except by restarting
+\fBb10\-stats\fR\&.)
+.RE
+.PP
+stats\&.start_time
+.RS 4
+This is the date and time (in ISO 8601 format) when this daemon started collecting data\&.
+.RE
+.PP
+stats\&.timestamp
+.RS 4
+The current date and time represented in seconds since UNIX epoch (1970\-01\-01T0 0:00:00Z) with precision (delimited with a period) up to one hundred thousandth of second\&.
+.RE
+.PP
+See other manual pages for explanations for their statistics that are kept track by
+\fBb10\-stats\fR\&.
 .SH "FILES"
 .PP
 /usr/local/share/bind10\-devel/stats\&.spec
@@ -66,10 +135,6 @@ switches to verbose mode\&. It sends verbose messages to STDOUT\&.
 \fBb10\-stats\fR\&. It contains commands for
 \fBb10\-stats\fR\&. They can be invoked via
 bindctl(1)\&.
-.PP
-/usr/local/share/bind10\-devel/stats\-schema\&.spec
-\(em This is a spec file for data schema of of BIND 10 statistics\&. This schema cannot be configured via
-bindctl(1)\&.
 .SH "SEE ALSO"
 .PP
 
@@ -82,7 +147,7 @@ BIND 10 Guide\&.
 .PP
 The
 \fBb10\-stats\fR
-daemon was initially designed and implemented by Naoki Kambe of JPRS in Oct 2010\&.
+daemon was initially designed and implemented by Naoki Kambe of JPRS in October 2010\&.
 .SH "COPYRIGHT"
 .br
 Copyright \(co 2010 Internet Systems Consortium, Inc. ("ISC")

src/bin/stats/b10-stats.xml

@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>Oct 15, 2010</date>
+    <date>August 11, 2011</date>
   </refentryinfo>
 
   <refmeta>
@@ -64,9 +64,10 @@
       send stats data to stats module independently from
       implementation of stats module, so the frequency of sending data
       may not be constant. Stats module collects data and aggregates
-      it. <command>b10-stats</command> invokes "sendstats" command
+      it. <command>b10-stats</command> invokes an internal command
       for <command>bind10</command> after its initial starting because it's
       sure to collect statistics data from <command>bind10</command>.
+<!-- TODO: reword that last sentence? -->
     </para>
   </refsect1>
 
@@ -87,6 +88,123 @@
   </refsect1>
 
   <refsect1>
+    <title>CONFIGURATION AND COMMANDS</title>
+
+    <para>
+      The <command>b10-stats</command> command does not have any
+      configurable settings.
+    </para>
+
+<!-- TODO: formating -->
+    <para>
+      The configuration commands are:
+    </para>
+
+    <para>
+<!-- TODO: remove is removed in trac930 -->
+      <command>remove</command> removes the named statistics name and data.
+    </para>
+
+    <para>
+<!-- TODO: reset is removed in trac930 -->
+      <command>reset</command> will reset all statistics data to
+      default values except for constant names.
+      This may re-add previously removed statistics names.
+    </para>
+
+    <para>
+      <command>set</command>
+<!-- TODO: document this -->
+    </para>
+
+    <para>
+      <command>show</command> will send the statistics data
+      in JSON format.
+      By default, it outputs all the statistics data it has collected.
+      An optional item name may be specified to receive individual output.
+    </para>
+
+<!-- TODO: document showschema -->
+
+    <para>
+      <command>shutdown</command> will shutdown the
+      <command>b10-stats</command> process.
+      (Note that the <command>bind10</command> parent may restart it.)
+    </para>
+
+    <para>
+      <command>status</command> simply indicates that the daemon is
+      running.
+    </para>
+
+  </refsect1>
+
+  <refsect1>
+    <title>STATISTICS DATA</title>
+
+    <para>
+      The <command>b10-stats</command> daemon contains these statistics:
+    </para>
+
+    <variablelist>
+
+      <varlistentry>
+        <term>report_time</term>
+<!-- TODO: why not named stats.report_time? -->
+        <listitem><simpara>The latest report date and time in
+          ISO 8601 format.</simpara></listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>stats.boot_time</term>
+        <listitem><simpara>The date and time when this daemon was
+          started in ISO 8601 format.
+          This is a constant which can't be reset except by restarting
+          <command>b10-stats</command>.
+        </simpara></listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>stats.last_update_time</term>
+        <listitem><simpara>The date and time (in ISO 8601 format)
+          when this daemon last received data from another component.
+        </simpara></listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>stats.lname</term>
+        <listitem><simpara>This is the name used for the
+          <command>b10-msgq</command> command-control channel.
+          (This is a constant which can't be reset except by restarting
+          <command>b10-stats</command>.)
+        </simpara></listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>stats.start_time</term>
+        <listitem><simpara>This is the date and time (in ISO 8601 format)
+          when this daemon started collecting data.
+        </simpara></listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>stats.timestamp</term>
+        <listitem><simpara>The current date and time represented in
+          seconds since UNIX epoch (1970-01-01T0 0:00:00Z) with
+          precision (delimited with a period) up to
+          one hundred thousandth of second.</simpara></listitem>
+      </varlistentry>
+
+    </variablelist>
+
+    <para>
+      See other manual pages for explanations for their statistics
+      that are kept track by <command>b10-stats</command>.
+    </para>
+
+  </refsect1>
+
+  <refsect1>
     <title>FILES</title>
     <para><filename>/usr/local/share/bind10-devel/stats.spec</filename>
       <!--TODO: The filename should be computed from prefix-->
@@ -95,12 +213,6 @@
       invoked
       via <refentrytitle>bindctl</refentrytitle><manvolnum>1</manvolnum>.
     </para>
-    <para><filename>/usr/local/share/bind10-devel/stats-schema.spec</filename>
-      <!--TODO: The filename should be computed from prefix-->
-      &mdash; This is a spec file for data schema of
-      of BIND 10 statistics. This schema cannot be configured 
-      via <refentrytitle>bindctl</refentrytitle><manvolnum>1</manvolnum>.
-    </para>
   </refsect1>
 
   <refsect1>
@@ -126,7 +238,7 @@
     <title>HISTORY</title>
     <para>
       The <command>b10-stats</command> daemon was initially designed
-      and implemented by Naoki Kambe of JPRS in Oct 2010.
+      and implemented by Naoki Kambe of JPRS in October 2010.
     </para>
   </refsect1>
 </refentry><!--

src/bin/stats/stats-httpd-xsl.tpl

@@ -44,6 +44,7 @@ td.title {
         <h1>BIND 10 Statistics</h1>
         <table>
           <tr>
+            <th>Owner</th>
             <th>Title</th>
             <th>Value</th>
           </tr>

src/bin/stats/stats-schema.spec

@@ -1,87 +0,0 @@
-{
-  "module_spec": {
-    "module_name": "Stats",
-    "module_description": "Statistics data schema",
-    "config_data": [
-      {
-        "item_name": "report_time",
-        "item_type": "string",
-        "item_optional": false,
-        "item_default": "1970-01-01T00:00:00Z",
-        "item_title": "Report time",
-        "item_description": "A date time when stats module reports",
-        "item_format": "date-time"
-      },
-      {
-        "item_name": "bind10.boot_time",
-        "item_type": "string",
-        "item_optional": false,
-        "item_default": "1970-01-01T00:00:00Z",
-        "item_title": "bind10.BootTime",
-        "item_description": "A date time when bind10 process starts initially",
-        "item_format": "date-time"
-      },
-      {
-        "item_name": "stats.boot_time",
-        "item_type": "string",
-        "item_optional": false,
-        "item_default": "1970-01-01T00:00:00Z",
-        "item_title": "stats.BootTime",
-        "item_description": "A date time when the stats module starts initially or when the stats module restarts",
-        "item_format": "date-time"
-      },
-      {
-        "item_name": "stats.start_time",
-        "item_type": "string",
-        "item_optional": false,
-        "item_default": "1970-01-01T00:00:00Z",
-        "item_title": "stats.StartTime",
-        "item_description": "A date time when the stats module starts collecting data or resetting values last time",
-        "item_format": "date-time"
-      },
-      {
-        "item_name": "stats.last_update_time",
-        "item_type": "string",
-        "item_optional": false,
-        "item_default": "1970-01-01T00:00:00Z",
-        "item_title": "stats.LastUpdateTime",
-        "item_description": "The latest date time when the stats module receives from other modules like auth server or boss process and so on",
-        "item_format": "date-time"
-      },
-      {
-        "item_name": "stats.timestamp",
-        "item_type": "real",
-        "item_optional": false,
-        "item_default": 0.0,
-        "item_title": "stats.Timestamp",
-        "item_description": "A current time stamp since epoch time (1970-01-01T00:00:00Z)",
-        "item_format": "second"
-      },
-      {
-        "item_name": "stats.lname",
-        "item_type": "string",
-        "item_optional": false,
-        "item_default": "",
-        "item_title": "stats.LocalName",
-        "item_description": "A localname of stats module given via CC protocol"
-      },
-      {
-        "item_name": "auth.queries.tcp",
-        "item_type": "integer",
-        "item_optional": false,
-        "item_default": 0,
-        "item_title": "auth.queries.tcp",
-        "item_description": "A number of total query counts which all auth servers receive over TCP since they started initially"
-      },
-      {
-        "item_name": "auth.queries.udp",
-        "item_type": "integer",
-        "item_optional": false,
-        "item_default": 0,
-        "item_title": "auth.queries.udp",
-        "item_description": "A number of total query counts which all auth servers receive over UDP since they started initially"
-      }
-    ],
-    "commands": []
-  }
-}

src/bin/stats/stats.py.in

@@ -15,399 +15,401 @@
 # NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
 # WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
+"""
+Statistics daemon in BIND 10
+
+"""
 import sys; sys.path.append ('@@PYTHONPATH@@')
 import os
-import signal
-import select
 from time import time, strftime, gmtime
 from optparse import OptionParser, OptionValueError
-from collections import defaultdict
-from isc.config.ccsession import ModuleCCSession, create_answer
-from isc.cc import Session, SessionError
 
-# for setproctitle
+import isc
 import isc.util.process
+import isc.log
+from isc.log_messages.stats_messages import *
+
+isc.log.init("b10-stats")
+logger = isc.log.Logger("stats")
+
+# Some constants for debug levels, these should be removed when we
+# have #1074
+DBG_STATS_MESSAGING = 30
+
+# This is for boot_time of Stats
+_BASETIME = gmtime()
+
+# for setproctitle
 isc.util.process.rename()
 
 # If B10_FROM_SOURCE is set in the environment, we use data files
 # from a directory relative to that, otherwise we use the ones
 # installed on the system
 if "B10_FROM_SOURCE" in os.environ:
-    BASE_LOCATION = os.environ["B10_FROM_SOURCE"] + os.sep + \
-        "src" + os.sep + "bin" + os.sep + "stats"
+    SPECFILE_LOCATION = os.environ["B10_FROM_SOURCE"] + os.sep + \
+        "src" + os.sep + "bin" + os.sep + "stats" + os.sep + "stats.spec"
 else:
     PREFIX = "@prefix@"
     DATAROOTDIR = "@datarootdir@"
-    BASE_LOCATION = "@datadir@" + os.sep + "@PACKAGE@"
-    BASE_LOCATION = BASE_LOCATION.replace("${datarootdir}", DATAROOTDIR).replace("${prefix}", PREFIX)
-SPECFILE_LOCATION = BASE_LOCATION + os.sep + "stats.spec"
-SCHEMA_SPECFILE_LOCATION = BASE_LOCATION + os.sep + "stats-schema.spec"
+    SPECFILE_LOCATION = "@datadir@" + os.sep + "@PACKAGE@" + os.sep + "stats.spec"
+    SPECFILE_LOCATION = SPECFILE_LOCATION.replace("${datarootdir}", DATAROOTDIR)\
+        .replace("${prefix}", PREFIX)
 
-class Singleton(type):
+def get_timestamp():
     """
-    A abstract class of singleton pattern
+    get current timestamp
     """
-    # Because of singleton pattern: 
-    #   At the beginning of coding, one UNIX domain socket is needed
-    #  for config manager, another socket is needed for stats module,
-    #  then stats module might need two sockets. So I adopted the
-    #  singleton pattern because I avoid creating multiple sockets in
-    #  one stats module. But in the initial version stats module
-    #  reports only via bindctl, so just one socket is needed. To use
-    #  the singleton pattern is not important now. :(
+    return time()
 
-    def __init__(self, *args, **kwargs):
-        type.__init__(self, *args, **kwargs)
-        self._instances = {}
+def get_datetime(gmt=None):
+    """
+    get current datetime
+    """
+    if not gmt: gmt = gmtime()
+    return strftime("%Y-%m-%dT%H:%M:%SZ", gmt)
 
-    def __call__(self, *args, **kwargs):
-        if args not in self._instances:
-            self._instances[args]={}
-        kw = tuple(kwargs.items())
-        if  kw not in self._instances[args]:
-            self._instances[args][kw] = type.__call__(self, *args, **kwargs)
-        return self._instances[args][kw]
+def get_spec_defaults(spec):
+    """
+    extracts the default values of the items from spec specified in
+    arg, and returns the dict-type variable which is a set of the item
+    names and the default values
+    """
+    if type(spec) is not list: return {}
+    def _get_spec_defaults(spec):
+        item_type = spec['item_type']
+        if item_type == "integer":
+            return int(spec.get('item_default', 0))
+        elif item_type == "real":
+            return float(spec.get('item_default', 0.0))
+        elif item_type == "boolean":
+            return bool(spec.get('item_default', False))
+        elif item_type == "string":
+            return str(spec.get('item_default', ""))
+        elif item_type == "list":
+            return spec.get(
+                    "item_default",
+                    [ _get_spec_defaults(spec["list_item_spec"]) ])
+        elif item_type == "map":
+            return spec.get(
+                    "item_default",
+                    dict([ (s["item_name"], _get_spec_defaults(s)) for s in spec["map_item_spec"] ]) )
+        else:
+            return spec.get("item_default", None)
+    return dict([ (s['item_name'], _get_spec_defaults(s)) for s in spec ])
 
 class Callback():
     """
     A Callback handler class
     """
-    def __init__(self, name=None, callback=None, args=(), kwargs={}):
-        self.name = name
-        self.callback = callback
+    def __init__(self, command=None, args=(), kwargs={}):
+        self.command = command
         self.args = args
         self.kwargs = kwargs
 
     def __call__(self, *args, **kwargs):
-        if not args:
-            args = self.args
-        if not kwargs:
-            kwargs = self.kwargs
-        if self.callback:
-            return self.callback(*args, **kwargs)
-
-class Subject():
-    """
-    A abstract subject class of observer pattern
-    """
-    # Because of observer pattern:
-    #   In the initial release, I'm also sure that observer pattern
-    #  isn't definitely needed because the interface between gathering
-    #  and reporting statistics data is single.  However in the future
-    #  release, the interfaces may be multiple, that is, multiple
-    #  listeners may be needed. For example, one interface, which
-    #  stats module has, is for between ''config manager'' and stats
-    #  module, another interface is for between ''HTTP server'' and
-    #  stats module, and one more interface is for between ''SNMP
-    #  server'' and stats module. So by considering that stats module
-    #  needs multiple interfaces in the future release, I adopted the
-    #  observer pattern in stats module. But I don't have concrete
-    #  ideas in case of multiple listener currently.
-
-    def __init__(self):
-        self._listeners = []
-
-    def attach(self, listener):
-        if not listener in self._listeners:
-            self._listeners.append(listener)
+        if not args: args = self.args
+        if not kwargs: kwargs = self.kwargs
+        if self.command: return self.command(*args, **kwargs)
 
-    def detach(self, listener):
-        try:
-            self._listeners.remove(listener)
-        except ValueError:
-            pass
+class StatsError(Exception):
+    """Exception class for Stats class"""
+    pass
 
-    def notify(self, event, modifier=None):
-        for listener in self._listeners:
-            if modifier != listener:
-                listener.update(event)
-
-class Listener():
+class Stats:
     """
-    A abstract listener class of observer pattern
+    Main class of stats module
     """
-    def __init__(self, subject):
-        self.subject = subject
-        self.subject.attach(self)
-        self.events = {}
-
-    def update(self, name):
-        if name in self.events:
-            callback = self.events[name]
-            return callback()
-
-    def add_event(self, event):
-        self.events[event.name]=event
-
-class SessionSubject(Subject, metaclass=Singleton):
-    """
-    A concrete subject class which creates CC session object
-    """
-    def __init__(self, session=None, verbose=False):
-        Subject.__init__(self)
-        self.verbose = verbose
-        self.session=session
-        self.running = False
-
-    def start(self):
-        self.running = True
-        self.notify('start')
-
-    def stop(self):
+    def __init__(self):
         self.running = False
-        self.notify('stop')
-
-    def check(self):
-        self.notify('check')
-
-class CCSessionListener(Listener):
-    """
-    A concrete listener class which creates SessionSubject object and
-    ModuleCCSession object
-    """
-    def __init__(self, subject, verbose=False):
-        Listener.__init__(self, subject)
-        self.verbose = verbose
-        self.session = subject.session
-        self.boot_time = get_datetime()
-
         # create ModuleCCSession object
-        self.cc_session = ModuleCCSession(SPECFILE_LOCATION,
-                                          self.config_handler,
-                                          self.command_handler,
-                                          self.session)
-
-        self.session = self.subject.session = self.cc_session._session
-
-        # initialize internal data
-        self.stats_spec = isc.config.module_spec_from_file(SCHEMA_SPECFILE_LOCATION).get_config_spec()
-        self.stats_data = self.initialize_data(self.stats_spec)
-
-        # add event handler invoked via SessionSubject object
-        self.add_event(Callback('start', self.start))
-        self.add_event(Callback('stop', self.stop))
-        self.add_event(Callback('check', self.check))
-        # don't add 'command_' suffix to the special commands in
-        # order to prevent executing internal command via bindctl
-
+        self.mccs = isc.config.ModuleCCSession(SPECFILE_LOCATION,
+                                               self.config_handler,
+                                               self.command_handler)
+        self.cc_session = self.mccs._session
+        # get module spec
+        self.module_name = self.mccs.get_module_spec().get_module_name()
+        self.modules = {}
+        self.statistics_data = {}
         # get commands spec
-        self.commands_spec = self.cc_session.get_module_spec().get_commands_spec()
-
+        self.commands_spec = self.mccs.get_module_spec().get_commands_spec()
         # add event handler related command_handler of ModuleCCSession
-        # invoked via bindctl
+        self.callbacks = {}
         for cmd in self.commands_spec:
+            # add prefix "command_"
+            name = "command_" + cmd["command_name"]
             try:
-                # add prefix "command_"
-                name = "command_" + cmd["command_name"]
                 callback = getattr(self, name)
-                kwargs = self.initialize_data(cmd["command_args"])
-                self.add_event(Callback(name=name, callback=callback, args=(), kwargs=kwargs))
-            except AttributeError as ae:
-                sys.stderr.write("[b10-stats] Caught undefined command while parsing spec file: "
-                                 +str(cmd["command_name"])+"\n")
+                kwargs = get_spec_defaults(cmd["command_args"])
+                self.callbacks[name] = Callback(command=callback, kwargs=kwargs)
+            except AttributeError:
+                raise StatsError(STATS_UNKNOWN_COMMAND_IN_SPEC, cmd["command_name"])
+        self.mccs.start()
 
     def start(self):
         """
-        start the cc chanel
+        Start stats module
         """
-        # set initial value
-        self.stats_data['stats.boot_time'] = self.boot_time
-        self.stats_data['stats.start_time'] = get_datetime()
-        self.stats_data['stats.last_update_time'] = get_datetime()
-        self.stats_data['stats.lname'] = self.session.lname
-        self.cc_session.start()
+        self.running = True
+        logger.info(STATS_STARTING)
+
         # request Bob to send statistics data
-        if self.verbose:
-            sys.stdout.write("[b10-stats] request Bob to send statistics data\n")
-        cmd = isc.config.ccsession.create_command("sendstats", None)
-        seq = self.session.group_sendmsg(cmd, 'Boss')
-        self.session.group_recvmsg(True, seq)
+        logger.debug(DBG_STATS_MESSAGING, STATS_SEND_REQUEST_BOSS)
+        cmd = isc.config.ccsession.create_command("getstats", None)
+        seq = self.cc_session.group_sendmsg(cmd, 'Boss')
+        try:
+            answer, env = self.cc_session.group_recvmsg(False, seq)
+            if answer:
+                rcode, args = isc.config.ccsession.parse_answer(answer)
+                if rcode == 0:
+                    errors = self.update_statistics_data(
+                        args["owner"], **args["data"])
+                    if errors:
+                        raise StatsError("boss spec file is incorrect: "
+                                         + ", ".join(errors))
+                    errors = self.update_statistics_data(
+                                self.module_name,
+                                last_update_time=get_datetime())
+                    if errors:
+                        raise StatsError("stats spec file is incorrect: "
+                                         + ", ".join(errors))
+        except isc.cc.session.SessionTimeout:
+            pass
 
-    def stop(self):
-        """
-        stop the cc chanel
-        """
-        return self.cc_session.close()
+        # initialized Statistics data
+        errors = self.update_statistics_data(
+            self.module_name,
+            lname=self.cc_session.lname,
+            boot_time=get_datetime(_BASETIME)
+            )
+        if errors:
+            raise StatsError("stats spec file is incorrect: "
+                             + ", ".join(errors))
 
-    def check(self):
-        """
-        check the cc chanel
-        """
-        return self.cc_session.check_command(False)
+        while self.running:
+            self.mccs.check_command(False)
 
     def config_handler(self, new_config):
         """
         handle a configure from the cc channel
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] newconfig received: "+str(new_config)+"\n")
-
+        logger.debug(DBG_STATS_MESSAGING, STATS_RECEIVED_NEW_CONFIG,
+                     new_config)
         # do nothing currently
-        return create_answer(0)
+        return isc.config.create_answer(0)
 
-    def command_handler(self, command, *args, **kwargs):
+    def command_handler(self, command, kwargs):
         """
         handle commands from the cc channel
         """
-        # add 'command_' suffix in order to executing command via bindctl
         name = 'command_' + command
-        
-        if name in self.events:
-            event = self.events[name]
-            return event(*args, **kwargs)
+        if name in self.callbacks:
+            callback = self.callbacks[name]
+            if kwargs:
+                return callback(**kwargs)
+            else:
+                return callback()
         else:
-            return self.command_unknown(command, args)
+            logger.error(STATS_RECEIVED_UNKNOWN_COMMAND, command)
+            return isc.config.create_answer(1, "Unknown command: '"+str(command)+"'")
 
-    def command_shutdown(self, args):
+    def update_modules(self):
         """
-        handle shutdown command
+        updates information of each module. This method gets each
+        module's information from the config manager and sets it into
+        self.modules. If its getting from the config manager fails, it
+        raises StatsError.
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] 'shutdown' command received\n")
-        self.subject.running = False
-        return create_answer(0)
+        modules = {}
+        seq = self.cc_session.group_sendmsg(
+            isc.config.ccsession.create_command(
+                isc.config.ccsession.COMMAND_GET_STATISTICS_SPEC),
+            'ConfigManager')
+        (answer, env) = self.cc_session.group_recvmsg(False, seq)
+        if answer:
+            (rcode, value) = isc.config.ccsession.parse_answer(answer)
+            if rcode == 0:
+                for mod in value:
+                    spec = { "module_name" : mod }
+                    if value[mod] and type(value[mod]) is list:
+                        spec["statistics"] = value[mod]
+                    modules[mod] = isc.config.module_spec.ModuleSpec(spec)
+            else:
+                raise StatsError("Updating module spec fails: " + str(value))
+        modules[self.module_name] = self.mccs.get_module_spec()
+        self.modules = modules
 
-    def command_set(self, args, stats_data={}):
+    def get_statistics_data(self, owner=None, name=None):
         """
-        handle set command
+        returns statistics data which stats module has of each
+        module. If it can't find specified statistics data, it raises
+        StatsError.
         """
-        # 'args' must be dictionary type
-        self.stats_data.update(args['stats_data'])
-
-        # overwrite "stats.LastUpdateTime"
-        self.stats_data['stats.last_update_time'] = get_datetime()
-
-        return create_answer(0)
+        self.update_statistics_data()
+        if owner and name:
+            try:
+                return self.statistics_data[owner][name]
+            except KeyError:
+                pass
+        elif owner:
+            try:
+                return self.statistics_data[owner]
+            except KeyError:
+                pass
+        elif name:
+            pass
+        else:
+            return self.statistics_data
+        raise StatsError("No statistics data found: "
+                         + "owner: " + str(owner) + ", "
+                         + "name: " + str(name))
 
-    def command_remove(self, args, stats_item_name=''):
+    def update_statistics_data(self, owner=None, **data):
         """
-        handle remove command
+        change statistics date of specified module into specified
+        data. It updates information of each module first, and it
+        updates statistics data. If specified data is invalid for
+        statistics spec of specified owner, it returns a list of error
+        messeges. If there is no error or if neither owner nor data is
+        specified in args, it returns None.
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] 'remove' command received, args: "+str(args)+"\n")
-
-        # 'args' must be dictionary type
-        if args and args['stats_item_name'] in self.stats_data:
-            stats_item_name = args['stats_item_name']
-
-        # just remove one item
-        self.stats_data.pop(stats_item_name)
-
-        return create_answer(0)
-
-    def command_show(self, args, stats_item_name=''):
+        self.update_modules()
+        statistics_data = {}
+        for (name, module) in self.modules.items():
+            value = get_spec_defaults(module.get_statistics_spec())
+            if module.validate_statistics(True, value):
+                statistics_data[name] = value
+        for (name, value) in self.statistics_data.items():
+            if name in statistics_data:
+                statistics_data[name].update(value)
+            else:
+                statistics_data[name] = value
+        self.statistics_data = statistics_data
+        if owner and data:
+            errors = []
+            try:
+                if self.modules[owner].validate_statistics(False, data, errors):
+                    self.statistics_data[owner].update(data)
+                    return
+            except KeyError:
+                errors.append("unknown module name: " + str(owner))
+            return errors
+
+    def command_status(self):
         """
-        handle show command
+        handle status command
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] 'show' command received, args: "+str(args)+"\n")
-
-        # always overwrite 'report_time' and 'stats.timestamp'
-        # if "show" command invoked
-        self.stats_data['report_time'] = get_datetime()
-        self.stats_data['stats.timestamp'] = get_timestamp()
-
-        # if with args
-        if args and args['stats_item_name'] in self.stats_data:
-            stats_item_name = args['stats_item_name']
-            return create_answer(0, {stats_item_name: self.stats_data[stats_item_name]})
+        logger.debug(DBG_STATS_MESSAGING, STATS_RECEIVED_STATUS_COMMAND)
+        return isc.config.create_answer(
+            0, "Stats is up. (PID " + str(os.getpid()) + ")")
 
-        return create_answer(0, self.stats_data)
-
-    def command_reset(self, args):
+    def command_shutdown(self):
         """
-        handle reset command
+        handle shutdown command
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] 'reset' command received\n")
-
-        # re-initialize internal variables
-        self.stats_data = self.initialize_data(self.stats_spec)
-
-        # reset initial value
-        self.stats_data['stats.boot_time'] = self.boot_time
-        self.stats_data['stats.start_time'] = get_datetime()
-        self.stats_data['stats.last_update_time'] = get_datetime()
-        self.stats_data['stats.lname'] = self.session.lname
-
-        return create_answer(0)
+        logger.info(STATS_RECEIVED_SHUTDOWN_COMMAND)
+        self.running = False
+        return isc.config.create_answer(0)
 
-    def command_status(self, args):
+    def command_show(self, owner=None, name=None):
         """
-        handle status command
+        handle show command
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] 'status' command received\n")
-        # just return "I'm alive."
-        return create_answer(0, "I'm alive.")
-
-    def command_unknown(self, command, args):
+        if owner or name:
+            logger.debug(DBG_STATS_MESSAGING,
+                         STATS_RECEIVED_SHOW_NAME_COMMAND,
+                         str(owner)+", "+str(name))
+        else:
+            logger.debug(DBG_STATS_MESSAGING,
+                         STATS_RECEIVED_SHOW_ALL_COMMAND)
+        errors = self.update_statistics_data(
+            self.module_name,
+            timestamp=get_timestamp(),
+            report_time=get_datetime()
+            )
+        if errors:
+            raise StatsError("stats spec file is incorrect: "
+                             + ", ".join(errors))
+        try:
+            return isc.config.create_answer(
+                0, self.get_statistics_data(owner, name))
+        except StatsError:
+            return isc.config.create_answer(
+                1, "specified arguments are incorrect: " \
+                    + "owner: " + str(owner) + ", name: " + str(name))
+
+    def command_showschema(self, owner=None, name=None):
         """
-        handle an unknown command
+        handle show command
         """
-        if self.verbose:
-            sys.stdout.write("[b10-stats] Unknown command received: '"
-                             + str(command) + "'\n")
-        return create_answer(1, "Unknown command: '"+str(command)+"'")
-
+        if owner or name:
+            logger.debug(DBG_STATS_MESSAGING,
+                         STATS_RECEIVED_SHOWSCHEMA_NAME_COMMAND,
+                         str(owner)+", "+str(name))
+        else:
+            logger.debug(DBG_STATS_MESSAGING,
+                         STATS_RECEIVED_SHOWSCHEMA_ALL_COMMAND)
+        self.update_modules()
+        schema = {}
+        schema_byname = {}
+        for mod in self.modules:
+            spec = self.modules[mod].get_statistics_spec()
+            schema_byname[mod] = {}
+            if spec:
+                schema[mod] = spec
+                for item in spec:
+                    schema_byname[mod][item['item_name']] = item
+        if owner:
+            try:
+                if name:
+                    return isc.config.create_answer(0, schema_byname[owner][name])
+                else:
+                    return isc.config.create_answer(0, schema[owner])
+            except KeyError:
+                pass
+        else:
+            if name:
+                return isc.config.create_answer(1, "module name is not specified")
+            else:
+                return isc.config.create_answer(0, schema)
+        return isc.config.create_answer(
+                1, "specified arguments are incorrect: " \
+                    + "owner: " + str(owner) + ", name: " + str(name))
 
-    def initialize_data(self, spec):
+    def command_set(self, owner, data):
         """
-        initialize stats data
+        handle set command
         """
-        def __get_init_val(spec):
-            if spec['item_type'] == 'null':
-                return None
-            elif spec['item_type'] == 'boolean':
-                return bool(spec.get('item_default', False))
-            elif spec['item_type'] == 'string':
-                return str(spec.get('item_default', ''))
-            elif spec['item_type'] in set(['number', 'integer']):
-                return int(spec.get('item_default', 0))
-            elif spec['item_type'] in set(['float', 'double', 'real']):
-                return float(spec.get('item_default', 0.0))
-            elif spec['item_type'] in set(['list', 'array']):
-                return spec.get('item_default',
-                                [ __get_init_val(s) for s in spec['list_item_spec'] ])
-            elif spec['item_type'] in set(['map', 'object']):
-                return spec.get('item_default',
-                                dict([ (s['item_name'], __get_init_val(s)) for s in spec['map_item_spec'] ]) )
-            else:
-                return spec.get('item_default')
-        return dict([ (s['item_name'], __get_init_val(s)) for s in spec ])
+        errors = self.update_statistics_data(owner, **data)
+        if errors:
+            return isc.config.create_answer(
+                1, "errors while setting statistics data: " \
+                    + ", ".join(errors))
+        errors = self.update_statistics_data(
+            self.module_name, last_update_time=get_datetime() )
+        if errors:
+            raise StatsError("stats spec file is incorrect: "
+                             + ", ".join(errors))
+        return isc.config.create_answer(0)
 
-def get_timestamp():
-    """
-    get current timestamp
-    """
-    return time()
-
-def get_datetime():
-    """
-    get current datetime
-    """
-    return strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())
-
-def main(session=None):
+if __name__ == "__main__":
     try:
         parser = OptionParser()
-        parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
-                      help="display more about what is going on")
+        parser.add_option(
+            "-v", "--verbose", dest="verbose", action="store_true",
+            help="display more about what is going on")
         (options, args) = parser.parse_args()
-        subject = SessionSubject(session=session, verbose=options.verbose)
-        listener = CCSessionListener(subject, verbose=options.verbose)
-        subject.start()
-        while subject.running:
-            subject.check()
-        subject.stop()
-
-    except OptionValueError:
-        sys.stderr.write("[b10-stats] Error parsing options\n")
-    except SessionError as se:
-        sys.stderr.write("[b10-stats] Error creating Stats module, "
-              + "is the command channel daemon running?\n")
+        if options.verbose:
+            isc.log.init("b10-stats", "DEBUG", 99)
+        stats = Stats()
+        stats.start()
+    except OptionValueError as ove:
+        logger.fatal(STATS_BAD_OPTION_VALUE, ove)
+        sys.exit(1)
+    except isc.cc.session.SessionError as se:
+        logger.fatal(STATS_CC_SESSION_ERROR, se)
+        sys.exit(1)
+    except StatsError as se:
+        logger.fatal(STATS_START_ERROR, se)
+        sys.exit(1)
     except KeyboardInterrupt as kie:
-        sys.stderr.write("[b10-stats] Interrupted, exiting\n")
-
-if __name__ == "__main__":
-    main()
+        logger.info(STATS_STOPPED_BY_KEYBOARD)

src/bin/stats/stats.spec

@@ -6,55 +6,119 @@
     "commands": [
       {
         "command_name": "status",
-        "command_description": "identify whether stats module is alive or not",
+        "command_description": "Show status of the stats daemon",
+        "command_args": []
+      },
+      {
+        "command_name": "shutdown",
+        "command_description": "Shut down the stats module",
         "command_args": []
       },
       {
         "command_name": "show",
-        "command_description": "show the specified/all statistics data",
+        "command_description": "Show the specified/all statistics data",
         "command_args": [
           {
-            "item_name": "stats_item_name",
+            "item_name": "owner",
             "item_type": "string",
             "item_optional": true,
-            "item_default": ""
+            "item_default": "",
+            "item_description": "module name of the owner of the statistics data"
+          },
+	  {
+	    "item_name": "name",
+            "item_type": "string",
+            "item_optional": true,
+            "item_default": "",
+            "item_description": "statistics item name of the owner"
           }
         ]
       },
       {
-        "command_name": "set",
-        "command_description": "set the value of specified name in statistics data",
+        "command_name": "showschema",
+        "command_description": "show the specified/all statistics shema",
         "command_args": [
           {
-            "item_name": "stats_data",
-            "item_type": "map",
-            "item_optional": false,
-            "item_default": {},
-            "map_item_spec": []
+            "item_name": "owner",
+            "item_type": "string",
+            "item_optional": true,
+            "item_default": "",
+            "item_description": "module name of the owner of the statistics data"
+          },
+	  {
+	    "item_name": "name",
+            "item_type": "string",
+            "item_optional": true,
+            "item_default": "",
+            "item_description": "statistics item name of the owner"
           }
         ]
       },
       {
-        "command_name": "remove",
-        "command_description": "remove the specified name from statistics data",
+        "command_name": "set",
+        "command_description": "set the value of specified name in statistics data",
         "command_args": [
           {
-            "item_name": "stats_item_name",
+            "item_name": "owner",
             "item_type": "string",
             "item_optional": false,
-            "item_default": ""
+            "item_default": "",
+            "item_description": "module name of the owner of the statistics data"
+          },
+	  {
+	    "item_name": "data",
+            "item_type": "map",
+            "item_optional": false,
+            "item_default": {},
+            "item_description": "statistics data set of the owner",
+            "map_item_spec": []
           }
         ]
+      }
+    ],
+    "statistics": [
+      {
+        "item_name": "report_time",
+        "item_type": "string",
+        "item_optional": false,
+        "item_default": "1970-01-01T00:00:00Z",
+        "item_title": "Report time",
+        "item_description": "A date time when stats module reports",
+        "item_format": "date-time"
       },
       {
-        "command_name": "reset",
-        "command_description": "reset all statistics data to default values except for several constant names",
-        "command_args": []
+        "item_name": "boot_time",
+        "item_type": "string",
+        "item_optional": false,
+        "item_default": "1970-01-01T00:00:00Z",
+        "item_title": "Boot time",
+        "item_description": "A date time when the stats module starts initially or when the stats module restarts",
+        "item_format": "date-time"
       },
       {
-        "command_name": "shutdown",
-        "command_description": "Shut down the stats module",
-        "command_args": []
+        "item_name": "last_update_time",
+        "item_type": "string",
+        "item_optional": false,
+        "item_default": "1970-01-01T00:00:00Z",
+        "item_title": "Last update time",
+        "item_description": "The latest date time when the stats module receives from other modules like auth server or boss process and so on",
+        "item_format": "date-time"
+      },
+      {
+        "item_name": "timestamp",
+        "item_type": "real",
+        "item_optional": false,
+        "item_default": 0.0,
+        "item_title": "Timestamp",
+        "item_description": "A current time stamp since epoch time (1970-01-01T00:00:00Z)"
+      },
+      {
+        "item_name": "lname",
+        "item_type": "string",
+        "item_optional": false,
+        "item_default": "",
+        "item_title": "Local Name",
+        "item_description": "A localname of stats module given via CC protocol"
       }
     ]
   }

src/bin/stats/stats_httpd.py.in

@@ -34,6 +34,17 @@ import isc.cc
 import isc.config
 import isc.util.process
 
+import isc.log
+from isc.log_messages.stats_httpd_messages import *
+
+isc.log.init("b10-stats-httpd")
+logger = isc.log.Logger("stats-httpd")
+
+# Some constants for debug levels, these should be removed when we
+# have #1074
+DBG_STATHTTPD_INIT = 10
+DBG_STATHTTPD_MESSAGING = 30
+
 # If B10_FROM_SOURCE is set in the environment, we use data files
 # from a directory relative to that, otherwise we use the ones
 # installed on the system
@@ -46,7 +57,6 @@ else:
     BASE_LOCATION = "@datadir@" + os.sep + "@PACKAGE@"
     BASE_LOCATION = BASE_LOCATION.replace("${datarootdir}", DATAROOTDIR).replace("${prefix}", PREFIX)
 SPECFILE_LOCATION = BASE_LOCATION + os.sep + "stats-httpd.spec"
-SCHEMA_SPECFILE_LOCATION = BASE_LOCATION + os.sep + "stats-schema.spec"
 XML_TEMPLATE_LOCATION = BASE_LOCATION + os.sep + "stats-httpd-xml.tpl"
 XSD_TEMPLATE_LOCATION = BASE_LOCATION + os.sep + "stats-httpd-xsd.tpl"
 XSL_TEMPLATE_LOCATION = BASE_LOCATION + os.sep + "stats-httpd-xsl.tpl"
@@ -58,7 +68,6 @@ XSD_URL_PATH = '/bind10/statistics/xsd'
 XSL_URL_PATH = '/bind10/statistics/xsl'
 # TODO: This should be considered later.
 XSD_NAMESPACE = 'http://bind10.isc.org' + XSD_URL_PATH
-DEFAULT_CONFIG = dict(listen_on=[('127.0.0.1', 8000)])
 
 # Assign this process name
 isc.util.process.rename()
@@ -98,9 +107,7 @@ class HttpHandler(http.server.BaseHTTPRequestHandler):
                     return None
         except StatsHttpdError as err:
             self.send_error(500)
-            if self.server.verbose:
-                self.server.log_writer(
-                    "[b10-stats-httpd] %s\n" % err)
+            logger.error(STATHTTPD_SERVER_ERROR, err)
             return None
         else:
             self.send_response(200)
@@ -109,15 +116,6 @@ class HttpHandler(http.server.BaseHTTPRequestHandler):
             self.end_headers()
             return body
 
-    def log_message(self, format, *args):
-        """Change the default log format"""
-        if self.server.verbose:
-            self.server.log_writer(
-                "[b10-stats-httpd] %s - - [%s] %s\n" %
-                (self.address_string(),
-                 self.log_date_time_string(),
-                 format%args))
-
 class HttpServerError(Exception):
     """Exception class for HttpServer class. It is intended to be
     passed from the HttpServer object to the StatsHttpd object."""
@@ -134,13 +132,12 @@ class HttpServer(http.server.HTTPServer):
     sys.stderr.write. They are intended to be referred by HttpHandler
     object."""
     def __init__(self, server_address, handler,
-                 xml_handler, xsd_handler, xsl_handler, log_writer, verbose=False):
+                 xml_handler, xsd_handler, xsl_handler, log_writer):
         self.server_address = server_address
         self.xml_handler = xml_handler
         self.xsd_handler = xsd_handler
         self.xsl_handler = xsl_handler
         self.log_writer = log_writer
-        self.verbose = verbose
         http.server.HTTPServer.__init__(self, server_address, handler)
 
 class StatsHttpdError(Exception):
@@ -154,37 +151,33 @@ class StatsHttpd:
     statistics module. It handles HTTP requests, and command channel
     and config channel CC session. It uses select.select function
     while waiting for clients requests."""
-    def __init__(self, verbose=False):
-        self.verbose = verbose
+    def __init__(self):
         self.running = False
         self.poll_intval = 0.5
         self.write_log = sys.stderr.write
         self.mccs = None
         self.httpd = []
         self.open_mccs()
+        self.config = {}
         self.load_config()
-        self.load_templates()
+        self.http_addrs = []
+        self.mccs.start()
         self.open_httpd()
 
     def open_mccs(self):
         """Opens a ModuleCCSession object"""
         # create ModuleCCSession
-        if self.verbose:
-            self.write_log("[b10-stats-httpd] Starting CC Session\n")
+        logger.debug(DBG_STATHTTPD_INIT, STATHTTPD_STARTING_CC_SESSION)
         self.mccs = isc.config.ModuleCCSession(
             SPECFILE_LOCATION, self.config_handler, self.command_handler)
         self.cc_session = self.mccs._session
-        # read spec file of stats module and subscribe 'Stats'
-        self.stats_module_spec = isc.config.module_spec_from_file(SCHEMA_SPECFILE_LOCATION)
-        self.stats_config_spec = self.stats_module_spec.get_config_spec()
-        self.stats_module_name = self.stats_module_spec.get_module_name()
 
     def close_mccs(self):
         """Closes a ModuleCCSession object"""
         if self.mccs is None:
             return
-        if self.verbose:
-            self.write_log("[b10-stats-httpd] Closing CC Session\n")
+
+        logger.debug(DBG_STATHTTPD_INIT, STATHTTPD_CLOSING_CC_SESSION)
         self.mccs.close()
         self.mccs = None
 
@@ -192,18 +185,19 @@ class StatsHttpd:
         """Loads configuration from spec file or new configuration
         from the config manager"""
         # load config
-        if len(new_config) > 0:
-            self.config.update(new_config)
-        else:
-            self.config = DEFAULT_CONFIG
-            self.config.update(
-                dict([
-                        (itm['item_name'], self.mccs.get_value(itm['item_name'])[0])
-                        for itm in self.mccs.get_module_spec().get_config_spec()
-                        ])
-                )
+        if len(self.config) == 0:
+            self.config = dict([
+                (itm['item_name'], self.mccs.get_value(itm['item_name'])[0])
+                for itm in self.mccs.get_module_spec().get_config_spec()
+                ])
+        self.config.update(new_config)
         # set addresses and ports for HTTP
-        self.http_addrs = [ (cf['address'], cf['port']) for cf in self.config['listen_on'] ]
+        addrs = []
+        if 'listen_on' in self.config:
+            for cf in self.config['listen_on']:
+                if 'address' in cf and 'port' in cf:
+                    addrs.append((cf['address'], cf['port']))
+        self.http_addrs = addrs
 
     def open_httpd(self):
         """Opens sockets for HTTP. Iterating each HTTP address to be
@@ -211,51 +205,44 @@ class StatsHttpd:
         for addr in self.http_addrs:
             self.httpd.append(self._open_httpd(addr))
 
-    def _open_httpd(self, server_address, address_family=None):
+    def _open_httpd(self, server_address):
+        httpd = None
         try:
-            # try IPv6 at first
-            if address_family is not None:
-                HttpServer.address_family = address_family
-            elif socket.has_ipv6:
-                HttpServer.address_family = socket.AF_INET6
+            # get address family for the server_address before
+            # creating HttpServer object. If a specified address is
+            # not numerical, gaierror may be thrown.
+            address_family = socket.getaddrinfo(
+                server_address[0], server_address[1], 0,
+                socket.SOCK_STREAM, socket.IPPROTO_TCP, socket.AI_NUMERICHOST
+                )[0][0]
+            HttpServer.address_family = address_family
             httpd = HttpServer(
                 server_address, HttpHandler,
                 self.xml_handler, self.xsd_handler, self.xsl_handler,
-                self.write_log, self.verbose)
+                self.write_log)
+            logger.info(STATHTTPD_STARTED, server_address[0],
+                        server_address[1])
+            return httpd
         except (socket.gaierror, socket.error,
                 OverflowError, TypeError) as err:
-            # try IPv4 next
-            if HttpServer.address_family == socket.AF_INET6:
-                httpd = self._open_httpd(server_address, socket.AF_INET)
-            else:
-                raise HttpServerError(
-                    "Invalid address %s, port %s: %s: %s" %
-                    (server_address[0], server_address[1],
-                     err.__class__.__name__, err))
-        else:
-            if self.verbose:
-                self.write_log(
-                    "[b10-stats-httpd] Started on address %s, port %s\n" %
-                    server_address)
-        return httpd
+           if httpd:
+                httpd.server_close()
+           raise HttpServerError(
+               "Invalid address %s, port %s: %s: %s" %
+               (server_address[0], server_address[1],
+                err.__class__.__name__, err))
 
     def close_httpd(self):
         """Closes sockets for HTTP"""
-        if len(self.httpd) == 0:
-            return
-        for ht in self.httpd:
-            if self.verbose:
-                self.write_log(
-                    "[b10-stats-httpd] Closing address %s, port %s\n" %
-                    (ht.server_address[0], ht.server_address[1])
-                    )
+        while len(self.httpd)>0:
+            ht = self.httpd.pop()
+            logger.info(STATHTTPD_CLOSING, ht.server_address[0],
+                        ht.server_address[1])
             ht.server_close()
-        self.httpd = []
 
     def start(self):
         """Starts StatsHttpd objects to run. Waiting for client
         requests by using select.select functions"""
-        self.mccs.start()
         self.running = True
         while self.running:
             try:
@@ -285,10 +272,10 @@ class StatsHttpd:
     def stop(self):
         """Stops the running StatsHttpd objects. Closes CC session and
         HTTP handling sockets"""
-        if self.verbose:
-            self.write_log("[b10-stats-httpd] Shutting down\n")
+        logger.info(STATHTTPD_SHUTDOWN)
         self.close_httpd()
         self.close_mccs()
+        self.running = False
 
     def get_sockets(self):
         """Returns sockets to select.select"""
@@ -303,29 +290,29 @@ class StatsHttpd:
     def config_handler(self, new_config):
         """Config handler for the ModuleCCSession object. It resets
         addresses and ports to listen HTTP requests on."""
-        if self.verbose:
-            self.write_log("[b10-stats-httpd] Loading config : %s\n" % str(new_config))
-        for key in new_config.keys():
-            if key not in DEFAULT_CONFIG:
-                if self.verbose:
-                    self.write_log(
-                        "[b10-stats-httpd] Unknown known config: %s" % key)
+        logger.debug(DBG_STATHTTPD_MESSAGING, STATHTTPD_HANDLE_CONFIG,
+                   new_config)
+        errors = []
+        if not self.mccs.get_module_spec().\
+                validate_config(False, new_config, errors):
                 return isc.config.ccsession.create_answer(
-                    1, "Unknown known config: %s" % key)
+                    1, ", ".join(errors))
         # backup old config
         old_config = self.config.copy()
-        self.close_httpd()
         self.load_config(new_config)
+        # If the http sockets aren't opened or
+        # if new_config doesn't have'listen_on', it returns
+        if len(self.httpd) == 0 or 'listen_on' not in new_config:
+            return isc.config.ccsession.create_answer(0)
+        self.close_httpd()
         try:
             self.open_httpd()
         except HttpServerError as err:
-            if self.verbose:
-                self.write_log("[b10-stats-httpd] %s\n" % err)
-                self.write_log("[b10-stats-httpd] Restoring old config\n")
+            logger.error(STATHTTPD_SERVER_ERROR, err)
             # restore old config
-            self.config_handler(old_config)
-            return isc.config.ccsession.create_answer(
-                1, "[b10-stats-httpd] %s" % err)
+            self.load_config(old_config)
+            self.open_httpd()
+            return isc.config.ccsession.create_answer(1, str(err))
         else:
             return isc.config.ccsession.create_answer(0)
 
@@ -333,19 +320,18 @@ class StatsHttpd:
         """Command handler for the ModuleCCSesson object. It handles
         "status" and "shutdown" commands."""
         if command == "status":
-            if self.verbose:
-                self.write_log("[b10-stats-httpd] Received 'status' command\n")
+            logger.debug(DBG_STATHTTPD_MESSAGING,
+                         STATHTTPD_RECEIVED_STATUS_COMMAND)
             return isc.config.ccsession.create_answer(
                 0, "Stats Httpd is up. (PID " + str(os.getpid()) + ")")
         elif command == "shutdown":
-            if self.verbose:
-                self.write_log("[b10-stats-httpd] Received 'shutdown' command\n")
+            logger.debug(DBG_STATHTTPD_MESSAGING,
+                         STATHTTPD_RECEIVED_SHUTDOWN_COMMAND)
             self.running = False
-            return isc.config.ccsession.create_answer(
-                0, "Stats Httpd is shutting down.")
+            return isc.config.ccsession.create_answer(0)
         else:
-            if self.verbose:
-                self.write_log("[b10-stats-httpd] Received unknown command\n")
+            logger.debug(DBG_STATHTTPD_MESSAGING,
+                         STATHTTPD_RECEIVED_UNKNOWN_COMMAND, command)
             return isc.config.ccsession.create_answer(
                 1, "Unknown command: " + str(command))
 
@@ -354,8 +340,7 @@ class StatsHttpd:
         the data which obtains from it"""
         try:
             seq = self.cc_session.group_sendmsg(
-                isc.config.ccsession.create_command('show'),
-                self.stats_module_name)
+                isc.config.ccsession.create_command('show'), 'Stats')
             (answer, env) = self.cc_session.group_recvmsg(False, seq)
             if answer:
                 (rcode, value) = isc.config.ccsession.parse_answer(answer)
@@ -370,73 +355,34 @@ class StatsHttpd:
                 raise StatsHttpdError("Stats module: %s" % str(value))
 
     def get_stats_spec(self):
-        """Just returns spec data"""
-        return self.stats_config_spec
-
-    def load_templates(self):
-        """Setup the bodies of XSD and XSL documents to be responds to
-        HTTP clients. Before that it also creates XML tag structures by
-        using xml.etree.ElementTree.Element class and substitutes
-        concrete strings with parameters embed in the string.Template
-        object."""
-        # for XSD
-        xsd_root = xml.etree.ElementTree.Element("all") # started with "all" tag
-        for item in self.get_stats_spec():
-            element = xml.etree.ElementTree.Element(
-                "element",
-                dict( name=item["item_name"],
-                      type=item["item_type"] if item["item_type"].lower() != 'real' else 'float',
-                      minOccurs="1",
-                      maxOccurs="1" ),
-                )
-            annotation = xml.etree.ElementTree.Element("annotation")
-            appinfo = xml.etree.ElementTree.Element("appinfo")
-            documentation = xml.etree.ElementTree.Element("documentation")
-            appinfo.text = item["item_title"]
-            documentation.text = item["item_description"]
-            annotation.append(appinfo)
-            annotation.append(documentation)
-            element.append(annotation)
-            xsd_root.append(element)
-        xsd_string = xml.etree.ElementTree.tostring(xsd_root)
-        self.xsd_body = self.open_template(XSD_TEMPLATE_LOCATION).substitute(
-            xsd_string=xsd_string,
-            xsd_namespace=XSD_NAMESPACE
-            )
-        assert self.xsd_body is not None
-
-        # for XSL
-        xsd_root = xml.etree.ElementTree.Element(
-            "xsl:template",
-            dict(match="*")) # started with xml:template tag
-        for item in self.get_stats_spec():
-            tr = xml.etree.ElementTree.Element("tr")
-            td1 = xml.etree.ElementTree.Element(
-                "td", { "class" : "title",
-                        "title" : item["item_description"] })
-            td1.text = item["item_title"]
-            td2 = xml.etree.ElementTree.Element("td")
-            xsl_valueof = xml.etree.ElementTree.Element(
-                "xsl:value-of",
-                dict(select=item["item_name"]))
-            td2.append(xsl_valueof)
-            tr.append(td1)
-            tr.append(td2)
-            xsd_root.append(tr)
-        xsl_string = xml.etree.ElementTree.tostring(xsd_root)
-        self.xsl_body = self.open_template(XSL_TEMPLATE_LOCATION).substitute(
-            xsl_string=xsl_string,
-            xsd_namespace=XSD_NAMESPACE)
-        assert self.xsl_body is not None
+        """Requests statistics data to the Stats daemon and returns
+        the data which obtains from it"""
+        try:
+            seq = self.cc_session.group_sendmsg(
+                isc.config.ccsession.create_command('showschema'), 'Stats')
+            (answer, env) = self.cc_session.group_recvmsg(False, seq)
+            if answer:
+                (rcode, value) = isc.config.ccsession.parse_answer(answer)
+                if rcode == 0:
+                    return value
+                else:
+                    raise StatsHttpdError("Stats module: %s" % str(value))
+        except (isc.cc.session.SessionTimeout,
+                isc.cc.session.SessionError) as err:
+            raise StatsHttpdError("%s: %s" %
+                                  (err.__class__.__name__, err))
 
     def xml_handler(self):
         """Handler which requests to Stats daemon to obtain statistics
         data and returns the body of XML document"""
         xml_list=[]
-        for (k, v) in self.get_stats_data().items():
-            (k, v) = (str(k), str(v))
-            elem = xml.etree.ElementTree.Element(k)
-            elem.text = v
+        for (mod, spec) in self.get_stats_data().items():
+            if not spec: continue
+            elem1 = xml.etree.ElementTree.Element(str(mod))
+            for (k, v) in spec.items():
+                elem2 = xml.etree.ElementTree.Element(str(k))
+                elem2.text = str(v)
+                elem1.append(elem2)
             # The coding conversion is tricky. xml..tostring() of Python 3.2
             # returns bytes (not string) regardless of the coding, while
             # tostring() of Python 3.1 returns a string.  To support both
@@ -444,7 +390,7 @@ class StatsHttpd:
             # bytes by specifying utf-8 and then convert the result to a
             # plain string (code below assume it).
             xml_list.append(
-                str(xml.etree.ElementTree.tostring(elem, encoding='utf-8'),
+                str(xml.etree.ElementTree.tostring(elem1, encoding='utf-8'),
                     encoding='us-ascii'))
         xml_string = "".join(xml_list)
         self.xml_body = self.open_template(XML_TEMPLATE_LOCATION).substitute(
@@ -457,18 +403,95 @@ class StatsHttpd:
 
     def xsd_handler(self):
         """Handler which just returns the body of XSD document"""
+        # for XSD
+        xsd_root = xml.etree.ElementTree.Element("all") # started with "all" tag
+        for (mod, spec) in self.get_stats_spec().items():
+            if not spec: continue
+            alltag = xml.etree.ElementTree.Element("all")
+            for item in spec:
+                element = xml.etree.ElementTree.Element(
+                    "element",
+                    dict( name=item["item_name"],
+                          type=item["item_type"] if item["item_type"].lower() != 'real' else 'float',
+                          minOccurs="1",
+                          maxOccurs="1" ),
+                    )
+                annotation = xml.etree.ElementTree.Element("annotation")
+                appinfo = xml.etree.ElementTree.Element("appinfo")
+                documentation = xml.etree.ElementTree.Element("documentation")
+                appinfo.text = item["item_title"]
+                documentation.text = item["item_description"]
+                annotation.append(appinfo)
+                annotation.append(documentation)
+                element.append(annotation)
+                alltag.append(element)
+
+            complextype = xml.etree.ElementTree.Element("complexType")
+            complextype.append(alltag)
+            mod_element = xml.etree.ElementTree.Element("element", { "name" : mod })
+            mod_element.append(complextype)
+            xsd_root.append(mod_element)
+        # The coding conversion is tricky. xml..tostring() of Python 3.2
+        # returns bytes (not string) regardless of the coding, while
+        # tostring() of Python 3.1 returns a string.  To support both
+        # cases transparently, we first make sure tostring() returns
+        # bytes by specifying utf-8 and then convert the result to a
+        # plain string (code below assume it).
+        xsd_string = str(xml.etree.ElementTree.tostring(xsd_root, encoding='utf-8'),
+                         encoding='us-ascii')
+        self.xsd_body = self.open_template(XSD_TEMPLATE_LOCATION).substitute(
+            xsd_string=xsd_string,
+            xsd_namespace=XSD_NAMESPACE
+            )
+        assert self.xsd_body is not None
         return self.xsd_body
 
     def xsl_handler(self):
         """Handler which just returns the body of XSL document"""
+        # for XSL
+        xsd_root = xml.etree.ElementTree.Element(
+            "xsl:template",
+            dict(match="*")) # started with xml:template tag
+        for (mod, spec) in self.get_stats_spec().items():
+            if not spec: continue
+            for item in spec:
+                tr = xml.etree.ElementTree.Element("tr")
+                td0 = xml.etree.ElementTree.Element("td")
+                td0.text = str(mod)
+                td1 = xml.etree.ElementTree.Element(
+                    "td", { "class" : "title",
+                            "title" : item["item_description"] })
+                td1.text = item["item_title"]
+                td2 = xml.etree.ElementTree.Element("td")
+                xsl_valueof = xml.etree.ElementTree.Element(
+                    "xsl:value-of",
+                    dict(select=mod+'/'+item["item_name"]))
+                td2.append(xsl_valueof)
+                tr.append(td0)
+                tr.append(td1)
+                tr.append(td2)
+                xsd_root.append(tr)
+        # The coding conversion is tricky. xml..tostring() of Python 3.2
+        # returns bytes (not string) regardless of the coding, while
+        # tostring() of Python 3.1 returns a string.  To support both
+        # cases transparently, we first make sure tostring() returns
+        # bytes by specifying utf-8 and then convert the result to a
+        # plain string (code below assume it).
+        xsl_string = str(xml.etree.ElementTree.tostring(xsd_root, encoding='utf-8'),
+                         encoding='us-ascii')
+        self.xsl_body = self.open_template(XSL_TEMPLATE_LOCATION).substitute(
+            xsl_string=xsl_string,
+            xsd_namespace=XSD_NAMESPACE)
+        assert self.xsl_body is not None
         return self.xsl_body
 
     def open_template(self, file_name):
         """It opens a template file, and it loads all lines to a
         string variable and returns string. Template object includes
         the variable. Limitation of a file size isn't needed there."""
-        lines = "".join(
-            open(file_name, 'r').readlines())
+        f = open(file_name, 'r')
+        lines = "".join(f.readlines())
+        f.close()
         assert lines is not None
         return string.Template(lines)
 
@@ -479,14 +502,18 @@ if __name__ == "__main__":
             "-v", "--verbose", dest="verbose", action="store_true",
             help="display more about what is going on")
         (options, args) = parser.parse_args()
-        stats_httpd = StatsHttpd(verbose=options.verbose)
+        if options.verbose:
+            isc.log.init("b10-stats-httpd", "DEBUG", 99)
+        stats_httpd = StatsHttpd()
         stats_httpd.start()
-    except OptionValueError:
-        sys.exit("[b10-stats-httpd] Error parsing options")
+    except OptionValueError as ove:
+        logger.fatal(STATHTTPD_BAD_OPTION_VALUE, ove)
+        sys.exit(1)
     except isc.cc.session.SessionError as se:
-        sys.exit("[b10-stats-httpd] Error creating module, "
-                 + "is the command channel daemon running?")
+        logger.fatal(STATHTTPD_CC_SESSION_ERROR, se)
+        sys.exit(1)
     except HttpServerError as hse:
-        sys.exit("[b10-stats-httpd] %s" % hse)
+        logger.fatal(STATHTTPD_START_SERVER_INIT_ERROR, hse)
+        sys.exit(1)
     except KeyboardInterrupt as kie:
-        sys.exit("[b10-stats-httpd] Interrupted, exiting")
+        logger.info(STATHTTPD_STOPPED_BY_KEYBOARD)

src/bin/stats/stats_httpd_messages.mes

@@ -0,0 +1,92 @@
+# Copyright (C) 2011  Internet Systems Consortium, Inc. ("ISC")
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
+# No namespace declaration - these constants go in the global namespace
+# of the stats_httpd_messages python module.
+
+% STATHTTPD_BAD_OPTION_VALUE bad command line argument: %1
+The stats-httpd module was called with a bad command-line argument
+and will not start.
+
+% STATHTTPD_CC_SESSION_ERROR error connecting to message bus: %1
+The stats-httpd module was unable to connect to the BIND 10 command
+and control bus. A likely problem is that the message bus daemon
+(b10-msgq) is not running. The stats-httpd module will now shut down.
+
+% STATHTTPD_CLOSING_CC_SESSION stopping cc session
+Debug message indicating that the stats-httpd module is disconnecting
+from the command and control bus.
+
+% STATHTTPD_CLOSING closing %1#%2
+The stats-httpd daemon will stop listening for requests on the given
+address and port number.
+
+% STATHTTPD_HANDLE_CONFIG reading configuration: %1
+The stats-httpd daemon has received new configuration data and will now
+process it. The (changed) data is printed.
+
+% STATHTTPD_RECEIVED_SHUTDOWN_COMMAND shutdown command received
+A shutdown command was sent to the stats-httpd module, and it will
+now shut down.
+
+% STATHTTPD_RECEIVED_STATUS_COMMAND received command to return status
+A status command was sent to the stats-httpd module, and it will
+respond with 'Stats Httpd is up.' and its PID.
+
+% STATHTTPD_RECEIVED_UNKNOWN_COMMAND received unknown command: %1
+An unknown command has been sent to the stats-httpd module. The
+stats-httpd module will respond with an error, and the command will
+be ignored.
+
+% STATHTTPD_SERVER_ERROR HTTP server error: %1
+An internal error occurred while handling an HTTP request. An HTTP 500
+response will be sent back, and the specific error is printed. This
+is an error condition that likely points to a module that is not
+responding correctly to statistic requests.
+
+% STATHTTPD_SERVER_INIT_ERROR HTTP server initialization error: %1
+There was a problem initializing the HTTP server in the stats-httpd
+module upon receiving its configuration data. The most likely cause
+is a port binding problem or a bad configuration value. The specific
+error is printed in the message. The new configuration is ignored,
+and an error is sent back.
+
+% STATHTTPD_SHUTDOWN shutting down
+The stats-httpd daemon is shutting down.
+
+% STATHTTPD_START_SERVER_INIT_ERROR HTTP server initialization error: %1
+There was a problem initializing the HTTP server in the stats-httpd
+module upon startup. The most likely cause is that it was not able
+to bind to the listening port. The specific error is printed, and the
+module will shut down.
+
+% STATHTTPD_STARTED listening on %1#%2
+The stats-httpd daemon will now start listening for requests on the
+given address and port number.
+
+% STATHTTPD_STARTING_CC_SESSION starting cc session
+Debug message indicating that the stats-httpd module is connecting to
+the command and control bus.
+
+% STATHTTPD_STOPPED_BY_KEYBOARD keyboard interrupt, shutting down
+There was a keyboard interrupt signal to stop the stats-httpd
+daemon. The daemon will now shut down.
+
+% STATHTTPD_UNKNOWN_CONFIG_ITEM unknown configuration item: %1
+The stats-httpd daemon received a configuration update from the
+configuration manager. However, one of the items in the
+configuration is unknown. The new configuration is ignored, and an
+error is sent back. As possible cause is that there was an upgrade
+problem, and the stats-httpd version is out of sync with the rest of
+the system.

src/bin/stats/stats_messages.mes

@@ -0,0 +1,76 @@
+# Copyright (C) 2011  Internet Systems Consortium, Inc. ("ISC")
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
+# No namespace declaration - these constants go in the global namespace
+# of the stats_messages python module.
+
+% STATS_BAD_OPTION_VALUE bad command line argument: %1
+The stats module was called with a bad command-line argument and will
+not start.
+
+% STATS_CC_SESSION_ERROR error connecting to message bus: %1
+The stats module was unable to connect to the BIND 10 command and
+control bus. A likely problem is that the message bus daemon
+(b10-msgq) is not running. The stats module will now shut down.
+
+% STATS_RECEIVED_NEW_CONFIG received new configuration: %1
+This debug message is printed when the stats module has received a
+configuration update from the configuration manager.
+
+% STATS_RECEIVED_SHOW_ALL_COMMAND received command to show all statistics
+The stats module received a command to show all statistics that it has
+collected.
+
+% STATS_RECEIVED_SHOW_NAME_COMMAND received command to show statistics for %1
+The stats module received a command to show the statistics that it has
+collected for the given item.
+
+% STATS_RECEIVED_SHUTDOWN_COMMAND shutdown command received
+A shutdown command was sent to the stats module and it will now shut down.
+
+% STATS_RECEIVED_STATUS_COMMAND received command to return status
+A status command was sent to the stats module. It will return a
+response indicating that it is running normally.
+
+% STATS_RECEIVED_UNKNOWN_COMMAND received unknown command: %1
+An unknown command has been sent to the stats module. The stats module
+will respond with an error and the command will be ignored.
+
+% STATS_SEND_REQUEST_BOSS requesting boss to send statistics
+This debug message is printed when a request is sent to the boss module
+to send its data to the stats module.
+
+% STATS_STOPPED_BY_KEYBOARD keyboard interrupt, shutting down
+There was a keyboard interrupt signal to stop the stats module. The
+daemon will now shut down.
+
+% STATS_UNKNOWN_COMMAND_IN_SPEC unknown command in specification file: %1
+The specification file for the stats module contains a command that
+is unknown in the implementation. The most likely cause is an
+installation problem, where the specification file stats.spec is
+from a different version of BIND 10 than the stats module itself.
+Please check your installation.
+
+% STATS_STARTING starting
+The stats module will be now starting.
+
+% STATS_RECEIVED_SHOWSCHEMA_ALL_COMMAND received command to show all statistics schema
+The stats module received a command to show all statistics schemas of all modules.
+
+% STATS_RECEIVED_SHOWSCHEMA_NAME_COMMAND received command to show statistics schema for %1
+The stats module received a command to show the specified statistics schema of the specified module.
+
+% STATS_START_ERROR stats module error: %1
+An internal error occurred while starting the stats module. The stats
+module will be now shutting down.

src/bin/stats/tests/Makefile.am

@@ -1,20 +1,28 @@
-SUBDIRS = isc http testdata
 PYCOVERAGE_RUN = @PYCOVERAGE_RUN@
 PYTESTS = b10-stats_test.py b10-stats-httpd_test.py
-EXTRA_DIST = $(PYTESTS) fake_time.py fake_socket.py fake_select.py
-CLEANFILES = fake_time.pyc fake_socket.pyc fake_select.pyc
+EXTRA_DIST = $(PYTESTS) test_utils.py
+CLEANFILES = test_utils.pyc
+
+# If necessary (rare cases), explicitly specify paths to dynamic libraries
+# required by loadable python modules.
+LIBRARY_PATH_PLACEHOLDER =
+if SET_ENV_LIBRARY_PATH
+LIBRARY_PATH_PLACEHOLDER += $(ENV_LIBRARY_PATH)=$(abs_top_builddir)/src/lib/cryptolink/.libs:$(abs_top_builddir)/src/lib/dns/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/dns/python/.libs:$(abs_top_builddir)/src/lib/cc/.libs:$(abs_top_builddir)/src/lib/config/.libs:$(abs_top_builddir)/src/lib/log/.libs:$(abs_top_builddir)/src/lib/util/.libs:$(abs_top_builddir)/src/lib/exceptions/.libs:$(abs_top_builddir)/src/lib/datasrc/.libs:$$$(ENV_LIBRARY_PATH)
+endif
 
 # test using command-line arguments, so use check-local target instead of TESTS
 check-local:
 if ENABLE_PYTHON_COVERAGE
-	touch $(abs_top_srcdir)/.coverage 
+	touch $(abs_top_srcdir)/.coverage
 	rm -f .coverage
 	${LN_S} $(abs_top_srcdir)/.coverage .coverage
 endif
 	for pytest in $(PYTESTS) ; do \
 	echo Running test: $$pytest ; \
-	env PYTHONPATH=$(abs_top_srcdir)/src/lib/python:$(abs_top_builddir)/src/lib/python:$(abs_top_builddir)/src/bin/stats:$(abs_top_builddir)/src/bin/stats/tests \
+	$(LIBRARY_PATH_PLACEHOLDER) \
+	PYTHONPATH=$(COMMON_PYTHON_PATH):$(abs_top_builddir)/src/bin/stats:$(abs_top_builddir)/src/bin/stats/tests:$(abs_top_builddir)/src/bin/msgq:$(abs_top_builddir)/src/lib/python/isc/config \
 	B10_FROM_SOURCE=$(abs_top_srcdir) \
+	CONFIG_TESTDATA_PATH=$(abs_top_srcdir)/src/lib/config/tests/testdata \
 	$(PYCOVERAGE_RUN) $(abs_srcdir)/$$pytest || exit ; \
 	done
 

src/bin/stats/tests/b10-stats-httpd_test.py

@@ -13,166 +13,269 @@
 # NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
 # WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
+"""
+In each of these tests we start several virtual components. They are
+not the real components, no external processes are started. They are
+just simple mock objects running each in its own thread and pretending
+to be bind10 modules. This helps testing the stats http server in a
+close to real environment.
+"""
+
 import unittest
 import os
-import http.server
-import string
-import fake_select
 import imp
-import sys
-import fake_socket
-
-import isc.cc
+import socket
+import errno
+import select
+import string
+import time
+import threading
+import http.client
+import xml.etree.ElementTree
+import random
 
+import isc
 import stats_httpd
-stats_httpd.socket = fake_socket
-stats_httpd.select = fake_select
+import stats
+from test_utils import BaseModules, ThreadingServerManager, MyStats, MyStatsHttpd, SignalHandler, send_command, send_shutdown
 
 DUMMY_DATA = {
-    "auth.queries.tcp": 10000,
-    "auth.queries.udp": 12000,
-    "bind10.boot_time": "2011-03-04T11:59:05Z",
-    "report_time": "2011-03-04T11:59:19Z",
-    "stats.boot_time": "2011-03-04T11:59:06Z",
-    "stats.last_update_time": "2011-03-04T11:59:07Z",
-    "stats.lname": "4d70d40a_c@host",
-    "stats.start_time": "2011-03-04T11:59:06Z",
-    "stats.timestamp": 1299239959.560846
+    'Boss' : {
+        "boot_time": "2011-03-04T11:59:06Z"
+        },
+    'Auth' : {
+        "queries.tcp": 2,
+        "queries.udp": 3
+        },
+    'Stats' : {
+        "report_time": "2011-03-04T11:59:19Z",
+        "boot_time": "2011-03-04T11:59:06Z",
+        "last_update_time": "2011-03-04T11:59:07Z",
+        "lname": "4d70d40a_c@host",
+        "timestamp": 1299239959.560846
+        }
     }
 
-def push_answer(stats_httpd):
-    stats_httpd.cc_session.group_sendmsg(
-        { 'result': 
-          [ 0, DUMMY_DATA ] }, "Stats")
-
-def pull_query(stats_httpd):
-    (msg, env) = stats_httpd.cc_session.group_recvmsg()
-    if 'result' in msg:
-        (ret, arg) = isc.config.ccsession.parse_answer(msg)
-    else:
-        (ret, arg) = isc.config.ccsession.parse_command(msg)
-    return (ret, arg, env)
+def get_availaddr(address='127.0.0.1', port=8001):
+    """returns a tuple of address and port which is available to
+    listen on the platform. The first argument is a address for
+    search. The second argument is a port for search. If a set of
+    address and port is failed on the search for the availability, the
+    port number is increased and it goes on the next trial until the
+    available set of address and port is looked up. If the port number
+    reaches over 65535, it may stop the search and raise a
+    OverflowError exception."""
+    while True:
+        for addr in socket.getaddrinfo(
+            address, port, 0,
+            socket.SOCK_STREAM, socket.IPPROTO_TCP):
+            sock = socket.socket(addr[0], socket.SOCK_STREAM)
+            try:
+                sock.bind((address, port))
+                return (address, port)
+            except socket.error:
+                continue
+            finally:
+                if sock: sock.close()
+        # This address and port number are already in use.
+        # next port number is added
+        port = port + 1
+
+def is_ipv6_enabled(address='::1', port=8001):
+    """checks IPv6 enabled on the platform. address for check is '::1'
+    and port for check is random number between 8001 and
+    65535. Retrying is 3 times even if it fails. The built-in socket
+    module provides a 'has_ipv6' parameter, but it's not used here
+    because there may be a situation where the value is True on an
+    environment where the IPv6 config is disabled."""
+    for p in random.sample(range(port, 65535), 3):
+        try:
+            sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
+            sock.bind((address, p))
+            return True
+        except socket.error:
+            continue
+        finally:
+            if sock: sock.close()
+    return False
 
 class TestHttpHandler(unittest.TestCase):
     """Tests for HttpHandler class"""
-
     def setUp(self):
-        self.verbose = True
-        self.stats_httpd = stats_httpd.StatsHttpd(self.verbose)
-        self.assertTrue(type(self.stats_httpd.httpd) is list)
-        self.httpd = self.stats_httpd.httpd
-        for ht in self.httpd:
-            self.assertTrue(ht.verbose)
-        self.stats_httpd.cc_session.verbose = False
+        # set the signal handler for deadlock
+        self.sig_handler = SignalHandler(self.fail)
+        self.base = BaseModules()
+        self.stats_server = ThreadingServerManager(MyStats)
+        self.stats = self.stats_server.server
+        self.stats_server.run()
+        (self.address, self.port) = get_availaddr()
+        self.stats_httpd_server = ThreadingServerManager(MyStatsHttpd, (self.address, self.port))
+        self.stats_httpd = self.stats_httpd_server.server
+        self.stats_httpd_server.run()
+        self.client = http.client.HTTPConnection(self.address, self.port)
+        self.client._http_vsn_str = 'HTTP/1.0\n'
+        self.client.connect()
 
-    def test_do_GET(self):
-        for ht in self.httpd:
-            self._test_do_GET(ht._handler)
+    def tearDown(self):
+        self.client.close()
+        self.stats_httpd_server.shutdown()
+        self.stats_server.shutdown()
+        self.base.shutdown()
+        # reset the signal handler
+        self.sig_handler.reset()
 
-    def _test_do_GET(self, handler):
+    def test_do_GET(self):
+        self.assertTrue(type(self.stats_httpd.httpd) is list)
+        self.assertEqual(len(self.stats_httpd.httpd), 1)
+        self.assertEqual((self.address, self.port), self.stats_httpd.http_addrs[0])
 
         # URL is '/bind10/statistics/xml'
-        handler.path = stats_httpd.XML_URL_PATH
-        push_answer(self.stats_httpd)
-        handler.do_GET()
-        (ret, arg, env) = pull_query(self.stats_httpd)
-        self.assertEqual(ret, "show")
-        self.assertIsNone(arg)
-        self.assertTrue('group' in env)
-        self.assertEqual(env['group'], 'Stats')
-        self.assertEqual(handler.response.code, 200)
-        self.assertEqual(handler.response.headers["Content-type"], "text/xml")
-        self.assertTrue(handler.response.headers["Content-Length"] > 0)
-        self.assertTrue(handler.response.wrote_headers)
-        self.assertTrue(handler.response.body.find(stats_httpd.XSD_NAMESPACE)>0)
-        self.assertTrue(handler.response.body.find(stats_httpd.XSD_URL_PATH)>0)
-        for (k, v) in DUMMY_DATA.items():
-            self.assertTrue(handler.response.body.find(str(k))>0)
-            self.assertTrue(handler.response.body.find(str(v))>0)
+        self.client.putrequest('GET', stats_httpd.XML_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.getheader("Content-type"), "text/xml")
+        self.assertTrue(int(response.getheader("Content-Length")) > 0)
+        self.assertEqual(response.status, 200)
+        root = xml.etree.ElementTree.parse(response).getroot()
+        self.assertTrue(root.tag.find('stats_data') > 0)
+        for (k,v) in root.attrib.items():
+            if k.find('schemaLocation') > 0:
+                self.assertEqual(v, stats_httpd.XSD_NAMESPACE + ' ' + stats_httpd.XSD_URL_PATH)
+        for mod in DUMMY_DATA:
+            for (item, value) in DUMMY_DATA[mod].items():
+                self.assertIsNotNone(root.find(mod + '/' + item))
 
         # URL is '/bind10/statitics/xsd'
-        handler.path = stats_httpd.XSD_URL_PATH
-        handler.do_GET()
-        self.assertEqual(handler.response.code, 200)
-        self.assertEqual(handler.response.headers["Content-type"], "text/xml")
-        self.assertTrue(handler.response.headers["Content-Length"] > 0)
-        self.assertTrue(handler.response.wrote_headers)
-        self.assertTrue(handler.response.body.find(stats_httpd.XSD_NAMESPACE)>0)
-        for (k, v) in DUMMY_DATA.items():
-            self.assertTrue(handler.response.body.find(str(k))>0)
+        self.client.putrequest('GET', stats_httpd.XSD_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.getheader("Content-type"), "text/xml")
+        self.assertTrue(int(response.getheader("Content-Length")) > 0)
+        self.assertEqual(response.status, 200)
+        root = xml.etree.ElementTree.parse(response).getroot()
+        url_xmlschema = '{http://www.w3.org/2001/XMLSchema}'
+        tags = [ url_xmlschema + t for t in [ 'element', 'complexType', 'all', 'element' ] ]
+        xsdpath = '/'.join(tags)
+        self.assertTrue(root.tag.find('schema') > 0)
+        self.assertTrue(hasattr(root, 'attrib'))
+        self.assertTrue('targetNamespace' in root.attrib)
+        self.assertEqual(root.attrib['targetNamespace'],
+                         stats_httpd.XSD_NAMESPACE)
+        for elm in root.findall(xsdpath):
+            self.assertIsNotNone(elm.attrib['name'])
+            self.assertTrue(elm.attrib['name'] in DUMMY_DATA)
 
         # URL is '/bind10/statitics/xsl'
-        handler.path = stats_httpd.XSL_URL_PATH
-        handler.do_GET()
-        self.assertEqual(handler.response.code, 200)
-        self.assertEqual(handler.response.headers["Content-type"], "text/xml")
-        self.assertTrue(handler.response.headers["Content-Length"] > 0)
-        self.assertTrue(handler.response.wrote_headers)
-        self.assertTrue(handler.response.body.find(stats_httpd.XSD_NAMESPACE)>0)
-        for (k, v) in DUMMY_DATA.items():
-            self.assertTrue(handler.response.body.find(str(k))>0)
+        self.client.putrequest('GET', stats_httpd.XSL_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.getheader("Content-type"), "text/xml")
+        self.assertTrue(int(response.getheader("Content-Length")) > 0)
+        self.assertEqual(response.status, 200)
+        root = xml.etree.ElementTree.parse(response).getroot()
+        url_trans = '{http://www.w3.org/1999/XSL/Transform}'
+        url_xhtml = '{http://www.w3.org/1999/xhtml}'
+        xslpath = url_trans + 'template/' + url_xhtml + 'tr'
+        self.assertEqual(root.tag, url_trans + 'stylesheet')
+        for tr in root.findall(xslpath):
+            tds = tr.findall(url_xhtml + 'td')
+            self.assertIsNotNone(tds)
+            self.assertEqual(type(tds), list)
+            self.assertTrue(len(tds) > 2)
+            self.assertTrue(hasattr(tds[0], 'text'))
+            self.assertTrue(tds[0].text in DUMMY_DATA)
+            valueof = tds[2].find(url_trans + 'value-of')
+            self.assertIsNotNone(valueof)
+            self.assertTrue(hasattr(valueof, 'attrib'))
+            self.assertIsNotNone(valueof.attrib)
+            self.assertTrue('select' in valueof.attrib)
+            self.assertTrue(valueof.attrib['select'] in \
+                                [ tds[0].text+'/'+item for item in DUMMY_DATA[tds[0].text].keys() ])
 
         # 302 redirect
-        handler.path = '/'
-        handler.headers = {'Host': 'my.host.domain'}
-        handler.do_GET()
-        self.assertEqual(handler.response.code, 302)
-        self.assertEqual(handler.response.headers["Location"],
-                         "http://my.host.domain%s" % stats_httpd.XML_URL_PATH)
+        self.client._http_vsn_str = 'HTTP/1.1'
+        self.client.putrequest('GET', '/')
+        self.client.putheader('Host', self.address)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 302)
+        self.assertEqual(response.getheader('Location'),
+                         "http://%s:%d%s" % (self.address, self.port, stats_httpd.XML_URL_PATH))
 
         # 404 NotFound
-        handler.path = '/path/to/foo/bar'
-        handler.headers = {}
-        handler.do_GET()
-        self.assertEqual(handler.response.code, 404)
-
-        # failure case(connection with Stats is down)
-        handler.path = stats_httpd.XML_URL_PATH
-        push_answer(self.stats_httpd)
-        self.assertFalse(self.stats_httpd.cc_session._socket._closed)
-        self.stats_httpd.cc_session._socket._closed = True
-        handler.do_GET()
-        self.stats_httpd.cc_session._socket._closed = False
-        self.assertEqual(handler.response.code, 500)
-        self.stats_httpd.cc_session._clear_queues()
-
-        # failure case(Stats module returns err)
-        handler.path = stats_httpd.XML_URL_PATH
-        self.stats_httpd.cc_session.group_sendmsg(
-            { 'result': [ 1, "I have an error." ] }, "Stats")
-        self.assertFalse(self.stats_httpd.cc_session._socket._closed)
-        self.stats_httpd.cc_session._socket._closed = False
-        handler.do_GET()
-        self.assertEqual(handler.response.code, 500)
-        self.stats_httpd.cc_session._clear_queues()
+        self.client._http_vsn_str = 'HTTP/1.0'
+        self.client.putrequest('GET', '/path/to/foo/bar')
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 404)
+
+
+    def test_do_GET_failed1(self):
+        # checks status
+        self.assertEqual(send_command("status", "Stats"),
+                         (0, "Stats is up. (PID " + str(os.getpid()) + ")"))
+        # failure case(Stats is down)
+        self.assertTrue(self.stats.running)
+        self.assertEqual(send_shutdown("Stats"), (0, None)) # Stats is down
+        self.assertFalse(self.stats.running)
+        self.stats_httpd.cc_session.set_timeout(milliseconds=100)
+
+        # request XML
+        self.client.putrequest('GET', stats_httpd.XML_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 500)
+
+        # request XSD
+        self.client.putrequest('GET', stats_httpd.XSD_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 500)
+
+        # request XSL
+        self.client.putrequest('GET', stats_httpd.XSL_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 500)
+
+    def test_do_GET_failed2(self):
+        # failure case(Stats replies an error)
+        self.stats.mccs.set_command_handler(
+            lambda cmd, args: \
+                isc.config.ccsession.create_answer(1, "I have an error.")
+            )
+
+        # request XML
+        self.client.putrequest('GET', stats_httpd.XML_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 500)
+
+        # request XSD
+        self.client.putrequest('GET', stats_httpd.XSD_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 500)
+
+        # request XSL
+        self.client.putrequest('GET', stats_httpd.XSL_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 500)
 
     def test_do_HEAD(self):
-        for ht in self.httpd:
-            self._test_do_HEAD(ht._handler)
-
-    def _test_do_HEAD(self, handler):
-        handler.path = '/path/to/foo/bar'
-        handler.do_HEAD()
-        self.assertEqual(handler.response.code, 404)
-
-    def test_log_message(self):
-        for ht in self.httpd:
-            self._test_log_message(ht._handler)
-
-    def _test_log_message(self, handler):
-        # switch write_log function
-        handler.server.log_writer = handler.response._write_log
-        log_message = 'ABCDEFG'
-        handler.log_message("%s", log_message)
-        self.assertEqual(handler.response.log, 
-                         "[b10-stats-httpd] %s - - [%s] %s\n" %
-                         (handler.address_string(),
-                          handler.log_date_time_string(),
-                          log_message))
+        self.client.putrequest('HEAD', stats_httpd.XML_URL_PATH)
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 200)
+
+        self.client.putrequest('HEAD', '/path/to/foo/bar')
+        self.client.endheaders()
+        response = self.client.getresponse()
+        self.assertEqual(response.status, 404)
 
 class TestHttpServerError(unittest.TestCase):
     """Tests for HttpServerError exception"""
-
     def test_raises(self):
         try:
             raise stats_httpd.HttpServerError('Nothing')
@@ -181,20 +284,24 @@ class TestHttpServerError(unittest.TestCase):
 
 class TestHttpServer(unittest.TestCase):
     """Tests for HttpServer class"""
+    def setUp(self):
+        # set the signal handler for deadlock
+        self.sig_handler = SignalHandler(self.fail)
+        self.base = BaseModules()
+
+    def tearDown(self):
+        if hasattr(self, "stats_httpd"):
+            self.stats_httpd.stop()
+        self.base.shutdown()
+        # reset the signal handler
+        self.sig_handler.reset()
 
     def test_httpserver(self):
-        self.verbose = True
-        self.stats_httpd = stats_httpd.StatsHttpd(self.verbose)
-        self.stats_httpd.cc_session.verbose = False
-        for ht in self.stats_httpd.httpd:
-            self.assertTrue(ht.server_address in self.stats_httpd.http_addrs)
-            self.assertEqual(ht.verbose, self.verbose)
-            self.assertEqual(ht.xml_handler, self.stats_httpd.xml_handler)
-            self.assertEqual(ht.xsd_handler, self.stats_httpd.xsd_handler)
-            self.assertEqual(ht.xsl_handler, self.stats_httpd.xsl_handler)
-            self.assertEqual(ht.log_writer, self.stats_httpd.write_log)
-            self.assertTrue(isinstance(ht._handler, stats_httpd.HttpHandler))
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.assertEqual(type(self.stats_httpd.httpd), list)
+        self.assertEqual(len(self.stats_httpd.httpd), 1)
+        for httpd in self.stats_httpd.httpd:
+            self.assertTrue(isinstance(httpd, stats_httpd.HttpServer))
 
 class TestStatsHttpdError(unittest.TestCase):
     """Tests for StatsHttpdError exception"""
@@ -209,136 +316,173 @@ class TestStatsHttpd(unittest.TestCase):
     """Tests for StatsHttpd class"""
 
     def setUp(self):
-        self.verbose = True
-        fake_socket._CLOSED = False
-        fake_socket.has_ipv6 = True
-        self.stats_httpd = stats_httpd.StatsHttpd(self.verbose)
-        self.stats_httpd.cc_session.verbose = False
+        # set the signal handler for deadlock
+        self.sig_handler = SignalHandler(self.fail)
+        self.base = BaseModules()
+        self.stats_server = ThreadingServerManager(MyStats)
+        self.stats_server.run()
+        # checking IPv6 enabled on this platform
+        self.ipv6_enabled = is_ipv6_enabled()
 
     def tearDown(self):
-        self.stats_httpd.stop()
+        if hasattr(self, "stats_httpd"):
+            self.stats_httpd.stop()
+        self.stats_server.shutdown()
+        self.base.shutdown()
+        # reset the signal handler
+        self.sig_handler.reset()
 
     def test_init(self):
-        self.assertTrue(self.stats_httpd.verbose)
-        self.assertFalse(self.stats_httpd.mccs.get_socket()._closed)
-        self.assertEqual(self.stats_httpd.mccs.get_socket().fileno(),
-                         id(self.stats_httpd.mccs.get_socket()))
-        for ht in self.stats_httpd.httpd:
-            self.assertFalse(ht.socket._closed)
-            self.assertEqual(ht.socket.fileno(), id(ht.socket))
-        fake_socket._CLOSED = True
-        self.assertRaises(isc.cc.session.SessionError,
-                          stats_httpd.StatsHttpd)
-        fake_socket._CLOSED = False
+        server_address = get_availaddr()
+        self.stats_httpd = MyStatsHttpd(server_address)
+        self.assertEqual(self.stats_httpd.running, False)
+        self.assertEqual(self.stats_httpd.poll_intval, 0.5)
+        self.assertNotEqual(len(self.stats_httpd.httpd), 0)
+        self.assertEqual(type(self.stats_httpd.mccs), isc.config.ModuleCCSession)
+        self.assertEqual(type(self.stats_httpd.cc_session), isc.cc.Session)
+        self.assertEqual(len(self.stats_httpd.config), 2)
+        self.assertTrue('listen_on' in self.stats_httpd.config)
+        self.assertEqual(len(self.stats_httpd.config['listen_on']), 1)
+        self.assertTrue('address' in self.stats_httpd.config['listen_on'][0])
+        self.assertTrue('port' in self.stats_httpd.config['listen_on'][0])
+        self.assertTrue(server_address in set(self.stats_httpd.http_addrs))
+
+    def test_openclose_mccs(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.stats_httpd.close_mccs()
+        self.assertEqual(self.stats_httpd.mccs, None)
+        self.stats_httpd.open_mccs()
+        self.assertIsNotNone(self.stats_httpd.mccs)
+        self.stats_httpd.mccs = None
+        self.assertEqual(self.stats_httpd.mccs, None)
+        self.assertEqual(self.stats_httpd.close_mccs(), None)
 
     def test_mccs(self):
-        self.stats_httpd.open_mccs()
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.assertIsNotNone(self.stats_httpd.mccs.get_socket())
         self.assertTrue(
-            isinstance(self.stats_httpd.mccs.get_socket(), fake_socket.socket))
+            isinstance(self.stats_httpd.mccs.get_socket(), socket.socket))
         self.assertTrue(
             isinstance(self.stats_httpd.cc_session, isc.cc.session.Session))
-        self.assertTrue(
-            isinstance(self.stats_httpd.stats_module_spec, isc.config.ModuleSpec))
-        for cfg in self.stats_httpd.stats_config_spec:
-            self.assertTrue('item_name' in cfg)
-            self.assertTrue(cfg['item_name'] in DUMMY_DATA)
-        self.assertTrue(len(self.stats_httpd.stats_config_spec), len(DUMMY_DATA))
-
-    def test_load_config(self):
-        self.stats_httpd.load_config()
-        self.assertTrue(('127.0.0.1', 8000) in set(self.stats_httpd.http_addrs))
+        statistics_spec = self.stats_httpd.get_stats_spec()
+        for mod in DUMMY_DATA:
+            self.assertTrue(mod in statistics_spec)
+            for cfg in statistics_spec[mod]:
+                self.assertTrue('item_name' in cfg)
+                self.assertTrue(cfg['item_name'] in DUMMY_DATA[mod])
+            self.assertTrue(len(statistics_spec[mod]), len(DUMMY_DATA[mod]))
+        self.stats_httpd.close_mccs()
+        self.assertIsNone(self.stats_httpd.mccs)
 
     def test_httpd(self):
         # dual stack (addresses is ipv4 and ipv6)
-        fake_socket.has_ipv6 = True
-        self.assertTrue(('127.0.0.1', 8000) in set(self.stats_httpd.http_addrs))
-        self.stats_httpd.http_addrs = [ ('::1', 8000), ('127.0.0.1', 8000) ]
-        self.assertTrue(
-            stats_httpd.HttpServer.address_family in set([fake_socket.AF_INET, fake_socket.AF_INET6]))
-        self.stats_httpd.open_httpd()
-        for ht in self.stats_httpd.httpd:
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
-        self.stats_httpd.close_httpd()
+        if self.ipv6_enabled:
+            server_addresses = (get_availaddr('::1'), get_availaddr())
+            self.stats_httpd = MyStatsHttpd(*server_addresses)
+            for ht in self.stats_httpd.httpd:
+                self.assertTrue(isinstance(ht, stats_httpd.HttpServer))
+                self.assertTrue(ht.address_family in set([socket.AF_INET, socket.AF_INET6]))
+                self.assertTrue(isinstance(ht.socket, socket.socket))
 
         # dual stack (address is ipv6)
-        fake_socket.has_ipv6 = True
-        self.stats_httpd.http_addrs = [ ('::1', 8000) ]
-        self.stats_httpd.open_httpd()
-        for ht in self.stats_httpd.httpd:
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
-        self.stats_httpd.close_httpd()
-
-        # dual stack (address is ipv4)
-        fake_socket.has_ipv6 = True
-        self.stats_httpd.http_addrs = [ ('127.0.0.1', 8000) ]
-        self.stats_httpd.open_httpd()
-        for ht in self.stats_httpd.httpd:
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
-        self.stats_httpd.close_httpd()
-
-        # only-ipv4 single stack
-        fake_socket.has_ipv6 = False
-        self.stats_httpd.http_addrs = [ ('127.0.0.1', 8000) ]
-        self.stats_httpd.open_httpd()
+        if self.ipv6_enabled:
+            server_addresses = get_availaddr('::1')
+            self.stats_httpd = MyStatsHttpd(server_addresses)
+            for ht in self.stats_httpd.httpd:
+                self.assertTrue(isinstance(ht, stats_httpd.HttpServer))
+                self.assertEqual(ht.address_family, socket.AF_INET6)
+                self.assertTrue(isinstance(ht.socket, socket.socket))
+
+        # dual/single stack (address is ipv4)
+        server_addresses = get_availaddr()
+        self.stats_httpd = MyStatsHttpd(server_addresses)
         for ht in self.stats_httpd.httpd:
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
-        self.stats_httpd.close_httpd()
-
-        # only-ipv4 single stack (force set ipv6 )
-        fake_socket.has_ipv6 = False
-        self.stats_httpd.http_addrs = [ ('::1', 8000) ]
-        self.assertRaises(stats_httpd.HttpServerError,
-            self.stats_httpd.open_httpd)
-
-        # hostname
-        self.stats_httpd.http_addrs = [ ('localhost', 8000) ]
-        self.stats_httpd.open_httpd()
-        for ht in self.stats_httpd.httpd:
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
-        self.stats_httpd.close_httpd()
+            self.assertTrue(isinstance(ht, stats_httpd.HttpServer))
+            self.assertEqual(ht.address_family, socket.AF_INET)
+            self.assertTrue(isinstance(ht.socket, socket.socket))
 
-        self.stats_httpd.http_addrs = [ ('my.host.domain', 8000) ]
-        self.stats_httpd.open_httpd()
+        # any address (IPv4)
+        server_addresses = get_availaddr(address='0.0.0.0')
+        self.stats_httpd = MyStatsHttpd(server_addresses)
         for ht in self.stats_httpd.httpd:
-            self.assertTrue(isinstance(ht.socket, fake_socket.socket))
-        self.stats_httpd.close_httpd()
+            self.assertTrue(isinstance(ht, stats_httpd.HttpServer))
+            self.assertEqual(ht.address_family,socket.AF_INET)
+            self.assertTrue(isinstance(ht.socket, socket.socket))
+
+        # any address (IPv6)
+        if self.ipv6_enabled:
+            server_addresses = get_availaddr(address='::')
+            self.stats_httpd = MyStatsHttpd(server_addresses)
+            for ht in self.stats_httpd.httpd:
+                self.assertTrue(isinstance(ht, stats_httpd.HttpServer))
+                self.assertEqual(ht.address_family,socket.AF_INET6)
+                self.assertTrue(isinstance(ht.socket, socket.socket))
+
+        # existent hostname
+        self.assertRaises(stats_httpd.HttpServerError, MyStatsHttpd,
+                          get_availaddr(address='localhost'))
+
+        # nonexistent hostname
+        self.assertRaises(stats_httpd.HttpServerError, MyStatsHttpd, ('my.host.domain', 8000))
 
         # over flow of port number
-        self.stats_httpd.http_addrs = [ ('', 80000) ]
-        self.assertRaises(stats_httpd.HttpServerError, self.stats_httpd.open_httpd)
+        self.assertRaises(stats_httpd.HttpServerError, MyStatsHttpd, ('127.0.0.1', 80000))
+
         # negative
-        self.stats_httpd.http_addrs = [ ('', -8000) ]
-        self.assertRaises(stats_httpd.HttpServerError, self.stats_httpd.open_httpd)
-        # alphabet
-        self.stats_httpd.http_addrs = [ ('', 'ABCDE') ]
-        self.assertRaises(stats_httpd.HttpServerError, self.stats_httpd.open_httpd)
-
-    def test_start(self):
-        self.stats_httpd.cc_session.group_sendmsg(
-            { 'command': [ "shutdown" ] }, "StatsHttpd")
-        self.stats_httpd.start()
-        self.stats_httpd = stats_httpd.StatsHttpd(self.verbose)
-        self.stats_httpd.cc_session.verbose = False
-        self.assertRaises(
-            fake_select.error, self.stats_httpd.start)
+        self.assertRaises(stats_httpd.HttpServerError, MyStatsHttpd, ('127.0.0.1', -8000))
 
-    def test_stop(self):
-        # success case
-        fake_socket._CLOSED = False
-        self.stats_httpd.stop()
+        # alphabet
+        self.assertRaises(stats_httpd.HttpServerError, MyStatsHttpd, ('127.0.0.1', 'ABCDE'))
+
+        # Address already in use
+        server_addresses = get_availaddr()
+        self.stats_httpd_server = ThreadingServerManager(MyStatsHttpd, server_addresses)
+        self.stats_httpd_server.run()
+        self.assertRaises(stats_httpd.HttpServerError, MyStatsHttpd, server_addresses)
+        send_shutdown("StatsHttpd")
+
+    def test_running(self):
+        self.stats_httpd_server = ThreadingServerManager(MyStatsHttpd, get_availaddr())
+        self.stats_httpd = self.stats_httpd_server.server
         self.assertFalse(self.stats_httpd.running)
-        self.assertIsNone(self.stats_httpd.mccs)
-        for ht in self.stats_httpd.httpd:
-            self.assertTrue(ht.socket._closed)
-        self.assertTrue(self.stats_httpd.cc_session._socket._closed)
+        self.stats_httpd_server.run()
+        self.assertEqual(send_command("status", "StatsHttpd"),
+                         (0, "Stats Httpd is up. (PID " + str(os.getpid()) + ")"))
+        self.assertTrue(self.stats_httpd.running)
+        self.assertEqual(send_shutdown("StatsHttpd"), (0, None))
+        self.assertFalse(self.stats_httpd.running)
+        self.stats_httpd_server.shutdown()
+
         # failure case
-        self.stats_httpd.cc_session._socket._closed = False
-        self.stats_httpd.open_mccs()
-        self.stats_httpd.cc_session._socket._closed = True
-        self.stats_httpd.stop() # No excetion raises
-        self.stats_httpd.cc_session._socket._closed = False
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.stats_httpd.cc_session.close()
+        self.assertRaises(ValueError, self.stats_httpd.start)
+
+    def test_failure_with_a_select_error (self):
+        """checks select.error is raised if the exception except
+        errno.EINTR is raised while it's selecting"""
+        def raise_select_except(*args):
+            raise select.error('dummy error')
+        orig_select = stats_httpd.select.select
+        stats_httpd.select.select = raise_select_except
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.assertRaises(select.error, self.stats_httpd.start)
+        stats_httpd.select.select = orig_select
+
+    def test_nofailure_with_errno_EINTR(self):
+        """checks no exception is raised if errno.EINTR is raised
+        while it's selecting"""
+        def raise_select_except(*args):
+            raise select.error(errno.EINTR)
+        orig_select = stats_httpd.select.select
+        stats_httpd.select.select = raise_select_except
+        self.stats_httpd_server = ThreadingServerManager(MyStatsHttpd, get_availaddr())
+        self.stats_httpd_server.run()
+        self.stats_httpd_server.shutdown()
+        stats_httpd.select.select = orig_select
 
     def test_open_template(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
         # successful conditions
         tmpl = self.stats_httpd.open_template(stats_httpd.XML_TEMPLATE_LOCATION)
         self.assertTrue(isinstance(tmpl, string.Template))
@@ -372,13 +516,13 @@ class TestStatsHttpd(unittest.TestCase):
             self.stats_httpd.open_template, '/path/to/foo/bar')
 
     def test_commands(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
         self.assertEqual(self.stats_httpd.command_handler("status", None),
                          isc.config.ccsession.create_answer(
                 0, "Stats Httpd is up. (PID " + str(os.getpid()) + ")"))
         self.stats_httpd.running = True
         self.assertEqual(self.stats_httpd.command_handler("shutdown", None),
-                         isc.config.ccsession.create_answer(
-                0, "Stats Httpd is shutting down."))
+                         isc.config.ccsession.create_answer(0))
         self.assertFalse(self.stats_httpd.running)
         self.assertEqual(
             self.stats_httpd.command_handler("__UNKNOWN_COMMAND__", None),
@@ -386,48 +530,153 @@ class TestStatsHttpd(unittest.TestCase):
                 1, "Unknown command: __UNKNOWN_COMMAND__"))
 
     def test_config(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
         self.assertEqual(
             self.stats_httpd.config_handler(dict(_UNKNOWN_KEY_=None)),
             isc.config.ccsession.create_answer(
-                    1, "Unknown known config: _UNKNOWN_KEY_"))
-        self.assertEqual(
-            self.stats_httpd.config_handler(
-                        dict(listen_on=[dict(address="::2",port=8000)])),
-            isc.config.ccsession.create_answer(0))
-        self.assertTrue("listen_on" in self.stats_httpd.config)
-        for addr in self.stats_httpd.config["listen_on"]:
-            self.assertTrue("address" in addr)
-            self.assertTrue("port" in addr)
-            self.assertTrue(addr["address"] == "::2")
-            self.assertTrue(addr["port"] == 8000)
+                1, "unknown item _UNKNOWN_KEY_"))
 
+        addresses = get_availaddr()
         self.assertEqual(
             self.stats_httpd.config_handler(
-                        dict(listen_on=[dict(address="::1",port=80)])),
+                dict(listen_on=[dict(address=addresses[0],port=addresses[1])])),
             isc.config.ccsession.create_answer(0))
         self.assertTrue("listen_on" in self.stats_httpd.config)
         for addr in self.stats_httpd.config["listen_on"]:
             self.assertTrue("address" in addr)
             self.assertTrue("port" in addr)
-            self.assertTrue(addr["address"] == "::1")
-            self.assertTrue(addr["port"] == 80)
-
+            self.assertTrue(addr["address"] == addresses[0])
+            self.assertTrue(addr["port"] == addresses[1])
+
+        if self.ipv6_enabled:
+            addresses = get_availaddr("::1")
+            self.assertEqual(
+                self.stats_httpd.config_handler(
+                dict(listen_on=[dict(address=addresses[0],port=addresses[1])])),
+                isc.config.ccsession.create_answer(0))
+            self.assertTrue("listen_on" in self.stats_httpd.config)
+            for addr in self.stats_httpd.config["listen_on"]:
+                self.assertTrue("address" in addr)
+                self.assertTrue("port" in addr)
+                self.assertTrue(addr["address"] == addresses[0])
+                self.assertTrue(addr["port"] == addresses[1])
+
+        addresses = get_availaddr()
         self.assertEqual(
             self.stats_httpd.config_handler(
-                        dict(listen_on=[dict(address="1.2.3.4",port=54321)])),
+                dict(listen_on=[dict(address=addresses[0],port=addresses[1])])),
             isc.config.ccsession.create_answer(0))
         self.assertTrue("listen_on" in self.stats_httpd.config)
         for addr in self.stats_httpd.config["listen_on"]:
             self.assertTrue("address" in addr)
             self.assertTrue("port" in addr)
-            self.assertTrue(addr["address"] == "1.2.3.4")
-            self.assertTrue(addr["port"] == 54321)
+            self.assertTrue(addr["address"] == addresses[0])
+            self.assertTrue(addr["port"] == addresses[1])
         (ret, arg) = isc.config.ccsession.parse_answer(
             self.stats_httpd.config_handler(
                 dict(listen_on=[dict(address="1.2.3.4",port=543210)]))
             )
         self.assertEqual(ret, 1)
 
+    def test_xml_handler(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.stats_httpd.get_stats_data = lambda: \
+            { 'Dummy' : { 'foo':'bar' } }
+        xml_body1 = self.stats_httpd.open_template(
+            stats_httpd.XML_TEMPLATE_LOCATION).substitute(
+            xml_string='<Dummy><foo>bar</foo></Dummy>',
+            xsd_namespace=stats_httpd.XSD_NAMESPACE,
+            xsd_url_path=stats_httpd.XSD_URL_PATH,
+            xsl_url_path=stats_httpd.XSL_URL_PATH)
+        xml_body2 = self.stats_httpd.xml_handler()
+        self.assertEqual(type(xml_body1), str)
+        self.assertEqual(type(xml_body2), str)
+        self.assertEqual(xml_body1, xml_body2)
+        self.stats_httpd.get_stats_data = lambda: \
+            { 'Dummy' : {'bar':'foo'} }
+        xml_body2 = self.stats_httpd.xml_handler()
+        self.assertNotEqual(xml_body1, xml_body2)
+
+    def test_xsd_handler(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.stats_httpd.get_stats_spec = lambda: \
+            { "Dummy" :
+                  [{
+                        "item_name": "foo",
+                        "item_type": "string",
+                        "item_optional": False,
+                        "item_default": "bar",
+                        "item_description": "foo is bar",
+                        "item_title": "Foo"
+                        }]
+              }
+        xsd_body1 = self.stats_httpd.open_template(
+            stats_httpd.XSD_TEMPLATE_LOCATION).substitute(
+            xsd_string=\
+                '<all><element name="Dummy"><complexType><all>' \
+                + '<element maxOccurs="1" minOccurs="1" name="foo" type="string">' \
+                + '<annotation><appinfo>Foo</appinfo>' \
+                + '<documentation>foo is bar</documentation>' \
+                + '</annotation></element></all>' \
+                + '</complexType></element></all>',
+            xsd_namespace=stats_httpd.XSD_NAMESPACE)
+        xsd_body2 = self.stats_httpd.xsd_handler()
+        self.assertEqual(type(xsd_body1), str)
+        self.assertEqual(type(xsd_body2), str)
+        self.assertEqual(xsd_body1, xsd_body2)
+        self.stats_httpd.get_stats_spec = lambda: \
+            { "Dummy" :
+                  [{
+                        "item_name": "bar",
+                        "item_type": "string",
+                        "item_optional": False,
+                        "item_default": "foo",
+                        "item_description": "bar is foo",
+                        "item_title": "bar"
+                        }]
+              }
+        xsd_body2 = self.stats_httpd.xsd_handler()
+        self.assertNotEqual(xsd_body1, xsd_body2)
+
+    def test_xsl_handler(self):
+        self.stats_httpd = MyStatsHttpd(get_availaddr())
+        self.stats_httpd.get_stats_spec = lambda: \
+            { "Dummy" :
+                  [{
+                        "item_name": "foo",
+                        "item_type": "string",
+                        "item_optional": False,
+                        "item_default": "bar",
+                        "item_description": "foo is bar",
+                        "item_title": "Foo"
+                        }]
+              }
+        xsl_body1 = self.stats_httpd.open_template(
+            stats_httpd.XSL_TEMPLATE_LOCATION).substitute(
+            xsl_string='<xsl:template match="*"><tr>' \
+                + '<td>Dummy</td>' \
+                + '<td class="title" title="foo is bar">Foo</td>' \
+                + '<td><xsl:value-of select="Dummy/foo" /></td>' \
+                + '</tr></xsl:template>',
+            xsd_namespace=stats_httpd.XSD_NAMESPACE)
+        xsl_body2 = self.stats_httpd.xsl_handler()
+        self.assertEqual(type(xsl_body1), str)
+        self.assertEqual(type(xsl_body2), str)
+        self.assertEqual(xsl_body1, xsl_body2)
+        self.stats_httpd.get_stats_spec = lambda: \
+            { "Dummy" :
+                  [{
+                        "item_name": "bar",
+                        "item_type": "string",
+                        "item_optional": False,
+                        "item_default": "foo",
+                        "item_description": "bar is foo",
+                        "item_title": "bar"
+                        }]
+              }
+        xsl_body2 = self.stats_httpd.xsl_handler()
+        self.assertNotEqual(xsl_body1, xsl_body2)
+
     def test_for_without_B10_FROM_SOURCE(self):
         # just lets it go through the code without B10_FROM_SOURCE env
         # variable
@@ -437,8 +686,6 @@ class TestStatsHttpd(unittest.TestCase):
             imp.reload(stats_httpd)
             os.environ["B10_FROM_SOURCE"] = tmppath
             imp.reload(stats_httpd)
-            stats_httpd.socket = fake_socket
-            stats_httpd.select = fake_select
 
 if __name__ == "__main__":
     unittest.main()

src/bin/stats/tests/fake_socket.py

@@ -1,70 +0,0 @@
-# Copyright (C) 2011  Internet Systems Consortium.
-#
-# Permission to use, copy, modify, and distribute this software for any
-# purpose with or without fee is hereby granted, provided that the above
-# copyright notice and this permission notice appear in all copies.
-#
-# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
-# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
-# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
-# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
-# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
-# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
-# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-"""
-A mock-up module of socket
-
-*** NOTE ***
-It is only for testing stats_httpd module and not reusable for
-external module.
-"""
-
-import re
-
-AF_INET = 'AF_INET'
-AF_INET6 = 'AF_INET6'
-_ADDRFAMILY = AF_INET
-has_ipv6 = True
-_CLOSED = False
-
-class gaierror(Exception):
-    pass
-
-class error(Exception):
-    pass
-
-class socket:
-
-    def __init__(self, family=None):
-        if family is None:
-            self.address_family = _ADDRFAMILY
-        else:
-            self.address_family = family
-        self._closed = _CLOSED
-        if self._closed:
-            raise error('socket is already closed!')
-        self._called = 0
-
-    def close(self):
-        self._closed = True
-
-    def fileno(self):
-        return id(self)
-
-    def bind(self, server_class):
-        (self.server_address, self.server_port) = server_class
-        if self.address_family not in set([AF_INET, AF_INET6]):
-            raise error("Address family not supported by protocol: %s" % self.address_family)
-        if self.address_family == AF_INET6 and not has_ipv6:
-            raise error("Address family not supported in this machine: %s has_ipv6: %s"
-                        % (self.address_family, str(has_ipv6)))
-        if self.address_family == AF_INET and re.search(':', self.server_address) is not None:
-            raise gaierror("Address family for hostname not supported : %s %s" % (self.server_address, self.address_family))
-        if self.address_family == AF_INET6 and re.search(':', self.server_address) is None:
-            raise error("Cannot assign requested address : %s" % str(self.server_address))
-        if type(self.server_port) is not int:
-            raise TypeError("an integer is required: %s" % str(self.server_port))
-        if self.server_port < 0 or self.server_port > 65535:
-            raise OverflowError("port number must be 0-65535.: %s" % str(self.server_port))

src/bin/stats/tests/fake_time.py

@@ -1,47 +0,0 @@
-# Copyright (C) 2010  Internet Systems Consortium.
-#
-# Permission to use, copy, modify, and distribute this software for any
-# purpose with or without fee is hereby granted, provided that the above
-# copyright notice and this permission notice appear in all copies.
-#
-# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
-# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
-# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
-# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
-# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
-# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
-# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-__version__ = "$Revision$"
-
-# This is a dummy time class against a Python standard time class.
-# It is just testing use only.
-# Other methods which time class has is not implemented.
-# (This class isn't orderloaded for time class.)
-
-# These variables are constant. These are example.
-_TEST_TIME_SECS = 1283364938.229088
-_TEST_TIME_STRF = '2010-09-01T18:15:38Z'
-
-def time():
-    """
-    This is a dummy time() method against time.time()
-    """
-    # return float constant value
-    return _TEST_TIME_SECS
-
-def gmtime():
-    """
-    This is a dummy gmtime() method against time.gmtime()
-    """
-    # always return nothing
-    return None
-
-def strftime(*arg):
-    """
-    This is a dummy gmtime() method against time.gmtime()
-    """
-    return _TEST_TIME_STRF
-
-

src/bin/stats/tests/http/Makefile.am

@@ -1,6 +0,0 @@
-EXTRA_DIST = __init__.py server.py
-CLEANFILES = __init__.pyc server.pyc
-CLEANDIRS = __pycache__
-
-clean-local:
-	rm -rf $(CLEANDIRS)

src/bin/stats/tests/http/server.py

@@ -1,96 +0,0 @@
-# Copyright (C) 2011  Internet Systems Consortium.
-#
-# Permission to use, copy, modify, and distribute this software for any
-# purpose with or without fee is hereby granted, provided that the above
-# copyright notice and this permission notice appear in all copies.
-#
-# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
-# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
-# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
-# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
-# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
-# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
-# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-"""
-A mock-up module of http.server
-
-*** NOTE ***
-It is only for testing stats_httpd module and not reusable for
-external module.
-"""
-
-import fake_socket
-
-class DummyHttpResponse:
-    def __init__(self, path):
-        self.path = path
-        self.headers={}
-        self.log = ""
-
-    def _write_log(self, msg):
-        self.log = self.log + msg
-
-class HTTPServer:
-    """
-    A mock-up class of http.server.HTTPServer
-    """
-    address_family = fake_socket.AF_INET
-    def __init__(self, server_class, handler_class):
-        self.socket = fake_socket.socket(self.address_family)
-        self.server_class = server_class
-        self.socket.bind(self.server_class)
-        self._handler = handler_class(None, None, self)
-
-    def handle_request(self):
-        pass
-
-    def server_close(self):
-        self.socket.close()
-
-class BaseHTTPRequestHandler:
-    """
-    A mock-up class of http.server.BaseHTTPRequestHandler
-    """
-
-    def __init__(self, request, client_address, server):
-        self.path = "/path/to"
-        self.headers = {}
-        self.server = server
-        self.response = DummyHttpResponse(path=self.path)
-        self.response.write = self._write
-        self.wfile = self.response
-
-    def send_response(self, code=0):
-        if self.path != self.response.path:
-            self.response = DummyHttpResponse(path=self.path)
-        self.response.code = code
-
-    def send_header(self, key, value):
-        if self.path != self.response.path:
-            self.response = DummyHttpResponse(path=self.path)
-        self.response.headers[key] = value
-
-    def end_headers(self):
-        if self.path != self.response.path:
-            self.response = DummyHttpResponse(path=self.path)
-        self.response.wrote_headers = True
-
-    def send_error(self, code, message=None):
-        if self.path != self.response.path:
-            self.response = DummyHttpResponse(path=self.path)
-        self.response.code = code
-        self.response.body = message
-
-    def address_string(self):
-        return 'dummyhost'
-
-    def log_date_time_string(self):
-        return '[DD/MM/YYYY HH:MI:SS]'
-
-    def _write(self, obj):
-        if self.path != self.response.path:
-            self.response = DummyHttpResponse(path=self.path)
-        self.response.body = obj.decode()
-

src/bin/stats/tests/isc/Makefile.am

@@ -1,8 +0,0 @@
-SUBDIRS = cc config util
-EXTRA_DIST = __init__.py
-CLEANFILES = __init__.pyc
-
-CLEANDIRS = __pycache__
-
-clean-local:
-	rm -rf $(CLEANDIRS)