Merge branch 'master' into trac1528

Conflicts:
	ChangeLog
	src/lib/dhcp/iface_mgr_linux.cc

.gitignore

@@ -0,0 +1,31 @@
+*.la
+*.lo
+*.o
+.deps/
+.libs/
+__pycache__/
+Makefile
+Makefile.in
+TAGS
+
+/aclocal.m4
+/autom4te.cache/
+/config.guess
+/config.h
+/config.h.in
+/config.log
+/config.report
+/config.status
+/config.sub
+/configure
+/cscope.files
+/cscope.out
+/depcomp
+/install-sh
+/libtool
+/ltmain.sh
+/missing
+/py-compile
+/stamp-h1
+
+/dns++.pc

COPYING

@@ -11,3 +11,9 @@ INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
 LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
 OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 PERFORMANCE OF THIS SOFTWARE.
+
+-----------------------------------------------------------------------------
+
+The ext/asio and ext/coroutine code is externally maintained and
+distributed under the Boost Software License, Version 1.0.
+(See accompanying file ext/LICENSE_1_0.txt.)

ChangeLog

@@ -1,11 +1,352 @@
-3XX.	[func]	tomek
+434.	[func]	tomek
 	libdhcp++: Linux interface detection refactored. The code is
 	now cleaner. Tests better support certain versions of ifconfig.
 	(Trac #1528, git TBD)
 
-382.	[func]	jelte
+433.	[func]		tomek
+	libdhcp++: Option6 and Pkt6 now follow the same design as
+	options and packet for DHCPv4. General code refactoring after
+	end of 2011 year release.
+	(Trac #1540, git a40b6c665617125eeb8716b12d92d806f0342396)
+
+432.	[bug]*		muks
+	BIND 10 now installs its header files in a BIND 10 specific
+	sub-directory in the install prefix.
+	(Trac #1930, git fcf2f08db9ebc2198236bfa25cf73286821cba6b)
+
+431.	[func]*		muks
+	BIND 10 no longer starts b10-stats-httpd by default.
+	(Trac #1885, git 5c8bbd7ab648b6b7c48e366e7510dedca5386f6c)
+
+430.	[bug]		jelte
+	When displaying configuration data, bindctl no longer treats
+	optional list items as an error, but shows them as an empty list.
+	(Trac #1520, git 0f18039bc751a8f498c1f832196e2ecc7b997b2a)
+
+429.	[func]		jelte
+	Added an 'execute' component to bindctl, which executes either a set
+	of commands from a file or a built-in set of commands. Currently,
+	only 'init_authoritative_server' is provided as a built-in set, but
+	it is expected that more will be added later.
+	(Trac #1843, git 551657702a4197ef302c567b5c0eaf2fded3e121)
+
+428.	[bug]		marcin
+	perfdhcp: bind to local address to allow reception of replies from IPv6
+	DHCP servers.
+	(Trac #1908, git 597e059afaa4a89e767f8f10d2a4d78223af3940)
+
+427.	[bug]		jinmei
+	libdatasrc, b10-xfrin: the zone updater for database-based data
+	sources now correctly distinguishes NSEC3-related RRs (NSEC3 and
+	NSEC3-covering RRSIG) from others, and the SQLite3 implementation
+	now manipulates them in the separate table for the NSEC3 namespace.
+	As a result b10-xfrin now correctly updates NSEC3-signed zones by
+	inbound zone transfers.
+	(Trac #1891, git 672f129700dae33b701bb02069cf276238d66be3)
+
+426.	[bug]		vorner
+	The NSEC3 records are now included when transferring a signed zone out.
+	(Trac #1782, git 36efa7d10ecc4efd39d2ce4dfffa0cbdeffa74b0)
+
+425.	[func]*		muks
+	Don't autostart b10-auth, b10-xfrin, b10-xfrout and b10-zonemgr in
+	the default configuration.
+	(Trac #1818, git 31de885ba0409f54d9a1615eff5a4b03ed420393)
+
+424.	[bug]		jelte
+	Fixed a bug in bindctl where in some cases, configuration settings
+	in a named set could disappear, if a child element is modified.
+	(Trac #1491, git 00a36e752802df3cc683023d256687bf222e256a)
+
+423.	[bug]		jinmei
+	The database based zone iterator now correctly resets mixed TTLs
+	of the same RRset (when that happens) to the lowest one.  The
+	previous implementation could miss lower ones if it appears in a
+	later part of the RRset.
+	(part of Trac #1791, git f1f0bc00441057e7050241415ee0367a09c35032)
+
+422.	[bug]		jinmei
+	The database based zone iterator now separates RRSIGs of the same
+	name and type but for different covered types.
+	(part of Trac #1791, git b4466188150a50872bc3c426242bc7bba4c5f38d)
+
+421.	[build]		jinmei
+	Made sure BIND 10 can be built with clang++ 3.1.  (It failed on
+	MacOS 10.7 using Xcode 4.3, but it's more likely to be a matter of
+	clang version.)
+	(Trac #1773, git ceaa247d89ac7d97594572bc17f005144c5efb8d)
+
+420.	[bug]*		jinmei, stephen
+	Updated the DB schema used in the SQLite3 data source so it can
+	use SQL indices more effectively.  The previous schema had several
+	issues in this sense and could be very slow for some queries on a
+	very large zone (especially for negative answers).  This change
+	requires a major version up of the schema; use b10-dbutil to
+	upgrade existing database files.  Note: 'make install' will fail
+	unless old DB files installed in the standard location have been
+	upgraded.
+	(Trac #324, git 8644866497053f91ada4e99abe444d7876ed00ff)
+
+419.	[bug]		jelte
+	JSON handler has been improved; escaping now works correctly
+	(including quotes in strings), and it now rejects more types of
+	malformed input.
+	(Trac #1626, git 3b09268518e4e90032218083bcfebf7821be7bd5)
+
+418.	[bug]		vorner
+	Fixed crash in bindctl when config unset was called.
+	(Trac #1715, git 098da24dddad497810aa2787f54126488bb1095c)
+
+417.	[bug]		jelte
+	The notify-out code now looks up notify targets in their correct
+	zones (and no longer just in the zone that the notify is about).
+	(Trac #1535, git 66300a3c4769a48b765f70e2d0dbf8bbb714435b)
+
+416.	[func]*		jelte
+	The implementations of ZoneFinder::find() now throw an OutOfZone
+	exception when the name argument is not in or below the zone this
+	zonefinder contains.
+	(Trac #1535, git 66300a3c4769a48b765f70e2d0dbf8bbb714435b)
+
+bind10-devel-20120329 released on March 29, 2012
+
+415.	[doc]		jinmei, jreed
+	BIND 10 Guide updated to now describe the in-memory data source
+	configurations for b10-auth.
+	(Trac #1732, git 434d8db8dfcd23a87b8e798e5702e91f0bbbdcf6)
+
+414.	[bug]		jinmei
+	b10-auth now correctly handles delegation from an unsigned zone
+	(defined in the in-memory data source) when the query has DNSSEC
+	DO bit on.  It previously returned SERVFAIL.
+	(Trac #1836, git 78bb8f4b9676d6345f3fdd1e5cc89039806a9aba)
+
+413.	[func]		stephen, jelte
+	Created a new tool b10-dbutil, that can check and upgrade database
+	schemas, to be used when incompatible changes are introduced in the
+	backend database schema. Currently it only supports sqlite3 databases.
+	Note: there's no schema change that requires this utility as of
+	the March 29th release.  While running it shouldn't break
+	an existing database file, it should be even more advisable not to
+	run it at the moment.
+	(Trac #963, git 49ba2cf8ac63246f389ab5e8ea3b3d081dba9adf)
+
+412.	[func]		jelte
+	Added a command-line option '--clear-config' to bind10, which causes
+	the system to create a backup of the existing configuration database
+	file, and start out with a clean default configuration. This can be
+	used if the configuration file is corrupted to the point where it
+	cannot be read anymore, and BIND 10 refuses to start. The name of
+	the backup file can be found in the logs (CFGMGR_RENAMED_CONFIG_FILE).
+	(Trac #1443, git 52b36c921ee59ec69deefb6123cbdb1b91dc3bc7)
+
+411.	[func]		muks
+	Add a -i/--no-kill command-line argument to bind10, which stops
+	it from sending SIGTERM and SIGKILL to other b10 processes when
+	they're shutting down.
+	(Trac #1819, git 774554f46b20ca5ec2ef6c6d5e608114f14e2102)
+
+410.	[bug]		jinmei
+	Python CC library now ensures write operations transmit all given
+	data (unless an error happens).  Previously it didn't check the
+	size of transmitted data, which could result in partial write on
+	some systems (notably on OpenBSD) and subsequently cause system
+	hang up or other broken state.  This fix specifically solves start
+	up failure on OpenBSD.
+	(Trac #1829, git 5e5a33213b60d89e146cd5e47d65f3f9833a9297)
+
+409.	[bug]		jelte
+	Fixed a parser bug in bindctl that could make bindctl crash. Also
+	improved 'command help' output; argument order is now shown
+	correctly, and parameter descriptions are shown as well.
+	(Trac #1172, git bec26c6137c9b0a59a3a8ca0f55a17cfcb8a23de)
+
+408.	[bug]		stephen, jinmei
+	b10-auth now filters out duplicate RRsets when building a
+	response message using the new query handling logic.  It's
+	currently only used with the in-memory data source, but will
+	also be used for others soon.
+	(Trac #1688, git b77baca56ffb1b9016698c00ae0a1496d603d197)
+
+407.	[build]		haikuo
+	Remove "--enable-boost-threads" switch in configure command. This
+	thread lock mechanism is useless for bind10 and causes performance 
+	hits. 
+	(Trac #1680, git 9c4d0cadf4adc802cc41a2610dc2c30b25aad728)
+
+406.	[bug]		muks
+	On platforms such as OpenBSD where pselect() is not available,
+	make a wrapper around select() in perfdhcp.
+	(Trac #1639, git 6ea0b1d62e7b8b6596209291aa6c8b34b8e73191)
+
+405.	[bug]		jinmei
+	Make sure disabling Boost threads if the default configuration is
+	to disable it for the system.  This fixes a crash and hang up
+	problem on OpenBSD, where the use of Boost thread could be
+	different in different program files depending on the order of
+	including various header files, and could introduce inconsistent
+	states between a library and a program.  Explicitly forcing the
+	original default throughout the BIND 10 build environment will
+	prevent this from happening.
+	(Trac #1727, git 23f9c3670b544c5f8105958ff148aeba050bc1b4)
+
+404.	[bug]		naokikambe
+	The statistic counters are now properly accumulated across multiple
+	instances of b10-auth (if there are multiple instances), instead of
+	providing result for random instance.
+	(Trac #1751, git 3285353a660e881ec2b645e1bc10d94e5020f357)
+
+403.	[build]*	jelte
+	The configure option for botan (--with-botan=PATH) is replaced by
+	--with-botan-config=PATH, which takes a full path to a botan-config
+	script, instead of the botan 'install' directory. Also, if not
+	provided, configure will try out config scripts and pkg-config
+	options until it finds one that works.
+	(Trac #1640, git 582bcd66dbd8d39f48aef952902f797260280637)
+
+402.	[func]		jelte
+	b10-xfrout now has a visible command to send out notifies for
+	a given zone, callable from bindctl. Xfrout notify <zone> [class]
+	(Trac #1321, git 0bb258f8610620191d75cfd5d2308b6fc558c280)
+
+401.	[func]*		jinmei
+	libdns++: updated the internal implementation of the
+	MessageRenderer class.  This is mostly a transparent change, but
+	the new version now doesn't allow changing compression mode in the
+	middle of rendering (which shouldn't be an issue in practice).
+	On the other hand, name compression performance was significantly
+	improved: depending on the number of names, micro benchmark tests
+	showed the new version is several times faster than the previous
+	version .
+	(Trac #1603, git 9a2a86f3f47b60ff017ce1a040941d0c145cfe16)
+
+400.	[bug]		stephen
+	Fix crash on Max OS X 10.7 by altering logging so as not to allocate
+	heap storage in the static initialization of logging objects.
+	(Trac #1698, git a8e53be7039ad50d8587c0972244029ff3533b6e)
+
+399.	[func]		muks
+	Add support for the SSHFP RR type (RFC 4255).
+	(Trac #1136, git ea5ac57d508a17611cfae9d9ea1c238f59d52c51)
+
+398.	[func]		jelte
+	The b10-xfrin module now logs more information on successful
+	incoming transfers. In the case of IXFR, it logs the number of
+	changesets, and the total number of added and deleted resource
+	records. For AXFR (or AXFR-style IXFR), it logs the number of
+	resource records. In both cases, the number of overhead DNS
+	messages, runtime, amount of wire data, and transfer speed are logged.
+	(Trac #1280, git 2b01d944b6a137f95d47673ea8367315289c205d)
+
+397.	[func]		muks
+	The boss process now gives more helpful description when a
+	sub-process exits due to a signal.
+	(Trac #1673, git 1cd0d0e4fc9324bbe7f8593478e2396d06337b1e)
+
+396.	[func]*		jinmei
+	libdatasrc: change the return type of ZoneFinder::find() so it can
+	contain more context of the search, which can be used for
+	optimizing post find() processing.  A new method getAdditional()
+	is added to it for finding additional RRsets based on the result
+	of find().  External behavior shouldn't change.  The query
+	handling code of b10-auth now uses the new interface.
+	(Trac #1607, git 2e940ea65d5b9f371c26352afd9e66719c38a6b9)
+
+395.	[bug]		jelte
+	The log message compiler now errors (resulting in build failures) if
+	duplicate log message identifiers are found in a single message file.
+	Renamed one duplicate that was found (RESOLVER_SHUTDOWN, renamed to
+	RESOLVER_SHUTDOWN_RECEIVED).
+	(Trac #1093, git f537c7e12fb7b25801408f93132ed33410edae76)
+	(Trac #1741, git b8960ab85c717fe70ad282e0052ac0858c5b57f7)
+
+394.	[bug]		jelte
+	b10-auth now catches any exceptions during response building; if any
+	datasource either throws an exception or causes an exception to be
+	thrown, the message processing code will now catch it, log a debug
+	message, and return a SERVFAIL response.
+	(Trac #1612, git b5740c6b3962a55e46325b3c8b14c9d64cf0d845)
+
+393.	[func]		jelte
+	Introduced a new class LabelSequence in libdns++, which provides
+	lightweight accessor functionality to the Name class, for more
+	efficient comparison of parts of names.
+	(Trac #1602, git b33929ed5df7c8f482d095e96e667d4a03180c78)
+
+392.	[func]*		jinmei
+	libdns++: revised the (Abstract)MessageRenderer class so that it
+	has a default internal buffer and the buffer can be temporarily
+	switched.  The constructor interface was modified, and a new
+	method setBuffer() was added.
+	(Trac #1697, git 9cabc799f2bf9a3579dae7f1f5d5467c8bb1aa40)
+
+391.	[bug]*		vorner
+	The long time unused configuration options of Xfrout "log_name",
+	"log_file", "log_severity", "log_version" and "log_max_bytes" were
+	removed, as they had no effect (Xfrout uses the global logging
+	framework).  However, if you have them set, you need to remove
+	them from the configuration file or the configuration will be
+	rejected.
+	(Trac #1090, git ef1eba02e4cf550e48e7318702cff6d67c1ec82e)
+
+bind10-devel-20120301 released on March 1, 2012
+
+390.	[bug]		vorner
+	The UDP IPv6 packets are now correctly fragmented for maximum
+	guaranteed MTU, so they won't get lost because being too large
+	for some hop.
+	(Trac #1534, git ff013364643f9bfa736b2d23fec39ac35872d6ad)
+
+389.	[func]*		vorner
+	Xfrout now uses the global TSIG keyring, instead of its own. This
+	means the keys need to be set only once (in tsig_keys/keys).
+	However, the old configuration of Xfrout/tsig_keys need to be
+	removed for Xfrout to work.
+	(Trac #1643, git 5a7953933a49a0ddd4ee1feaddc908cd2285522d)
+
+388.	[func]		jreed
+	Use prefix "sockcreator-" for the private temporary directory
+	used for b10-sockcreator communication.
+	(git b98523c1260637cb33436964dc18e9763622a242)
+
+387.	[build]		muks
+	Accept a --without-werror configure switch so that some builders can
+	disable the use of -Werror in CFLAGS when building.
+	(Trac #1671, git 8684a411d7718a71ad9fb616f56b26436c4f03e5)
+
+386.	[bug]		jelte
+	Upon initial sqlite3 database creation, the 'diffs' table is now
+	always created. This already happened most of the time, but there
+	are a few cases where it was skipped, resulting in potential errors
+	in xfrout later.
+	(Trac #1717, git 30d7686cb6e2fa64866c983e0cfb7b8fabedc7a2)
+
+385.	[bug]		jinmei
+	libdns++: masterLoad() didn't accept comments placed at the end of
+	an RR.  Due to this the in-memory data source cannot load a master
+	file for a signed zone even if it's preprocessed with BIND 9's
+	named-compilezone.
+	Note: this fix is considered temporary and still only accepts some
+	limited form of such comments.  The main purpose is to allow the
+	in-memory data source to load any signed or unsigned zone files as
+	long as they are at least normalized with named-compilezone.
+	(Trac #1667, git 6f771b28eea25c693fe93a0e2379af924464a562)
+
+384.	[func]		jinmei, jelte, vorner, haikuo, kevin
+	b10-auth now supports NSEC3-signed zones in the in-memory data
+	source.
+	(Trac #1580, #1581, #1582, #1583, #1584, #1585, #1587, and
+	other related changes to the in-memory data source)
+
+383.	[build]		jinmei
+	Fixed build failure on MacOS 10.7 (Lion) due to the use of
+	IPV6_PKTINFO; the OS requires a special definition to make it
+	visible to the compiler.
+	(Trac #1633, git 19ba70c7cc3da462c70e8c4f74b321b8daad0100)
+
+382.	[func]		jelte
 	b10-auth now also experimentally supports statistics counters of
-	the rcode reponses it sends. The counters can be shown as
+	the rcode responses it sends. The counters can be shown as
 	rcode.<code name>, where code name is the lowercase textual
 	representation of the rcode (e.g. "noerror", "formerr", etc.).
 	Same note applies as for opcodes, see changelog entry 364.
@@ -55,11 +396,11 @@
 	(Trac #1570, git 2858b2098a10a8cc2d34bf87463ace0629d3670e)
 
 375.	[func]		jelte
-	Modules now inform the system when they are stopping. As a result, they
-	are removed from the 'active modules' list in bindctl, which can then
-	inform the user directly when it tries to send them a command or
-	configuration update. Previously this would result in a 'not
-	responding' error instead of 'not running'.
+	Modules now inform the system when they are stopping. As a result,
+	they are removed from the 'active modules' list in bindctl, which
+	can then inform the user directly when it tries to send them a
+	command or configuration update.  Previously this would result
+	in a 'not responding' error instead of 'not running'.
 	(Trac #640, git 17e78fa1bb1227340aa9815e91ed5c50d174425d)
 
 374.	[func]*		stephen
@@ -95,10 +436,11 @@
 	(Trac #1575, git 2c421b58e810028b303d328e4e2f5b74ea124839)
 
 369.	[func]		vorner
-	The SocketRequestor provides more information about what error happened
-	when it throws, by using subclasses of the original exception. This way
-	a user not interested in the difference can still use the original
-	exception, while it can be recognized if necessary.
+	The SocketRequestor provides more information about what error
+	happened when it throws, by using subclasses of the original
+	exception. This way a user not interested in the difference can
+	still use the original exception, while it can be recognized if
+	necessary.
 	(Trac #1542, git 2080e0316a339fa3cadea00e10b1ec4bc322ada0)
 
 368.	[func]*		jinmei
@@ -156,7 +498,8 @@ bind10-devel-20120119 released on January 19, 2012
 	configuration.  If your b10-config.db contains "setuid" for
 	Boss.components, you'll need to remove that entry by hand before
 	starting BIND 10.
-	(Trac #1508-#1510, git edc5b3c12eb45437361484c843794416ad86bb00)
+	(Trac #1508, #1509, #1510,
+	git edc5b3c12eb45437361484c843794416ad86bb00)
 
 361.	[func]		vorner,jelte,jinmei
 	The socket creator is now used to provide sockets. It means you can

Makefile.am

@@ -1,3 +1,7 @@
+ACLOCAL_AMFLAGS = -I m4macros ${ACLOCAL_FLAGS}
+# ^^^^^^^^ This has to be the first line and cannot come later in this
+# Makefile.am due to some bork in some versions of autotools.
+
 SUBDIRS = compatcheck doc src tests
 USE_LCOV=@USE_LCOV@
 LCOV=@LCOV@
@@ -79,7 +83,7 @@ report-coverage: report-cpp-coverage report-python-coverage
 
 # for static C++ check using cppcheck (when available)
 cppcheck:
-	cppcheck --enable=all --suppressions src/cppcheck-suppress.lst \
+	cppcheck --enable=all --suppressions src/cppcheck-suppress.lst --inline-suppr \
 		--quiet --error-exitcode=1 \
 		--template '{file}:{line}: check_fail: {message} ({severity},{id})' \
 		src
@@ -398,3 +402,6 @@ EXTRA_DIST += ext/asio/asio/system_error.hpp
 EXTRA_DIST += ext/asio/asio/deadline_timer.hpp
 EXTRA_DIST += ext/asio/asio/stream_socket_service.hpp
 EXTRA_DIST += ext/coroutine/coroutine.h
+
+pkgconfigdir = $(libdir)/pkgconfig
+pkgconfig_DATA = dns++.pc

compatcheck/.gitignore

@@ -0,0 +1 @@
+/sqlite3-difftbl-check.py

compatcheck/Makefile.am

@@ -1,8 +1,12 @@
-noinst_SCRIPTS = sqlite3-difftbl-check.py
-
 # We're going to abuse install-data-local for a pre-install check.
 # This is to be considered a short term hack and is expected to be removed
 # in a near future version.
 install-data-local:
-	$(PYTHON) sqlite3-difftbl-check.py \
-	$(localstatedir)/$(PACKAGE)/zone.sqlite3
+	if test -e $(localstatedir)/$(PACKAGE)/zone.sqlite3; then \
+		$(SHELL) $(top_builddir)/src/bin/dbutil/run_dbutil.sh --check \
+		$(localstatedir)/$(PACKAGE)/zone.sqlite3 || \
+		(echo "\nSQLite3 DB file schema version is old.  " \
+		"Please run: " \
+		"$(abs_top_builddir)/src/bin/dbutil/run_dbutil.sh --upgrade " \
+		"$(localstatedir)/$(PACKAGE)/zone.sqlite3";  exit 1) \
+	fi

compatcheck/sqlite3-difftbl-check.py.in

@@ -1,60 +0,0 @@
-#!@PYTHON@
-
-# Copyright (C) 2011  Internet Systems Consortium.
-#
-# Permission to use, copy, modify, and distribute this software for any
-# purpose with or without fee is hereby granted, provided that the above
-# copyright notice and this permission notice appear in all copies.
-#
-# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
-# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
-# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
-# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
-# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
-# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
-# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-import os, sqlite3, sys
-from optparse import OptionParser
-
-usage = 'usage: %prog [options] db_file'
-parser = OptionParser(usage=usage)
-parser.add_option("-u", "--upgrade", action="store_true",
-                  dest="upgrade", default=False,
-                  help="Upgrade the database file [default: %default]")
-(options, args) = parser.parse_args()
-if len(args) == 0:
-    parser.error('missing argument')
-
-db_file = args[0]
-
-# If the file doesn't exist, there's nothing to do
-if not os.path.exists(db_file):
-    sys.exit(0)
-
-conn = sqlite3.connect(db_file)
-cur = conn.cursor()
-try:
-    # This can be anything that works iff the "diffs" table exists
-    cur.execute('SELECT name FROM diffs DESC LIMIT 1')
-except sqlite3.OperationalError as ex:
-    # If it fails with 'no such table', create a new one or fail with
-    # warning depending on the --upgrade command line option.
-    if str(ex) == 'no such table: diffs':
-        if options.upgrade:
-            cur.execute('CREATE TABLE diffs (id INTEGER PRIMARY KEY, ' +
-                        'zone_id INTEGER NOT NULL, ' +
-                        'version INTEGER NOT NULL, ' +
-                        'operation INTEGER NOT NULL, ' +
-                        'name STRING NOT NULL COLLATE NOCASE, ' +
-                        'rrtype STRING NOT NULL COLLATE NOCASE, ' +
-                        'ttl INTEGER NOT NULL, rdata STRING NOT NULL)')
-        else:
-            sys.stdout.write('Found an older version of SQLite3 DB file: ' +
-                             db_file + '\n' + "Perform '" + os.getcwd() +
-                             "/sqlite3-difftbl-check.py --upgrade " +
-                             db_file + "'\n" +
-                             'before continuing install.\n')
-            sys.exit(1)
-conn.close()

configure.ac

@@ -2,10 +2,12 @@
 # Process this file with autoconf to produce a configure script.
 
 AC_PREREQ([2.59])
-AC_INIT(bind10-devel, 20120127, bind10-dev@isc.org)
+AC_INIT(bind10-devel, 20120405, bind10-dev@isc.org)
 AC_CONFIG_SRCDIR(README)
 AM_INIT_AUTOMAKE
+m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])dnl be backward compatible
 AC_CONFIG_HEADERS([config.h])
+AC_CONFIG_MACRO_DIR([m4macros])
 
 # Checks for programs.
 AC_PROG_CXX
@@ -84,11 +86,6 @@ if test $enable_shared = no; then
 	AC_MSG_ERROR([BIND 10 requires shared libraries to be built])
 fi
 
-AC_ARG_ENABLE(boost-threads,
-AC_HELP_STRING([--enable-boost-threads],
-  [use boost threads. Currently this only means using its locks instead of dummy locks, in the cache and NSAS]),
-  use_boost_threads=$enableval, use_boost_threads=no)
-
 # allow configuring without setproctitle.
 AC_ARG_ENABLE(setproctitle-check,
 AC_HELP_STRING([--disable-setproctitle-check],
@@ -108,6 +105,10 @@ case "$host" in
 	LDFLAGS="$LDFLAGS -z now"
 	;;
 *-apple-darwin*)
+	# Starting with OSX 10.7 (Lion) we must choose which IPv6 API to use
+	# (RFC2292 or RFC3542).
+	CPPFLAGS="$CPPFLAGS -D__APPLE_USE_RFC_3542"
+
 	# libtool doesn't work perfectly with Darwin: libtool embeds the
 	# final install path in dynamic libraries and our loadable python
 	# modules always refer to that path even if it's loaded within the
@@ -123,6 +124,9 @@ case "$host" in
 *-netbsd*)
 	SET_ENV_LIBRARY_PATH=yes
 	;;
+*-openbsd*)
+	SET_ENV_LIBRARY_PATH=yes
+	;;
 esac
 AM_CONDITIONAL(SET_ENV_LIBRARY_PATH, test $SET_ENV_LIBRARY_PATH = yes)
 AC_SUBST(SET_ENV_LIBRARY_PATH)
@@ -270,7 +274,7 @@ AC_DEFUN([BIND10_CXX_TRY_FLAG], [
   bind10_save_CXXFLAGS="$CXXFLAGS"
   CXXFLAGS="$CXXFLAGS $1"
 
-  AC_LINK_IFELSE([int main(void){ return 0;} ],
+  AC_LINK_IFELSE([AC_LANG_SOURCE([int main(void){ return 0;}])],
                  [bind10_cxx_flag=yes], [bind10_cxx_flag=no])
   CXXFLAGS="$bind10_save_CXXFLAGS"
 
@@ -283,8 +287,6 @@ AC_DEFUN([BIND10_CXX_TRY_FLAG], [
   AC_MSG_RESULT([$bind10_cxx_flag])
 ])
 
-werror_ok=0
-
 # SunStudio compiler requires special compiler options for boost
 # (http://blogs.sun.com/sga/entry/boost_mini_howto)
 if test "$SUNCXX" = "yes"; then
@@ -292,7 +294,7 @@ CXXFLAGS="$CXXFLAGS -library=stlport4 -features=tmplife -features=tmplrefstatic"
 MULTITHREADING_FLAG="-mt"
 fi
 
-BIND10_CXX_TRY_FLAG(-Wno-missing-field-initializers,
+BIND10_CXX_TRY_FLAG([-Wno-missing-field-initializers],
 	[WARNING_NO_MISSING_FIELD_INITIALIZERS_CFLAG="-Wno-missing-field-initializers"])
 AC_SUBST(WARNING_NO_MISSING_FIELD_INITIALIZERS_CFLAG)
 
@@ -310,19 +312,34 @@ case "$host" in
 	;;
 esac
 
+# Don't use -Werror if configured not to
+AC_ARG_WITH(werror,
+    AC_HELP_STRING([--with-werror], [Compile using -Werror (default=yes)]),
+    [
+     case "${withval}" in
+         yes) with_werror=1 ;;
+         no)  with_werror=0 ;;
+         *)   AC_MSG_ERROR(bad value ${withval} for --with-werror) ;;
+     esac],
+     [with_werror=1])
+
+werror_ok=0
+
 # Certain versions of gcc (g++) have a bug that incorrectly warns about
 # the use of anonymous name spaces even if they're closed in a single
 # translation unit.  For these versions we have to disable -Werror.
-CXXFLAGS_SAVED="$CXXFLAGS"
-CXXFLAGS="$CXXFLAGS $B10_CXXFLAGS -Werror"
-AC_MSG_CHECKING(for in-TU anonymous namespace breakage)
-AC_TRY_COMPILE([namespace { class Foo {}; }
-namespace isc {class Bar {Foo foo_;};} ],,
+if test $with_werror = 1; then
+   CXXFLAGS_SAVED="$CXXFLAGS"
+   CXXFLAGS="$CXXFLAGS $B10_CXXFLAGS -Werror"
+   AC_MSG_CHECKING(for in-TU anonymous namespace breakage)
+   AC_TRY_COMPILE([namespace { class Foo {}; }
+   namespace isc {class Bar {Foo foo_;};} ],,
 	[AC_MSG_RESULT(no)
 	 werror_ok=1
 	 B10_CXXFLAGS="$B10_CXXFLAGS -Werror"],
 	[AC_MSG_RESULT(yes)])
-CXXFLAGS="$CXXFLAGS_SAVED"
+   CXXFLAGS="$CXXFLAGS_SAVED"
+fi
 
 # Python 3.2 has an unused parameter in one of its headers. This
 # has been reported, but not fixed as of yet, so we check if we need
@@ -477,61 +494,163 @@ if test "$lcov" != "no"; then
 fi
 AC_SUBST(USE_LCOV)
 
+# Simplified, non-caching AC_CHECK_PROG
+# Searches $PATH for the existence of argument 2,
+# and sets the full path to the variable in argument 1.
+# if not found, and a third argument is given, the value
+# is set to that. If not, the value is untouched.
+# Does not take absolute paths into account at this point,
+# and also works for single files only (arguments are not
+# stripped like in AC_CHECK_PROG)
+AC_DEFUN([ACX_CHECK_PROG_NONCACHE], [
+    RESULT=""
+    IFS_SAVED="$IFS"
+    IFS=${PATH_SEPARATOR}
+    for cur_path in ${PATH} ; do
+      if test -e "${cur_path}/$2" ; then
+          RESULT="${cur_path}/$2"
+      fi
+    done
+    if test "$RESULT" = "" ; then
+        :
+        m4_ifvaln([$3], [$1=$3])
+    else
+        $1=$RESULT
+    fi
+    IFS="$IFS_SAVED"
+])
+
+# Botan helper test function
+# Tries to compile a botan program, given the output of the given
+# config tool
+# Arguments:
+# - name of tool (checked for path), must support --libs and --cflags
+# - fixed argument(s) for tool
+# - action if successful
+AC_DEFUN([ACX_TRY_BOTAN_TOOL], [
+    TOOL=$1
+    TOOL_ARG=$2
+    BOTAN_TOOL=""
+    ACX_CHECK_PROG_NONCACHE([BOTAN_TOOL], [${TOOL}])
+    AC_MSG_CHECKING([usability of ${TOOL} ${TOOL_ARG}])
+    if test "$BOTAN_TOOL" != "" ; then
+        if test -x ${BOTAN_TOOL}; then
+            BOTAN_LIBS=`$BOTAN_TOOL $TOOL_ARG --libs`
+            LIBS_SAVED=${LIBS}
+            LIBS="$LIBS $BOTAN_LIBS"
+            BOTAN_INCLUDES=`$BOTAN_TOOL $TOOL_ARG --cflags`
+            CPPFLAGS_SAVED=${CPPFLAGS}
+            CPPFLAGS="$BOTAN_INCLUDES $CPPFLAGS"
+            #AC_MSG_RESULT([found])
+            AC_LINK_IFELSE(
+                [AC_LANG_PROGRAM([#include <botan/botan.h>
+                                  #include <botan/hash.h>
+                                 ],
+                                 [using namespace Botan;
+                                  LibraryInitializer::initialize();
+                                  HashFunction *h = get_hash("MD5");
+                                 ])],
+                [ AC_MSG_RESULT([ok])
+                  $3
+                ],
+                [ AC_MSG_RESULT([not usable]) ]
+            )
+            LIBS=${LIBS_SAVED}
+            CPPFLAGS=${CPPFLAGS_SAVED}
+        else
+            AC_MSG_RESULT([not executable])
+        fi
+    else
+        AC_MSG_RESULT([not found])
+    fi
+    BOTAN_TOOL=""
+    AC_SUBST(BOTAN_TOOL)
+    ]
+)
+
 # Check for Botan
-botan_path="yes"
-AC_ARG_WITH([botan],
-  AC_HELP_STRING([--with-botan=PATH],
-    [specify the path to botan-config (PATH/bin/botan-config will be used)]),
-    [botan_path="$withval"])
-if test "${botan_path}" = "no" ; then
+#
+# Unless --with-botan-config is given, we first try to find these config
+# scripts ourselves. Unfortunately, on some systems, these scripts do not
+# provide the correct implementation, so for each script found, we try
+# a compilation test (ACX_TRY_BOTAN_TOOL). If none are found, or none of
+# them work, we see if pkg-config is available. If so, we try the several
+# potential pkg-config .pc files. Again, on some systems, these can return
+# incorrect information as well, so the try-compile test is repeated for
+# each.
+#
+# If a working config script or pkgconfig file is found, we then munge its
+# output for use in our Makefiles, and to make sure it works, another header
+# and compilation test is done (this should also check whether we can compile
+# against botan should neither -config scripts nor pkgconfig data exist).
+#
+botan_config="yes"
+AC_ARG_WITH([botan-config],
+  AC_HELP_STRING([--with-botan-config=PATH],
+    [specify the path to the botan-config script]),
+    [botan_config="$withval"])
+if test "${botan_config}" = "no" ; then
     AC_MSG_ERROR([Need botan for libcryptolink])
 fi
-if test "${botan_path}" != "yes" ; then
-    if test -x "${botan_path}/bin/botan-config" ; then
-        BOTAN_CONFIG="${botan_path}/bin/botan-config"
+if test "${botan_config}" != "yes" ; then
+    if test -x "${botan_config}" ; then
+        if test -d "${botan_config}" ; then
+            AC_MSG_ERROR([${botan_config} is a directory])
+        else
+            BOTAN_CONFIG="${botan_config}"
+        fi
     else
-        AC_MSG_ERROR([${botan_path}/bin/botan-config not found])
+        AC_MSG_ERROR([--with-botan-config should point to a botan-config program and not a directory (${botan_config})])
     fi
 else
-    # First see if pkg-config knows of it.
-    # Unfortunately, the botan.pc files have their minor version in them
-    # too, so we need to try them one by one
     BOTAN_CONFIG=""
-    AC_PATH_PROG([PKG_CONFIG], [pkg-config])
-    if test "$PKG_CONFIG" != "" ; then
-        BOTAN_VERSIONS="botan-1.10 botan-1.9 botan-1.8"
-        for version in $BOTAN_VERSIONS; do
-            AC_MSG_CHECKING([Checking botan version with pkg-config $version])
-            
-            if [ $PKG_CONFIG --exists ${version} ]; then
-                AC_MSG_RESULT([found])
-                BOTAN_CONFIG="$PKG_CONFIG ${version}"
+    # first try several possible names of the config script
+    # (botan-config-1.8 is there just in case, the official name change
+    # came later)
+    BOTAN_CONFIG_VERSIONS="botan-config-1.10 botan-config-1.9 botan-config-1.8 botan-config"
+    for botan_config in $BOTAN_CONFIG_VERSIONS; do
+        ACX_TRY_BOTAN_TOOL([$botan_config],,
+                           [ BOTAN_CONFIG="$botan_config"  ]
+                          )
+        if test "$BOTAN_CONFIG" != "" ; then
+            break
+        fi
+    done
+    if test "$BOTAN_CONFIG" = "" ; then
+        AC_PATH_PROG([PKG_CONFIG], [pkg-config])
+        if test "$PKG_CONFIG" != "" ; then
+            # Ok so no script found, see if pkg-config knows of it.
+            # Unfortunately, the botan.pc files also have their minor version
+            # in their name, so we need to try them one by one
+            BOTAN_VERSIONS="botan-1.10 botan-1.9 botan-1.8"
+            for version in $BOTAN_VERSIONS; do
+                ACX_TRY_BOTAN_TOOL([pkg-config], ["$version --silence-errors"],
+                                   [ BOTAN_CONFIG="$PKG_CONFIG $version" ]
+                                  )
+            if test "$BOTAN_CONFIG" != "" ; then
                 break
-            else
-                AC_MSG_RESULT([not found])
             fi
-        done
-    fi
-    # If we had no pkg-config, or it didn't know about botan, use botan-config
-    if test "$BOTAN_CONFIG" = "" ; then
-        AC_PATH_PROG([BOTAN_CONFIG], [botan-config])
+            done
+        fi
     fi
 fi
 
-BOTAN_LIBS=`${BOTAN_CONFIG} --libs`
-BOTAN_INCLUDES=`${BOTAN_CONFIG} --cflags`
-
-# We expect botan-config --libs to contain -L<path_to_libbotan>, but
-# this is not always the case.  As a heuristics workaround we add
-# -L`botan-config --prefix/lib` in this case (if not present already).
-# Same for BOTAN_INCLUDES (but using include instead of lib) below.
-if [ $BOTAN_CONFIG --prefix >/dev/null 2>&1 ] ; then
-    echo ${BOTAN_LIBS} | grep -- -L > /dev/null || \
-        BOTAN_LIBS="-L`${BOTAN_CONFIG} --prefix`/lib ${BOTAN_LIBS}"
-    echo ${BOTAN_INCLUDES} | grep -- -I > /dev/null || \
-        BOTAN_INCLUDES="-I`${BOTAN_CONFIG} --prefix`/include ${BOTAN_INCLUDES}"
+if test "x${BOTAN_CONFIG}" != "x"
+then
+    BOTAN_LIBS=`${BOTAN_CONFIG} --libs`
+    BOTAN_INCLUDES=`${BOTAN_CONFIG} --cflags`
+
+    # We expect botan-config --libs to contain -L<path_to_libbotan>, but
+    # this is not always the case.  As a heuristics workaround we add
+    # -L`botan-config --prefix/lib` in this case (if not present already).
+    # Same for BOTAN_INCLUDES (but using include instead of lib) below.
+    if [ ${BOTAN_CONFIG} --prefix >/dev/null 2>&1 ] ; then
+        echo ${BOTAN_LIBS} | grep -- -L > /dev/null || \
+            BOTAN_LIBS="-L`${BOTAN_CONFIG} --prefix`/lib ${BOTAN_LIBS}"
+        echo ${BOTAN_INCLUDES} | grep -- -I > /dev/null || \
+            BOTAN_INCLUDES="-I`${BOTAN_CONFIG} --prefix`/include ${BOTAN_INCLUDES}"
+    fi
 fi
-
 # botan-config script (and the way we call pkg-config) returns -L and -l
 # as one string, but we need them in separate values
 BOTAN_LDFLAGS=
@@ -563,6 +682,9 @@ AC_SUBST(BOTAN_LDFLAGS)
 AC_SUBST(BOTAN_LIBS)
 AC_SUBST(BOTAN_INCLUDES)
 
+# Even though chances are high we already performed a real compilation check
+# in the search for the right (pkg)config data, we try again here, to
+# be sure.
 CPPFLAGS_SAVED=$CPPFLAGS
 CPPFLAGS="$BOTAN_INCLUDES $CPPFLAGS"
 LIBS_SAVED="$LIBS"
@@ -578,7 +700,11 @@ AC_LINK_IFELSE(
                          ])],
         [AC_MSG_RESULT([checking for Botan library... yes])],
         [AC_MSG_RESULT([checking for Botan library... no])
-         AC_MSG_ERROR([Needs Botan library 1.8 or higher])]
+         AC_MSG_ERROR([Needs Botan library 1.8 or higher. On some systems,
+         the botan package has a few missing dependencies (libbz2 and
+         libgmp), if libbotan has been installed and you see this error,
+         try upgrading to a higher version of botan or installing libbz2
+         and libgmp.])]
 )
 CPPFLAGS=$CPPFLAGS_SAVED
 LIBS=$LIBS_SAVED
@@ -658,70 +784,23 @@ if test "${boost_include_path}" ; then
 fi
 AC_CHECK_HEADERS([boost/shared_ptr.hpp boost/foreach.hpp boost/interprocess/sync/interprocess_upgradable_mutex.hpp boost/date_time/posix_time/posix_time_types.hpp boost/bind.hpp boost/function.hpp],,
   AC_MSG_ERROR([Missing required header files.]))
-CPPFLAGS="$CPPFLAGS_SAVES"
-AC_SUBST(BOOST_INCLUDES)
-
 
-if test "${use_boost_threads}" = "yes" ; then
-    AC_DEFINE([USE_BOOST_THREADS], [], [Use boost threads])
-
-    # Using boost::mutex can result in requiring libboost_thread with older
-    # versions of Boost.  We'd like to avoid relying on a compiled Boost library
-    # whenever possible, so we need to check for it step by step.
-    #
-    # NOTE: another fix of this problem is to simply require newer versions of
-    # boost.  If we choose that solution we should simplify the following tricky
-    # checks accordingly and all Makefile.am's that refer to NEED_LIBBOOST_THREAD.
-    AC_MSG_CHECKING(for boost::mutex)
-    CPPFLAGS_SAVES="$CPPFLAGS"
-    LIBS_SAVES="$LIBS"
-    CPPFLAGS="$BOOST_INCLUDES $CPPFLAGS $MULTITHREADING_FLAG"
-    need_libboost_thread=0
-    need_sunpro_workaround=0
-    AC_TRY_LINK([
-    #include <boost/thread.hpp>
-    ],[
-    boost::mutex m;
-    ],
-        [ AC_MSG_RESULT(yes (without libboost_thread)) ],
-        # there is one specific problem with SunStudio 5.10
-        # where including boost/thread causes a compilation failure
-        # There is a workaround in boost but it checks the version not being 5.10
-        # This will probably be fixed in the future, in which case this
-        # is only a temporary workaround
-        [ AC_TRY_LINK([
-    #if defined(__SUNPRO_CC) && __SUNPRO_CC == 0x5100
-    #undef __SUNPRO_CC
-    #define __SUNPRO_CC 0x5090
-    #endif
-    #include <boost/thread.hpp>
-    ],[
-    boost::mutex m;
-    ],
-        [ AC_MSG_RESULT(yes (with SUNOS workaround))
-          need_sunpro_workaround=1 ],
-            [ LIBS=" $LIBS -lboost_thread"
-          AC_TRY_LINK([
-    #include <boost/thread.hpp>
-    ],[
-    boost::mutex m;
-    ],
-              [ AC_MSG_RESULT(yes (with libboost_thread))
-                need_libboost_thread=1 ],
-              [ AC_MSG_RESULT(no)
-                AC_MSG_ERROR([boost::mutex cannot be linked in this build environment.
-    Perhaps you are using an older version of Boost that requires libboost_thread for the mutex support, which does not appear to be available.
-    You may want to check the availability of the library or to upgrade Boost.])
-              ])])])
-    CPPFLAGS="$CPPFLAGS_SAVES"
-    LIBS="$LIBS_SAVES"
-    AM_CONDITIONAL(NEED_LIBBOOST_THREAD, test $need_libboost_thread = 1)
-    if test $need_sunpro_workaround = 1; then
-        AC_DEFINE([NEED_SUNPRO_WORKAROUND], [], [Need boost sunstudio workaround])
-    fi
-else
-    AM_CONDITIONAL(NEED_LIBBOOST_THREAD, test "${use_boost_threads}" = "yes")
-fi
+# Detect whether Boost tries to use threads by default, and, if not,
+# make it sure explicitly.  In some systems the automatic detection
+# may depend on preceding header files, and if inconsistency happens
+# it could lead to a critical disruption.
+AC_MSG_CHECKING([whether Boost tries to use threads])
+AC_TRY_COMPILE([
+#include <boost/config.hpp>
+#ifdef BOOST_HAS_THREADS
+#error "boost will use threads"
+#endif],,
+[AC_MSG_RESULT(no)
+ CPPFLAGS_BOOST_THREADCONF="-DBOOST_DISABLE_THREADS=1"],
+[AC_MSG_RESULT(yes)])
+
+CPPFLAGS="$CPPFLAGS_SAVES $CPPFLAGS_BOOST_THREADCONF"
+AC_SUBST(BOOST_INCLUDES)
 
 # I can't get some of the #include <asio.hpp> right without this
 # TODO: find the real cause of asio/boost wanting pthreads
@@ -846,28 +925,9 @@ CPPFLAGS="$CPPFLAGS -I\$(top_srcdir)/ext/coroutine"
 #
 # Disable threads: Currently we don't use them.
 CPPFLAGS="$CPPFLAGS -DASIO_DISABLE_THREADS=1"
-#
-# kqueue portability: ASIO uses kqueue by default if it's available (it's
-# generally available in BSD variants).  Unfortunately, some public
-# implementation of kqueue forces a conversion from a pointer to an integer,
-# which is prohibited in C++ unless reinterpret_cast, C++'s most evil beast
-# (and ASIO doesn't use it anyway) is used.  This will cause build error for
-# some of our C++ files including ASIO header files.  The following check
-# detects such cases and tells ASIO not to use kqueue if so.
-AC_CHECK_FUNC(kqueue, ac_cv_have_kqueue=yes, ac_cv_have_kqueue=no)
-if test "X$ac_cv_have_kqueue" = "Xyes"; then
-	AC_MSG_CHECKING([whether kqueue EV_SET compiles in C++])
-	AC_TRY_COMPILE([
-#include <sys/types.h>
-#include <sys/param.h>
-#include <sys/event.h>],
-[char* udata;
-EV_SET(NULL, 0, 0, 0, 0, 0, udata);],
-	[AC_MSG_RESULT(yes)],
-	[AC_MSG_RESULT([no, disable kqueue for ASIO])
-	 CPPFLAGS="$CPPFLAGS -DASIO_DISABLE_KQUEUE=1"
-	])
-fi
+
+# Check for functions that are not available on all platforms
+AC_CHECK_FUNCS([pselect])
 
 # perfdhcp: If the clock_gettime() function does not exist on the system,
 # use an alternative supplied in the code based on gettimeofday().
@@ -917,6 +977,11 @@ AC_ARG_ENABLE(install-configurations,
 
 AM_CONDITIONAL(INSTALL_CONFIGURATIONS, test x$install_configurations = xyes || test x$install_configurations = xtrue)
 
+AC_ARG_ENABLE(logger-checks, [AC_HELP_STRING([--enable-logger-checks],
+  [check logger messages [default=no]])], enable_logger_checks=$enableval, enable_logger_checks=no)
+AM_CONDITIONAL(ENABLE_LOGGER_CHECKS, test x$enable_logger_checks != xno)
+AM_COND_IF([ENABLE_LOGGER_CHECKS], [AC_DEFINE([ENABLE_LOGGER_CHECKS], [1], [Check logger messages?])])
+
 AC_CONFIG_FILES([Makefile
                  doc/Makefile
                  doc/guide/Makefile
@@ -933,6 +998,9 @@ AC_CONFIG_FILES([Makefile
                  src/bin/cfgmgr/plugins/Makefile
                  src/bin/cfgmgr/plugins/tests/Makefile
                  src/bin/cfgmgr/tests/Makefile
+                 src/bin/dbutil/Makefile
+                 src/bin/dbutil/tests/Makefile
+                 src/bin/dbutil/tests/testdata/Makefile
                  src/bin/host/Makefile
                  src/bin/loadzone/Makefile
                  src/bin/loadzone/tests/correct/Makefile
@@ -946,8 +1014,8 @@ AC_CONFIG_FILES([Makefile
                  src/bin/ddns/tests/Makefile
                  src/bin/dhcp6/Makefile
                  src/bin/dhcp6/tests/Makefile
-		 src/bin/dhcp4/Makefile
-		 src/bin/dhcp4/tests/Makefile
+                 src/bin/dhcp4/Makefile
+                 src/bin/dhcp4/tests/Makefile
                  src/bin/resolver/Makefile
                  src/bin/resolver/tests/Makefile
                  src/bin/sockcreator/Makefile
@@ -1001,6 +1069,8 @@ AC_CONFIG_FILES([Makefile
                  src/lib/python/isc/bind10/tests/Makefile
                  src/lib/python/isc/xfrin/Makefile
                  src/lib/python/isc/xfrin/tests/Makefile
+                 src/lib/python/isc/server_common/Makefile
+                 src/lib/python/isc/server_common/tests/Makefile
                  src/lib/config/Makefile
                  src/lib/config/tests/Makefile
                  src/lib/config/tests/testdata/Makefile
@@ -1050,15 +1120,18 @@ AC_CONFIG_FILES([Makefile
                  tests/tools/badpacket/Makefile
                  tests/tools/badpacket/tests/Makefile
                  tests/tools/perfdhcp/Makefile
+                 dns++.pc
                ])
 AC_OUTPUT([doc/version.ent
-           compatcheck/sqlite3-difftbl-check.py
            src/bin/cfgmgr/b10-cfgmgr.py
            src/bin/cfgmgr/tests/b10-cfgmgr_test.py
            src/bin/cmdctl/cmdctl.py
            src/bin/cmdctl/run_b10-cmdctl.sh
            src/bin/cmdctl/tests/cmdctl_test
            src/bin/cmdctl/cmdctl.spec.pre
+           src/bin/dbutil/dbutil.py
+           src/bin/dbutil/run_dbutil.sh
+           src/bin/dbutil/tests/dbutil_test.sh
            src/bin/ddns/ddns.py
            src/bin/xfrin/tests/xfrin_test
            src/bin/xfrin/xfrin.py
@@ -1135,13 +1208,14 @@ AC_OUTPUT([doc/version.ent
            tests/system/ixfr/in-3/setup.sh
            tests/system/ixfr/in-4/setup.sh
           ], [
-           chmod +x compatcheck/sqlite3-difftbl-check.py
            chmod +x src/bin/cmdctl/run_b10-cmdctl.sh
            chmod +x src/bin/xfrin/run_b10-xfrin.sh
            chmod +x src/bin/xfrout/run_b10-xfrout.sh
            chmod +x src/bin/zonemgr/run_b10-zonemgr.sh
            chmod +x src/bin/bind10/run_bind10.sh
            chmod +x src/bin/cmdctl/tests/cmdctl_test
+           chmod +x src/bin/dbutil/run_dbutil.sh
+           chmod +x src/bin/dbutil/tests/dbutil_test.sh
            chmod +x src/bin/xfrin/tests/xfrin_test
            chmod +x src/bin/xfrout/tests/xfrout_test
            chmod +x src/bin/zonemgr/tests/zonemgr_test

depcomp

@@ -1,630 +0,0 @@
-#! /bin/sh
-# depcomp - compile a program generating dependencies as side-effects
-
-scriptversion=2009-04-28.21; # UTC
-
-# Copyright (C) 1999, 2000, 2003, 2004, 2005, 2006, 2007, 2009 Free
-# Software Foundation, Inc.
-
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-# As a special exception to the GNU General Public License, if you
-# distribute this file as part of a program that contains a
-# configuration script generated by Autoconf, you may include it under
-# the same distribution terms that you use for the rest of that program.
-
-# Originally written by Alexandre Oliva <oliva@dcc.unicamp.br>.
-
-case $1 in
-  '')
-     echo "$0: No command.  Try \`$0 --help' for more information." 1>&2
-     exit 1;
-     ;;
-  -h | --h*)
-    cat <<\EOF
-Usage: depcomp [--help] [--version] PROGRAM [ARGS]
-
-Run PROGRAMS ARGS to compile a file, generating dependencies
-as side-effects.
-
-Environment variables:
-  depmode     Dependency tracking mode.
-  source      Source file read by `PROGRAMS ARGS'.
-  object      Object file output by `PROGRAMS ARGS'.
-  DEPDIR      directory where to store dependencies.
-  depfile     Dependency file to output.
-  tmpdepfile  Temporary file to use when outputing dependencies.
-  libtool     Whether libtool is used (yes/no).
-
-Report bugs to <bug-automake@gnu.org>.
-EOF
-    exit $?
-    ;;
-  -v | --v*)
-    echo "depcomp $scriptversion"
-    exit $?
-    ;;
-esac
-
-if test -z "$depmode" || test -z "$source" || test -z "$object"; then
-  echo "depcomp: Variables source, object and depmode must be set" 1>&2
-  exit 1
-fi
-
-# Dependencies for sub/bar.o or sub/bar.obj go into sub/.deps/bar.Po.
-depfile=${depfile-`echo "$object" |
-  sed 's|[^\\/]*$|'${DEPDIR-.deps}'/&|;s|\.\([^.]*\)$|.P\1|;s|Pobj$|Po|'`}
-tmpdepfile=${tmpdepfile-`echo "$depfile" | sed 's/\.\([^.]*\)$/.T\1/'`}
-
-rm -f "$tmpdepfile"
-
-# Some modes work just like other modes, but use different flags.  We
-# parameterize here, but still list the modes in the big case below,
-# to make depend.m4 easier to write.  Note that we *cannot* use a case
-# here, because this file can only contain one case statement.
-if test "$depmode" = hp; then
-  # HP compiler uses -M and no extra arg.
-  gccflag=-M
-  depmode=gcc
-fi
-
-if test "$depmode" = dashXmstdout; then
-   # This is just like dashmstdout with a different argument.
-   dashmflag=-xM
-   depmode=dashmstdout
-fi
-
-cygpath_u="cygpath -u -f -"
-if test "$depmode" = msvcmsys; then
-   # This is just like msvisualcpp but w/o cygpath translation.
-   # Just convert the backslash-escaped backslashes to single forward
-   # slashes to satisfy depend.m4
-   cygpath_u="sed s,\\\\\\\\,/,g"
-   depmode=msvisualcpp
-fi
-
-case "$depmode" in
-gcc3)
-## gcc 3 implements dependency tracking that does exactly what
-## we want.  Yay!  Note: for some reason libtool 1.4 doesn't like
-## it if -MD -MP comes after the -MF stuff.  Hmm.
-## Unfortunately, FreeBSD c89 acceptance of flags depends upon
-## the command line argument order; so add the flags where they
-## appear in depend2.am.  Note that the slowdown incurred here
-## affects only configure: in makefiles, %FASTDEP% shortcuts this.
-  for arg
-  do
-    case $arg in
-    -c) set fnord "$@" -MT "$object" -MD -MP -MF "$tmpdepfile" "$arg" ;;
-    *)  set fnord "$@" "$arg" ;;
-    esac
-    shift # fnord
-    shift # $arg
-  done
-  "$@"
-  stat=$?
-  if test $stat -eq 0; then :
-  else
-    rm -f "$tmpdepfile"
-    exit $stat
-  fi
-  mv "$tmpdepfile" "$depfile"
-  ;;
-
-gcc)
-## There are various ways to get dependency output from gcc.  Here's
-## why we pick this rather obscure method:
-## - Don't want to use -MD because we'd like the dependencies to end
-##   up in a subdir.  Having to rename by hand is ugly.
-##   (We might end up doing this anyway to support other compilers.)
-## - The DEPENDENCIES_OUTPUT environment variable makes gcc act like
-##   -MM, not -M (despite what the docs say).
-## - Using -M directly means running the compiler twice (even worse
-##   than renaming).
-  if test -z "$gccflag"; then
-    gccflag=-MD,
-  fi
-  "$@" -Wp,"$gccflag$tmpdepfile"
-  stat=$?
-  if test $stat -eq 0; then :
-  else
-    rm -f "$tmpdepfile"
-    exit $stat
-  fi
-  rm -f "$depfile"
-  echo "$object : \\" > "$depfile"
-  alpha=ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
-## The second -e expression handles DOS-style file names with drive letters.
-  sed -e 's/^[^:]*: / /' \
-      -e 's/^['$alpha']:\/[^:]*: / /' < "$tmpdepfile" >> "$depfile"
-## This next piece of magic avoids the `deleted header file' problem.
-## The problem is that when a header file which appears in a .P file
-## is deleted, the dependency causes make to die (because there is
-## typically no way to rebuild the header).  We avoid this by adding
-## dummy dependencies for each header file.  Too bad gcc doesn't do
-## this for us directly.
-  tr ' ' '
-' < "$tmpdepfile" |
-## Some versions of gcc put a space before the `:'.  On the theory
-## that the space means something, we add a space to the output as
-## well.
-## Some versions of the HPUX 10.20 sed can't process this invocation
-## correctly.  Breaking it into two sed invocations is a workaround.
-    sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' | sed -e 's/$/ :/' >> "$depfile"
-  rm -f "$tmpdepfile"
-  ;;
-
-hp)
-  # This case exists only to let depend.m4 do its work.  It works by
-  # looking at the text of this script.  This case will never be run,
-  # since it is checked for above.
-  exit 1
-  ;;
-
-sgi)
-  if test "$libtool" = yes; then
-    "$@" "-Wp,-MDupdate,$tmpdepfile"
-  else
-    "$@" -MDupdate "$tmpdepfile"
-  fi
-  stat=$?
-  if test $stat -eq 0; then :
-  else
-    rm -f "$tmpdepfile"
-    exit $stat
-  fi
-  rm -f "$depfile"
-
-  if test -f "$tmpdepfile"; then  # yes, the sourcefile depend on other files
-    echo "$object : \\" > "$depfile"
-
-    # Clip off the initial element (the dependent).  Don't try to be
-    # clever and replace this with sed code, as IRIX sed won't handle
-    # lines with more than a fixed number of characters (4096 in
-    # IRIX 6.2 sed, 8192 in IRIX 6.5).  We also remove comment lines;
-    # the IRIX cc adds comments like `#:fec' to the end of the
-    # dependency line.
-    tr ' ' '
-' < "$tmpdepfile" \
-    | sed -e 's/^.*\.o://' -e 's/#.*$//' -e '/^$/ d' | \
-    tr '
-' ' ' >> "$depfile"
-    echo >> "$depfile"
-
-    # The second pass generates a dummy entry for each header file.
-    tr ' ' '
-' < "$tmpdepfile" \
-   | sed -e 's/^.*\.o://' -e 's/#.*$//' -e '/^$/ d' -e 's/$/:/' \
-   >> "$depfile"
-  else
-    # The sourcefile does not contain any dependencies, so just
-    # store a dummy comment line, to avoid errors with the Makefile
-    # "include basename.Plo" scheme.
-    echo "#dummy" > "$depfile"
-  fi
-  rm -f "$tmpdepfile"
-  ;;
-
-aix)
-  # The C for AIX Compiler uses -M and outputs the dependencies
-  # in a .u file.  In older versions, this file always lives in the
-  # current directory.  Also, the AIX compiler puts `$object:' at the
-  # start of each line; $object doesn't have directory information.
-  # Version 6 uses the directory in both cases.
-  dir=`echo "$object" | sed -e 's|/[^/]*$|/|'`
-  test "x$dir" = "x$object" && dir=
-  base=`echo "$object" | sed -e 's|^.*/||' -e 's/\.o$//' -e 's/\.lo$//'`
-  if test "$libtool" = yes; then
-    tmpdepfile1=$dir$base.u
-    tmpdepfile2=$base.u
-    tmpdepfile3=$dir.libs/$base.u
-    "$@" -Wc,-M
-  else
-    tmpdepfile1=$dir$base.u
-    tmpdepfile2=$dir$base.u
-    tmpdepfile3=$dir$base.u
-    "$@" -M
-  fi
-  stat=$?
-
-  if test $stat -eq 0; then :
-  else
-    rm -f "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
-    exit $stat
-  fi
-
-  for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
-  do
-    test -f "$tmpdepfile" && break
-  done
-  if test -f "$tmpdepfile"; then
-    # Each line is of the form `foo.o: dependent.h'.
-    # Do two passes, one to just change these to
-    # `$object: dependent.h' and one to simply `dependent.h:'.
-    sed -e "s,^.*\.[a-z]*:,$object:," < "$tmpdepfile" > "$depfile"
-    # That's a tab and a space in the [].
-    sed -e 's,^.*\.[a-z]*:[	 ]*,,' -e 's,$,:,' < "$tmpdepfile" >> "$depfile"
-  else
-    # The sourcefile does not contain any dependencies, so just
-    # store a dummy comment line, to avoid errors with the Makefile
-    # "include basename.Plo" scheme.
-    echo "#dummy" > "$depfile"
-  fi
-  rm -f "$tmpdepfile"
-  ;;
-
-icc)
-  # Intel's C compiler understands `-MD -MF file'.  However on
-  #    icc -MD -MF foo.d -c -o sub/foo.o sub/foo.c
-  # ICC 7.0 will fill foo.d with something like
-  #    foo.o: sub/foo.c
-  #    foo.o: sub/foo.h
-  # which is wrong.  We want:
-  #    sub/foo.o: sub/foo.c
-  #    sub/foo.o: sub/foo.h
-  #    sub/foo.c:
-  #    sub/foo.h:
-  # ICC 7.1 will output
-  #    foo.o: sub/foo.c sub/foo.h
-  # and will wrap long lines using \ :
-  #    foo.o: sub/foo.c ... \
-  #     sub/foo.h ... \
-  #     ...
-
-  "$@" -MD -MF "$tmpdepfile"
-  stat=$?
-  if test $stat -eq 0; then :
-  else
-    rm -f "$tmpdepfile"
-    exit $stat
-  fi
-  rm -f "$depfile"
-  # Each line is of the form `foo.o: dependent.h',
-  # or `foo.o: dep1.h dep2.h \', or ` dep3.h dep4.h \'.
-  # Do two passes, one to just change these to
-  # `$object: dependent.h' and one to simply `dependent.h:'.
-  sed "s,^[^:]*:,$object :," < "$tmpdepfile" > "$depfile"
-  # Some versions of the HPUX 10.20 sed can't process this invocation
-  # correctly.  Breaking it into two sed invocations is a workaround.
-  sed 's,^[^:]*: \(.*\)$,\1,;s/^\\$//;/^$/d;/:$/d' < "$tmpdepfile" |
-    sed -e 's/$/ :/' >> "$depfile"
-  rm -f "$tmpdepfile"
-  ;;
-
-hp2)
-  # The "hp" stanza above does not work with aCC (C++) and HP's ia64
-  # compilers, which have integrated preprocessors.  The correct option
-  # to use with these is +Maked; it writes dependencies to a file named
-  # 'foo.d', which lands next to the object file, wherever that
-  # happens to be.
-  # Much of this is similar to the tru64 case; see comments there.
-  dir=`echo "$object" | sed -e 's|/[^/]*$|/|'`
-  test "x$dir" = "x$object" && dir=
-  base=`echo "$object" | sed -e 's|^.*/||' -e 's/\.o$//' -e 's/\.lo$//'`
-  if test "$libtool" = yes; then
-    tmpdepfile1=$dir$base.d
-    tmpdepfile2=$dir.libs/$base.d
-    "$@" -Wc,+Maked
-  else
-    tmpdepfile1=$dir$base.d
-    tmpdepfile2=$dir$base.d
-    "$@" +Maked
-  fi
-  stat=$?
-  if test $stat -eq 0; then :
-  else
-     rm -f "$tmpdepfile1" "$tmpdepfile2"
-     exit $stat
-  fi
-
-  for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2"
-  do
-    test -f "$tmpdepfile" && break
-  done
-  if test -f "$tmpdepfile"; then
-    sed -e "s,^.*\.[a-z]*:,$object:," "$tmpdepfile" > "$depfile"
-    # Add `dependent.h:' lines.
-    sed -ne '2,${
-	       s/^ *//
-	       s/ \\*$//
-	       s/$/:/
-	       p
-	     }' "$tmpdepfile" >> "$depfile"
-  else
-    echo "#dummy" > "$depfile"
-  fi
-  rm -f "$tmpdepfile" "$tmpdepfile2"
-  ;;
-
-tru64)
-   # The Tru64 compiler uses -MD to generate dependencies as a side
-   # effect.  `cc -MD -o foo.o ...' puts the dependencies into `foo.o.d'.
-   # At least on Alpha/Redhat 6.1, Compaq CCC V6.2-504 seems to put
-   # dependencies in `foo.d' instead, so we check for that too.
-   # Subdirectories are respected.
-   dir=`echo "$object" | sed -e 's|/[^/]*$|/|'`
-   test "x$dir" = "x$object" && dir=
-   base=`echo "$object" | sed -e 's|^.*/||' -e 's/\.o$//' -e 's/\.lo$//'`
-
-   if test "$libtool" = yes; then
-      # With Tru64 cc, shared objects can also be used to make a
-      # static library.  This mechanism is used in libtool 1.4 series to
-      # handle both shared and static libraries in a single compilation.
-      # With libtool 1.4, dependencies were output in $dir.libs/$base.lo.d.
-      #
-      # With libtool 1.5 this exception was removed, and libtool now
-      # generates 2 separate objects for the 2 libraries.  These two
-      # compilations output dependencies in $dir.libs/$base.o.d and
-      # in $dir$base.o.d.  We have to check for both files, because
-      # one of the two compilations can be disabled.  We should prefer
-      # $dir$base.o.d over $dir.libs/$base.o.d because the latter is
-      # automatically cleaned when .libs/ is deleted, while ignoring
-      # the former would cause a distcleancheck panic.
-      tmpdepfile1=$dir.libs/$base.lo.d   # libtool 1.4
-      tmpdepfile2=$dir$base.o.d          # libtool 1.5
-      tmpdepfile3=$dir.libs/$base.o.d    # libtool 1.5
-      tmpdepfile4=$dir.libs/$base.d      # Compaq CCC V6.2-504
-      "$@" -Wc,-MD
-   else
-      tmpdepfile1=$dir$base.o.d
-      tmpdepfile2=$dir$base.d
-      tmpdepfile3=$dir$base.d
-      tmpdepfile4=$dir$base.d
-      "$@" -MD
-   fi
-
-   stat=$?
-   if test $stat -eq 0; then :
-   else
-      rm -f "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3" "$tmpdepfile4"
-      exit $stat
-   fi
-
-   for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3" "$tmpdepfile4"
-   do
-     test -f "$tmpdepfile" && break
-   done
-   if test -f "$tmpdepfile"; then
-      sed -e "s,^.*\.[a-z]*:,$object:," < "$tmpdepfile" > "$depfile"
-      # That's a tab and a space in the [].
-      sed -e 's,^.*\.[a-z]*:[	 ]*,,' -e 's,$,:,' < "$tmpdepfile" >> "$depfile"
-   else
-      echo "#dummy" > "$depfile"
-   fi
-   rm -f "$tmpdepfile"
-   ;;
-
-#nosideeffect)
-  # This comment above is used by automake to tell side-effect
-  # dependency tracking mechanisms from slower ones.
-
-dashmstdout)
-  # Important note: in order to support this mode, a compiler *must*
-  # always write the preprocessed file to stdout, regardless of -o.
-  "$@" || exit $?
-
-  # Remove the call to Libtool.
-  if test "$libtool" = yes; then
-    while test "X$1" != 'X--mode=compile'; do
-      shift
-    done
-    shift
-  fi
-
-  # Remove `-o $object'.
-  IFS=" "
-  for arg
-  do
-    case $arg in
-    -o)
-      shift
-      ;;
-    $object)
-      shift
-      ;;
-    *)
-      set fnord "$@" "$arg"
-      shift # fnord
-      shift # $arg
-      ;;
-    esac
-  done
-
-  test -z "$dashmflag" && dashmflag=-M
-  # Require at least two characters before searching for `:'
-  # in the target name.  This is to cope with DOS-style filenames:
-  # a dependency such as `c:/foo/bar' could be seen as target `c' otherwise.
-  "$@" $dashmflag |
-    sed 's:^[  ]*[^: ][^:][^:]*\:[    ]*:'"$object"'\: :' > "$tmpdepfile"
-  rm -f "$depfile"
-  cat < "$tmpdepfile" > "$depfile"
-  tr ' ' '
-' < "$tmpdepfile" | \
-## Some versions of the HPUX 10.20 sed can't process this invocation
-## correctly.  Breaking it into two sed invocations is a workaround.
-    sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' | sed -e 's/$/ :/' >> "$depfile"
-  rm -f "$tmpdepfile"
-  ;;
-
-dashXmstdout)
-  # This case only exists to satisfy depend.m4.  It is never actually
-  # run, as this mode is specially recognized in the preamble.
-  exit 1
-  ;;
-
-makedepend)
-  "$@" || exit $?
-  # Remove any Libtool call
-  if test "$libtool" = yes; then
-    while test "X$1" != 'X--mode=compile'; do
-      shift
-    done
-    shift
-  fi
-  # X makedepend
-  shift
-  cleared=no eat=no
-  for arg
-  do
-    case $cleared in
-    no)
-      set ""; shift
-      cleared=yes ;;
-    esac
-    if test $eat = yes; then
-      eat=no
-      continue
-    fi
-    case "$arg" in
-    -D*|-I*)
-      set fnord "$@" "$arg"; shift ;;
-    # Strip any option that makedepend may not understand.  Remove
-    # the object too, otherwise makedepend will parse it as a source file.
-    -arch)
-      eat=yes ;;
-    -*|$object)
-      ;;
-    *)
-      set fnord "$@" "$arg"; shift ;;
-    esac
-  done
-  obj_suffix=`echo "$object" | sed 's/^.*\././'`
-  touch "$tmpdepfile"
-  ${MAKEDEPEND-makedepend} -o"$obj_suffix" -f"$tmpdepfile" "$@"
-  rm -f "$depfile"
-  cat < "$tmpdepfile" > "$depfile"
-  sed '1,2d' "$tmpdepfile" | tr ' ' '
-' | \
-## Some versions of the HPUX 10.20 sed can't process this invocation
-## correctly.  Breaking it into two sed invocations is a workaround.
-    sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' | sed -e 's/$/ :/' >> "$depfile"
-  rm -f "$tmpdepfile" "$tmpdepfile".bak
-  ;;
-
-cpp)
-  # Important note: in order to support this mode, a compiler *must*
-  # always write the preprocessed file to stdout.
-  "$@" || exit $?
-
-  # Remove the call to Libtool.
-  if test "$libtool" = yes; then
-    while test "X$1" != 'X--mode=compile'; do
-      shift
-    done
-    shift
-  fi
-
-  # Remove `-o $object'.
-  IFS=" "
-  for arg
-  do
-    case $arg in
-    -o)
-      shift
-      ;;
-    $object)
-      shift
-      ;;
-    *)
-      set fnord "$@" "$arg"
-      shift # fnord
-      shift # $arg
-      ;;
-    esac
-  done
-
-  "$@" -E |
-    sed -n -e '/^# [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' \
-       -e '/^#line [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' |
-    sed '$ s: \\$::' > "$tmpdepfile"
-  rm -f "$depfile"
-  echo "$object : \\" > "$depfile"
-  cat < "$tmpdepfile" >> "$depfile"
-  sed < "$tmpdepfile" '/^$/d;s/^ //;s/ \\$//;s/$/ :/' >> "$depfile"
-  rm -f "$tmpdepfile"
-  ;;
-
-msvisualcpp)
-  # Important note: in order to support this mode, a compiler *must*
-  # always write the preprocessed file to stdout.
-  "$@" || exit $?
-
-  # Remove the call to Libtool.
-  if test "$libtool" = yes; then
-    while test "X$1" != 'X--mode=compile'; do
-      shift
-    done
-    shift
-  fi
-
-  IFS=" "
-  for arg
-  do
-    case "$arg" in
-    -o)
-      shift
-      ;;
-    $object)
-      shift
-      ;;
-    "-Gm"|"/Gm"|"-Gi"|"/Gi"|"-ZI"|"/ZI")
-	set fnord "$@"
-	shift
-	shift
-	;;
-    *)
-	set fnord "$@" "$arg"
-	shift
-	shift
-	;;
-    esac
-  done
-  "$@" -E 2>/dev/null |
-  sed -n '/^#line [0-9][0-9]* "\([^"]*\)"/ s::\1:p' | $cygpath_u | sort -u > "$tmpdepfile"
-  rm -f "$depfile"
-  echo "$object : \\" > "$depfile"
-  sed < "$tmpdepfile" -n -e 's% %\\ %g' -e '/^\(.*\)$/ s::	\1 \\:p' >> "$depfile"
-  echo "	" >> "$depfile"
-  sed < "$tmpdepfile" -n -e 's% %\\ %g' -e '/^\(.*\)$/ s::\1\::p' >> "$depfile"
-  rm -f "$tmpdepfile"
-  ;;
-
-msvcmsys)
-  # This case exists only to let depend.m4 do its work.  It works by
-  # looking at the text of this script.  This case will never be run,
-  # since it is checked for above.
-  exit 1
-  ;;
-
-none)
-  exec "$@"
-  ;;
-
-*)
-  echo "Unknown depmode $depmode" 1>&2
-  exit 1
-  ;;
-esac
-
-exit 0
-
-# Local Variables:
-# mode: shell-script
-# sh-indentation: 2
-# eval: (add-hook 'write-file-hooks 'time-stamp)
-# time-stamp-start: "scriptversion="
-# time-stamp-format: "%:y-%02m-%02d.%02H"
-# time-stamp-time-zone: "UTC"
-# time-stamp-end: "; # UTC"
-# End:

dns++.pc.in

@@ -0,0 +1,11 @@
+prefix=@prefix@
+exec_prefix=@exec_prefix@
+libdir=@libdir@
+includedir=@includedir@
+
+Name: dns++
+Description: BIND 10 DNS library
+Version: @PACKAGE_VERSION@
+Requires: botan-1.8
+Cflags: -I${includedir}/@PACKAGE_NAME@
+Libs: -L${libdir} -ldns++ -lcryptolink -lutil -lexceptions -lm

doc/.gitignore

@@ -0,0 +1,2 @@
+/version.ent
+html

doc/Doxyfile

@@ -25,13 +25,17 @@ DOXYFILE_ENCODING      = UTF-8
 # The PROJECT_NAME tag is a single word (or a sequence of words surrounded
 # by quotes) that should identify the project.
 
-PROJECT_NAME           = BIND
+PROJECT_NAME           = BIND10
 
 # The PROJECT_NUMBER tag can be used to enter a project or revision number.
 # This could be handy for archiving the generated documentation or
 # if some version control system is used.
 
-PROJECT_NUMBER         = 10.0.0
+# Currently this variable is overwritten (see devel target in Makefile.am)
+# If the number of parameters to overwrite increases, we should generate
+# Doxyfile (rename it to Doxyfile.in and generate during configure phase)
+
+PROJECT_NUMBER         =
 
 # The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute)
 # base path where the generated documentation will be put.
@@ -47,7 +51,7 @@ OUTPUT_DIRECTORY       = html
 # source files, where putting all generated files in the same directory would
 # otherwise cause performance problems for the file system.
 
-CREATE_SUBDIRS         = NO
+CREATE_SUBDIRS         = YES
 
 # The OUTPUT_LANGUAGE tag is used to specify the language in which all
 # documentation generated by doxygen is written. Doxygen will use this
@@ -281,7 +285,7 @@ TYPEDEF_HIDES_STRUCT   = NO
 # causing a significant performance penality.
 # If the system has enough physical memory increasing the cache will improve the
 # performance by keeping more symbols in memory. Note that the value works on
-# a logarithmic scale so increasing the size by one will rougly double the
+# a logarithmic scale so increasing the size by one will roughly double the
 # memory usage. The cache size is given by this formula:
 # 2^(16+SYMBOL_CACHE_SIZE). The valid range is 0..9, the default is 0,
 # corresponding to a cache size of 2^16 = 65536 symbols
@@ -574,7 +578,8 @@ INPUT                  = ../src/lib/exceptions ../src/lib/cc \
     ../src/lib/log/compiler ../src/lib/asiolink/ ../src/lib/nsas \
     ../src/lib/testutils ../src/lib/cache ../src/lib/server_common/ \
     ../src/bin/sockcreator/ ../src/lib/util/ ../src/lib/util/io/ \
-    ../src/lib/resolve ../src/lib/acl ../src/bin/dhcp6 ../src/lib/dhcp
+    ../src/lib/resolve ../src/lib/acl ../src/bin/dhcp6 ../src/lib/dhcp \
+    ../src/bin/dhcp4 devel
 
 # This tag can be used to specify the character encoding of the source files
 # that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is
@@ -591,7 +596,7 @@ INPUT_ENCODING         = UTF-8
 # *.c *.cc *.cxx *.cpp *.c++ *.java *.ii *.ixx *.ipp *.i++ *.inl *.h *.hh *.hxx
 # *.hpp *.h++ *.idl *.odl *.cs *.php *.php3 *.inc *.m *.mm *.py *.f90
 
-FILE_PATTERNS          =
+FILE_PATTERNS          = *.c *.cc *.h *.hpp *.dox
 
 # The RECURSIVE tag can be used to turn specify whether or not subdirectories
 # should be searched for input files as well. Possible values are YES and NO.
@@ -651,7 +656,7 @@ EXAMPLE_RECURSIVE      = NO
 # directories that contain image that are included in the documentation (see
 # the \image command).
 
-IMAGE_PATH             =
+IMAGE_PATH             = ../doc/images
 
 # The INPUT_FILTER tag can be used to specify a program that doxygen should
 # invoke to filter for each input file. Doxygen will invoke the filter program
@@ -773,7 +778,7 @@ GENERATE_HTML          = YES
 # If a relative path is entered the value of OUTPUT_DIRECTORY will be
 # put in front of it. If left blank `html' will be used as the default path.
 
-HTML_OUTPUT            = cpp
+HTML_OUTPUT            = ../html
 
 # The HTML_FILE_EXTENSION tag can be used to specify the file extension for
 # each generated HTML page (for example: .htm,.php,.asp). If it is left blank
@@ -954,7 +959,7 @@ ENUM_VALUES_PER_LINE   = 4
 # JavaScript, DHTML, CSS and frames is required (i.e. any modern browser).
 # Windows users are probably better off using the HTML help feature.
 
-GENERATE_TREEVIEW      = NO
+GENERATE_TREEVIEW      = YES
 
 # By enabling USE_INLINE_TREES, doxygen will generate the Groups, Directories,
 # and Class Hierarchy pages using a tree view instead of an ordered list.
@@ -965,7 +970,7 @@ USE_INLINE_TREES       = NO
 # used to set the initial width (in pixels) of the frame in which the tree
 # is shown.
 
-TREEVIEW_WIDTH         = 250
+TREEVIEW_WIDTH         = 180
 
 # Use this tag to change the font size of Latex formulas included
 # as images in the HTML documentation. The default is 10. Note that

doc/Makefile.am

@@ -1,3 +1,16 @@
 SUBDIRS = guide
 
 EXTRA_DIST = version.ent.in
+
+devel:
+	mkdir -p html
+	(cat Doxyfile; echo PROJECT_NUMBER=$(PACKAGE_VERSION)) | doxygen - > html/doxygen.log 2> html/doxygen-error.log
+	echo `grep -i ": warning:" html/doxygen-error.log | wc -l` warnings/errors detected.
+
+clean:
+	rm -rf html
+
+# That's a bit of a hack, but we are making sure that devel target
+# is always valid. The alternative is to make devel depend on all
+# *.cc *.h files in the whole tree.
+.PHONY: devel

doc/devel/01-dns.dox

@@ -0,0 +1,14 @@
+/**
+ *
+ * @page dns BIND10 DNS
+ *
+ * @section dns-auth b10-auth
+ *
+ * @todo: Describe b10-auth here.
+ *
+ * @section b10-cfgmgr b10-cfgmgr Overview
+ *
+ * @todo: Descibe b10-cfgmgr here.
+ *
+ *
+ */

doc/devel/02-dhcp.dox

@@ -0,0 +1,117 @@
+/**
+ * @page dhcpv4 DHCPv4 Server Component
+ *
+ * BIND10 offers DHCPv4 server implementation. It is implemented as
+ * b10-dhcp4 component.  Its primary code is located in
+ * isc::dhcp::Dhcpv4Srv class. It uses \ref libdhcp extensively,
+ * especially isc::dhcp::Pkt4, isc::dhcp::Option and
+ * isc::dhcp::IfaceMgr classes. Currently this code offers skeleton
+ * functionality, i.e. it is able to receive and process incoming
+ * requests and trasmit responses. However, it does not have database
+ * management, so it returns only one, hardcoded lease to whoever asks
+ * for it.
+ *
+ * DHCPv4 server component does not support direct traffic (relayed
+ * only), as support for transmission to hosts without IPv4 address
+ * assigned is not implemented in IfaceMgr yet.
+ *
+ * DHCPv4 server component does not listen to BIND10 message queue.
+ *
+ * DHCPv4 server component does not use BIND10 logging yet.
+ *
+ * DHCPv4 server component is not integrated with boss yet.
+ *
+ * @page dhcpv6 DHCPv6 Server Component
+ *
+ * BIND10 offers DHCPv6 server implementation. It is implemented as
+ * b10-dhcp6 component. Its primary code is located in
+ * isc::dhcp::Dhcpv6Srv class. It uses \ref libdhcp extensively,
+ * especially lib::dhcp::Pkt6, isc::dhcp::Option and
+ * isc::dhcp::IfaceMgr classes. Currently this code offers skeleton
+ * functionality, i.e. it is able to receive and process incoming
+ * requests and trasmit responses. However, it does not have database
+ * management, so it returns only one, hardcoded lease to whoever asks
+ * for it.
+ *
+ * DHCPv6 server component does not support relayed traffic yet, as
+ * support for relay decapsulation is not implemented yet.
+ *
+ * DHCPv6 server component does not listen to BIND10 message queue.
+ *
+ * DHCPv6 server component does not use BIND10 logging yet.
+ *
+ * DHCPv6 server component is not integrated with boss yet.
+ *
+ * @page libdhcp libdhcp++ library
+ *
+ * @section libdhcpIntro Libdhcp++ Introduction
+ *
+ * libdhcp++ is an all-purpose DHCP-manipulation library, written in
+ * C++. It offers packet parsing and assembly, DHCPv4 and DHCPv6
+ * options parsing and ssembly, interface detection (currently on
+ * Linux systems only) and socket operations. Following classes are
+ * implemented:
+ *
+ * - isc::dhcp::Pkt4 - represents DHCPv4 packet.
+ * - isc::dhcp::Pkt6 - represents DHCPv6 packet.
+ *
+ * There are two pointer types defined: Pkt4Ptr and Pkt6Ptr. They are
+ * smart pointer and are using boost::shared_ptr. There are not const
+ * versions defined, as we assume that hooks can modify any aspect of
+ * the packet at almost any stage of processing.
+ *
+ * Both packets use collection of Option objects to represent DHCPv4
+ * and DHCPv6 options. The base class -- Option -- can be used to
+ * represent generic option that contains collection of
+ * bytes. Depending on if the option is instantiated as v4 or v6
+ * option, it will adjust its header (DHCPv4 options use 1 octet for
+ * type and 1 octet for length, while DHCPv6 options use 2 bytes for
+ * each).
+ *
+ * There are many specialized classes that are intended to handle options with
+ * specific content:
+ * - isc::dhcp::Option4AddrLst -- DHCPv4 option, contains one or more IPv4 addresses;
+ * - isc::dhcp::Option6AddrLst -- DHCPv6 option, contains one or more IPv6 addresses;
+ * - isc::dhcp::Option6IAAddr -- DHCPv6 option, represents IAADDR_OPTION (an option that
+ *                     contains IPv6 address with extra parameters);
+ * - isc::dhcp::Option6IA -- DHCPv6 option used to store IA_NA and its suboptions.
+ *
+ * All options can store sub-options (i.e. options that are stored within option
+ * rather than in a message directly). This functionality is commonly used in
+ * DHCPv6, but is rarely used in DHCPv4. isc::dhcp::Option::addOption(),
+ * isc::dhcp::Option::delOption(), isc::dhcp::Option::getOption() can be used
+ * for that purpose.
+ *
+ * @section lidhcpIfaceMgr Interface Manager
+ *
+ * Interface Manager (or IfaceMgr) is an abstraction layer about low-level
+ * network operations. In particlar, it provides information about existing
+ * network interfaces See isc::dhcp::IfaceMgr::Iface class and
+ * isc::dhcp::IfaceMgr::detectIfaces() and isc::dhcp::IfaceMgr::getIface().
+ *
+ * Currently there is interface detection is implemented in Linux only. There
+ * are plans to implement such support for other OSes, but they remain low
+ * priority for now.
+ *
+ * Generic parts of the code are isc::dhcp::IfaceMgr class in
+ * src/lib/dhcp/iface_mgr.cc file. OS-specific code is located in separate
+ * files, e.g. iface_mgr_linux.cc. Such separation should be maintained when
+ * additional code will be developed.
+ *
+ * For systems that interface detection is not supported on, there is a stub
+ * mechanism implemented. It assumes that interface name is read from a text
+ * file. This is a temporary solution and will be removed as soon as proper
+ * interface detection is implemented. It is not going to be developed further.
+ * To use this feature, store interfaces.txt file. It uses a simple syntax.
+ * Each line represents an interface name, followed by IPv4 or IPv6 address
+ * that follows it. This is usually link-local IPv6 address that the server
+ * should bind to. In theory this mechanism also supports IPv4, but it was
+ * never tested. The code currently supports only a single interface defined
+ * that way.
+ *
+ * Another useful methods are dedicated to transmission
+ * (isc::dhcp::IfaceMgr::send(), 2 overloads) and reception
+ * (isc::dhcp::IfaceMgr::receive4() and isc::dhcp::IfaceMgr::receive6()).
+ * Note that receive4() and receive6() methods may return NULL, e.g.
+ * when timeout is reached or if dhcp daemon receives a signal.
+ */

doc/devel/mainpage.dox

@@ -0,0 +1,36 @@
+/**
+ *
+ * @mainpage BIND10 Developer's Guide
+ *
+ * Welcome to BIND10 Developer's Guide. This documentation is addressed
+ * at existing and prospecting developers and programmers, who would like
+ * to gain insight into internal workings of BIND 10. It could also be useful
+ * for existing and prospective contributors.
+ *
+ * If you are a user or system administrator, rather than software engineer,
+ * you should read <a href="http://bind10.isc.org/docs/bind10-guide.html">BIND10
+ * Guide (Administrator Reference for BIND10)</a> instead.
+ *
+ * Regardless of your field of expertise, you are encouraged to visit
+ * <a href="http://bind10.isc.org/">BIND10 webpage (http://bind10.isc.org)</a>
+ *
+ * @section DNS
+ * - @subpage DataScrubbing
+ *
+ * @section DHCP
+ * - @subpage dhcpv4
+ * - @subpage dhcpv6
+ * - @subpage libdhcp
+ *
+ * @section misc Miscellaneous topics
+ * - @subpage LoggingApi
+ *   - @subpage LoggingApiOverview
+ *   - @subpage LoggingApiLoggerNames
+ *   - @subpage LoggingApiLoggingMessages
+ * - @subpage SocketSessionUtility
+ * - <a href="./doxygen-error.log">Documentation warnings and errors</a>
+ *
+ * @todo: Move this logo to the right (and possibly up). Not sure what
+ * is the best way to do it in Doxygen, without using CSS hacks.
+ * @image html isc-logo.png
+ */

doc/guide/bind10-guide.xml

@@ -172,15 +172,6 @@
 
           <listitem>
             <simpara>
-              <command>b10-msgq</command> &mdash;
-              Message bus daemon.
-              This process coordinates communication between all of the other
-              BIND 10 processes.
-            </simpara>
-          </listitem>
-
-          <listitem>
-            <simpara>
               <command>b10-auth</command> &mdash;
               Authoritative DNS server.
               This process serves DNS requests.
@@ -205,6 +196,15 @@
 
           <listitem>
             <simpara>
+              <command>b10-msgq</command> &mdash;
+              Message bus daemon.
+              This process coordinates communication between all of the other
+              BIND 10 processes.
+            </simpara>
+          </listitem>
+
+          <listitem>
+            <simpara>
               <command>b10-resolver</command> &mdash;
               Recursive name server.
               This process handles incoming queries.
@@ -214,6 +214,15 @@
 
           <listitem>
             <simpara>
+              <command>b10-sockcreator</command> &mdash;
+              Socket creator daemon.
+              This process creates sockets used by
+              network-listening BIND 10 processes.
+            </simpara>
+          </listitem>
+
+          <listitem>
+            <simpara>
               <command>b10-stats</command> &mdash;
               Statistics collection daemon.
               This process collects and reports statistics data.
@@ -222,6 +231,14 @@
 
           <listitem>
             <simpara>
+              <command>b10-stats-httpd</command> &mdash;
+              HTTP server for statistics reporting.
+              This process reports statistics data in XML format over HTTP.
+            </simpara>
+          </listitem>
+
+          <listitem>
+            <simpara>
               <command>b10-xfrin</command> &mdash;
               Incoming zone transfer service.
               This process is used to transfer a new copy
@@ -269,8 +286,9 @@
             <simpara>
               <command>bindctl</command> &mdash;
               interactive administration interface.
-              This is a command-line tool which allows an administrator
-              to control BIND 10.
+              This is a low-level command-line tool which allows
+              a developer or an experienced administrator to control
+              BIND 10.
             </simpara>
           </listitem>
           <listitem>
@@ -751,12 +769,9 @@ as a dependency earlier -->
     <para>
       In its default configuration, the <command>bind10</command>
       master process will also start up
-      <command>b10-cmdctl</command> for admins to communicate with the
-      system, <command>b10-auth</command> for authoritative DNS service,
-      <command>b10-stats</command> for statistics collection,
-      <command>b10-xfrin</command> for inbound DNS zone transfers,
-      <command>b10-xfrout</command> for outbound DNS zone transfers,
-      and <command>b10-zonemgr</command> for secondary service.
+      <command>b10-cmdctl</command> for administration tools to
+      communicate with the system, and
+      <command>b10-stats</command> for statistics collection.
     </para>
 
     <section id="start">
@@ -790,12 +805,7 @@ as a dependency earlier -->
         The configuration is in the Boss/components section. Each element
         represents one component, which is an abstraction of a process
         (currently there's also one component which doesn't represent
-        a process). If you didn't want to transfer out at all (your server
-        is a slave only), you would just remove the corresponding component
-        from the set, like this and the process would be stopped immediately
-        (and not started on the next startup):
-      <screen>&gt; <userinput>config remove Boss/components b10-xfrout</userinput>
-&gt; <userinput>config commit</userinput></screen>
+        a process).
       </para>
 
       <para>
@@ -889,7 +899,7 @@ address, but the usual ones don't." mean? -->
           This system allows you to start the same component multiple times
           (by including it in the configuration with different names, but the
           same process setting). However, the rest of the system doesn't expect
-          such situation, so it would probably not do what you want. Such
+          such a situation, so it would probably not do what you want. Such
           support is yet to be implemented.
         </para>
       </note>
@@ -901,10 +911,10 @@ address, but the usual ones don't." mean? -->
           <command>b10-cmdctl</command>, but then you couldn't
           change it back the usual way, as it would require it to
           be running (you would have to find and edit the configuration
-          directly).  Also, some modules might have dependencies
-          -- <command>b10-stats-httpd</command> need
+          directly).  Also, some modules might have dependencies:
+          <command>b10-stats-httpd</command> needs
           <command>b10-stats</command>, <command>b10-xfrout</command>
-          needs the <command>b10-auth</command> to be running, etc.
+          needs <command>b10-auth</command> to be running, etc.
 
 <!-- TODO: should we define dependencies? -->
 
@@ -999,7 +1009,7 @@ Unix domain sockets
 <!-- TODO -->
       <note>
         <para>
-          The development prototype release only provides the
+          The development prototype release only provides
           <command>bindctl</command> as a user interface to
           <command>b10-cmdctl</command>.
           Upcoming releases will provide another interactive command-line
@@ -1190,7 +1200,7 @@ or accounts database -->
       The port can be set by using the <option>--port</option> command line option.
       The address to listen on can be set using the <option>--address</option> command
       line argument.
-      Each HTTPS connection is stateless and timesout in 1200 seconds
+      Each HTTPS connection is stateless and times out in 1200 seconds
       by default.  This can be
       redefined by using the <option>--idle-timeout</option> command line argument.
     </para>
@@ -1281,7 +1291,7 @@ since we used bind10 -->
         <command>b10-auth</command> is configured via the
         <command>b10-cfgmgr</command> configuration manager.
         The module name is <quote>Auth</quote>.
-        The configuration data item is:
+        The configuration data items are:
 
         <variablelist>
 
@@ -1297,22 +1307,119 @@ This may be a temporary setting until then.
             </listitem>
           </varlistentry>
 
+<!-- NOTE: docs pulled in verbatim from the b10-auth.xml manual page.
+     TODO: automate this if want this or rewrite
+-->
+          <varlistentry>
+            <term>datasources</term>
+            <listitem>
+              <simpara>
+      <varname>datasources</varname> configures data sources.
+      The list items include:
+      <varname>type</varname> to define the required data source type
+      (such as <quote>memory</quote>);
+      <varname>class</varname> to optionally select the class
+      (it defaults to <quote>IN</quote>);
+      and
+      <varname>zones</varname> to define the
+      <varname>file</varname> path name and the
+      <varname>origin</varname> (default domain).
+
+      By default, this is empty.
+
+      <note><simpara>
+        In this development version, currently this is only used for the
+        memory data source.
+        Only the IN class is supported at this time.
+        By default, the memory data source is disabled.
+        Also, currently the zone file must be canonical such as
+        generated by <command>named-compilezone -D</command>.
+      </simpara></note>
+
+              </simpara>
+            </listitem>
+          </varlistentry>
+
+          <varlistentry>
+            <term>listen_on</term>
+            <listitem>
+              <simpara>
+      <varname>listen_on</varname> is a list of addresses and ports for
+      <command>b10-auth</command> to listen on.
+      The list items are the <varname>address</varname> string
+      and <varname>port</varname> number.
+      By default, <command>b10-auth</command> listens on port 53
+      on the IPv6 (::) and IPv4 (0.0.0.0) wildcard addresses.
+              </simpara>
+            </listitem>
+          </varlistentry>
+
+          <varlistentry>
+            <term>statistics-interval</term>
+            <listitem>
+              <simpara>
+      <varname>statistics-interval</varname> is the timer interval
+      in seconds for <command>b10-auth</command> to share its
+      statistics information to
+      <citerefentry><refentrytitle>b10-stats</refentrytitle><manvolnum>8</manvolnum></citerefentry>.
+      Statistics updates can be disabled by setting this to 0.
+      The default is 60.
+              </simpara>
+            </listitem>
+          </varlistentry>
+
         </variablelist>
 
       </para>
 
       <para>
 
-        The configuration command is:
+        The configuration commands are:
 
         <variablelist>
 
           <varlistentry>
+            <term>loadzone</term>
+            <listitem>
+              <simpara>
+      <command>loadzone</command> tells <command>b10-auth</command>
+      to load or reload a zone file. The arguments include:
+      <varname>class</varname> which optionally defines the class
+      (it defaults to <quote>IN</quote>);
+      <varname>origin</varname> is the domain name of the zone;
+      and
+      <varname>datasrc</varname> optionally defines the type of datasource
+      (it defaults to <quote>memory</quote>).
+
+      <note><simpara>
+        In this development version, currently this only supports the
+        IN class and the memory data source.
+      </simpara></note>
+              </simpara>
+            </listitem>
+          </varlistentry>
+
+          <varlistentry>
+            <term>sendstats</term>
+            <listitem>
+              <simpara>
+      <command>sendstats</command> tells <command>b10-auth</command>
+      to send its statistics data to
+      <citerefentry><refentrytitle>b10-stats</refentrytitle><manvolnum>8</manvolnum></citerefentry>
+      immediately.
+              </simpara>
+            </listitem>
+          </varlistentry>
+
+          <varlistentry>
             <term>shutdown</term>
             <listitem>
               <simpara>Stop the authoritative DNS server.
+      This has an optional <varname>pid</varname> argument to
+      select the process ID to stop.
+      (Note that the BIND 10 boss process may restart this service
+      if configured.)
               </simpara>
-<!-- TODO: what happens when this is sent, will bind10 restart? -->
             </listitem>
           </varlistentry>
 
@@ -1342,10 +1449,79 @@ This may be a temporary setting until then.
         (The full path is what was defined at build configure time for
         <option>--localstatedir</option>.
         The default is <filename>/usr/local/var/</filename>.)
-  This data file location may be changed by defining the
-  <quote>database_file</quote> configuration.
+	This data file location may be changed by defining the
+	<quote>database_file</quote> configuration.
       </para>
 
+      <section id="in-memory-datasource">
+	<title>In-memory Data Source</title>
+
+	<para>
+<!--	  How to configure it. -->
+	  The following commands to <command>bindctl</command>
+	  provide an example of configuring an in-memory data
+	  source containing the <quote>example.com</quote> zone
+	  with the zone file named <quote>example.com.zone</quote>:
+
+<!--
+	  <screen>&gt; <userinput> config set Auth/datasources/ [{"type": "memory", "zones": [{"origin": "example.com", "file": "example.com.zone"}]}]</userinput></screen>
+-->
+
+          <screen>&gt; <userinput>config add Auth/datasources</userinput>
+&gt; <userinput>config set Auth/datasources[0]/type "<option>memory</option>"</userinput>
+&gt; <userinput>config add Auth/datasources[0]/zones</userinput>
+&gt; <userinput>config set Auth/datasources[0]/zones[0]/origin "<option>example.com</option>"</userinput>
+&gt; <userinput>config set Auth/datasources[0]/zones[0]/file "<option>example.com.zone</option>"</userinput>
+&gt; <userinput>config commit</userinput></screen>
+
+	  The authoritative server will begin serving it immediately
+	  after it is loaded.
+	</para>
+
+	<para>
+	  Use the <command>Auth loadzone</command> command in
+	  <command>bindctl</command> to reload a changed master
+	  file into memory; for example:
+
+	  <screen>&gt; <userinput>Auth loadzone origin="example.com"</userinput>
+</screen>
+
+	</para>
+
+<!--
+        <para>
+          The <varname>file</varname> may be an absolute path to the
+          master zone file or it is relative to the directory BIND 10 is
+          started from.
+	</para>
+-->
+
+        <para>
+	By default, the memory data source is disabled; it must be
+	configured explicitly.  To disable all the in-memory zones,
+	specify a null list for <varname>Auth/datasources</varname>:
+
+<!-- TODO: this assumes that Auth/datasources is for memory only -->
+
+	  <screen>&gt; <userinput>config set Auth/datasources/ []</userinput>
+&gt; <userinput>config commit</userinput></screen>
+	</para>
+
+	<para>
+          The following example stops serving a specific zone:
+
+	  <screen>&gt; <userinput>config remove Auth/datasources[<option>0</option>]/zones[<option>0</option>]</userinput>
+&gt; <userinput>config commit</userinput></screen>
+
+	  (Replace the list number(s) in
+	  <varname>datasources[<replaceable>0</replaceable>]</varname>
+	  and/or <varname>zones[<replaceable>0</replaceable>]</varname>
+	  for the relevant zone as needed.)
+
+	</para>
+
+      </section>
+
     </section>
 
     <section>
@@ -1353,7 +1529,7 @@ This may be a temporary setting until then.
 
       <para>
         RFC 1035 style DNS master zone files may imported
-        into a BIND 10 data source by using the
+        into a BIND 10 SQLite3 data source by using the
         <command>b10-loadzone</command> utility.
       </para>
 
@@ -1400,7 +1576,7 @@ This may be a temporary setting until then.
       <note>
       <para>
         In the development prototype release, only the SQLite3 back
-        end is used.
+        end is used by <command>b10-loadzone</command>.
         By default, it stores the zone data in
         <filename>/usr/local/var/bind10-devel/zone.sqlite3</filename>
         unless the <option>-d</option> switch is used to set the
@@ -1629,31 +1805,23 @@ Xfrout/transfer_acl[0]	{"action": "ACCEPT"}	any	(default)</screen>
     </simpara></note>
 
     <para>
-      If you want to require TSIG in access control, a separate TSIG
-      "key ring" must be configured specifically
-      for <command>b10-xfrout</command> as well as a system wide
-      key ring, both containing a consistent set of keys.
+      If you want to require TSIG in access control, a system wide TSIG
+      "key ring" must be configured.
       For example, to change the previous example to allowing requests
       from 192.0.2.1 signed by a TSIG with a key name of
       "key.example", you'll need to do this:
     </para>
 
     <screen>&gt; <userinput>config set tsig_keys/keys ["key.example:&lt;base64-key&gt;"]</userinput>
-&gt; <userinput>config set Xfrout/tsig_keys/keys ["key.example:&lt;base64-key&gt;"]</userinput>
 &gt; <userinput>config set Xfrout/zone_config[0]/transfer_acl [{"action": "ACCEPT", "from": "192.0.2.1", "key": "key.example"}]</userinput>
 &gt; <userinput>config commit</userinput></screen>
 
-    <para>
-      The first line of configuration defines a system wide key ring.
-      This is necessary because the <command>b10-auth</command> server
-      also checks TSIGs and it uses the system wide configuration.
-    </para>
+    <para>Both Xfrout and Auth will use the system wide keyring to check
+    TSIGs in the incoming messages and to sign responses.</para>
 
     <note><simpara>
-        In a future version, <command>b10-xfrout</command> will also
-        use the system wide TSIG configuration.
         The way to specify zone specific configuration (ACLs, etc) is
-        likely to be changed, too.
+        likely to be changed.
     </simpara></note>
 
 <!--
@@ -1683,15 +1851,10 @@ what is XfroutClient xfr_client??
     <para>
       The main <command>bind10</command> process can be configured
       to select to run either the authoritative or resolver or both.
-      By default, it starts the authoritative service.
-<!-- TODO: later both -->
-
-      You may change this using <command>bindctl</command>, for example:
+      By default, it doesn't start either one. You may change this using
+      <command>bindctl</command>, for example:
 
       <screen>
-&gt; <userinput>config remove Boss/components b10-xfrout</userinput>
-&gt; <userinput>config remove Boss/components b10-xfrin</userinput>
-&gt; <userinput>config remove Boss/components b10-auth</userinput>
 &gt; <userinput>config add Boss/components b10-resolver</userinput>
 &gt; <userinput>config set Boss/components/b10-resolver/special resolver</userinput>
 &gt; <userinput>config set Boss/components/b10-resolver/kind needed</userinput>
@@ -2359,7 +2522,7 @@ eth0 fe80::21e:8cff:fe9b:7349
 
           In the Logging module, you can specify the configuration
           for zero or more loggers; any that are not specified will
-          take appropriate default values..
+          take appropriate default values.
 
         </para>
 
@@ -2645,34 +2808,43 @@ TODO; there's a ticket to determine these levels, see #1074
           <varlistentry>
             <term><option>destination</option> is <quote>console</quote></term>
             <listitem>
-              <simpara>
+              <para>
                  The value of output must be one of <quote>stdout</quote>
                  (messages printed to standard output) or
                  <quote>stderr</quote> (messages printed to standard
                  error).
-              </simpara>
+              </para>
+              <para>
+                Note: if output is set to <quote>stderr</quote> and a lot of
+                messages are produced in a short time (e.g. if the logging
+                level is set to DEBUG), you may occasionally see some messages
+                jumbled up together.  This is due to a combination of the way
+                that messages are written to the screen and the unbuffered
+                nature of the standard error stream.  If this occurs, it is
+                recommended that output be set to <quote>stdout</quote>.
+              </para>
             </listitem>
           </varlistentry>
 
           <varlistentry>
             <term><option>destination</option> is <quote>file</quote></term>
             <listitem>
-              <simpara>
+              <para>
                 The value of output is interpreted as a file name;
                 log messages will be appended to this file.
-              </simpara>
+              </para>
             </listitem>
           </varlistentry>
 
           <varlistentry>
             <term><option>destination</option> is <quote>syslog</quote></term>
             <listitem>
-              <simpara>
+              <para>
                 The value of output is interpreted as the
                 <command>syslog</command> facility (e.g.
                 <emphasis>local0</emphasis>) that should be used
                 for log messages.
-              </simpara>
+              </para>
             </listitem>
           </varlistentry>
 
@@ -2853,7 +3025,7 @@ Logging/loggers[0]/output_options[0]/maxver	0	integer	(default)
 
           <screen>&gt; <userinput> config set Logging/loggers[0]/output_options[0]/destination file</userinput>
 &gt; <userinput> config set Logging/loggers[0]/output_options[0]/output /var/log/bind10.log</userinput>
-&gt; <userinput> config set Logging/loggers[0]/output_options[0]/maxsize 30000</userinput>
+&gt; <userinput> config set Logging/loggers[0]/output_options[0]/maxsize 204800</userinput>
 &gt; <userinput> config set Logging/loggers[0]/output_options[0]/maxver 8</userinput>
 </screen>
 
@@ -2876,7 +3048,7 @@ Logging/loggers[0]/additive	false	boolean	(default)
 Logging/loggers[0]/output_options[0]/destination	"file"	string	(modified)
 Logging/loggers[0]/output_options[0]/output	"/var/log/bind10.log"	string	(modified)
 Logging/loggers[0]/output_options[0]/flush	false	boolean	(default)
-Logging/loggers[0]/output_options[0]/maxsize	30000	integer	(modified)
+Logging/loggers[0]/output_options[0]/maxsize	204800	integer	(modified)
 Logging/loggers[0]/output_options[0]/maxver	8	integer	(modified)
 </screen>
 

doc/guide/bind10-messages.xml

@@ -405,6 +405,27 @@ message associated with it has its own code.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="AUTH_RESPONSE_FAILURE">
+<term>AUTH_RESPONSE_FAILURE exception while building response to query: %1</term>
+<listitem><para>
+This is a debug message, generated by the authoritative server when an
+attempt to create a response to a received DNS packet has failed. The
+reason for the failure is given in the log message. A SERVFAIL response
+is sent back. The most likely cause of this is an error in the data
+source implementation; it is either creating bad responses or raising
+exceptions itself.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="AUTH_RESPONSE_FAILURE_UNKNOWN">
+<term>AUTH_RESPONSE_FAILURE_UNKNOWN unknown exception while building response to query</term>
+<listitem><para>
+This debug message is similar to AUTH_RESPONSE_FAILURE, but further
+details about the error are unknown, because it was signaled by something
+which is not an exception. This is definitely a bug.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="AUTH_RESPONSE_RECEIVED">
 <term>AUTH_RESPONSE_RECEIVED received response message, ignoring</term>
 <listitem><para>
@@ -467,6 +488,14 @@ and it is entering the main loop, waiting for queries to arrive.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="AUTH_SHUTDOWN">
+<term>AUTH_SHUTDOWN asked to stop, doing so</term>
+<listitem><para>
+This is a debug message indicating the server was asked to shut down and it is
+complying to the request.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="AUTH_SQLITE3">
 <term>AUTH_SQLITE3 nothing to do for loading sqlite3</term>
 <listitem><para>
@@ -590,7 +619,7 @@ needs a dedicated message bus.
 </varlistentry>
 
 <varlistentry id="BIND10_COMPONENT_FAILED">
-<term>BIND10_COMPONENT_FAILED component %1 (pid %2) failed with %3 exit status</term>
+<term>BIND10_COMPONENT_FAILED component %1 (pid %2) failed: %3</term>
 <listitem><para>
 The process terminated, but the bind10 boss didn't expect it to, which means
 it must have failed.
@@ -1610,6 +1639,15 @@ configuration is not stored.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="CFGMGR_RENAMED_CONFIG_FILE">
+<term>CFGMGR_RENAMED_CONFIG_FILE renamed configuration file %1 to %2, will create new %1</term>
+<listitem><para>
+BIND 10 has been started with the command to clear the configuration file.
+The existing file is backed up to the given file name, so that data is not
+immediately lost if this was done by accident.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="CFGMGR_STOPPED_BY_KEYBOARD">
 <term>CFGMGR_STOPPED_BY_KEYBOARD keyboard interrupt, shutting down</term>
 <listitem><para>
@@ -1766,6 +1804,26 @@ a bug report.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="CONFIG_CCSESSION_STOPPING">
+<term>CONFIG_CCSESSION_STOPPING error sending stopping message: %1</term>
+<listitem><para>
+There was a problem when sending a message signaling that the module using
+this CCSession is stopping. This message is sent so that the rest of the
+system is aware that the module is no longer running. Apart from logging
+this message, the error itself is ignored, and the ModuleCCSession is
+still stopped. The specific exception message is printed.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="CONFIG_CCSESSION_STOPPING_UNKNOWN">
+<term>CONFIG_CCSESSION_STOPPING_UNKNOWN unknown error sending stopping message</term>
+<listitem><para>
+Similar to CONFIG_CCSESSION_STOPPING, but in this case the exception that
+is seen is not a standard exception, and further information is unknown.
+This is a bug.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="CONFIG_GET_FAIL">
 <term>CONFIG_GET_FAIL error getting configuration from cfgmgr: %1</term>
 <listitem><para>
@@ -1873,6 +1931,28 @@ is included in the message.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="CONFIG_SESSION_STOPPING_FAILED">
+<term>CONFIG_SESSION_STOPPING_FAILED error sending stopping message: %1</term>
+<listitem><para>
+There was a problem when sending a message signaling that the module using
+this CCSession is stopping. This message is sent so that the rest of the
+system is aware that the module is no longer running. Apart from logging
+this message, the error itself is ignored, and the ModuleCCSession is
+still stopped. The specific exception message is printed.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DATASRC_BAD_NSEC3_NAME">
+<term>DATASRC_BAD_NSEC3_NAME NSEC3 record has a bad owner name '%1'</term>
+<listitem><para>
+The software refuses to load NSEC3 records into a wildcard domain or
+the owner name has two or more labels below the zone origin.
+It isn't explicitly forbidden, but no sane zone wouldn have such names
+for NSEC3.  BIND 9 also refuses NSEC3 at wildcard, so this behavior is
+compatible with BIND 9.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DATASRC_CACHE_CREATE">
 <term>DATASRC_CACHE_CREATE creating the hotspot cache</term>
 <listitem><para>
@@ -2177,15 +2257,6 @@ its class and the database name are printed.
 </para></listitem>
 </varlistentry>
 
-<varlistentry id="DATASRC_DATABASE_UPDATER_COMMIT (1)">
-<term>DATASRC_DATABASE_UPDATER_COMMIT (1) updates committed for '%1/%2' on %3</term>
-<listitem><para>
-Debug information.  A set of updates to a zone has been successfully
-committed to the corresponding database backend.  The zone name,
-its class and the database name are printed.
-</para></listitem>
-</varlistentry>
-
 <varlistentry id="DATASRC_DATABASE_UPDATER_CREATED">
 <term>DATASRC_DATABASE_UPDATER_CREATED zone updater created for '%1/%2' on %3</term>
 <listitem><para>
@@ -2194,14 +2265,6 @@ the shown zone on the shown backend database.
 </para></listitem>
 </varlistentry>
 
-<varlistentry id="DATASRC_DATABASE_UPDATER_CREATED (1)">
-<term>DATASRC_DATABASE_UPDATER_CREATED (1) zone updater created for '%1/%2' on %3</term>
-<listitem><para>
-Debug information.  A zone updater object is created to make updates to
-the shown zone on the shown backend database.
-</para></listitem>
-</varlistentry>
-
 <varlistentry id="DATASRC_DATABASE_UPDATER_DESTROYED">
 <term>DATASRC_DATABASE_UPDATER_DESTROYED zone updater destroyed for '%1/%2' on %3</term>
 <listitem><para>
@@ -2211,15 +2274,6 @@ database.
 </para></listitem>
 </varlistentry>
 
-<varlistentry id="DATASRC_DATABASE_UPDATER_DESTROYED (1)">
-<term>DATASRC_DATABASE_UPDATER_DESTROYED (1) zone updater destroyed for '%1/%2' on %3</term>
-<listitem><para>
-Debug information.  A zone updater object is destroyed, either successfully
-or after failure of, making updates to the shown zone on the shown backend
-database.
-</para></listitem>
-</varlistentry>
-
 <varlistentry id="DATASRC_DATABASE_UPDATER_ROLLBACK">
 <term>DATASRC_DATABASE_UPDATER_ROLLBACK zone updates roll-backed for '%1/%2' on %3</term>
 <listitem><para>
@@ -2232,18 +2286,6 @@ the underlying database name are shown in the log message.
 </para></listitem>
 </varlistentry>
 
-<varlistentry id="DATASRC_DATABASE_UPDATER_ROLLBACK (1)">
-<term>DATASRC_DATABASE_UPDATER_ROLLBACK (1) zone updates roll-backed for '%1/%2' on %3</term>
-<listitem><para>
-A zone updater is being destroyed without committing the changes.
-This would typically mean the update attempt was aborted due to some
-error, but may also be a bug of the application that forgets committing
-the changes.  The intermediate changes made through the updater won't
-be applied to the underlying database.  The zone name, its class, and
-the underlying database name are shown in the log message.
-</para></listitem>
-</varlistentry>
-
 <varlistentry id="DATASRC_DATABASE_UPDATER_ROLLBACKFAIL">
 <term>DATASRC_DATABASE_UPDATER_ROLLBACKFAIL failed to roll back zone updates for '%1/%2' on %3: %4</term>
 <listitem><para>
@@ -2261,23 +2303,6 @@ database module are shown in the log message.
 </para></listitem>
 </varlistentry>
 
-<varlistentry id="DATASRC_DATABASE_UPDATER_ROLLBACKFAIL (1)">
-<term>DATASRC_DATABASE_UPDATER_ROLLBACKFAIL (1) failed to roll back zone updates for '%1/%2' on %3: %4</term>
-<listitem><para>
-A zone updater is being destroyed without committing the changes to
-the database, and attempts to rollback incomplete updates, but it
-unexpectedly fails.  The higher level implementation does not expect
-it to fail, so this means either a serious operational error in the
-underlying data source (such as a system failure of a database) or
-software bug in the underlying data source implementation.  In either
-case if this message is logged the administrator should carefully
-examine the underlying data source to see what exactly happens and
-whether the data is still valid.  The zone name, its class, and the
-underlying database name as well as the error message thrown from the
-database module are shown in the log message.
-</para></listitem>
-</varlistentry>
-
 <varlistentry id="DATASRC_DATABASE_WILDCARD_ANY">
 <term>DATASRC_DATABASE_WILDCARD_ANY search in datasource %1 resulted in wildcard match type ANY on %2</term>
 <listitem><para>
@@ -2497,6 +2522,46 @@ Debug information. A search for the requested RRset is being started.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="DATASRC_MEM_FINDNSEC3">
+<term>DATASRC_MEM_FINDNSEC3 finding NSEC3 for %1, mode %2</term>
+<listitem><para>
+Debug information. A search in an in-memory data source for NSEC3 that
+matches or covers the given name is being started.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DATASRC_MEM_FINDNSEC3_COVER">
+<term>DATASRC_MEM_FINDNSEC3_COVER found a covering NSEC3 for %1: %2</term>
+<listitem><para>
+Debug information. An NSEC3 that covers the given name is found and
+being returned.  The found NSEC3 RRset is also displayed.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DATASRC_MEM_FINDNSEC3_MATCH">
+<term>DATASRC_MEM_FINDNSEC3_MATCH found a matching NSEC3 for %1 at label count %2: %3</term>
+<listitem><para>
+Debug information. An NSEC3 that matches (a possibly superdomain of)
+the given name is found and being returned.  When the shown label
+count is smaller than that of the given name, the matching NSEC3 is
+for a superdomain of the given name (see DATASRC_MEM_FINDNSEC3_TRYHASH).
+The found NSEC3 RRset is also displayed.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DATASRC_MEM_FINDNSEC3_TRYHASH">
+<term>DATASRC_MEM_FINDNSEC3_TRYHASH looking for NSEC3 for %1 at label count %2 (hash %3)</term>
+<listitem><para>
+Debug information. In an attempt of finding an NSEC3 for the give name,
+(a possibly superdomain of) the name is hashed and searched for in the
+NSEC3 name space.  When the shown label count is smaller than that of the
+shown name, the search tries the superdomain name that share the shown
+(higher) label count of the shown name (e.g., for
+www.example.com. with shown label count of 3, example.com. is being
+tried).
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DATASRC_MEM_FIND_ZONE">
 <term>DATASRC_MEM_FIND_ZONE looking for zone '%1'</term>
 <listitem><para>
@@ -2519,6 +2584,18 @@ Debug information. The requested domain does not exist.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="DATASRC_MEM_NO_NSEC3PARAM">
+<term>DATASRC_MEM_NO_NSEC3PARAM NSEC3PARAM is missing for NSEC3-signed zone %1/%2</term>
+<listitem><para>
+The in-memory data source has loaded a zone signed with NSEC3 RRs,
+but it doesn't have a NSEC3PARAM RR at the zone origin.  It's likely that
+the zone is somehow broken, but this RR is not necessarily needed for
+handling lookups with NSEC3 in this data source, so it accepts the given
+content of the zone.  Nevertheless the administrator should look into
+the integrity of the zone data.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DATASRC_MEM_NS_ENCOUNTERED">
 <term>DATASRC_MEM_NS_ENCOUNTERED encountered a NS</term>
 <listitem><para>
@@ -2570,11 +2647,13 @@ Debug information. The requested record was found.
 </varlistentry>
 
 <varlistentry id="DATASRC_MEM_SUPER_STOP">
-<term>DATASRC_MEM_SUPER_STOP stopped at superdomain '%1', domain '%2' is empty</term>
+<term>DATASRC_MEM_SUPER_STOP stopped as '%1' is superdomain of a zone node, meaning it's empty</term>
 <listitem><para>
-Debug information. The search stopped at a superdomain of the requested
-domain. The domain is an empty nonterminal, therefore it is treated  as NXRRSET
-case (eg. the domain exists, but it doesn't have the requested record type).
+Debug information. The search stopped because the requested domain was
+detected to be a superdomain of some existing node of zone (while there
+was no exact match).  This means that the domain is an empty nonterminal,
+therefore it is treated  as NXRRSET case (eg. the domain exists, but it
+doesn't have the requested record type).
 </para></listitem>
 </varlistentry>
 
@@ -2894,8 +2973,10 @@ not have any DS record. This indicates problem with the provided data.
 <varlistentry id="DATASRC_QUERY_NO_ZONE">
 <term>DATASRC_QUERY_NO_ZONE no zone containing '%1' in class '%2'</term>
 <listitem><para>
-Lookup of domain failed because the data have no zone that contain the
-domain. Maybe someone sent a query to the wrong server for some reason.
+Debug information. Lookup of domain failed because the datasource
+has no zone that contains the domain. Maybe someone sent a query
+to the wrong server for some reason. This may also happen when
+looking in the datasource for addresses for NS records.
 </para></listitem>
 </varlistentry>
 
@@ -2925,7 +3006,7 @@ the specific error already.
 </varlistentry>
 
 <varlistentry id="DATASRC_QUERY_RRSIG">
-<term>DATASRC_QUERY_RRSIG unable to answer RRSIG query</term>
+<term>DATASRC_QUERY_RRSIG unable to answer RRSIG query for %1</term>
 <listitem><para>
 The server is unable to answer a direct query for RRSIG type, but was asked
 to do so.
@@ -3232,6 +3313,210 @@ generated.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="DBUTIL_BACKUP">
+<term>DBUTIL_BACKUP created backup of %1 in %2</term>
+<listitem><para>
+A backup for the given database file was created. Same of original file and
+backup are given in the output message.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_CHECK_ERROR">
+<term>DBUTIL_CHECK_ERROR unable to check database version: %1</term>
+<listitem><para>
+There was an error while trying to check the current version of the database
+schema. The error is shown in the message.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_CHECK_NOCONFIRM">
+<term>DBUTIL_CHECK_NOCONFIRM --noconfirm is not compatible with --check</term>
+<listitem><para>
+b10-dbutil was called with --check and --noconfirm. --noconfirm only has
+meaning with --upgrade, so this is considered an error.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_CHECK_OK">
+<term>DBUTIL_CHECK_OK this is the latest version of the database schema. No upgrade is required</term>
+<listitem><para>
+The database schema version has been checked, and is up to date.
+No action is required.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_CHECK_UPGRADE_NEEDED">
+<term>DBUTIL_CHECK_UPGRADE_NEEDED re-run this program with the --upgrade switch to upgrade</term>
+<listitem><para>
+The database schema version is not up to date, and an update is required.
+Please run the dbutil tool again, with the --upgrade argument.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_COMMAND_NONE">
+<term>DBUTIL_COMMAND_NONE must select one of --check or --upgrade</term>
+<listitem><para>
+b10-dbutil was called with neither --check nor --upgrade. One action must be
+provided.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_COMMAND_UPGRADE_CHECK">
+<term>DBUTIL_COMMAND_UPGRADE_CHECK --upgrade is not compatible with --check</term>
+<listitem><para>
+b10-dbutil was called with both the commands --upgrade and --check. Only one
+action can be performed at a time.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_DATABASE_MAY_BE_CORRUPT">
+<term>DBUTIL_DATABASE_MAY_BE_CORRUPT database file %1 may be corrupt, restore it from backup (%2)</term>
+<listitem><para>
+The upgrade failed while it was in progress; the database may now be in an
+inconsistent state, and it is advised to restore it from the backup that was
+created when b10-dbutil started.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_EXECUTE">
+<term>DBUTIL_EXECUTE Executing SQL statement: %1</term>
+<listitem><para>
+Debug message; the given SQL statement is executed
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_FILE">
+<term>DBUTIL_FILE Database file: %1</term>
+<listitem><para>
+The database file that is being checked.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_NO_FILE">
+<term>DBUTIL_NO_FILE must supply name of the database file to upgrade</term>
+<listitem><para>
+b10-dbutil was called without a database file. Currently, it cannot find this
+file on its own, and it must be provided.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_STATEMENT_ERROR">
+<term>DBUTIL_STATEMENT_ERROR failed to execute %1: %2</term>
+<listitem><para>
+The given database statement failed to execute. The error is shown in the
+message.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_TOO_MANY_ARGUMENTS">
+<term>DBUTIL_TOO_MANY_ARGUMENTS too many arguments to the command, maximum of one expected</term>
+<listitem><para>
+There were too many command-line arguments to b10-dbutil
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_CANCELED">
+<term>DBUTIL_UPGRADE_CANCELED upgrade canceled; database has not been changed</term>
+<listitem><para>
+The user aborted the upgrade, and b10-dbutil will now exit.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_DBUTIL">
+<term>DBUTIL_UPGRADE_DBUTIL please get the latest version of b10-dbutil and re-run</term>
+<listitem><para>
+A database schema was found that was newer than this version of dbutil, which
+is apparently out of date and should be upgraded itself.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_FAILED">
+<term>DBUTIL_UPGRADE_FAILED upgrade failed: %1</term>
+<listitem><para>
+While the upgrade was in progress, an unexpected error occurred. The error
+is shown in the message.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_NOT_ATTEMPTED">
+<term>DBUTIL_UPGRADE_NOT_ATTEMPTED database upgrade was not attempted</term>
+<listitem><para>
+Due to the earlier failure, the database schema upgrade was not attempted,
+and b10-dbutil will now exit.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_NOT_NEEDED">
+<term>DBUTIL_UPGRADE_NOT_NEEDED database already at latest version, no upgrade necessary</term>
+<listitem><para>
+b10-dbutil was told to upgrade the database schema, but it is already at the
+latest version.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_NOT_POSSIBLE">
+<term>DBUTIL_UPGRADE_NOT_POSSIBLE database at a later version than this utility can support</term>
+<listitem><para>
+b10-dbutil was told to upgrade the database schema, but it is at a higher
+version than this tool currently supports. Please update b10-dbutil and try
+again.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_PREPARATION_FAILED">
+<term>DBUTIL_UPGRADE_PREPARATION_FAILED upgrade preparation failed: %1</term>
+<listitem><para>
+An unexpected error occurred while b10-dbutil was preparing to upgrade the
+database schema. The error is shown in the message
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADE_SUCCESFUL">
+<term>DBUTIL_UPGRADE_SUCCESFUL database upgrade successfully completed</term>
+<listitem><para>
+The database schema update was completed successfully.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_UPGRADING">
+<term>DBUTIL_UPGRADING upgrading database from %1 to %2</term>
+<listitem><para>
+An upgrade is in progress, the versions of the current upgrade action are shown.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_VERSION_CURRENT">
+<term>DBUTIL_VERSION_CURRENT database version %1</term>
+<listitem><para>
+The current version of the database schema.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_VERSION_HIGH">
+<term>DBUTIL_VERSION_HIGH database is at a later version (%1) than this program can cope with (%2)</term>
+<listitem><para>
+The database schema is at a higher version than b10-dbutil knows about.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DBUTIL_VERSION_LOW">
+<term>DBUTIL_VERSION_LOW database version %1, latest version is %2.</term>
+<listitem><para>
+The database schema is not up to date, the current version and the latest
+version are in the message.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="DDNS_ACCEPT_FAILURE">
+<term>DDNS_ACCEPT_FAILURE error accepting a connection: %1</term>
+<listitem><para>
+There was a low-level error when we tried to accept an incoming connection
+(probably coming from b10-auth). We continue serving on whatever other
+connections we already have, but this connection is dropped. The reason
+is logged.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DDNS_CC_SESSION_ERROR">
 <term>DDNS_CC_SESSION_ERROR error reading from cc channel: %1</term>
 <listitem><para>
@@ -3257,6 +3542,17 @@ startup time.  Details of the error are included in the log message.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="DDNS_DROP_CONN">
+<term>DDNS_DROP_CONN dropping connection on file descriptor %1 because of error %2</term>
+<listitem><para>
+There was an error on a connection with the b10-auth server (or whatever
+connects to the ddns daemon). This might be OK, for example when the
+authoritative server shuts down, the connection would get closed. It also
+can mean the system is busy and can't keep up or that the other side got
+confused and sent bad data.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DDNS_MODULECC_SESSION_ERROR">
 <term>DDNS_MODULECC_SESSION_ERROR error encountered by configuration/command module: %1</term>
 <listitem><para>
@@ -3268,6 +3564,16 @@ will also be displayed.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="DDNS_NEW_CONN">
+<term>DDNS_NEW_CONN new connection on file descriptor %1 from %2</term>
+<listitem><para>
+Debug message. We received a connection and we are going to start handling
+requests from it. The file descriptor number and the address where the request
+comes from is logged. The connection is over a unix domain socket and is likely
+coming from a b10-auth process.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DDNS_RECEIVED_SHUTDOWN_COMMAND">
 <term>DDNS_RECEIVED_SHUTDOWN_COMMAND shutdown command received</term>
 <listitem><para>
@@ -3284,6 +3590,15 @@ and updates.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="DDNS_SESSION">
+<term>DDNS_SESSION session arrived on file descriptor %1</term>
+<listitem><para>
+A debug message, informing there's some activity on the given file descriptor.
+It will be either a request or the file descriptor will be closed. See
+following log messages to see what of it.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="DDNS_SHUTDOWN">
 <term>DDNS_SHUTDOWN ddns server shutting down</term>
 <listitem><para>
@@ -3826,6 +4141,32 @@ bug report.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="PYSERVER_COMMON_TSIG_KEYRING_DEINIT">
+<term>PYSERVER_COMMON_TSIG_KEYRING_DEINIT Deinitializing global TSIG keyring</term>
+<listitem><para>
+A debug message noting that the global TSIG keyring is being removed from
+memory. Most programs don't do that, they just exit, which is OK.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="PYSERVER_COMMON_TSIG_KEYRING_INIT">
+<term>PYSERVER_COMMON_TSIG_KEYRING_INIT Initializing global TSIG keyring</term>
+<listitem><para>
+A debug message noting the TSIG keyring storage is being prepared. It should
+appear at most once in the lifetime of a program. The keyring still needs
+to be loaded from configuration.
+</para></listitem>
+</varlistentry>
+
+<varlistentry id="PYSERVER_COMMON_TSIG_KEYRING_UPDATE">
+<term>PYSERVER_COMMON_TSIG_KEYRING_UPDATE Updating global TSIG keyring</term>
+<listitem><para>
+A debug message. The TSIG keyring is being (re)loaded from configuration.
+This happens at startup or when the configuration changes. The old keyring
+is removed and new one created with all the keys.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="RESLIB_ANSWER">
 <term>RESLIB_ANSWER answer received in response to query for &lt;%1&gt;</term>
 <listitem><para>
@@ -4571,6 +4912,14 @@ This informational message is output when the resolver has shut down.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="RESOLVER_SHUTDOWN_RECEIVED">
+<term>RESOLVER_SHUTDOWN_RECEIVED received command to shut down</term>
+<listitem><para>
+A debug message noting that the server was asked to terminate and is
+complying to the request.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="RESOLVER_STARTED">
 <term>RESOLVER_STARTED resolver started</term>
 <listitem><para>
@@ -4605,7 +4954,7 @@ a message to the sender with the RCODE set to NOTIMP.
 </varlistentry>
 
 <varlistentry id="SOCKETREQUESTOR_CREATED">
-<term>SOCKETREQUESTOR_CREATED Socket requestor created</term>
+<term>SOCKETREQUESTOR_CREATED Socket requestor created for application %1</term>
 <listitem><para>
 Debug message.  A socket requesor (client of the socket creator) is created
 for the corresponding application.  Normally this should happen at most
@@ -4706,6 +5055,21 @@ be hidden, as it has higher debug level.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="SRVCOMM_EXCEPTION_ALLOC">
+<term>SRVCOMM_EXCEPTION_ALLOC exception when allocating a socket: %1</term>
+<listitem><para>
+The process tried to allocate a socket using the socket creator, but an error
+occurred. But it is not one of the errors we are sure are "safe". In this case
+it is unclear if the unsuccessful communication left the process and the bind10
+process in inconsistent state, so the process is going to abort to prevent
+further problems in that area.
+</para><para>
+This is probably a bug in the code, but it could be caused by other unusual
+conditions (like insufficient memory, deleted socket file used for
+communication).
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="SRVCOMM_KEYS_DEINIT">
 <term>SRVCOMM_KEYS_DEINIT deinitializing TSIG keyring</term>
 <listitem><para>
@@ -4745,6 +5109,15 @@ different set of IP addresses and ports than before.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="SRVCOMM_UNKNOWN_EXCEPTION_ALLOC">
+<term>SRVCOMM_UNKNOWN_EXCEPTION_ALLOC unknown exception when allocating a socket</term>
+<listitem><para>
+The situation is the same as in the SRVCOMM_EXCEPTION_ALLOC case, but further
+details about the error are unknown, because it was signaled by throwing
+something not being an exception. This is definitely a bug.
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="STATHTTPD_BAD_OPTION_VALUE">
 <term>STATHTTPD_BAD_OPTION_VALUE bad command line argument: %1</term>
 <listitem><para>
@@ -5117,6 +5490,35 @@ likely cause is a PYTHONPATH problem.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="XFRIN_IXFR_TRANSFER_SUCCESS">
+<term>XFRIN_IXFR_TRANSFER_SUCCESS incremental IXFR transfer of zone %1 succeeded (messages: %2, changesets: %3, deletions: %4, additions: %5, bytes: %6, run time: %7 seconds, %8 bytes/second)</term>
+<listitem><para>
+The IXFR transfer for the given zone was successful.
+The provided information contains the following values:
+</para><para>
+messages: Number of overhead DNS messages in the transfer.
+</para><para>
+changesets: Number of difference sequences.
+</para><para>
+deletions: Number of Resource Records deleted by all the changesets combined,
+including the SOA records.
+</para><para>
+additions: Number of Resource Records added by all the changesets combined,
+including the SOA records.
+</para><para>
+bytes: Full size of the transfer data on the wire.
+</para><para>
+run time: Time (in seconds) the complete ixfr took.
+</para><para>
+bytes/second: Transfer speed.
+</para><para>
+Note that there is no cross-checking of additions and deletions; if the same
+RR gets added and deleted in multiple changesets, it is counted each time;
+therefore, for each changeset, there should at least be 1 deletion and 1
+addition (the updated SOA record).
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="XFRIN_IXFR_UPTODATE">
 <term>XFRIN_IXFR_UPTODATE IXFR requested serial for %1 is %2, master has %3, not updating</term>
 <listitem><para>
@@ -5183,6 +5585,25 @@ daemon will now shut down.
 </para></listitem>
 </varlistentry>
 
+<varlistentry id="XFRIN_TRANSFER_SUCCESS">
+<term>XFRIN_TRANSFER_SUCCESS full %1 transfer of zone %2 succeeded (messages: %3, records: %4, bytes: %5, run time: %6 seconds, %7 bytes/second)</term>
+<listitem><para>
+The AXFR transfer of the given zone was successful.
+The provided information contains the following values:
+</para><para>
+messages: Number of overhead DNS messages in the transfer
+</para><para>
+records: Number of Resource Records in the full transfer, excluding the
+final SOA record that marks the end of the AXFR.
+</para><para>
+bytes: Full size of the transfer data on the wire.
+</para><para>
+run time: Time (in seconds) the complete axfr took
+</para><para>
+bytes/second: Transfer speed
+</para></listitem>
+</varlistentry>
+
 <varlistentry id="XFRIN_UNKNOWN_ERROR">
 <term>XFRIN_UNKNOWN_ERROR unknown error: %1</term>
 <listitem><para>
@@ -5259,13 +5680,6 @@ the SOA record has been checked, and a zone transfer has been started.
 </para></listitem>
 </varlistentry>
 
-<varlistentry id="XFRIN_XFR_TRANSFER_SUCCESS">
-<term>XFRIN_XFR_TRANSFER_SUCCESS %1 transfer of zone %2 succeeded</term>
-<listitem><para>
-The XFR transfer of the given zone was successfully completed.
-</para></listitem>
-</varlistentry>
-
 <varlistentry id="XFRIN_ZONE_CREATED">
 <term>XFRIN_ZONE_CREATED Zone %1 not found in the given data source, newly created</term>
 <listitem><para>
@@ -5865,9 +6279,11 @@ a bug report.
 <term>ZONEMGR_UNKNOWN_ZONE_FAIL zone %1 (class %2) is not known to the zone manager</term>
 <listitem><para>
 An XFRIN operation has failed but the zone that was the subject of the
-operation is not being managed by the zone manager.  This may indicate
-an error in the program (as the operation should not have been initiated
-if this were the case).  Please submit a bug report.
+operation is not being managed by the zone manager. This can be either the
+result of a bindctl command to transfer in a currently unknown (or mistyped)
+zone, or, if this error appears without the administrator giving transfer
+commands, it can indicate an error in the program, as it should not have
+initiated transfers of unknown zones on its own.
 </para></listitem>
 </varlistentry>
 

ext/LICENSE_1_0.txt

@@ -0,0 +1,23 @@
+Boost Software License - Version 1.0 - August 17th, 2003
+
+Permission is hereby granted, free of charge, to any person or organization
+obtaining a copy of the software and accompanying documentation covered by
+this license (the "Software") to use, reproduce, display, distribute,
+execute, and transmit the Software, and to prepare derivative works of the
+Software, and to permit third-parties to whom the Software is furnished to
+do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including
+the above license grant, this restriction and the following disclaimer,
+must be included in all copies of the Software, in whole or in part, and
+all derivative works of the Software, unless such copies or derivative
+works are solely in the form of machine-executable object code generated by
+a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

ext/asio/asio/detail/impl/kqueue_reactor.ipp

@@ -301,12 +301,14 @@ void kqueue_reactor::run(bool block, op_queue<operation>& ops)
               EV_ADD | EV_ONESHOT, EV_OOBAND, 0, descriptor_data);
         else
           continue;
+        break;
       case EVFILT_WRITE:
         if (!descriptor_data->op_queue_[write_op].empty())
           ASIO_KQUEUE_EV_SET(&event, descriptor, EVFILT_WRITE,
               EV_ADD | EV_ONESHOT, 0, 0, descriptor_data);
         else
           continue;
+        break;
       default:
         break;
       }

ext/asio/asio/detail/impl/socket_ops.ipp

@@ -282,15 +282,26 @@ int close(socket_type s, state_type& state,
   int result = 0;
   if (s != invalid_socket)
   {
-#if defined(BOOST_WINDOWS) || defined(__CYGWIN__)
-    if ((state & non_blocking) && (state & user_set_linger))
+    if (destruction && (state & user_set_linger))
     {
-      ioctl_arg_type arg = 0;
-      ::ioctlsocket(s, FIONBIO, &arg);
-      state &= ~non_blocking;
+      ::linger opt;
+      opt.l_onoff = 0;
+      opt.l_linger = 0;
+      asio::error_code ignored_ec;
+      socket_ops::setsockopt(s, state, SOL_SOCKET,
+          SO_LINGER, &opt, sizeof(opt), ignored_ec);
     }
+
+    clear_last_error();
+#if defined(BOOST_WINDOWS) || defined(__CYGWIN__)
+    result = error_wrapper(::closesocket(s), ec);
 #else // defined(BOOST_WINDOWS) || defined(__CYGWIN__)
-    if (state & non_blocking)
+    result = error_wrapper(::close(s), ec);
+#endif // defined(BOOST_WINDOWS) || defined(__CYGWIN__)
+
+    if (result != 0
+        && (ec == asio::error::would_block
+          || ec == asio::error::try_again))
     {
 #if defined(__SYMBIAN32__)
       int flags = ::fcntl(s, F_GETFL, 0);
@@ -301,18 +312,6 @@ int close(socket_type s, state_type& state,
       ::ioctl(s, FIONBIO, &arg);
 #endif // defined(__SYMBIAN32__)
       state &= ~non_blocking;
-    }
-#endif // defined(BOOST_WINDOWS) || defined(__CYGWIN__)
-
-    if (destruction && (state & user_set_linger))
-    {
-      ::linger opt;
-      opt.l_onoff = 0;
-      opt.l_linger = 0;
-      asio::error_code ignored_ec;
-      socket_ops::setsockopt(s, state, SOL_SOCKET,
-          SO_LINGER, &opt, sizeof(opt), ignored_ec);
-    }
 
     clear_last_error();
 #if defined(BOOST_WINDOWS) || defined(__CYGWIN__)
@@ -320,6 +319,7 @@ int close(socket_type s, state_type& state,
 #else // defined(BOOST_WINDOWS) || defined(__CYGWIN__)
     result = error_wrapper(::close(s), ec);
 #endif // defined(BOOST_WINDOWS) || defined(__CYGWIN__)
+    }
   }
 
   if (result == 0)

install-sh

@@ -1,520 +0,0 @@
-#!/bin/sh
-# install - install a program, script, or datafile
-
-scriptversion=2009-04-28.21; # UTC
-
-# This originates from X11R5 (mit/util/scripts/install.sh), which was
-# later released in X11R6 (xc/config/util/install.sh) with the
-# following copyright and license.
-#
-# Copyright (C) 1994 X Consortium
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to
-# deal in the Software without restriction, including without limitation the
-# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-# sell copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
-# X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
-# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC-
-# TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-#
-# Except as contained in this notice, the name of the X Consortium shall not
-# be used in advertising or otherwise to promote the sale, use or other deal-
-# ings in this Software without prior written authorization from the X Consor-
-# tium.
-#
-#
-# FSF changes to this file are in the public domain.
-#
-# Calling this script install-sh is preferred over install.sh, to prevent
-# `make' implicit rules from creating a file called install from it
-# when there is no Makefile.
-#
-# This script is compatible with the BSD install script, but was written
-# from scratch.
-
-nl='
-'
-IFS=" ""	$nl"
-
-# set DOITPROG to echo to test this script
-
-# Don't use :- since 4.3BSD and earlier shells don't like it.
-doit=${DOITPROG-}
-if test -z "$doit"; then
-  doit_exec=exec
-else
-  doit_exec=$doit
-fi
-
-# Put in absolute file names if you don't have them in your path;
-# or use environment vars.
-
-chgrpprog=${CHGRPPROG-chgrp}
-chmodprog=${CHMODPROG-chmod}
-chownprog=${CHOWNPROG-chown}
-cmpprog=${CMPPROG-cmp}
-cpprog=${CPPROG-cp}
-mkdirprog=${MKDIRPROG-mkdir}
-mvprog=${MVPROG-mv}
-rmprog=${RMPROG-rm}
-stripprog=${STRIPPROG-strip}
-
-posix_glob='?'
-initialize_posix_glob='
-  test "$posix_glob" != "?" || {
-    if (set -f) 2>/dev/null; then
-      posix_glob=
-    else
-      posix_glob=:
-    fi
-  }
-'
-
-posix_mkdir=
-
-# Desired mode of installed file.
-mode=0755
-
-chgrpcmd=
-chmodcmd=$chmodprog
-chowncmd=
-mvcmd=$mvprog
-rmcmd="$rmprog -f"
-stripcmd=
-
-src=
-dst=
-dir_arg=
-dst_arg=
-
-copy_on_change=false
-no_target_directory=
-
-usage="\
-Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE
-   or: $0 [OPTION]... SRCFILES... DIRECTORY
-   or: $0 [OPTION]... -t DIRECTORY SRCFILES...
-   or: $0 [OPTION]... -d DIRECTORIES...
-
-In the 1st form, copy SRCFILE to DSTFILE.
-In the 2nd and 3rd, copy all SRCFILES to DIRECTORY.
-In the 4th, create DIRECTORIES.
-
-Options:
-     --help     display this help and exit.
-     --version  display version info and exit.
-
-  -c            (ignored)
-  -C            install only if different (preserve the last data modification time)
-  -d            create directories instead of installing files.
-  -g GROUP      $chgrpprog installed files to GROUP.
-  -m MODE       $chmodprog installed files to MODE.
-  -o USER       $chownprog installed files to USER.
-  -s            $stripprog installed files.
-  -t DIRECTORY  install into DIRECTORY.
-  -T            report an error if DSTFILE is a directory.
-
-Environment variables override the default commands:
-  CHGRPPROG CHMODPROG CHOWNPROG CMPPROG CPPROG MKDIRPROG MVPROG
-  RMPROG STRIPPROG
-"
-
-while test $# -ne 0; do
-  case $1 in
-    -c) ;;
-
-    -C) copy_on_change=true;;
-
-    -d) dir_arg=true;;
-
-    -g) chgrpcmd="$chgrpprog $2"
-	shift;;
-
-    --help) echo "$usage"; exit $?;;
-
-    -m) mode=$2
-	case $mode in
-	  *' '* | *'	'* | *'
-'*	  | *'*'* | *'?'* | *'['*)
-	    echo "$0: invalid mode: $mode" >&2
-	    exit 1;;
-	esac
-	shift;;
-
-    -o) chowncmd="$chownprog $2"
-	shift;;
-
-    -s) stripcmd=$stripprog;;
-
-    -t) dst_arg=$2
-	shift;;
-
-    -T) no_target_directory=true;;
-
-    --version) echo "$0 $scriptversion"; exit $?;;
-
-    --)	shift
-	break;;
-
-    -*)	echo "$0: invalid option: $1" >&2
-	exit 1;;
-
-    *)  break;;
-  esac
-  shift
-done
-
-if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then
-  # When -d is used, all remaining arguments are directories to create.
-  # When -t is used, the destination is already specified.
-  # Otherwise, the last argument is the destination.  Remove it from $@.
-  for arg
-  do
-    if test -n "$dst_arg"; then
-      # $@ is not empty: it contains at least $arg.
-      set fnord "$@" "$dst_arg"
-      shift # fnord
-    fi
-    shift # arg
-    dst_arg=$arg
-  done
-fi
-
-if test $# -eq 0; then
-  if test -z "$dir_arg"; then
-    echo "$0: no input file specified." >&2
-    exit 1
-  fi
-  # It's OK to call `install-sh -d' without argument.
-  # This can happen when creating conditional directories.
-  exit 0
-fi
-
-if test -z "$dir_arg"; then
-  trap '(exit $?); exit' 1 2 13 15
-
-  # Set umask so as not to create temps with too-generous modes.
-  # However, 'strip' requires both read and write access to temps.
-  case $mode in
-    # Optimize common cases.
-    *644) cp_umask=133;;
-    *755) cp_umask=22;;
-
-    *[0-7])
-      if test -z "$stripcmd"; then
-	u_plus_rw=
-      else
-	u_plus_rw='% 200'
-      fi
-      cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;;
-    *)
-      if test -z "$stripcmd"; then
-	u_plus_rw=
-      else
-	u_plus_rw=,u+rw
-      fi
-      cp_umask=$mode$u_plus_rw;;
-  esac
-fi
-
-for src
-do
-  # Protect names starting with `-'.
-  case $src in
-    -*) src=./$src;;
-  esac
-
-  if test -n "$dir_arg"; then
-    dst=$src
-    dstdir=$dst
-    test -d "$dstdir"
-    dstdir_status=$?
-  else
-
-    # Waiting for this to be detected by the "$cpprog $src $dsttmp" command
-    # might cause directories to be created, which would be especially bad
-    # if $src (and thus $dsttmp) contains '*'.
-    if test ! -f "$src" && test ! -d "$src"; then
-      echo "$0: $src does not exist." >&2
-      exit 1
-    fi
-
-    if test -z "$dst_arg"; then
-      echo "$0: no destination specified." >&2
-      exit 1
-    fi
-
-    dst=$dst_arg
-    # Protect names starting with `-'.
-    case $dst in
-      -*) dst=./$dst;;
-    esac
-
-    # If destination is a directory, append the input filename; won't work
-    # if double slashes aren't ignored.
-    if test -d "$dst"; then
-      if test -n "$no_target_directory"; then
-	echo "$0: $dst_arg: Is a directory" >&2
-	exit 1
-      fi
-      dstdir=$dst
-      dst=$dstdir/`basename "$src"`
-      dstdir_status=0
-    else
-      # Prefer dirname, but fall back on a substitute if dirname fails.
-      dstdir=`
-	(dirname "$dst") 2>/dev/null ||
-	expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
-	     X"$dst" : 'X\(//\)[^/]' \| \
-	     X"$dst" : 'X\(//\)$' \| \
-	     X"$dst" : 'X\(/\)' \| . 2>/dev/null ||
-	echo X"$dst" |
-	    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
-		   s//\1/
-		   q
-		 }
-		 /^X\(\/\/\)[^/].*/{
-		   s//\1/
-		   q
-		 }
-		 /^X\(\/\/\)$/{
-		   s//\1/
-		   q
-		 }
-		 /^X\(\/\).*/{
-		   s//\1/
-		   q
-		 }
-		 s/.*/./; q'
-      `
-
-      test -d "$dstdir"
-      dstdir_status=$?
-    fi
-  fi
-
-  obsolete_mkdir_used=false
-
-  if test $dstdir_status != 0; then
-    case $posix_mkdir in
-      '')
-	# Create intermediate dirs using mode 755 as modified by the umask.
-	# This is like FreeBSD 'install' as of 1997-10-28.
-	umask=`umask`
-	case $stripcmd.$umask in
-	  # Optimize common cases.
-	  *[2367][2367]) mkdir_umask=$umask;;
-	  .*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;;
-
-	  *[0-7])
-	    mkdir_umask=`expr $umask + 22 \
-	      - $umask % 100 % 40 + $umask % 20 \
-	      - $umask % 10 % 4 + $umask % 2
-	    `;;
-	  *) mkdir_umask=$umask,go-w;;
-	esac
-
-	# With -d, create the new directory with the user-specified mode.
-	# Otherwise, rely on $mkdir_umask.
-	if test -n "$dir_arg"; then
-	  mkdir_mode=-m$mode
-	else
-	  mkdir_mode=
-	fi
-
-	posix_mkdir=false
-	case $umask in
-	  *[123567][0-7][0-7])
-	    # POSIX mkdir -p sets u+wx bits regardless of umask, which
-	    # is incompatible with FreeBSD 'install' when (umask & 300) != 0.
-	    ;;
-	  *)
-	    tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
-	    trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0
-
-	    if (umask $mkdir_umask &&
-		exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1
-	    then
-	      if test -z "$dir_arg" || {
-		   # Check for POSIX incompatibilities with -m.
-		   # HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
-		   # other-writeable bit of parent directory when it shouldn't.
-		   # FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
-		   ls_ld_tmpdir=`ls -ld "$tmpdir"`
-		   case $ls_ld_tmpdir in
-		     d????-?r-*) different_mode=700;;
-		     d????-?--*) different_mode=755;;
-		     *) false;;
-		   esac &&
-		   $mkdirprog -m$different_mode -p -- "$tmpdir" && {
-		     ls_ld_tmpdir_1=`ls -ld "$tmpdir"`
-		     test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
-		   }
-		 }
-	      then posix_mkdir=:
-	      fi
-	      rmdir "$tmpdir/d" "$tmpdir"
-	    else
-	      # Remove any dirs left behind by ancient mkdir implementations.
-	      rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null
-	    fi
-	    trap '' 0;;
-	esac;;
-    esac
-
-    if
-      $posix_mkdir && (
-	umask $mkdir_umask &&
-	$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir"
-      )
-    then :
-    else
-
-      # The umask is ridiculous, or mkdir does not conform to POSIX,
-      # or it failed possibly due to a race condition.  Create the
-      # directory the slow way, step by step, checking for races as we go.
-
-      case $dstdir in
-	/*) prefix='/';;
-	-*) prefix='./';;
-	*)  prefix='';;
-      esac
-
-      eval "$initialize_posix_glob"
-
-      oIFS=$IFS
-      IFS=/
-      $posix_glob set -f
-      set fnord $dstdir
-      shift
-      $posix_glob set +f
-      IFS=$oIFS
-
-      prefixes=
-
-      for d
-      do
-	test -z "$d" && continue
-
-	prefix=$prefix$d
-	if test -d "$prefix"; then
-	  prefixes=
-	else
-	  if $posix_mkdir; then
-	    (umask=$mkdir_umask &&
-	     $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break
-	    # Don't fail if two instances are running concurrently.
-	    test -d "$prefix" || exit 1
-	  else
-	    case $prefix in
-	      *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
-	      *) qprefix=$prefix;;
-	    esac
-	    prefixes="$prefixes '$qprefix'"
-	  fi
-	fi
-	prefix=$prefix/
-      done
-
-      if test -n "$prefixes"; then
-	# Don't fail if two instances are running concurrently.
-	(umask $mkdir_umask &&
-	 eval "\$doit_exec \$mkdirprog $prefixes") ||
-	  test -d "$dstdir" || exit 1
-	obsolete_mkdir_used=true
-      fi
-    fi
-  fi
-
-  if test -n "$dir_arg"; then
-    { test -z "$chowncmd" || $doit $chowncmd "$dst"; } &&
-    { test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } &&
-    { test "$obsolete_mkdir_used$chowncmd$chgrpcmd" = false ||
-      test -z "$chmodcmd" || $doit $chmodcmd $mode "$dst"; } || exit 1
-  else
-
-    # Make a couple of temp file names in the proper directory.
-    dsttmp=$dstdir/_inst.$$_
-    rmtmp=$dstdir/_rm.$$_
-
-    # Trap to clean up those temp files at exit.
-    trap 'ret=$?; rm -f "$dsttmp" "$rmtmp" && exit $ret' 0
-
-    # Copy the file name to the temp name.
-    (umask $cp_umask && $doit_exec $cpprog "$src" "$dsttmp") &&
-
-    # and set any options; do chmod last to preserve setuid bits.
-    #
-    # If any of these fail, we abort the whole thing.  If we want to
-    # ignore errors from any of these, just make sure not to ignore
-    # errors from the above "$doit $cpprog $src $dsttmp" command.
-    #
-    { test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } &&
-    { test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } &&
-    { test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } &&
-    { test -z "$chmodcmd" || $doit $chmodcmd $mode "$dsttmp"; } &&
-
-    # If -C, don't bother to copy if it wouldn't change the file.
-    if $copy_on_change &&
-       old=`LC_ALL=C ls -dlL "$dst"	2>/dev/null` &&
-       new=`LC_ALL=C ls -dlL "$dsttmp"	2>/dev/null` &&
-
-       eval "$initialize_posix_glob" &&
-       $posix_glob set -f &&
-       set X $old && old=:$2:$4:$5:$6 &&
-       set X $new && new=:$2:$4:$5:$6 &&
-       $posix_glob set +f &&
-
-       test "$old" = "$new" &&
-       $cmpprog "$dst" "$dsttmp" >/dev/null 2>&1
-    then
-      rm -f "$dsttmp"
-    else
-      # Rename the file to the real destination.
-      $doit $mvcmd -f "$dsttmp" "$dst" 2>/dev/null ||
-
-      # The rename failed, perhaps because mv can't rename something else
-      # to itself, or perhaps because mv is so ancient that it does not
-      # support -f.
-      {
-	# Now remove or move aside any old file at destination location.
-	# We try this two ways since rm can't unlink itself on some
-	# systems and the destination file might be busy for other
-	# reasons.  In this case, the final cleanup might fail but the new
-	# file should still install successfully.
-	{
-	  test ! -f "$dst" ||
-	  $doit $rmcmd -f "$dst" 2>/dev/null ||
-	  { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null &&
-	    { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }
-	  } ||
-	  { echo "$0: cannot unlink or rename $dst" >&2
-	    (exit 1); exit 1
-	  }
-	} &&
-
-	# Now rename the file to the real destination.
-	$doit $mvcmd "$dsttmp" "$dst"
-      }
-    fi || exit 1
-
-    trap '' 0
-  fi
-done
-
-# Local variables:
-# eval: (add-hook 'write-file-hooks 'time-stamp)
-# time-stamp-start: "scriptversion="
-# time-stamp-format: "%:y-%02m-%02d.%02H"
-# time-stamp-time-zone: "UTC"
-# time-stamp-end: "; # UTC"
-# End:

m4macros/.gitignore

@@ -0,0 +1,5 @@
+/libtool.m4
+/lt~obsolete.m4
+/ltoptions.m4
+/ltsugar.m4
+/ltversion.m4

missing

@@ -1,376 +0,0 @@
-#! /bin/sh
-# Common stub for a few missing GNU programs while installing.
-
-scriptversion=2009-04-28.21; # UTC
-
-# Copyright (C) 1996, 1997, 1999, 2000, 2002, 2003, 2004, 2005, 2006,
-# 2008, 2009 Free Software Foundation, Inc.
-# Originally by Fran,cois Pinard <pinard@iro.umontreal.ca>, 1996.
-
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-# As a special exception to the GNU General Public License, if you
-# distribute this file as part of a program that contains a
-# configuration script generated by Autoconf, you may include it under
-# the same distribution terms that you use for the rest of that program.
-
-if test $# -eq 0; then
-  echo 1>&2 "Try \`$0 --help' for more information"
-  exit 1
-fi
-
-run=:
-sed_output='s/.* --output[ =]\([^ ]*\).*/\1/p'
-sed_minuso='s/.* -o \([^ ]*\).*/\1/p'
-
-# In the cases where this matters, `missing' is being run in the
-# srcdir already.
-if test -f configure.ac; then
-  configure_ac=configure.ac
-else
-  configure_ac=configure.in
-fi
-
-msg="missing on your system"
-
-case $1 in
---run)
-  # Try to run requested program, and just exit if it succeeds.
-  run=
-  shift
-  "$@" && exit 0
-  # Exit code 63 means version mismatch.  This often happens
-  # when the user try to use an ancient version of a tool on
-  # a file that requires a minimum version.  In this case we
-  # we should proceed has if the program had been absent, or
-  # if --run hadn't been passed.
-  if test $? = 63; then
-    run=:
-    msg="probably too old"
-  fi
-  ;;
-
-  -h|--h|--he|--hel|--help)
-    echo "\
-$0 [OPTION]... PROGRAM [ARGUMENT]...
-
-Handle \`PROGRAM [ARGUMENT]...' for when PROGRAM is missing, or return an
-error status if there is no known handling for PROGRAM.
-
-Options:
-  -h, --help      display this help and exit
-  -v, --version   output version information and exit
-  --run           try to run the given command, and emulate it if it fails
-
-Supported PROGRAM values:
-  aclocal      touch file \`aclocal.m4'
-  autoconf     touch file \`configure'
-  autoheader   touch file \`config.h.in'
-  autom4te     touch the output file, or create a stub one
-  automake     touch all \`Makefile.in' files
-  bison        create \`y.tab.[ch]', if possible, from existing .[ch]
-  flex         create \`lex.yy.c', if possible, from existing .c
-  help2man     touch the output file
-  lex          create \`lex.yy.c', if possible, from existing .c
-  makeinfo     touch the output file
-  tar          try tar, gnutar, gtar, then tar without non-portable flags
-  yacc         create \`y.tab.[ch]', if possible, from existing .[ch]
-
-Version suffixes to PROGRAM as well as the prefixes \`gnu-', \`gnu', and
-\`g' are ignored when checking the name.
-
-Send bug reports to <bug-automake@gnu.org>."
-    exit $?
-    ;;
-
-  -v|--v|--ve|--ver|--vers|--versi|--versio|--version)
-    echo "missing $scriptversion (GNU Automake)"
-    exit $?
-    ;;
-
-  -*)
-    echo 1>&2 "$0: Unknown \`$1' option"
-    echo 1>&2 "Try \`$0 --help' for more information"
-    exit 1
-    ;;
-
-esac
-
-# normalize program name to check for.
-program=`echo "$1" | sed '
-  s/^gnu-//; t
-  s/^gnu//; t
-  s/^g//; t'`
-
-# Now exit if we have it, but it failed.  Also exit now if we
-# don't have it and --version was passed (most likely to detect
-# the program).  This is about non-GNU programs, so use $1 not
-# $program.
-case $1 in
-  lex*|yacc*)
-    # Not GNU programs, they don't have --version.
-    ;;
-
-  tar*)
-    if test -n "$run"; then
-       echo 1>&2 "ERROR: \`tar' requires --run"
-       exit 1
-    elif test "x$2" = "x--version" || test "x$2" = "x--help"; then
-       exit 1
-    fi
-    ;;
-
-  *)
-    if test -z "$run" && ($1 --version) > /dev/null 2>&1; then
-       # We have it, but it failed.
-       exit 1
-    elif test "x$2" = "x--version" || test "x$2" = "x--help"; then
-       # Could not run --version or --help.  This is probably someone
-       # running `$TOOL --version' or `$TOOL --help' to check whether
-       # $TOOL exists and not knowing $TOOL uses missing.
-       exit 1
-    fi
-    ;;
-esac
-
-# If it does not exist, or fails to run (possibly an outdated version),
-# try to emulate it.
-case $program in
-  aclocal*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-         you modified \`acinclude.m4' or \`${configure_ac}'.  You might want
-         to install the \`Automake' and \`Perl' packages.  Grab them from
-         any GNU archive site."
-    touch aclocal.m4
-    ;;
-
-  autoconf*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-         you modified \`${configure_ac}'.  You might want to install the
-         \`Autoconf' and \`GNU m4' packages.  Grab them from any GNU
-         archive site."
-    touch configure
-    ;;
-
-  autoheader*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-         you modified \`acconfig.h' or \`${configure_ac}'.  You might want
-         to install the \`Autoconf' and \`GNU m4' packages.  Grab them
-         from any GNU archive site."
-    files=`sed -n 's/^[ ]*A[CM]_CONFIG_HEADER(\([^)]*\)).*/\1/p' ${configure_ac}`
-    test -z "$files" && files="config.h"
-    touch_files=
-    for f in $files; do
-      case $f in
-      *:*) touch_files="$touch_files "`echo "$f" |
-				       sed -e 's/^[^:]*://' -e 's/:.*//'`;;
-      *) touch_files="$touch_files $f.in";;
-      esac
-    done
-    touch $touch_files
-    ;;
-
-  automake*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-         you modified \`Makefile.am', \`acinclude.m4' or \`${configure_ac}'.
-         You might want to install the \`Automake' and \`Perl' packages.
-         Grab them from any GNU archive site."
-    find . -type f -name Makefile.am -print |
-	   sed 's/\.am$/.in/' |
-	   while read f; do touch "$f"; done
-    ;;
-
-  autom4te*)
-    echo 1>&2 "\
-WARNING: \`$1' is needed, but is $msg.
-         You might have modified some files without having the
-         proper tools for further handling them.
-         You can get \`$1' as part of \`Autoconf' from any GNU
-         archive site."
-
-    file=`echo "$*" | sed -n "$sed_output"`
-    test -z "$file" && file=`echo "$*" | sed -n "$sed_minuso"`
-    if test -f "$file"; then
-	touch $file
-    else
-	test -z "$file" || exec >$file
-	echo "#! /bin/sh"
-	echo "# Created by GNU Automake missing as a replacement of"
-	echo "#  $ $@"
-	echo "exit 0"
-	chmod +x $file
-	exit 1
-    fi
-    ;;
-
-  bison*|yacc*)
-    echo 1>&2 "\
-WARNING: \`$1' $msg.  You should only need it if
-         you modified a \`.y' file.  You may need the \`Bison' package
-         in order for those modifications to take effect.  You can get
-         \`Bison' from any GNU archive site."
-    rm -f y.tab.c y.tab.h
-    if test $# -ne 1; then
-        eval LASTARG="\${$#}"
-	case $LASTARG in
-	*.y)
-	    SRCFILE=`echo "$LASTARG" | sed 's/y$/c/'`
-	    if test -f "$SRCFILE"; then
-	         cp "$SRCFILE" y.tab.c
-	    fi
-	    SRCFILE=`echo "$LASTARG" | sed 's/y$/h/'`
-	    if test -f "$SRCFILE"; then
-	         cp "$SRCFILE" y.tab.h
-	    fi
-	  ;;
-	esac
-    fi
-    if test ! -f y.tab.h; then
-	echo >y.tab.h
-    fi
-    if test ! -f y.tab.c; then
-	echo 'main() { return 0; }' >y.tab.c
-    fi
-    ;;
-
-  lex*|flex*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-         you modified a \`.l' file.  You may need the \`Flex' package
-         in order for those modifications to take effect.  You can get
-         \`Flex' from any GNU archive site."
-    rm -f lex.yy.c
-    if test $# -ne 1; then
-        eval LASTARG="\${$#}"
-	case $LASTARG in
-	*.l)
-	    SRCFILE=`echo "$LASTARG" | sed 's/l$/c/'`
-	    if test -f "$SRCFILE"; then
-	         cp "$SRCFILE" lex.yy.c
-	    fi
-	  ;;
-	esac
-    fi
-    if test ! -f lex.yy.c; then
-	echo 'main() { return 0; }' >lex.yy.c
-    fi
-    ;;
-
-  help2man*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-	 you modified a dependency of a manual page.  You may need the
-	 \`Help2man' package in order for those modifications to take
-	 effect.  You can get \`Help2man' from any GNU archive site."
-
-    file=`echo "$*" | sed -n "$sed_output"`
-    test -z "$file" && file=`echo "$*" | sed -n "$sed_minuso"`
-    if test -f "$file"; then
-	touch $file
-    else
-	test -z "$file" || exec >$file
-	echo ".ab help2man is required to generate this page"
-	exit $?
-    fi
-    ;;
-
-  makeinfo*)
-    echo 1>&2 "\
-WARNING: \`$1' is $msg.  You should only need it if
-         you modified a \`.texi' or \`.texinfo' file, or any other file
-         indirectly affecting the aspect of the manual.  The spurious
-         call might also be the consequence of using a buggy \`make' (AIX,
-         DU, IRIX).  You might want to install the \`Texinfo' package or
-         the \`GNU make' package.  Grab either from any GNU archive site."
-    # The file to touch is that specified with -o ...
-    file=`echo "$*" | sed -n "$sed_output"`
-    test -z "$file" && file=`echo "$*" | sed -n "$sed_minuso"`
-    if test -z "$file"; then
-      # ... or it is the one specified with @setfilename ...
-      infile=`echo "$*" | sed 's/.* \([^ ]*\) *$/\1/'`
-      file=`sed -n '
-	/^@setfilename/{
-	  s/.* \([^ ]*\) *$/\1/
-	  p
-	  q
-	}' $infile`
-      # ... or it is derived from the source name (dir/f.texi becomes f.info)
-      test -z "$file" && file=`echo "$infile" | sed 's,.*/,,;s,.[^.]*$,,'`.info
-    fi
-    # If the file does not exist, the user really needs makeinfo;
-    # let's fail without touching anything.
-    test -f $file || exit 1
-    touch $file
-    ;;
-
-  tar*)
-    shift
-
-    # We have already tried tar in the generic part.
-    # Look for gnutar/gtar before invocation to avoid ugly error
-    # messages.
-    if (gnutar --version > /dev/null 2>&1); then
-       gnutar "$@" && exit 0
-    fi
-    if (gtar --version > /dev/null 2>&1); then
-       gtar "$@" && exit 0
-    fi
-    firstarg="$1"
-    if shift; then
-	case $firstarg in
-	*o*)
-	    firstarg=`echo "$firstarg" | sed s/o//`
-	    tar "$firstarg" "$@" && exit 0
-	    ;;
-	esac
-	case $firstarg in
-	*h*)
-	    firstarg=`echo "$firstarg" | sed s/h//`
-	    tar "$firstarg" "$@" && exit 0
-	    ;;
-	esac
-    fi
-
-    echo 1>&2 "\
-WARNING: I can't seem to be able to run \`tar' with the given arguments.
-         You may want to install GNU tar or Free paxutils, or check the
-         command line arguments."
-    exit 1
-    ;;
-
-  *)
-    echo 1>&2 "\
-WARNING: \`$1' is needed, and is $msg.
-         You might have modified some files without having the
-         proper tools for further handling them.  Check the \`README' file,
-         it often tells you about the needed prerequisites for installing
-         this package.  You may also peek at any GNU archive site, in case
-         some other package would contain this missing \`$1' program."
-    exit 1
-    ;;
-esac
-
-exit 0
-
-# Local variables:
-# eval: (add-hook 'write-file-hooks 'time-stamp)
-# time-stamp-start: "scriptversion="
-# time-stamp-format: "%:y-%02m-%02d.%02H"
-# time-stamp-time-zone: "UTC"
-# time-stamp-end: "; # UTC"
-# End:

src/bin/Makefile.am

@@ -1,4 +1,4 @@
 SUBDIRS = bind10 bindctl cfgmgr ddns loadzone msgq host cmdctl auth xfrin \
-	xfrout usermgr zonemgr stats tests resolver sockcreator dhcp4 dhcp6
+	xfrout usermgr zonemgr stats tests resolver sockcreator dhcp4 dhcp6 dbutil
 
 check-recursive: all-recursive

src/bin/auth/.gitignore

@@ -0,0 +1,7 @@
+/auth.spec
+/auth.spec.pre
+/auth_messages.cc
+/auth_messages.h
+/b10-auth
+/spec_config.h
+/spec_config.h.pre

src/bin/auth/auth.spec.pre.in

@@ -47,6 +47,10 @@
                 "item_type": "string",
                 "item_optional": false,
                 "item_default": ""
+              },
+              { "item_name": "filetype",
+                "item_type": "string",
+                "item_optional": true
               }]
             }
           }]

src/bin/auth/auth_config.cc

@@ -12,14 +12,6 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <set>
-#include <string>
-#include <utility>
-#include <vector>
-
-#include <boost/foreach.hpp>
-#include <boost/shared_ptr.hpp>
-
 #include <dns/name.h>
 #include <dns/rrclass.h>
 
@@ -27,6 +19,7 @@
 
 #include <datasrc/memory_datasrc.h>
 #include <datasrc/zonetable.h>
+#include <datasrc/factory.h>
 
 #include <auth/auth_srv.h>
 #include <auth/auth_config.h>
@@ -34,6 +27,15 @@
 
 #include <server_common/portconfig.h>
 
+#include <boost/foreach.hpp>
+#include <boost/shared_ptr.hpp>
+#include <boost/scoped_ptr.hpp>
+
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
 using namespace std;
 using namespace isc::dns;
 using namespace isc::data;
@@ -155,17 +157,48 @@ MemoryDatasourceConfig::build(ConstElementPtr config_value) {
 
     BOOST_FOREACH(ConstElementPtr zone_config, zones_config->listValue()) {
         ConstElementPtr origin = zone_config->get("origin");
-        if (!origin) {
+        const string origin_txt = origin ? origin->stringValue() : "";
+        if (origin_txt.empty()) {
             isc_throw(AuthConfigError, "Missing zone origin");
         }
         ConstElementPtr file = zone_config->get("file");
-        if (!file) {
+        const string file_txt = file ? file->stringValue() : "";
+        if (file_txt.empty()) {
             isc_throw(AuthConfigError, "Missing zone file for zone: "
-                      << origin->str());
+                      << origin_txt);
         }
-        boost::shared_ptr<InMemoryZoneFinder> zone_finder(new
-                                                   InMemoryZoneFinder(rrclass_,
-            Name(origin->stringValue())));
+
+        // We support the traditional text type and SQLite3 backend.  For the
+        // latter we create a client for the underlying SQLite3 data source,
+        // and build the in-memory zone using an iterator of the underlying
+        // zone.
+        ConstElementPtr filetype = zone_config->get("filetype");
+        const string filetype_txt = filetype ? filetype->stringValue() :
+            "text";
+        boost::scoped_ptr<DataSourceClientContainer> container;
+        if (filetype_txt == "sqlite3") {
+            container.reset(new DataSourceClientContainer(
+                                "sqlite3",
+                                Element::fromJSON("{\"database_file\": \"" +
+                                                  file_txt + "\"}")));
+        } else if (filetype_txt != "text") {
+            isc_throw(AuthConfigError, "Invalid filetype for zone "
+                      << origin_txt << ": " << filetype_txt);
+        }
+
+        // Note: we don't want to have such small try-catch blocks for each
+        // specific error.  We may eventually want to introduce some unified
+        // error handling framework as we have more configuration parameters.
+        // See bug #1627 for the relevant discussion.
+        InMemoryZoneFinder* imzf = NULL;
+        try {
+            imzf = new InMemoryZoneFinder(rrclass_, Name(origin_txt));
+        } catch (const isc::dns::NameParserException& ex) {
+            isc_throw(AuthConfigError, "unable to parse zone's origin: " <<
+                      ex.what());
+        }
+
+        boost::shared_ptr<InMemoryZoneFinder> zone_finder(imzf);
         const result::Result result = memory_client_->addZone(zone_finder);
         if (result == result::EXIST) {
             isc_throw(AuthConfigError, "zone "<< origin->str()
@@ -178,7 +211,12 @@ MemoryDatasourceConfig::build(ConstElementPtr config_value) {
          * need the load method to be split into some kind of build and
          * commit/abort parts.
          */
-        zone_finder->load(file->stringValue());
+        if (filetype_txt == "text") {
+            zone_finder->load(file_txt);
+        } else {
+            zone_finder->load(*container->getInstance().getIterator(
+                                  Name(origin_txt)));
+        }
     }
 }
 

src/bin/auth/auth_messages.mes

@@ -73,6 +73,10 @@ attempt to parse the header of a received DNS packet has failed. (The
 reason for the failure is given in the message.) The server will drop the
 packet.
 
+% AUTH_INVALID_STATISTICS_DATA invalid specification of statistics data specified
+An error was encountered when the authoritiative server specified
+statistics data which is invalid for the auth specification file.
+
 % AUTH_LOAD_TSIG loading TSIG keys
 This is a debug message indicating that the authoritative server
 has requested the keyring holding TSIG keys from the configuration
@@ -92,6 +96,18 @@ discovered that the memory data source is disabled for the given class.
 This is a debug message reporting that the authoritative server has
 discovered that the memory data source is enabled for the given class.
 
+% AUTH_NOTIFY_QUESTIONS invalid number of questions (%1) in incoming NOTIFY
+This debug message is logged by the authoritative server when it receives
+a NOTIFY packet that contains zero or more than one question. (A valid
+NOTIFY packet contains one question.) The server will return a FORMERR
+error to the sender.
+
+% AUTH_NOTIFY_RRTYPE invalid question RR type (%1) in incoming NOTIFY
+This debug message is logged by the authoritative server when it receives
+a NOTIFY packet that an RR type of something other than SOA in the
+question section. (The RR type received is included in the message.) The
+server will return a FORMERR error to the sender.
+
 % AUTH_NO_STATS_SESSION session interface for statistics is not available
 The authoritative server had no session with the statistics module at the
 time it attempted to send it data: the attempt has been abandoned. This
@@ -102,18 +118,6 @@ This is a debug message produced by the authoritative server when it receives
 a NOTIFY packet but the XFRIN process is not running. The packet will be
 dropped and nothing returned to the sender.
 
-% AUTH_NOTIFY_RRTYPE invalid question RR type (%1) in incoming NOTIFY
-This debug message is logged by the authoritative server when it receives
-a NOTIFY packet that an RR type of something other than SOA in the
-question section. (The RR type received is included in the message.) The
-server will return a FORMERR error to the sender.
-
-% AUTH_NOTIFY_QUESTIONS invalid number of questions (%1) in incoming NOTIFY
-This debug message is logged by the authoritative server when it receives
-a NOTIFY packet that contains zero or more than one question. (A valid
-NOTIFY packet contains one question.) The server will return a FORMERR
-error to the sender.
-
 % AUTH_PACKET_PARSE_ERROR unable to parse received DNS packet: %1
 This is a debug message, generated by the authoritative server when an
 attempt to parse a received DNS packet has failed due to something other
@@ -154,6 +158,19 @@ a command from the statistics module to send it data. The 'sendstats'
 command is handled differently to other commands, which is why the debug
 message associated with it has its own code.
 
+% AUTH_RESPONSE_FAILURE exception while building response to query: %1
+This is a debug message, generated by the authoritative server when an
+attempt to create a response to a received DNS packet has failed. The
+reason for the failure is given in the log message. A SERVFAIL response
+is sent back. The most likely cause of this is an error in the data
+source implementation; it is either creating bad responses or raising
+exceptions itself.
+
+% AUTH_RESPONSE_FAILURE_UNKNOWN unknown exception while building response to query
+This debug message is similar to AUTH_RESPONSE_FAILURE, but further
+details about the error are unknown, because it was signaled by something
+which is not an exception. This is definitely a bug.
+
 % AUTH_RESPONSE_RECEIVED received response message, ignoring
 This is a debug message, this is output if the authoritative server
 receives a DNS packet with the QR bit set, i.e. a DNS response. The
@@ -260,7 +277,3 @@ This is a debug message output during the processing of a NOTIFY
 request. The zone manager component has been informed of the request,
 but has returned an error response (which is included in the message). The
 NOTIFY request will not be honored.
-
-% AUTH_INVALID_STATISTICS_DATA invalid specification of statistics data specified
-An error was encountered when the authoritiative server specified
-statistics data which is invalid for the auth specification file.

src/bin/auth/auth_srv.cc

@@ -14,6 +14,7 @@
 
 #include <config.h>
 
+#include <sys/types.h>
 #include <netinet/in.h>
 
 #include <algorithm>
@@ -46,6 +47,8 @@
 #include <dns/message.h>
 #include <dns/tsig.h>
 
+#include <asiodns/dns_service.h>
+
 #include <datasrc/query.h>
 #include <datasrc/data_source.h>
 #include <datasrc/memory_datasrc.h>
@@ -77,6 +80,35 @@ using namespace isc::asiolink;
 using namespace isc::asiodns;
 using namespace isc::server_common::portconfig;
 
+namespace {
+// A helper class for cleaning up message renderer.
+//
+// A temporary object of this class is expected to be created before starting
+// response message rendering.  On construction, it (re)initialize the given
+// message renderer with the given buffer.  On destruction, it releases
+// the previously set buffer and then release any internal resource in the
+// renderer, no matter what happened during the rendering, especially even
+// when it resulted in an exception.
+//
+// Note: if we need this helper in many other places we might consider making
+// it visible to other modules.  As of this implementation this is the only
+// user of this class, so we hide it within the implementation.
+class RendererHolder {
+public:
+    RendererHolder(MessageRenderer& renderer, OutputBuffer* buffer) :
+        renderer_(renderer)
+    {
+        renderer.setBuffer(buffer);
+    }
+    ~RendererHolder() {
+        renderer_.setBuffer(NULL);
+        renderer_.clear();
+    }
+private:
+    MessageRenderer& renderer_;
+};
+}
+
 class AuthSrvImpl {
 private:
     // prohibit copy
@@ -87,18 +119,19 @@ public:
     ~AuthSrvImpl();
     isc::data::ConstElementPtr setDbFile(isc::data::ConstElementPtr config);
 
-    bool processNormalQuery(const IOMessage& io_message, MessagePtr message,
-                            OutputBufferPtr buffer,
+    bool processNormalQuery(const IOMessage& io_message, Message& message,
+                            OutputBuffer& buffer,
                             auto_ptr<TSIGContext> tsig_context);
-    bool processXfrQuery(const IOMessage& io_message, MessagePtr message,
-                         OutputBufferPtr buffer,
+    bool processXfrQuery(const IOMessage& io_message, Message& message,
+                         OutputBuffer& buffer,
                          auto_ptr<TSIGContext> tsig_context);
-    bool processNotify(const IOMessage& io_message, MessagePtr message,
-                       OutputBufferPtr buffer,
+    bool processNotify(const IOMessage& io_message, Message& message,
+                       OutputBuffer& buffer,
                        auto_ptr<TSIGContext> tsig_context);
 
     IOService io_service_;
 
+    MessageRenderer renderer_;
     /// Currently non-configurable, but will be.
     static const uint16_t DEFAULT_LOCAL_UDPSIZE = 4096;
 
@@ -142,7 +175,7 @@ public:
     /// \param done If true, the Rcode from the given message is counted,
     ///             this value is then passed to server->resume(bool)
     void resumeServer(isc::asiodns::DNSServer* server,
-                      isc::dns::MessagePtr message,
+                      isc::dns::Message& message,
                       bool done);
 private:
     std::string db_file_;
@@ -161,6 +194,8 @@ private:
 
     // validateStatistics
     bool validateStatistics(isc::data::ConstElementPtr data) const;
+
+    auth::Query query_;
 };
 
 AuthSrvImpl::AuthSrvImpl(const bool use_cache,
@@ -200,12 +235,11 @@ public:
     MessageLookup(AuthSrv* srv) : server_(srv) {}
     virtual void operator()(const IOMessage& io_message,
                             MessagePtr message,
-                            MessagePtr answer_message,
+                            MessagePtr, // Not used here
                             OutputBufferPtr buffer,
                             DNSServer* server) const
     {
-        (void) answer_message;
-        server_->processMessage(io_message, message, buffer, server);
+        server_->processMessage(io_message, *message, *buffer, server);
     }
 private:
     AuthSrv* server_;
@@ -266,55 +300,56 @@ AuthSrv::~AuthSrv() {
 namespace {
 class QuestionInserter {
 public:
-    QuestionInserter(MessagePtr message) : message_(message) {}
+    QuestionInserter(Message& message) : message_(message) {}
     void operator()(const QuestionPtr question) {
-        message_->addQuestion(question);
+        message_.addQuestion(question);
     }
-    MessagePtr message_;
+    Message& message_;
 };
 
 void
-makeErrorMessage(MessagePtr message, OutputBufferPtr buffer,
-                 const Rcode& rcode, 
+makeErrorMessage(MessageRenderer& renderer, Message& message,
+                 OutputBuffer& buffer, const Rcode& rcode,
                  std::auto_ptr<TSIGContext> tsig_context =
                  std::auto_ptr<TSIGContext>())
 {
     // extract the parameters that should be kept.
     // XXX: with the current implementation, it's not easy to set EDNS0
     // depending on whether the query had it.  So we'll simply omit it.
-    const qid_t qid = message->getQid();
-    const bool rd = message->getHeaderFlag(Message::HEADERFLAG_RD);
-    const bool cd = message->getHeaderFlag(Message::HEADERFLAG_CD);
-    const Opcode& opcode = message->getOpcode();
+    const qid_t qid = message.getQid();
+    const bool rd = message.getHeaderFlag(Message::HEADERFLAG_RD);
+    const bool cd = message.getHeaderFlag(Message::HEADERFLAG_CD);
+    const Opcode& opcode = message.getOpcode();
     vector<QuestionPtr> questions;
 
     // If this is an error to a query or notify, we should also copy the
     // question section.
     if (opcode == Opcode::QUERY() || opcode == Opcode::NOTIFY()) {
-        questions.assign(message->beginQuestion(), message->endQuestion());
+        questions.assign(message.beginQuestion(), message.endQuestion());
     }
 
-    message->clear(Message::RENDER);
-    message->setQid(qid);
-    message->setOpcode(opcode);
-    message->setHeaderFlag(Message::HEADERFLAG_QR);
+    message.clear(Message::RENDER);
+    message.setQid(qid);
+    message.setOpcode(opcode);
+    message.setHeaderFlag(Message::HEADERFLAG_QR);
     if (rd) {
-        message->setHeaderFlag(Message::HEADERFLAG_RD);
+        message.setHeaderFlag(Message::HEADERFLAG_RD);
     }
     if (cd) {
-        message->setHeaderFlag(Message::HEADERFLAG_CD);
+        message.setHeaderFlag(Message::HEADERFLAG_CD);
     }
     for_each(questions.begin(), questions.end(), QuestionInserter(message));
-    message->setRcode(rcode);
 
-    MessageRenderer renderer(*buffer);
+    message.setRcode(rcode);
+    
+    RendererHolder holder(renderer, &buffer);
     if (tsig_context.get() != NULL) {
-        message->toWire(renderer, *tsig_context);
+        message.toWire(renderer, *tsig_context);
     } else {
-        message->toWire(renderer);
+        message.toWire(renderer);
     }
     LOG_DEBUG(auth_logger, DBG_AUTH_MESSAGES, AUTH_SEND_ERROR_RESPONSE)
-              .arg(renderer.getLength()).arg(*message);
+              .arg(renderer.getLength()).arg(message);
 }
 }
 
@@ -412,18 +447,18 @@ AuthSrv::setStatisticsTimerInterval(uint32_t interval) {
 }
 
 void
-AuthSrv::processMessage(const IOMessage& io_message, MessagePtr message,
-                        OutputBufferPtr buffer, DNSServer* server)
+AuthSrv::processMessage(const IOMessage& io_message, Message& message,
+                        OutputBuffer& buffer, DNSServer* server)
 {
     InputBuffer request_buffer(io_message.getData(), io_message.getDataSize());
 
     // First, check the header part.  If we fail even for the base header,
     // just drop the message.
     try {
-        message->parseHeader(request_buffer);
+        message.parseHeader(request_buffer);
 
         // Ignore all responses.
-        if (message->getHeaderFlag(Message::HEADERFLAG_QR)) {
+        if (message.getHeaderFlag(Message::HEADERFLAG_QR)) {
             LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_RESPONSE_RECEIVED);
             impl_->resumeServer(server, message, false);
             return;
@@ -437,29 +472,29 @@ AuthSrv::processMessage(const IOMessage& io_message, MessagePtr message,
 
     try {
         // Parse the message.
-        message->fromWire(request_buffer);
+        message.fromWire(request_buffer);
     } catch (const DNSProtocolError& error) {
         LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_PACKET_PROTOCOL_ERROR)
                   .arg(error.getRcode().toText()).arg(error.what());
-        makeErrorMessage(message, buffer, error.getRcode());
+        makeErrorMessage(impl_->renderer_, message, buffer, error.getRcode());
         impl_->resumeServer(server, message, true);
         return;
     } catch (const Exception& ex) {
         LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_PACKET_PARSE_ERROR)
                   .arg(ex.what());
-        makeErrorMessage(message, buffer, Rcode::SERVFAIL());
+        makeErrorMessage(impl_->renderer_, message, buffer, Rcode::SERVFAIL());
         impl_->resumeServer(server, message, true);
         return;
     } // other exceptions will be handled at a higher layer.
 
     LOG_DEBUG(auth_logger, DBG_AUTH_MESSAGES, AUTH_PACKET_RECEIVED)
-              .arg(message->toText());
+              .arg(message);
 
     // Perform further protocol-level validation.
     // TSIG first
     // If this is set to something, we know we need to answer with TSIG as well
     std::auto_ptr<TSIGContext> tsig_context;
-    const TSIGRecord* tsig_record(message->getTSIGRecord());
+    const TSIGRecord* tsig_record(message.getTSIGRecord());
     TSIGError tsig_error(TSIGError::NOERROR());
 
     // Do we do TSIG?
@@ -474,56 +509,67 @@ AuthSrv::processMessage(const IOMessage& io_message, MessagePtr message,
     }
 
     if (tsig_error != TSIGError::NOERROR()) {
-        makeErrorMessage(message, buffer, tsig_error.toRcode(), tsig_context);
+        makeErrorMessage(impl_->renderer_, message, buffer,
+                         tsig_error.toRcode(), tsig_context);
         impl_->resumeServer(server, message, true);
         return;
     }
 
-    // update per opcode statistics counter.  This can only be reliable after
-    // TSIG check succeeds.
-    impl_->counters_.inc(message->getOpcode());
-
     bool send_answer = true;
-    if (message->getOpcode() == Opcode::NOTIFY()) {
-        send_answer = impl_->processNotify(io_message, message, buffer,
-                                           tsig_context);
-    } else if (message->getOpcode() != Opcode::QUERY()) {
-        LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_UNSUPPORTED_OPCODE)
-                  .arg(message->getOpcode().toText());
-        makeErrorMessage(message, buffer, Rcode::NOTIMP(), tsig_context);
-    } else if (message->getRRCount(Message::SECTION_QUESTION) != 1) {
-        makeErrorMessage(message, buffer, Rcode::FORMERR(), tsig_context);
-    } else {
-        ConstQuestionPtr question = *message->beginQuestion();
-        const RRType &qtype = question->getType();
-        if (qtype == RRType::AXFR()) {
-            send_answer = impl_->processXfrQuery(io_message, message, buffer,
-                                                 tsig_context);
-        } else if (qtype == RRType::IXFR()) {
-            send_answer = impl_->processXfrQuery(io_message, message, buffer,
-                                                 tsig_context);
+    try {
+        // update per opcode statistics counter.  This can only be reliable
+        // after TSIG check succeeds.
+        impl_->counters_.inc(message.getOpcode());
+
+        if (message.getOpcode() == Opcode::NOTIFY()) {
+            send_answer = impl_->processNotify(io_message, message, buffer,
+                                               tsig_context);
+        } else if (message.getOpcode() != Opcode::QUERY()) {
+            LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_UNSUPPORTED_OPCODE)
+                      .arg(message.getOpcode().toText());
+            makeErrorMessage(impl_->renderer_, message, buffer,
+                             Rcode::NOTIMP(), tsig_context);
+        } else if (message.getRRCount(Message::SECTION_QUESTION) != 1) {
+            makeErrorMessage(impl_->renderer_, message, buffer,
+                             Rcode::FORMERR(), tsig_context);
         } else {
-            send_answer = impl_->processNormalQuery(io_message, message,
-                                                    buffer, tsig_context);
+            ConstQuestionPtr question = *message.beginQuestion();
+            const RRType &qtype = question->getType();
+            if (qtype == RRType::AXFR()) {
+                send_answer = impl_->processXfrQuery(io_message, message,
+                                                     buffer, tsig_context);
+            } else if (qtype == RRType::IXFR()) {
+                send_answer = impl_->processXfrQuery(io_message, message,
+                                                     buffer, tsig_context);
+            } else {
+                send_answer = impl_->processNormalQuery(io_message, message,
+                                                        buffer, tsig_context);
+            }
         }
+    } catch (const std::exception& ex) {
+        LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_RESPONSE_FAILURE)
+                  .arg(ex.what());
+        makeErrorMessage(impl_->renderer_, message, buffer, Rcode::SERVFAIL());
+    } catch (...) {
+        LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_RESPONSE_FAILURE_UNKNOWN);
+        makeErrorMessage(impl_->renderer_, message, buffer, Rcode::SERVFAIL());
     }
-
     impl_->resumeServer(server, message, send_answer);
 }
 
 bool
-AuthSrvImpl::processNormalQuery(const IOMessage& io_message, MessagePtr message,
-                                OutputBufferPtr buffer,
+AuthSrvImpl::processNormalQuery(const IOMessage& io_message, Message& message,
+                                OutputBuffer& buffer,
                                 auto_ptr<TSIGContext> tsig_context)
 {
-    ConstEDNSPtr remote_edns = message->getEDNS();
+    ConstEDNSPtr remote_edns = message.getEDNS();
     const bool dnssec_ok = remote_edns && remote_edns->getDNSSECAwareness();
     const uint16_t remote_bufsize = remote_edns ? remote_edns->getUDPSize() :
         Message::DEFAULT_MAX_UDPSIZE;
 
-    message->makeResponse();
-    message->setHeaderFlag(Message::HEADERFLAG_AA);
-    message->setRcode(Rcode::NOERROR());
+    message.makeResponse();
+    message.setHeaderFlag(Message::HEADERFLAG_AA);
+    message.setRcode(Rcode::NOERROR());
 
     // Increment query counter.
     incCounter(io_message.getSocket().getProtocol());
@@ -532,46 +578,44 @@ AuthSrvImpl::processNormalQuery(const IOMessage& io_message, MessagePtr message,
         EDNSPtr local_edns = EDNSPtr(new EDNS());
         local_edns->setDNSSECAwareness(dnssec_ok);
         local_edns->setUDPSize(AuthSrvImpl::DEFAULT_LOCAL_UDPSIZE);
-        message->setEDNS(local_edns);
+        message.setEDNS(local_edns);
     }
 
     try {
         // If a memory data source is configured call the separate
         // Query::process()
-        const ConstQuestionPtr question = *message->beginQuestion();
+        const ConstQuestionPtr question = *message.beginQuestion();
         if (memory_client_ && memory_client_class_ == question->getClass()) {
             const RRType& qtype = question->getType();
             const Name& qname = question->getName();
-            auth::Query(*memory_client_, qname, qtype, *message,
-                        dnssec_ok).process();
+            query_.process(*memory_client_, qname, qtype, message, dnssec_ok);
         } else {
-            datasrc::Query query(*message, cache_, dnssec_ok);
+            datasrc::Query query(message, cache_, dnssec_ok);
             data_sources_.doQuery(query);
         }
     } catch (const Exception& ex) {
         LOG_ERROR(auth_logger, AUTH_PROCESS_FAIL).arg(ex.what());
-        makeErrorMessage(message, buffer, Rcode::SERVFAIL());
+        makeErrorMessage(renderer_, message, buffer, Rcode::SERVFAIL());
         return (true);
     }
 
-    MessageRenderer renderer(*buffer);
+    RendererHolder holder(renderer_, &buffer);
     const bool udp_buffer =
         (io_message.getSocket().getProtocol() == IPPROTO_UDP);
-    renderer.setLengthLimit(udp_buffer ? remote_bufsize : 65535);
+    renderer_.setLengthLimit(udp_buffer ? remote_bufsize : 65535);
     if (tsig_context.get() != NULL) {
-        message->toWire(renderer, *tsig_context);
+        message.toWire(renderer_, *tsig_context);
     } else {
-        message->toWire(renderer);
+        message.toWire(renderer_);
     }
     LOG_DEBUG(auth_logger, DBG_AUTH_MESSAGES, AUTH_SEND_NORMAL_RESPONSE)
-              .arg(renderer.getLength()).arg(message->toText());
-
+              .arg(renderer_.getLength()).arg(message);
     return (true);
 }
 
 bool
-AuthSrvImpl::processXfrQuery(const IOMessage& io_message, MessagePtr message,
-                             OutputBufferPtr buffer,
+AuthSrvImpl::processXfrQuery(const IOMessage& io_message, Message& message,
+                             OutputBuffer& buffer,
                              auto_ptr<TSIGContext> tsig_context)
 {
     // Increment query counter.
@@ -579,7 +623,8 @@ AuthSrvImpl::processXfrQuery(const IOMessage& io_message, MessagePtr message,
 
     if (io_message.getSocket().getProtocol() == IPPROTO_UDP) {
         LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_AXFR_UDP);
-        makeErrorMessage(message, buffer, Rcode::FORMERR(), tsig_context);
+        makeErrorMessage(renderer_, message, buffer, Rcode::FORMERR(),
+                         tsig_context);
         return (true);
     }
 
@@ -604,7 +649,8 @@ AuthSrvImpl::processXfrQuery(const IOMessage& io_message, MessagePtr message,
 
         LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_AXFR_ERROR)
                   .arg(err.what());
-        makeErrorMessage(message, buffer, Rcode::SERVFAIL(), tsig_context);
+        makeErrorMessage(renderer_, message, buffer, Rcode::SERVFAIL(),
+                         tsig_context);
         return (true);
     }
 
@@ -612,23 +658,25 @@ AuthSrvImpl::processXfrQuery(const IOMessage& io_message, MessagePtr message,
 }
 
 bool
-AuthSrvImpl::processNotify(const IOMessage& io_message, MessagePtr message, 
-                           OutputBufferPtr buffer,
+AuthSrvImpl::processNotify(const IOMessage& io_message, Message& message,
+                           OutputBuffer& buffer,
                            std::auto_ptr<TSIGContext> tsig_context)
 {
     // The incoming notify must contain exactly one question for SOA of the
     // zone name.
-    if (message->getRRCount(Message::SECTION_QUESTION) != 1) {
+    if (message.getRRCount(Message::SECTION_QUESTION) != 1) {
         LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_NOTIFY_QUESTIONS)
-                  .arg(message->getRRCount(Message::SECTION_QUESTION));
-        makeErrorMessage(message, buffer, Rcode::FORMERR(), tsig_context);
+                  .arg(message.getRRCount(Message::SECTION_QUESTION));
+        makeErrorMessage(renderer_, message, buffer, Rcode::FORMERR(),
+                         tsig_context);
         return (true);
     }
-    ConstQuestionPtr question = *message->beginQuestion();
+    ConstQuestionPtr question = *message.beginQuestion();
     if (question->getType() != RRType::SOA()) {
         LOG_DEBUG(auth_logger, DBG_AUTH_DETAIL, AUTH_NOTIFY_RRTYPE)
                   .arg(question->getType().toText());
-        makeErrorMessage(message, buffer, Rcode::FORMERR(), tsig_context);
+        makeErrorMessage(renderer_, message, buffer, Rcode::FORMERR(),
+                         tsig_context);
         return (true);
     }
 
@@ -679,15 +727,15 @@ AuthSrvImpl::processNotify(const IOMessage& io_message, MessagePtr message,
         return (false);
     }
 
-    message->makeResponse();
-    message->setHeaderFlag(Message::HEADERFLAG_AA);
-    message->setRcode(Rcode::NOERROR());
+    message.makeResponse();
+    message.setHeaderFlag(Message::HEADERFLAG_AA);
+    message.setRcode(Rcode::NOERROR());
 
-    MessageRenderer renderer(*buffer);
+    RendererHolder holder(renderer_, &buffer);
     if (tsig_context.get() != NULL) {
-        message->toWire(renderer, *tsig_context);
+        message.toWire(renderer_, *tsig_context);
     } else {
-        message->toWire(renderer);
+        message.toWire(renderer_);
     }
     return (true);
 }
@@ -772,9 +820,9 @@ AuthSrvImpl::setDbFile(ConstElementPtr config) {
 }
 
 void
-AuthSrvImpl::resumeServer(DNSServer* server, MessagePtr message, bool done) {
+AuthSrvImpl::resumeServer(DNSServer* server, Message& message, bool done) {
     if (done) {
-        counters_.inc(message->getRcode());
+        counters_.inc(message.getRcode());
     }
     server->resume(done);
 }
@@ -820,11 +868,14 @@ AuthSrv::getListenAddresses() const {
 
 void
 AuthSrv::setListenAddresses(const AddressList& addresses) {
-    installListenAddresses(addresses, impl_->listen_addresses_, *dnss_);
+    // For UDP servers we specify the "SYNC_OK" option because in our usage
+    // it can act in the synchronous mode.
+    installListenAddresses(addresses, impl_->listen_addresses_, *dnss_,
+                           DNSService::SERVER_SYNC_OK);
 }
 
 void
-AuthSrv::setDNSService(isc::asiodns::DNSService& dnss) {
+AuthSrv::setDNSService(isc::asiodns::DNSServiceBase& dnss) {
     dnss_ = &dnss;
 }
 

src/bin/auth/auth_srv.h

@@ -28,6 +28,7 @@
 #include <util/buffer.h>
 
 #include <asiodns/dns_server.h>
+#include <asiodns/dns_service.h>
 #include <asiodns/dns_lookup.h>
 #include <asiodns/dns_answer.h>
 #include <asiolink/io_message.h>
@@ -115,14 +116,14 @@ public:
     /// send the reply.
     ///
     /// \param io_message The raw message received
-    /// \param message Pointer to the \c Message object
-    /// \param buffer Pointer to an \c OutputBuffer for the resposne
+    /// \param message the \c Message object
+    /// \param buffer an \c OutputBuffer for the resposne
     /// \param server Pointer to the \c DNSServer
     ///
     /// \throw isc::Unexpected Protocol type of \a message is unexpected
     void processMessage(const isc::asiolink::IOMessage& io_message,
-                        isc::dns::MessagePtr message,
-                        isc::util::OutputBufferPtr buffer,
+                        isc::dns::Message& message,
+                        isc::util::OutputBuffer& buffer,
                         isc::asiodns::DNSServer* server);
 
     /// \brief Updates the data source for the \c AuthSrv object.
@@ -276,7 +277,7 @@ public:
     /// in-memory data source.
     ///
     /// \param rrclass The RR class of the in-memory data source to be set.
-    /// \param memory_datasrc A (shared) pointer to \c InMemoryClient to be set.
+    /// \param memory_client A (shared) pointer to \c InMemoryClient to be set.
     void setInMemoryClient(const isc::dns::RRClass& rrclass,
                            InMemoryClientPtr memory_client);
 
@@ -384,7 +385,7 @@ public:
         const;
 
     /// \brief Assign an ASIO DNS Service queue to this Auth object
-    void setDNSService(isc::asiodns::DNSService& dnss);
+    void setDNSService(isc::asiodns::DNSServiceBase& dnss);
 
     /// \brief Sets the keyring used for verifying and signing
     ///
@@ -400,7 +401,7 @@ private:
     isc::asiolink::SimpleCallback* checkin_;
     isc::asiodns::DNSLookup* dns_lookup_;
     isc::asiodns::DNSAnswer* dns_answer_;
-    isc::asiodns::DNSService* dnss_;
+    isc::asiodns::DNSServiceBase* dnss_;
 };
 
 #endif // __AUTH_SRV_H

src/bin/auth/b10-auth.8

@@ -2,12 +2,12 @@
 .\"     Title: b10-auth
 .\"    Author: [FIXME: author] [see http://docbook.sf.net/el/author]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: December 28, 2011
+.\"      Date: March 28, 2012
 .\"    Manual: BIND10
 .\"    Source: BIND10
 .\"  Language: English
 .\"
-.TH "B10\-AUTH" "8" "December 28, 2011" "BIND10" "BIND10"
+.TH "B10\-AUTH" "8" "March 28, 2012" "BIND10" "BIND10"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -64,7 +64,7 @@ defines the path to the SQLite3 zone file when using the sqlite datasource\&. Th
 \fIdatasources\fR
 configures data sources\&. The list items include:
 \fItype\fR
-to optionally choose the data source type (such as
+to define the required data source type (such as
 \(lqmemory\(rq);
 \fIclass\fR
 to optionally select the class (it defaults to
@@ -154,21 +154,25 @@ immediately\&.
 
 \fBshutdown\fR
 exits
-\fBb10\-auth\fR\&. (Note that the BIND 10 boss process will restart this service\&.)
+\fBb10\-auth\fR\&. This has an optional
+\fIpid\fR
+argument to select the process ID to stop\&. (Note that the BIND 10 boss process may restart this service if configured\&.)
 .SH "STATISTICS DATA"
 .PP
 The statistics data collected by the
 \fBb10\-stats\fR
-daemon include:
+daemon for
+\(lqAuth\(rq
+include:
 .PP
-auth\&.queries\&.tcp
+queries\&.tcp
 .RS 4
 Total count of queries received by the
 \fBb10\-auth\fR
 server over TCP since startup\&.
 .RE
 .PP
-auth\&.queries\&.udp
+queries\&.udp
 .RS 4
 Total count of queries received by the
 \fBb10\-auth\fR
@@ -198,5 +202,5 @@ The
 daemon was first coded in October 2009\&.
 .SH "COPYRIGHT"
 .br
-Copyright \(co 2010 Internet Systems Consortium, Inc. ("ISC")
+Copyright \(co 2010-2012 Internet Systems Consortium, Inc. ("ISC")
 .br

src/bin/auth/b10-auth.xml

@@ -2,7 +2,7 @@
                "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
 	       [<!ENTITY mdash "&#8212;">]>
 <!--
- - Copyright (C) 2010-2011  Internet Systems Consortium, Inc. ("ISC")
+ - Copyright (C) 2010-2012  Internet Systems Consortium, Inc. ("ISC")
  -
  - Permission to use, copy, modify, and/or distribute this software for any
  - purpose with or without fee is hereby granted, provided that the above
@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>December 28, 2011</date>
+    <date>March 28, 2012</date>
   </refentryinfo>
 
   <refmeta>
@@ -36,7 +36,7 @@
 
   <docinfo>
     <copyright>
-      <year>2010</year>
+      <year>2010-2012</year>
       <holder>Internet Systems Consortium, Inc. ("ISC")</holder>
     </copyright>
   </docinfo>
@@ -119,7 +119,7 @@
     <para>
       <varname>datasources</varname> configures data sources.
       The list items include:
-      <varname>type</varname> to optionally choose the data source type
+      <varname>type</varname> to define the required data source type
       (such as <quote>memory</quote>);
       <varname>class</varname> to optionally select the class
       (it defaults to <quote>IN</quote>);
@@ -188,7 +188,10 @@
 
     <para>
       <command>shutdown</command> exits <command>b10-auth</command>.
-      (Note that the BIND 10 boss process will restart this service.)
+      This has an optional <varname>pid</varname> argument to
+      select the process ID to stop.
+      (Note that the BIND 10 boss process may restart this service
+      if configured.)
     </para>
 
   </refsect1>
@@ -198,20 +201,20 @@
 
     <para>
       The statistics data collected by the <command>b10-stats</command>
-      daemon include:
+      daemon for <quote>Auth</quote> include:
     </para>
 
     <variablelist>
 
       <varlistentry>
-        <term>auth.queries.tcp</term>
+        <term>queries.tcp</term>
         <listitem><simpara>Total count of queries received by the
           <command>b10-auth</command> server over TCP since startup.
         </simpara></listitem>
       </varlistentry>
 
       <varlistentry>
-        <term>auth.queries.udp</term>
+        <term>queries.udp</term>
         <listitem><simpara>Total count of queries received by the
           <command>b10-auth</command> server over UDP since startup.
         </simpara></listitem>
@@ -219,6 +222,8 @@
 
     </variablelist>
 
+<!-- TODO: missing stats docs. See ticket #1721 -->
+
   </refsect1>
 
   <refsect1>

src/bin/auth/benchmarks/.gitignore

@@ -0,0 +1 @@
+/query_bench

src/bin/auth/benchmarks/query_bench.cc

@@ -76,8 +76,8 @@ private:
     typedef boost::shared_ptr<const IOEndpoint> IOEndpointPtr;
 protected:
     QueryBenchMark(const bool enable_cache,
-                   const BenchQueries& queries, MessagePtr query_message,
-                   OutputBufferPtr buffer) :
+                   const BenchQueries& queries, Message& query_message,
+                   OutputBuffer& buffer) :
         server_(new AuthSrv(enable_cache, xfrout_client)),
         queries_(queries),
         query_message_(query_message),
@@ -95,8 +95,8 @@ public:
         for (query = queries_.begin(); query != query_end; ++query) {
             IOMessage io_message(&(*query)[0], (*query).size(), dummy_socket,
                                  *dummy_endpoint);
-            query_message_->clear(Message::PARSE);
-            buffer_->clear();
+            query_message_.clear(Message::PARSE);
+            buffer_.clear();
             server_->processMessage(io_message, query_message_, buffer_,
                                     &server);
         }
@@ -107,8 +107,8 @@ protected:
     AuthSrvPtr server_;
 private:
     const BenchQueries& queries_;
-    MessagePtr query_message_;
-    OutputBufferPtr buffer_;
+    Message& query_message_;
+    OutputBuffer& buffer_;
     IOSocket& dummy_socket;
     IOEndpointPtr dummy_endpoint;
 };
@@ -118,8 +118,8 @@ public:
     Sqlite3QueryBenchMark(const int cache_slots,
                           const char* const datasrc_file,
                           const BenchQueries& queries,
-                          MessagePtr query_message,
-                          OutputBufferPtr buffer) :
+                          Message& query_message,
+                          OutputBuffer& buffer) :
         QueryBenchMark(cache_slots >= 0 ? true : false, queries,
                        query_message, buffer)
     {
@@ -136,8 +136,8 @@ public:
     MemoryQueryBenchMark(const char* const zone_file,
                          const char* const zone_origin,
                           const BenchQueries& queries,
-                          MessagePtr query_message,
-                          OutputBufferPtr buffer) :
+                          Message& query_message,
+                          OutputBuffer& buffer) :
         QueryBenchMark(false, queries, query_message, buffer)
     {
         configureAuthServer(*server_,
@@ -255,8 +255,8 @@ main(int argc, char* argv[]) {
 
     BenchQueries queries;
     loadQueryData(query_data_file, queries, RRClass::IN());
-    OutputBufferPtr buffer(new OutputBuffer(4096));
-    MessagePtr message(new Message(Message::PARSE));
+    OutputBuffer buffer(4096);
+    Message message(Message::PARSE);
 
     cout << "Parameters:" << endl;
     cout << "  Iterations: " << iteration << endl;

src/bin/auth/command.cc

@@ -18,6 +18,7 @@
 
 #include <cc/data.h>
 #include <datasrc/memory_datasrc.h>
+#include <datasrc/factory.h>
 #include <config/ccsession.h>
 #include <exceptions/exceptions.h>
 #include <dns/rrclass.h>
@@ -149,26 +150,39 @@ public:
             return;
         }
 
+        const ConstElementPtr zone_config = getZoneConfig(server);
+
         // Load a new zone and replace the current zone with the new one.
         // TODO: eventually this should be incremental or done in some way
         // that doesn't block other server operations.
         // TODO: we may (should?) want to check the "last load time" and
         // the timestamp of the file and skip loading if the file isn't newer.
+        const ConstElementPtr type(zone_config->get("filetype"));
         boost::shared_ptr<InMemoryZoneFinder> zone_finder(
-            new InMemoryZoneFinder(old_zone_finder->getClass(),
-                                   old_zone_finder->getOrigin()));
-        zone_finder->load(old_zone_finder->getFileName());
-        old_zone_finder->swap(*zone_finder);
+            new InMemoryZoneFinder(old_zone_finder_->getClass(),
+                                   old_zone_finder_->getOrigin()));
+        if (type && type->stringValue() == "sqlite3") {
+            scoped_ptr<DataSourceClientContainer>
+                container(new DataSourceClientContainer("sqlite3",
+                                                        Element::fromJSON(
+                    "{\"database_file\": \"" +
+                    zone_config->get("file")->stringValue() + "\"}")));
+            zone_finder->load(*container->getInstance().getIterator(
+                old_zone_finder_->getOrigin()));
+        } else {
+            zone_finder->load(old_zone_finder_->getFileName());
+        }
+        old_zone_finder_->swap(*zone_finder);
         LOG_DEBUG(auth_logger, DBG_AUTH_OPS, AUTH_LOAD_ZONE)
                   .arg(zone_finder->getOrigin()).arg(zone_finder->getClass());
     }
 
 private:
     // zone finder to be updated with the new file.
-    boost::shared_ptr<InMemoryZoneFinder> old_zone_finder;
+    boost::shared_ptr<InMemoryZoneFinder> old_zone_finder_;
 
     // A helper private method to parse and validate command parameters.
-    // On success, it sets 'old_zone_finder' to the zone to be updated.
+    // On success, it sets 'old_zone_finder_' to the zone to be updated.
     // It returns true if everything is okay; and false if the command is
     // valid but there's no need for further process.
     bool validate(AuthSrv& server, isc::data::ConstElementPtr args) {
@@ -193,10 +207,11 @@ private:
         }
 
         ConstElementPtr class_elem = args->get("class");
-        const RRClass zone_class = class_elem ?
-            RRClass(class_elem->stringValue()) : RRClass::IN();
+        const RRClass zone_class =
+            class_elem ? RRClass(class_elem->stringValue()) : RRClass::IN();
 
-        AuthSrv::InMemoryClientPtr datasrc(server.getInMemoryClient(zone_class));
+        AuthSrv::InMemoryClientPtr datasrc(server.
+                                           getInMemoryClient(zone_class));
         if (datasrc == NULL) {
             isc_throw(AuthCommandError, "Memory data source is disabled");
         }
@@ -205,7 +220,7 @@ private:
         if (!origin_elem) {
             isc_throw(AuthCommandError, "Zone origin is missing");
         }
-        const Name origin(origin_elem->stringValue());
+        const Name origin = Name(origin_elem->stringValue());
 
         // Get the current zone
         const InMemoryClient::FindResult result = datasrc->findZone(origin);
@@ -214,11 +229,70 @@ private:
                       " is not found in data source");
         }
 
-        old_zone_finder = boost::dynamic_pointer_cast<InMemoryZoneFinder>(
+        old_zone_finder_ = boost::dynamic_pointer_cast<InMemoryZoneFinder>(
             result.zone_finder);
 
         return (true);
     }
+
+    ConstElementPtr getZoneConfig(const AuthSrv &server) {
+        if (!server.getConfigSession()) {
+            // FIXME: This is a hack to make older tests pass. We should
+            // update these tests as well sometime and remove this hack.
+            // (note that under normal situation, the
+            // server.getConfigSession() does not return NULL)
+
+            // We provide an empty map, which means no configuration --
+            // defaults.
+            return (ConstElementPtr(new MapElement()));
+        }
+
+        // Find the config corresponding to the zone.
+        // We expect the configuration to be valid, as we have it and we
+        // accepted it before, therefore it must be validated.
+        const ConstElementPtr config(server.getConfigSession()->
+                                     getValue("datasources"));
+        ConstElementPtr zone_list;
+        // Unfortunately, we need to walk the list to find the correct data
+        // source.
+        // TODO: Make it named sets. These lists are uncomfortable.
+        for (size_t i(0); i < config->size(); ++i) {
+            // We use the getValue to get defaults as well
+            const ConstElementPtr dsrc_config(config->get(i));
+            const ConstElementPtr class_config(dsrc_config->get("class"));
+            const string class_type(class_config ?
+                                    class_config->stringValue() : "IN");
+            // It is in-memory and our class matches.
+            // FIXME: Is it allowed to have two datasources for the same
+            // type and class at once? It probably would not work now
+            // anyway and we may want to change the configuration of
+            // datasources somehow.
+            if (dsrc_config->get("type")->stringValue() == "memory" &&
+                RRClass(class_type) == old_zone_finder_->getClass()) {
+                zone_list = dsrc_config->get("zones");
+                break;
+            }
+        }
+
+        if (!zone_list) {
+            isc_throw(AuthCommandError,
+                      "Corresponding data source configuration was not found");
+        }
+
+        // Now we need to walk the zones and find the correct one.
+        for (size_t i(0); i < zone_list->size(); ++i) {
+            const ConstElementPtr zone_config(zone_list->get(i));
+            if (Name(zone_config->get("origin")->stringValue()) ==
+                old_zone_finder_->getOrigin()) {
+                // The origins are the same, so we consider this config to be
+                // for the zone.
+                return (zone_config);
+            }
+        }
+
+        isc_throw(AuthCommandError,
+                  "Corresponding zone configuration was not found");
+    }
 };
 
 // The factory of command objects.

src/bin/auth/query.cc

@@ -12,99 +12,103 @@
 // OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
 // PERFORMANCE OF THIS SOFTWARE.
 
-#include <algorithm>            // for std::max
-#include <vector>
-#include <boost/foreach.hpp>
-#include <boost/bind.hpp>
-#include <boost/function.hpp>
-
 #include <dns/message.h>
 #include <dns/rcode.h>
+#include <dns/rrtype.h>
+#include <dns/rrset.h>
 #include <dns/rdataclass.h>
 
 #include <datasrc/client.h>
 
 #include <auth/query.h>
 
+#include <boost/foreach.hpp>
+#include <boost/bind.hpp>
+#include <boost/function.hpp>
+
+#include <cassert>
+#include <algorithm>            // for std::max
+#include <functional>
+#include <vector>
+
+using namespace std;
 using namespace isc::dns;
 using namespace isc::datasrc;
 using namespace isc::dns::rdata;
 
+// This is a "constant" vector storing desired RR types for the additional
+// section.  The vector is filled first time it's used.
+namespace {
+const vector<RRType>&
+A_AND_AAAA() {
+    static vector<RRType> needed_types;
+    if (needed_types.empty()) {
+        needed_types.push_back(RRType::A());
+        needed_types.push_back(RRType::AAAA());
+    }
+    return (needed_types);
+}
+}
+
 namespace isc {
 namespace auth {
 
 void
-Query::addAdditional(ZoneFinder& zone, const AbstractRRset& rrset) {
-    RdataIteratorPtr rdata_iterator(rrset.getRdataIterator());
-    for (; !rdata_iterator->isLast(); rdata_iterator->next()) {
-        const Rdata& rdata(rdata_iterator->getCurrent());
-        if (rrset.getType() == RRType::NS()) {
-            // Need to perform the search in the "GLUE OK" mode.
-            const generic::NS& ns = dynamic_cast<const generic::NS&>(rdata);
-            addAdditionalAddrs(zone, ns.getNSName(), ZoneFinder::FIND_GLUE_OK);
-        } else if (rrset.getType() == RRType::MX()) {
-            const generic::MX& mx(dynamic_cast<const generic::MX&>(rdata));
-            addAdditionalAddrs(zone, mx.getMXName());
-        }
+Query::ResponseCreator::addRRset(isc::dns::Message& message,
+                                 const isc::dns::Message::Section section,
+                                 const ConstRRsetPtr& rrset, const bool dnssec)
+{
+    /// Is this RRset already in the list of RRsets added to the message?
+    const std::vector<const AbstractRRset*>::const_iterator i =
+        std::find_if(added_.begin(), added_.end(),
+                     std::bind1st(Query::ResponseCreator::IsSameKind(),
+                                  rrset.get()));
+    if (i == added_.end()) {
+        // No - add it to both the message and the list of RRsets processed.
+        // The const-cast is wrong, but the message interface seems to insist.
+        message.addRRset(section,
+                         boost::const_pointer_cast<AbstractRRset>(rrset),
+                         dnssec);
+        added_.push_back(rrset.get());
     }
 }
 
 void
-Query::addAdditionalAddrs(ZoneFinder& zone, const Name& qname,
-                          const ZoneFinder::FindOptions options)
+Query::ResponseCreator::create(Message& response,
+                               const vector<ConstRRsetPtr>& answers,
+                               const vector<ConstRRsetPtr>& authorities,
+                               const vector<ConstRRsetPtr>& additionals,
+                               const bool dnssec)
 {
-    // Out of zone name
-    NameComparisonResult result = zone.getOrigin().compare(qname);
-    if ((result.getRelation() != NameComparisonResult::SUPERDOMAIN) &&
-        (result.getRelation() != NameComparisonResult::EQUAL))
-        return;
-
-    // Omit additional data which has already been provided in the answer
-    // section from the additional.
-    //
-    // All the address rrset with the owner name of qname have been inserted
-    // into ANSWER section.
-    if (qname_ == qname && qtype_ == RRType::ANY())
-        return;
-
-    // Find A rrset
-    if (qname_ != qname || qtype_ != RRType::A()) {
-        ZoneFinder::FindResult a_result = zone.find(qname, RRType::A(),
-                                                    options | dnssec_opt_);
-        if (a_result.code == ZoneFinder::SUCCESS) {
-            response_.addRRset(Message::SECTION_ADDITIONAL,
-                    boost::const_pointer_cast<AbstractRRset>(a_result.rrset), dnssec_);
-        }
+    // Inserter should be reset each time the query is reset, so should be
+    // empty at this point.
+    assert(added_.empty());
+
+    // Add the RRsets to the message.  The order of sections is important,
+    // as the ResponseCreator remembers RRsets added and will not add
+    // duplicates.  Adding in the order answer, authory, additional will
+    // guarantee that if there are duplicates, the single RRset added will
+    // appear in the most important section.
+    BOOST_FOREACH(const ConstRRsetPtr& rrset, answers) {
+        addRRset(response, Message::SECTION_ANSWER, rrset, dnssec);
     }
-
-    // Find AAAA rrset
-    if (qname_ != qname || qtype_ != RRType::AAAA()) {
-        ZoneFinder::FindResult aaaa_result = zone.find(qname, RRType::AAAA(),
-                                                       options | dnssec_opt_);
-        if (aaaa_result.code == ZoneFinder::SUCCESS) {
-            response_.addRRset(Message::SECTION_ADDITIONAL,
-                    boost::const_pointer_cast<AbstractRRset>(aaaa_result.rrset),
-                    dnssec_);
-        }
+    BOOST_FOREACH(const ConstRRsetPtr& rrset, authorities) {
+        addRRset(response, Message::SECTION_AUTHORITY, rrset, dnssec);
+    }
+    BOOST_FOREACH(const ConstRRsetPtr& rrset, additionals) {
+        addRRset(response, Message::SECTION_ADDITIONAL, rrset, dnssec);
     }
 }
 
 void
 Query::addSOA(ZoneFinder& finder) {
-    ZoneFinder::FindResult soa_result = finder.find(finder.getOrigin(),
-                                                    RRType::SOA(),
-                                                    dnssec_opt_);
-    if (soa_result.code != ZoneFinder::SUCCESS) {
+    ZoneFinderContextPtr soa_ctx = finder.find(finder.getOrigin(),
+                                               RRType::SOA(), dnssec_opt_);
+    if (soa_ctx->code != ZoneFinder::SUCCESS) {
         isc_throw(NoSOA, "There's no SOA record in zone " <<
             finder.getOrigin().toText());
     } else {
-        /*
-         * FIXME:
-         * The const-cast is wrong, but the Message interface seems
-         * to insist.
-         */
-        response_.addRRset(Message::SECTION_AUTHORITY,
-            boost::const_pointer_cast<AbstractRRset>(soa_result.rrset), dnssec_);
+        authorities_.push_back(soa_ctx->rrset);
     }
 }
 
@@ -123,8 +127,7 @@ Query::addNXDOMAINProofByNSEC(ZoneFinder& finder, ConstRRsetPtr nsec) {
     }
 
     // Add the NSEC proving NXDOMAIN to the authority section.
-    response_.addRRset(Message::SECTION_AUTHORITY,
-                       boost::const_pointer_cast<AbstractRRset>(nsec), dnssec_);
+    authorities_.push_back(nsec);
 
     // Next, identify the best possible wildcard name that would match
     // the query name.  It's the longer common suffix with the qname
@@ -135,109 +138,111 @@ Query::addNXDOMAINProofByNSEC(ZoneFinder& finder, ConstRRsetPtr nsec) {
     // and the best possible wildcard is *.b.example.com.  If the NXDOMAIN
     // NSEC is a.example.com. NSEC c.b.example.com., the longer suffix
     // is the next domain of the NSEC, and we get the same wildcard name.
-    const int qlabels = qname_.getLabelCount();
-    const int olabels = qname_.compare(nsec->getName()).getCommonLabels();
-    const int nlabels = qname_.compare(
+    const int qlabels = qname_->getLabelCount();
+    const int olabels = qname_->compare(nsec->getName()).getCommonLabels();
+    const int nlabels = qname_->compare(
         dynamic_cast<const generic::NSEC&>(nsec->getRdataIterator()->
                                            getCurrent()).
         getNextName()).getCommonLabels();
     const int common_labels = std::max(olabels, nlabels);
-    const Name wildname(Name("*").concatenate(qname_.split(qlabels -
+    const Name wildname(Name("*").concatenate(qname_->split(qlabels -
                                                            common_labels)));
 
     // Confirm the wildcard doesn't exist (this should result in NXDOMAIN;
     // otherwise we shouldn't have got NXDOMAIN for the original query in
     // the first place).
-    const ZoneFinder::FindResult fresult =
+    ConstZoneFinderContextPtr fcontext =
         finder.find(wildname, RRType::NSEC(), dnssec_opt_);
-    if (fresult.code != ZoneFinder::NXDOMAIN || !fresult.rrset ||
-        fresult.rrset->getRdataCount() == 0) {
+    if (fcontext->code != ZoneFinder::NXDOMAIN || !fcontext->rrset ||
+        fcontext->rrset->getRdataCount() == 0) {
         isc_throw(BadNSEC, "Unexpected result for wildcard NXDOMAIN proof");
     }
 
-    // Add the (no-) wildcard proof only when it's different from the NSEC
-    // that proves NXDOMAIN; sometimes they can be the same.
-    // Note: name comparison is relatively expensive.  When we are at the
-    // stage of performance optimization, we should consider optimizing this
-    // for some optimized data source implementations.
-    if (nsec->getName() != fresult.rrset->getName()) {
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(fresult.rrset),
-                           dnssec_);
+    // Add the (no-) wildcard proof.  This can be the same NSEC we already
+    // added, but we'd add it here anyway; duplicate checks will take place
+    // later in a unified manner.
+    authorities_.push_back(fcontext->rrset);
+}
+
+uint8_t
+Query::addClosestEncloserProof(ZoneFinder& finder, const Name& name,
+                               bool exact_ok, bool add_closest)
+{
+    const ZoneFinder::FindNSEC3Result result = finder.findNSEC3(name, true);
+
+    // Validity check (see the method description).  Note that a completely
+    // broken findNSEC3 implementation could even return NULL RRset in
+    // closest_proof.  We don't explicitly check such case; addRRset() will
+    // throw an exception, and it will be converted to SERVFAIL at the caller.
+    if (!exact_ok && !result.next_proof) {
+        isc_throw(BadNSEC3, "Matching NSEC3 found for a non existent name: "
+                  << qname_);
+    }
+
+    if (add_closest) {
+        authorities_.push_back(result.closest_proof);
     }
+    if (result.next_proof) {
+        authorities_.push_back(result.next_proof);
+    }
+    return (result.closest_labels);
+}
+
+void
+Query::addNSEC3ForName(ZoneFinder& finder, const Name& name, bool match) {
+    const ZoneFinder::FindNSEC3Result result = finder.findNSEC3(name, false);
+
+    // See the comment for addClosestEncloserProof().  We don't check a
+    // totally bogus case where closest_proof is NULL here.
+    if (match != result.matched) {
+        isc_throw(BadNSEC3, "Unexpected "
+                  << (result.matched ? "matching" : "covering")
+                  << " NSEC3 found for " << name);
+    }
+    authorities_.push_back(result.closest_proof);
 }
 
 void
 Query::addNXDOMAINProofByNSEC3(ZoneFinder& finder) {
     // Firstly get the NSEC3 proves for Closest Encloser Proof
-    // See section 7.2.1 of RFC 5155.
-    // Since this is a Name Error case both closest and next proofs should
-    // be available (see addNXRRsetProof).
-    const ZoneFinder::FindNSEC3Result fresult1 = finder.findNSEC3(qname_,
-                                                                  true);
-    response_.addRRset(Message::SECTION_AUTHORITY,
-                       boost::const_pointer_cast<AbstractRRset>(
-                           fresult1.closest_proof),
-                       dnssec_);
-    response_.addRRset(Message::SECTION_AUTHORITY,
-                       boost::const_pointer_cast<AbstractRRset>(
-                       fresult1.next_proof),
-                       dnssec_);
+    // See Section 7.2.1 of RFC 5155.
+    const uint8_t closest_labels =
+        addClosestEncloserProof(finder, *qname_, false);
 
     // Next, construct the wildcard name at the closest encloser, i.e.,
     // '*' followed by the closest encloser, and add NSEC3 for it.
     const Name wildname(Name("*").concatenate(
-               qname_.split(qname_.getLabelCount() -
-                            fresult1.closest_labels)));
-    const ZoneFinder::FindNSEC3Result fresult2 =
-        finder.findNSEC3(wildname, false);
-    if (fresult2.matched) {
-        isc_throw(BadNSEC3, "Matching NSEC3 found for nonexistent domain "
-                  << wildname);
-    }
-    response_.addRRset(Message::SECTION_AUTHORITY,
-                       boost::const_pointer_cast<AbstractRRset>(
-                           fresult2.closest_proof),
-                       dnssec_);
+               qname_->split(qname_->getLabelCount() - closest_labels)));
+    addNSEC3ForName(finder, wildname, false);
 }
 
 void
 Query::addWildcardProof(ZoneFinder& finder,
-                        const ZoneFinder::FindResult& db_result)
+                        const ZoneFinder::Context& db_context)
 {
-    if (db_result.isNSECSigned()) {
+    if (db_context.isNSECSigned()) {
         // Case for RFC4035 Section 3.1.3.3.
         //
         // The query name shouldn't exist in the zone if there were no wildcard
         // substitution.  Confirm that by specifying NO_WILDCARD.  It should
         // result in NXDOMAIN and an NSEC RR that proves it should be returned.
-        const ZoneFinder::FindResult fresult =
-            finder.find(qname_, RRType::NSEC(),
+        ConstZoneFinderContextPtr fcontext =
+            finder.find(*qname_, RRType::NSEC(),
                         dnssec_opt_ | ZoneFinder::NO_WILDCARD);
-        if (fresult.code != ZoneFinder::NXDOMAIN || !fresult.rrset ||
-            fresult.rrset->getRdataCount() == 0) {
+        if (fcontext->code != ZoneFinder::NXDOMAIN || !fcontext->rrset ||
+            fcontext->rrset->getRdataCount() == 0) {
             isc_throw(BadNSEC,
                       "Unexpected NSEC result for wildcard proof");
         }
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               fresult.rrset),
-                           dnssec_);
-    } else if (db_result.isNSEC3Signed()) {
-        // Case for RFC5155 Section 7.2.6.
+        authorities_.push_back(fcontext->rrset);
+    } else if (db_context.isNSEC3Signed()) {
+        // Case for RFC 5155 Section 7.2.6.
         //
         // Note that the closest encloser must be the immediate ancestor
-        // of the matching wildcard, so NSEC3 for its next closer is what
-        // we are expected to provided per the RFC (if this assumption isn't
-        // met the zone is broken anyway).
-        const ZoneFinder::FindNSEC3Result NSEC3Result(
-            finder.findNSEC3(qname_, true));
-        // Note that at this point next_proof must not be NULL unless it's
-        // a run time collision (or zone/findNSEC3() is broken).  The
-        // unexpected case will be caught in addRRset() and result in SERVFAIL.
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               NSEC3Result.next_proof), dnssec_);
+        // of the matching wildcard, so NSEC3 for its next closer (and only
+        // that NSEC3) is what we are expected to provided per the RFC (if
+        // this assumption isn't met the zone is broken anyway).
+        addClosestEncloserProof(finder, *qname_, false, false);
     }
 }
 
@@ -249,142 +254,84 @@ Query::addWildcardNXRRSETProof(ZoneFinder& finder, ConstRRsetPtr nsec) {
         isc_throw(BadNSEC, "NSEC for WILDCARD_NXRRSET is empty");
     }
     
-    const ZoneFinder::FindResult fresult =
-        finder.find(qname_, RRType::NSEC(),
+    ConstZoneFinderContextPtr fcontext =
+        finder.find(*qname_, RRType::NSEC(),
                     dnssec_opt_ | ZoneFinder::NO_WILDCARD);
-    if (fresult.code != ZoneFinder::NXDOMAIN || !fresult.rrset ||
-        fresult.rrset->getRdataCount() == 0) {
+    if (fcontext->code != ZoneFinder::NXDOMAIN || !fcontext->rrset ||
+        fcontext->rrset->getRdataCount() == 0) {
         isc_throw(BadNSEC, "Unexpected result for no match QNAME proof");
     }
-   
-    if (nsec->getName() != fresult.rrset->getName()) {
-        // one NSEC RR proves wildcard_nxrrset that no matched QNAME.
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(fresult.rrset),
-                           dnssec_);
-    }
+
+    authorities_.push_back(fcontext->rrset);
 }
 
 void
 Query::addDS(ZoneFinder& finder, const Name& dname) {
-    ZoneFinder::FindResult ds_result =
+    ConstZoneFinderContextPtr ds_context =
         finder.find(dname, RRType::DS(), dnssec_opt_);
-    if (ds_result.code == ZoneFinder::SUCCESS) {
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               ds_result.rrset),
-                           dnssec_);
-    } else if (ds_result.code == ZoneFinder::NXRRSET &&
-               ds_result.isNSECSigned()) {
-        addNXRRsetProof(finder, ds_result);
-    } else if (ds_result.code == ZoneFinder::NXRRSET &&
-               ds_result.isNSEC3Signed()) {
-        // Add no DS proof with NSEC3 as specified in RFC5155 Section 7.2.7.
-        // Depending on whether the zone is optout or not, findNSEC3() may
-        // return non-NULL or NULL next_proof (respectively).  The Opt-Out flag
-        // must be set or cleared accordingly, but we don't check that
-        // in this level (as long as the zone signed validly and findNSEC3()
-        // is valid, the condition should be met; otherwise we'd let the
-        // validator detect the error).
-        const ZoneFinder::FindNSEC3Result nsec3_result =
-            finder.findNSEC3(dname, true);
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               nsec3_result.closest_proof), dnssec_);
-        if (nsec3_result.next_proof) {
-            response_.addRRset(Message::SECTION_AUTHORITY,
-                               boost::const_pointer_cast<AbstractRRset>(
-                                   nsec3_result.next_proof), dnssec_);
-        }
-    } else {
-        // Any other case should be an error
+    if (ds_context->code == ZoneFinder::SUCCESS) {
+        authorities_.push_back(ds_context->rrset);
+    } else if (ds_context->code == ZoneFinder::NXRRSET &&
+               ds_context->isNSECSigned()) {
+        addNXRRsetProof(finder, *ds_context);
+    } else if (ds_context->code == ZoneFinder::NXRRSET &&
+               ds_context->isNSEC3Signed()) {
+        // Add no DS proof with NSEC3 as specified in RFC 5155 Section 7.2.7.
+        addClosestEncloserProof(finder, dname, true);
+    } else if (ds_context->code != ZoneFinder::NXRRSET) {
+        // We know this domain should exist, so the result must be NXRRSET.
+        // If not, the zone is broken, so we'll return SERVFAIL by triggering
+        // an exception.
         isc_throw(BadDS, "Unexpected result for DS lookup for delegation");
     }
 }
 
 void
 Query::addNXRRsetProof(ZoneFinder& finder,
-                       const ZoneFinder::FindResult& db_result)
+                       const ZoneFinder::Context& db_context)
 {
-    if (db_result.isNSECSigned() && db_result.rrset) {
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               db_result.rrset),
-                           dnssec_);
-        if (db_result.isWildcard()) {
-            addWildcardNXRRSETProof(finder, db_result.rrset);
+    if (db_context.isNSECSigned() && db_context.rrset) {
+        authorities_.push_back(db_context.rrset);
+        if (db_context.isWildcard()) {
+            addWildcardNXRRSETProof(finder, db_context.rrset);
         }
-    } else if (db_result.isNSEC3Signed() && !db_result.isWildcard()) {
-        // Handling depends on whether query type is DS or not
-        // (see RFC5155, 7.2.3 and 7.2.4):  If qtype == DS, do
-        // recursive search (and add next_proof, if necessary),
-        // otherwise, do non-recursive search
-        const bool qtype_ds = (qtype_ == RRType::DS());
-        ZoneFinder::FindNSEC3Result result(finder.findNSEC3(qname_, qtype_ds));
-        if (result.matched) {
-            response_.addRRset(Message::SECTION_AUTHORITY,
-                               boost::const_pointer_cast<AbstractRRset>(
-                                   result.closest_proof), dnssec_);
-            // For qtype == DS, next_proof could be set
-            // (We could check for opt-out here, but that's really the
-            // responsibility of the datasource)
-            if (qtype_ds && result.next_proof != ConstRRsetPtr()) {
-                response_.addRRset(Message::SECTION_AUTHORITY,
-                                   boost::const_pointer_cast<AbstractRRset>(
-                                       result.next_proof), dnssec_);
-            }
+    } else if (db_context.isNSEC3Signed() && !db_context.isWildcard()) {
+        if (*qtype_ == RRType::DS()) {
+            // RFC 5155, Section 7.2.4.  Add either NSEC3 for the qname or
+            // closest (provable) encloser proof in case of optout.
+            addClosestEncloserProof(finder, *qname_, true);
         } else {
-            isc_throw(BadNSEC3, "No matching NSEC3 found for existing domain "
-                      << qname_);
+            // RFC 5155, Section 7.2.3.  Just add NSEC3 for the qname.
+            addNSEC3ForName(finder, *qname_, true);
         }
-    } else if (db_result.isNSEC3Signed() && db_result.isWildcard()) {
-        // Case for RFC5155 Section 7.2.5
-        const ZoneFinder::FindNSEC3Result result(finder.findNSEC3(qname_,
-                                                                  true));
-        // We know there's no exact match for the qname, so findNSEC3() should
-        // return both closest and next proofs.  If the latter is NULL, it
-        // means a run time collision (or the zone is broken in other way).
-        // In that case addRRset() will throw, and it will be converted to
-        // SERVFAIL.
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               result.closest_proof), dnssec_);
-        response_.addRRset(Message::SECTION_AUTHORITY,
-                           boost::const_pointer_cast<AbstractRRset>(
-                               result.next_proof), dnssec_);
-
-        // Construct the matched wildcard name and add NSEC3 for it.
+    } else if (db_context.isNSEC3Signed() && db_context.isWildcard()) {
+        // Case for RFC 5155 Section 7.2.5: add closest encloser proof for the
+        // qname, construct the matched wildcard name and add NSEC3 for it.
+        const uint8_t closest_labels =
+            addClosestEncloserProof(finder, *qname_, false);
         const Name wname = Name("*").concatenate(
-            qname_.split(qname_.getLabelCount() - result.closest_labels));
-        const ZoneFinder::FindNSEC3Result wresult(finder.findNSEC3(wname,
-                                                                   false));
-        if (wresult.matched) {
-            response_.addRRset(Message::SECTION_AUTHORITY,
-                               boost::const_pointer_cast<AbstractRRset>(
-                                   wresult.closest_proof), dnssec_);
-        } else {
-            isc_throw(BadNSEC3, "No matching NSEC3 found for existing domain "
-                      << wname);
-        }
+            qname_->split(qname_->getLabelCount() - closest_labels));
+        addNSEC3ForName(finder, wname, true);
     }
 }
 
 void
-Query::addAuthAdditional(ZoneFinder& finder) {
+Query::addAuthAdditional(ZoneFinder& finder,
+                         vector<ConstRRsetPtr>& additionals)
+{
+    const Name& origin = finder.getOrigin();
+
     // Fill in authority and addtional sections.
-    ZoneFinder::FindResult ns_result =
-        finder.find(finder.getOrigin(), RRType::NS(), dnssec_opt_);
+    ConstZoneFinderContextPtr ns_context = finder.find(origin, RRType::NS(),
+                                                       dnssec_opt_);
 
     // zone origin name should have NS records
-    if (ns_result.code != ZoneFinder::SUCCESS) {
+    if (ns_context->code != ZoneFinder::SUCCESS) {
         isc_throw(NoApexNS, "There's no apex NS records in zone " <<
-                finder.getOrigin().toText());
-    } else {
-        response_.addRRset(Message::SECTION_AUTHORITY,
-            boost::const_pointer_cast<AbstractRRset>(ns_result.rrset), dnssec_);
-        // Handle additional for authority section
-        addAdditional(finder, *ns_result.rrset);
+                  finder.getOrigin().toText());
     }
+    authorities_.push_back(ns_context->rrset);
+    ns_context->getAdditional(A_AND_AAAA(), additionals);
 }
 
 namespace {
@@ -404,10 +351,20 @@ findZone(const DataSourceClient& client, const Name& qname, RRType qtype) {
 }
 
 void
-Query::process() {
+Query::process(datasrc::DataSourceClient& datasrc_client,
+               const isc::dns::Name& qname, const isc::dns::RRType& qtype,
+               isc::dns::Message& response, bool dnssec)
+{
+    // Set up the cleaner object so internal pointers and vectors are
+    // always reset after scope leaves this method
+    QueryCleaner cleaner(*this);
+
+    // Set up query parameters for the rest of the (internal) methods
+    initialize(datasrc_client, qname, qtype, response, dnssec);
+
     // Found a zone which is the nearest ancestor to QNAME
-    const DataSourceClient::FindResult result = findZone(datasrc_client_,
-                                                         qname_, qtype_);
+    const DataSourceClient::FindResult result = findZone(*datasrc_client_,
+                                                         *qname_, *qtype_);
 
     // If we have no matching authoritative zone for the query name, return
     // REFUSED.  In short, this is to be compatible with BIND 9, but the
@@ -419,51 +376,48 @@ Query::process() {
         // If we tried to find a "parent zone" for a DS query and failed,
         // we may still have authority at the child side.  If we do, the query
         // has to be handled there.
-        if (qtype_ == RRType::DS() && qname_.getLabelCount() > 1 &&
+        if (*qtype_ == RRType::DS() && qname_->getLabelCount() > 1 &&
             processDSAtChild()) {
             return;
         }
-        response_.setHeaderFlag(Message::HEADERFLAG_AA, false);
-        response_.setRcode(Rcode::REFUSED());
+        response_->setHeaderFlag(Message::HEADERFLAG_AA, false);
+        response_->setRcode(Rcode::REFUSED());
         return;
     }
     ZoneFinder& zfinder = *result.zone_finder;
 
     // We have authority for a zone that contain the query name (possibly
     // indirectly via delegation).  Look into the zone.
-    response_.setHeaderFlag(Message::HEADERFLAG_AA);
-    response_.setRcode(Rcode::NOERROR());
-    std::vector<ConstRRsetPtr> target;
-    boost::function0<ZoneFinder::FindResult> find;
-    const bool qtype_is_any = (qtype_ == RRType::ANY());
+    response_->setHeaderFlag(Message::HEADERFLAG_AA);
+    response_->setRcode(Rcode::NOERROR());
+    boost::function0<ZoneFinderContextPtr> find;
+    const bool qtype_is_any = (*qtype_ == RRType::ANY());
     if (qtype_is_any) {
-        find = boost::bind(&ZoneFinder::findAll, &zfinder, qname_,
-                           boost::ref(target), dnssec_opt_);
+        find = boost::bind(&ZoneFinder::findAll, &zfinder, *qname_,
+                           boost::ref(answers_), dnssec_opt_);
     } else {
-        find = boost::bind(&ZoneFinder::find, &zfinder, qname_, qtype_,
+        find = boost::bind(&ZoneFinder::find, &zfinder, *qname_, *qtype_,
                            dnssec_opt_);
     }
-    ZoneFinder::FindResult db_result(find());
-    switch (db_result.code) {
+    ZoneFinderContextPtr db_context(find());
+    switch (db_context->code) {
         case ZoneFinder::DNAME: {
             // First, put the dname into the answer
-            response_.addRRset(Message::SECTION_ANSWER,
-                boost::const_pointer_cast<AbstractRRset>(db_result.rrset),
-                dnssec_);
+            answers_.push_back(db_context->rrset);
             /*
              * Empty DNAME should never get in, as it is impossible to
              * create one in master file.
              *
              * FIXME: Other way to prevent this should be done
              */
-            assert(db_result.rrset->getRdataCount() > 0);
+            assert(db_context->rrset->getRdataCount() > 0);
             // Get the data of DNAME
             const rdata::generic::DNAME& dname(
                 dynamic_cast<const rdata::generic::DNAME&>(
-                db_result.rrset->getRdataIterator()->getCurrent()));
+                db_context->rrset->getRdataIterator()->getCurrent()));
             // The yet unmatched prefix dname
-            const Name prefix(qname_.split(0, qname_.getLabelCount() -
-                db_result.rrset->getName().getLabelCount()));
+            const Name prefix(qname_->split(0, qname_->getLabelCount() -
+                db_context->rrset->getName().getLabelCount()));
             // If we put it together, will it be too long?
             // (The prefix contains trailing ., which will be removed
             if (prefix.getLength() - Name::ROOT_NAME().getLength() +
@@ -472,20 +426,20 @@ Query::process() {
                  * In case the synthesized name is too long, section 4.1
                  * of RFC 2672 mandates we return YXDOMAIN.
                  */
-                response_.setRcode(Rcode::YXDOMAIN());
-                return;
+                response_->setRcode(Rcode::YXDOMAIN());
+                break;
             }
             // The new CNAME we are creating (it will be unsigned even
             // with DNSSEC, the DNAME is signed and it can be validated
             // by that)
-            RRsetPtr cname(new RRset(qname_, db_result.rrset->getClass(),
-                RRType::CNAME(), db_result.rrset->getTTL()));
+            RRsetPtr cname(new RRset(*qname_, db_context->rrset->getClass(),
+                RRType::CNAME(), db_context->rrset->getTTL()));
             // Construct the new target by replacing the end
-            cname->addRdata(rdata::generic::CNAME(qname_.split(0,
-                qname_.getLabelCount() -
-                db_result.rrset->getName().getLabelCount()).
+            cname->addRdata(rdata::generic::CNAME(qname_->split(0,
+                qname_->getLabelCount() -
+                db_context->rrset->getName().getLabelCount()).
                 concatenate(dname.getDname())));
-            response_.addRRset(Message::SECTION_ANSWER, cname, dnssec_);
+            answers_.push_back(cname);
             break;
         }
         case ZoneFinder::CNAME:
@@ -498,48 +452,40 @@ Query::process() {
              *
              * So, just put it there.
              */
-            response_.addRRset(Message::SECTION_ANSWER,
-                boost::const_pointer_cast<AbstractRRset>(db_result.rrset),
-                dnssec_);
+            answers_.push_back(db_context->rrset);
 
             // If the answer is a result of wildcard substitution,
             // add a proof that there's no closer name.
-            if (dnssec_ && db_result.isWildcard()) {
-                addWildcardProof(*result.zone_finder,db_result);
+            if (dnssec_ && db_context->isWildcard()) {
+                addWildcardProof(*result.zone_finder, *db_context);
             }
             break;
         case ZoneFinder::SUCCESS:
-            if (qtype_is_any) {
-                // If quety type is ANY, insert all RRs under the domain
-                // into answer section.
-                BOOST_FOREACH(ConstRRsetPtr rrset, target) {
-                    response_.addRRset(Message::SECTION_ANSWER,
-                        boost::const_pointer_cast<AbstractRRset>(rrset), dnssec_);
-                    // Handle additional for answer section
-                    addAdditional(*result.zone_finder, *rrset.get());
-                }
-            } else {
-                response_.addRRset(Message::SECTION_ANSWER,
-                    boost::const_pointer_cast<AbstractRRset>(db_result.rrset),
-                    dnssec_);
-                // Handle additional for answer section
-                addAdditional(*result.zone_finder, *db_result.rrset);
+            // If query type is ANY, the rrs have already been added
+            if (!qtype_is_any) {
+                answers_.push_back(db_context->rrset);
             }
+
+            // Retrieve additional records for the answer
+            db_context->getAdditional(A_AND_AAAA(), additionals_);
+
             // If apex NS records haven't been provided in the answer
             // section, insert apex NS records into the authority section
             // and AAAA/A RRS of each of the NS RDATA into the additional
             // section.
-            if (qname_ != result.zone_finder->getOrigin() ||
-                db_result.code != ZoneFinder::SUCCESS ||
-                (qtype_ != RRType::NS() && !qtype_is_any))
+            // Checking the findZone() is a lightweight check to see if
+            // qname is the zone origin.
+            if (result.code != result::SUCCESS ||
+                db_context->code != ZoneFinder::SUCCESS ||
+                (*qtype_ != RRType::NS() && !qtype_is_any))
             {
-                addAuthAdditional(*result.zone_finder);
+                addAuthAdditional(*result.zone_finder, additionals_);
             }
 
             // If the answer is a result of wildcard substitution,
             // add a proof that there's no closer name.
-            if (dnssec_ && db_result.isWildcard()) {
-                addWildcardProof(*result.zone_finder,db_result);
+            if (dnssec_ && db_context->isWildcard()) {
+                addWildcardProof(*result.zone_finder, *db_context);
             }
             break;
         case ZoneFinder::DELEGATION:
@@ -547,28 +493,28 @@ Query::process() {
             // if we are an authority of the child, too.  If so, we need to
             // complete the process in the child as specified in Section
             // 2.2.1.2. of RFC3658.
-            if (qtype_ == RRType::DS() && processDSAtChild()) {
+            if (*qtype_ == RRType::DS() && processDSAtChild()) {
                 return;
             }
 
-            response_.setHeaderFlag(Message::HEADERFLAG_AA, false);
-            response_.addRRset(Message::SECTION_AUTHORITY,
-                boost::const_pointer_cast<AbstractRRset>(db_result.rrset),
-                dnssec_);
+            response_->setHeaderFlag(Message::HEADERFLAG_AA, false);
+            authorities_.push_back(db_context->rrset);
+            // Retrieve additional records for the name servers
+            db_context->getAdditional(A_AND_AAAA(), additionals_);
+
             // If DNSSEC is requested, see whether there is a DS
             // record for this delegation.
             if (dnssec_) {
-                addDS(*result.zone_finder, db_result.rrset->getName());
+                addDS(*result.zone_finder, db_context->rrset->getName());
             }
-            addAdditional(*result.zone_finder, *db_result.rrset);
             break;
         case ZoneFinder::NXDOMAIN:
-            response_.setRcode(Rcode::NXDOMAIN());
+            response_->setRcode(Rcode::NXDOMAIN());
             addSOA(*result.zone_finder);
             if (dnssec_) {
-                if (db_result.isNSECSigned() && db_result.rrset) {
-                    addNXDOMAINProofByNSEC(zfinder, db_result.rrset);
-                } else if (db_result.isNSEC3Signed()) {
+                if (db_context->isNSECSigned() && db_context->rrset) {
+                    addNXDOMAINProofByNSEC(zfinder, db_context->rrset);
+                } else if (db_context->isNSEC3Signed()) {
                     addNXDOMAINProofByNSEC3(zfinder);
                 }
             }
@@ -576,7 +522,7 @@ Query::process() {
         case ZoneFinder::NXRRSET:
             addSOA(*result.zone_finder);
             if (dnssec_) {
-                addNXRRsetProof(zfinder, db_result);
+                addNXRRsetProof(zfinder, *db_context);
             }
             break;
         default:
@@ -586,12 +532,41 @@ Query::process() {
             isc_throw(isc::NotImplemented, "Unknown result code");
             break;
     }
+
+    response_creator_.create(*response_, answers_, authorities_, additionals_,
+                             dnssec_);
+}
+
+void
+Query::initialize(datasrc::DataSourceClient& datasrc_client,
+                  const isc::dns::Name& qname, const isc::dns::RRType& qtype,
+                  isc::dns::Message& response, bool dnssec)
+{
+    datasrc_client_ = &datasrc_client;
+    qname_ = &qname;
+    qtype_ = &qtype;
+    response_ = &response;
+    dnssec_ = dnssec;
+    dnssec_opt_ = (dnssec ? isc::datasrc::ZoneFinder::FIND_DNSSEC :
+                   isc::datasrc::ZoneFinder::FIND_DEFAULT);
+}
+
+void
+Query::reset() {
+    datasrc_client_ = NULL;
+    qname_ = NULL;
+    qtype_ = NULL;
+    response_ = NULL;
+    answers_.clear();
+    authorities_.clear();
+    additionals_.clear();
+    response_creator_.clear();
 }
 
 bool
 Query::processDSAtChild() {
     const DataSourceClient::FindResult zresult =
-        datasrc_client_.findZone(qname_);
+        datasrc_client_->findZone(*qname_);
 
     if (zresult.code != result::SUCCESS) {
         return (false);
@@ -606,17 +581,19 @@ Query::processDSAtChild() {
     // The important point in this case is to return SOA so that the resolver
     // that happens to contact us can hunt for the appropriate parent zone
     // by seeing the SOA.
-    response_.setHeaderFlag(Message::HEADERFLAG_AA);
-    response_.setRcode(Rcode::NOERROR());
+    response_->setHeaderFlag(Message::HEADERFLAG_AA);
+    response_->setRcode(Rcode::NOERROR());
     addSOA(*zresult.zone_finder);
-    const ZoneFinder::FindResult ds_result =
-        zresult.zone_finder->find(qname_, RRType::DS(), dnssec_opt_);
-    if (ds_result.code == ZoneFinder::NXRRSET) {
+    ConstZoneFinderContextPtr ds_context =
+        zresult.zone_finder->find(*qname_, RRType::DS(), dnssec_opt_);
+    if (ds_context->code == ZoneFinder::NXRRSET) {
         if (dnssec_) {
-            addNXRRsetProof(*zresult.zone_finder, ds_result);
+            addNXRRsetProof(*zresult.zone_finder, *ds_context);
         }
     }
 
+    response_creator_.create(*response_, answers_, authorities_, additionals_,
+                             dnssec_);
     return (true);
 }
 

src/bin/auth/query.h

@@ -15,8 +15,14 @@
  */
 
 #include <exceptions/exceptions.h>
+#include <dns/rrset.h>
 #include <datasrc/zone.h>
 
+#include <boost/noncopyable.hpp>
+
+#include <functional>
+#include <vector>
+
 namespace isc {
 namespace dns {
 class Message;
@@ -61,8 +67,14 @@ namespace auth {
 /// likely to misuse one of the classes instead of the other
 /// accidentally, and since it's considered a temporary development state,
 /// we keep this name at the moment.
-class Query {
+class Query : boost::noncopyable {
 private:
+    /// \brief Initial reserved size for the vectors in Query
+    ///
+    /// The value is larger than we expect the vectors to even become, and
+    /// has been chosen arbitrarily. The reason to set them quite high is
+    /// to prevent reallocation on addition.
+    static const size_t RESERVE_RRSETS = 64;
 
     /// \brief Adds a SOA.
     ///
@@ -96,7 +108,7 @@ private:
     ///               data
     /// \param db_result The ZoneFinder::FindResult returned by find()
     void addNXRRsetProof(isc::datasrc::ZoneFinder& finder,
-        const isc::datasrc::ZoneFinder::FindResult& db_result);
+                         const isc::datasrc::ZoneFinder::Context& db_context);
 
     /// Add NSEC RRs that prove an NXDOMAIN result.
     ///
@@ -115,7 +127,7 @@ private:
     /// of RFC5155.
     void addWildcardProof(
         isc::datasrc::ZoneFinder& finder,
-        const isc::datasrc::ZoneFinder::FindResult& dbResult);
+        const isc::datasrc::ZoneFinder::Context& db_context);
 
     /// \brief Adds one NSEC RR proved no matched QNAME,one NSEC RR proved no
     /// matched <QNAME,QTYPE> through wildcard extension.
@@ -129,44 +141,6 @@ private:
     void addWildcardNXRRSETProof(isc::datasrc::ZoneFinder& finder,
                                  isc::dns::ConstRRsetPtr nsec);
 
-    /// \brief Look up additional data (i.e., address records for the names
-    /// included in NS or MX records) and add them to the additional section.
-    ///
-    /// Note: Any additional data which has already been provided in the
-    /// answer section (i.e., if the original query happend to be for the
-    /// address of the DNS server), it should be omitted from the additional.
-    ///
-    /// This method may throw a exception because its underlying methods may
-    /// throw exceptions.
-    ///
-    /// \param zone The ZoneFinder through which the additional data for the
-    /// query is to be found.
-    /// \param rrset The RRset (i.e., NS or MX rrset) which require additional
-    /// processing.
-    void addAdditional(isc::datasrc::ZoneFinder& zone,
-                       const isc::dns::AbstractRRset& rrset);
-
-    /// \brief Find address records for a specified name.
-    ///
-    /// Search the specified zone for AAAA/A RRs of each of the NS/MX RDATA
-    /// (domain name), and insert the found ones into the additional section
-    /// if address records are available. By default the search will stop
-    /// once it encounters a zone cut.
-    ///
-    /// Note: we need to perform the search in the "GLUE OK" mode for NS RDATA,
-    /// which means that we should include A/AAAA RRs under a zone cut.
-    /// The glue records must exactly match the name in the NS RDATA, without
-    /// CNAME or wildcard processing.
-    ///
-    /// \param zone The \c ZoneFinder through which the address records is to
-    /// be found.
-    /// \param qname The name in rrset RDATA.
-    /// \param options The search options.
-    void addAdditionalAddrs(isc::datasrc::ZoneFinder& zone,
-                            const isc::dns::Name& qname,
-                            const isc::datasrc::ZoneFinder::FindOptions options
-                            = isc::datasrc::ZoneFinder::FIND_DEFAULT);
-
     /// \brief Look up a zone's NS RRset and their address records for an
     /// authoritative answer, and add them to the additional section.
     ///
@@ -185,7 +159,8 @@ private:
     ///
     /// \param finder The \c ZoneFinder through which the NS and additional
     /// data for the query are to be found.
-    void addAuthAdditional(isc::datasrc::ZoneFinder& finder);
+    void addAuthAdditional(isc::datasrc::ZoneFinder& finder,
+                           std::vector<isc::dns::ConstRRsetPtr>& additionals);
 
     /// \brief Process a DS query possible at the child side of zone cut.
     ///
@@ -204,10 +179,66 @@ private:
     /// within this method.
     bool processDSAtChild();
 
-public:
-    /// Constructor from query parameters.
+    /// \brief Add NSEC3 to the response for a closest encloser proof for a
+    /// given name.
+    ///
+    /// This method calls \c findNSEC3() of the given zone finder for the
+    /// given name in the recursive mode, and adds the returned NSEC3(s) to
+    /// the authority section of the response message associated with the
+    /// \c Query object.
+    ///
+    /// It returns the number of labels of the closest encloser (returned via
+    /// the \c findNSEC3() call) in case the caller needs to use that value
+    /// for subsequent processing, i.e, constructing the best possible wildcard
+    /// name that (would) match the query name.
+    ///
+    /// Unless \c exact_ok is true, \c name is expected to be non existent,
+    /// in which case findNSEC3() in the recursive mode must return both
+    /// closest and next proofs.  If the latter is NULL, it means a run time
+    /// collision (or the zone is broken in other way), and this method throws
+    /// a BadNSEC3 exception.
+    ///
+    /// If \c exact_ok is true, this method takes into account the case
+    /// where the name exists and may or may not be at a zone cut to an
+    /// optout zone.  In this case, depending on whether the zone is optout
+    /// or not, findNSEC3() may return non-NULL or NULL next_proof
+    /// (respectively).  This method adds the next proof if and only if
+    /// findNSEC3() returns non NULL value for it.  The Opt-Out flag
+    /// must be set or cleared accordingly, but this method doesn't check that
+    /// in this level (as long as the zone is signed validly and findNSEC3()
+    /// for the data source is implemented as documented, the condition
+    /// should be met; otherwise we'd let the validator detect the error).
+    ///
+    /// By default this method always adds the closest proof.
+    /// If \c add_closest is false, it only adds the next proof to the message.
+    /// This correspond to the case of "wildcard answer responses" as described
+    /// in Section 7.2.6 of RFC5155.
+    uint8_t addClosestEncloserProof(isc::datasrc::ZoneFinder& finder,
+                                    const isc::dns::Name& name, bool exact_ok,
+                                    bool add_closest = true);
+
+    /// \brief Add matching or covering NSEC3 to the response for a give name.
+    ///
+    /// This method calls \c findNSEC3() of the given zone finder for the
+    /// given name in the non recursive mode, and adds the returned NSEC3 to
+    /// the authority section of the response message associated with the
+    /// \c Query object.
+    ///
+    /// Depending on the caller's context, the returned NSEC3 is one and
+    /// only one of matching or covering NSEC3.  If \c match is true the
+    /// returned NSEC3 must be a matching one; otherwise it must be a covering
+    /// one.  If this assumption isn't met this method throws a BadNSEC3
+    /// exception (if it must be a matching NSEC3 but is not, it means a broken
+    /// zone, maybe with incorrect optout NSEC3s; if it must be a covering
+    /// NSEC3 but is not, it means a run time collision; or the \c findNSEC3()
+    /// implementation is broken for both cases.)
+    void addNSEC3ForName(isc::datasrc::ZoneFinder& finder,
+                         const isc::dns::Name& name, bool match);
+
+    /// Set up the Query object for a new query lookup
     ///
-    /// This constructor never throws an exception.
+    /// This is the first step of the process() method, and initializes
+    /// the member data
     ///
     /// \param datasrc_client The datasource wherein the answer to the query is
     /// to be found.
@@ -216,14 +247,49 @@ public:
     /// \param response The response message to store the answer to the query.
     /// \param dnssec If the answer should include signatures and NSEC/NSEC3 if
     ///     possible.
-    Query(const isc::datasrc::DataSourceClient& datasrc_client,
-          const isc::dns::Name& qname, const isc::dns::RRType& qtype,
-          isc::dns::Message& response, bool dnssec = false) :
-        datasrc_client_(datasrc_client), qname_(qname), qtype_(qtype),
-        response_(response), dnssec_(dnssec),
-        dnssec_opt_(dnssec ?  isc::datasrc::ZoneFinder::FIND_DNSSEC :
-                    isc::datasrc::ZoneFinder::FIND_DEFAULT)
-    {}
+    void initialize(datasrc::DataSourceClient& datasrc_client,
+                    const isc::dns::Name& qname, const isc::dns::RRType& qtype,
+                    isc::dns::Message& response, bool dnssec = false);
+
+    /// \brief Resets any partly built response data, and internal pointers
+    ///
+    /// Called by the QueryCleaner object upon its destruction
+    void reset();
+
+    /// \brief Internal class used for cleanup of Query members
+    ///
+    /// The process() call creates an object of this class, which
+    /// upon its destruction, calls Query::reset(), so that outside
+    /// of single calls to process(), the query state is always clean.
+    class QueryCleaner {
+    public:
+        QueryCleaner(isc::auth::Query& query) : query_(query) {}
+        ~QueryCleaner() { query_.reset(); }
+    private:
+        isc::auth::Query& query_;
+    };
+
+protected:
+    // Following methods declared protected so they can be accessed
+    // by unit tests.
+
+    void createResponse();
+
+public:
+    /// Default constructor.
+    ///
+    /// Query parameters will be set by the call to process()
+    ///
+    Query() :
+        datasrc_client_(NULL), qname_(NULL), qtype_(NULL),
+        dnssec_(false), dnssec_opt_(isc::datasrc::ZoneFinder::FIND_DEFAULT),
+        response_(NULL)
+    {
+        answers_.reserve(RESERVE_RRSETS);
+        authorities_.reserve(RESERVE_RRSETS);
+        additionals_.reserve(RESERVE_RRSETS);
+    }
+
 
     /// Process the query.
     ///
@@ -251,7 +317,17 @@ public:
     /// This might throw BadZone or any of its specific subclasses, but that
     /// shouldn't happen in real-life (as BadZone means wrong data, it should
     /// have been rejected upon loading).
-    void process();
+    ///
+    /// \param datasrc_client The datasource wherein the answer to the query is
+    /// to be found.
+    /// \param qname The query name
+    /// \param qtype The RR type of the query
+    /// \param response The response message to store the answer to the query.
+    /// \param dnssec If the answer should include signatures and NSEC/NSEC3 if
+    ///     possible.
+    void process(datasrc::DataSourceClient& datasrc_client,
+                 const isc::dns::Name& qname, const isc::dns::RRType& qtype,
+                 isc::dns::Message& response, bool dnssec = false);
 
     /// \short Bad zone data encountered.
     ///
@@ -319,13 +395,105 @@ public:
         {}
     };
 
+    /// \brief Response Creator Class
+    ///
+    /// This is a helper class of Query, and is expected to be used during the
+    /// construction of the response message. This class performs the
+    /// duplicate RRset detection check.  It keeps a list of RRsets added
+    /// to the message and does not add an RRset if it is the same as one
+    /// already added.
+    ///
+    /// This class is essentially private to Query, but is visible to public
+    /// for testing purposes.  It's not expected to be used from a normal
+    /// application.
+    class ResponseCreator {
+    public:
+        /// \brief Constructor
+        ///
+        /// Reserves space for the list of RRsets.  Although the
+        /// ResponseCreator will be used to create a message from the
+        /// contents of the Query object's answers_, authorities_ and
+        /// additionals_ elements, and each of these are sized to
+        /// RESERVE_RRSETS, it is _extremely_ unlikely that all three will be
+        /// filled to capacity.  So we reserve more elements than in each of
+        /// these components, but not three times the amount.
+        ///
+        /// As with the answers_, authorities_ and additionals_ elements, the
+        /// reservation is made in the constructor to avoid dynamic allocation
+        /// of memory.  The ResponseCreator is a member variable of the Query
+        /// object so is constructed once and lasts as long as that object.
+        /// Internal state is cleared through the clear() method.
+        ResponseCreator() {
+            added_.reserve(2 * RESERVE_RRSETS);
+        }
+
+        /// \brief Reset internal state
+        void clear() {
+            added_.clear();
+        }
+
+        /// \brief Complete the response message with filling in the
+        /// response sections.
+        ///
+        /// This is the final step of the Query::process() method, and within
+        /// that method, it should be called before it returns (if any
+        /// response data is to be added)
+        ///
+        /// This will take a message to build and each RRsets for the answer,
+        /// authority, and additional sections, and add them to their
+        /// corresponding sections in the given message.  The RRsets are
+        /// filtered such that a particular RRset appears only once in the
+        /// message.
+        ///
+        /// If \c dnssec is true, it tells the message to include any RRSIGs
+        /// attached to the RRsets.
+        void create(
+            isc::dns::Message& message,
+            const std::vector<isc::dns::ConstRRsetPtr>& answers_,
+            const std::vector<isc::dns::ConstRRsetPtr>& authorities_,
+            const std::vector<isc::dns::ConstRRsetPtr>& additionals_,
+            const bool dnssec);
+
+    private:
+        // \brief RRset comparison functor.
+        struct IsSameKind : public std::binary_function<
+                            const isc::dns::AbstractRRset*,
+                            const isc::dns::AbstractRRset*,
+                            bool> {
+            bool operator()(const isc::dns::AbstractRRset* r1,
+                            const isc::dns::AbstractRRset* r2) const {
+                return (r1->isSameKind(*r2));
+            }
+        };
+
+        /// Insertion operation
+        ///
+        /// \param message Message to which the RRset is to be added
+        /// \param section Section of the message in which the RRset is put
+        /// \param rrset Pointer to RRset to be added to the message
+        /// \param dnssec Whether RRSIG records should be added as well
+        void addRRset(isc::dns::Message& message,
+                      const isc::dns::Message::Section section,
+                      const isc::dns::ConstRRsetPtr& rrset, const bool dnssec);
+
+
+    private:
+        /// List of RRsets already added to the message
+        std::vector<const isc::dns::AbstractRRset*> added_;
+    };
+
 private:
-    const isc::datasrc::DataSourceClient& datasrc_client_;
-    const isc::dns::Name& qname_;
-    const isc::dns::RRType& qtype_;
-    isc::dns::Message& response_;
-    const bool dnssec_;
-    const isc::datasrc::ZoneFinder::FindOptions dnssec_opt_;
+    const isc::datasrc::DataSourceClient* datasrc_client_;
+    const isc::dns::Name* qname_;
+    const isc::dns::RRType* qtype_;
+    bool dnssec_;
+    isc::datasrc::ZoneFinder::FindOptions dnssec_opt_;
+    ResponseCreator response_creator_;
+
+    isc::dns::Message* response_;
+    std::vector<isc::dns::ConstRRsetPtr> answers_;
+    std::vector<isc::dns::ConstRRsetPtr> authorities_;
+    std::vector<isc::dns::ConstRRsetPtr> additionals_;
 };
 
 }

src/bin/auth/statistics.cc

@@ -112,9 +112,13 @@ AuthCountersImpl::submitStatistics() const {
         return (false);
     }
     std::stringstream statistics_string;
+    // add pid in order for stats to identify which auth sends
+    // statistics in the situation that multiple auth instances are
+    // working
     statistics_string << "{\"command\": [\"set\","
                       <<   "{ \"owner\": \"Auth\","
-                      <<   "  \"data\":"
+                      <<   "  \"pid\":" << getpid()
+                      <<   ", \"data\":"
                       <<     "{ \"queries.udp\": "
                       <<     server_counter_.get(AuthCounters::SERVER_UDP_QUERY)
                       <<     ", \"queries.tcp\": "

src/bin/auth/tests/.gitignore

@@ -0,0 +1 @@
+/run_unittests

src/bin/auth/tests/Makefile.am

@@ -3,6 +3,7 @@ AM_CPPFLAGS += -I$(top_builddir)/src/bin # for generated spec_config.h header
 AM_CPPFLAGS += -I$(top_builddir)/src/lib/dns -I$(top_srcdir)/src/bin
 AM_CPPFLAGS += -I$(top_builddir)/src/lib/cc
 AM_CPPFLAGS += $(BOOST_INCLUDES)
+AM_CPPFLAGS += -DAUTH_OBJ_DIR=\"$(abs_top_builddir)/src/bin/auth\"
 AM_CPPFLAGS += -DTEST_DATA_DIR=\"$(abs_top_srcdir)/src/lib/testutils/testdata\"
 AM_CPPFLAGS += -DTEST_DATA_BUILDDIR=\"$(abs_top_builddir)/src/lib/testutils/testdata\"
 AM_CPPFLAGS += -DINSTALL_PROG=\"$(abs_top_srcdir)/install-sh\"
@@ -11,14 +12,18 @@ AM_CXXFLAGS = $(B10_CXXFLAGS)
 
 if USE_STATIC_LINK
 AM_LDFLAGS = -static
+# Some test cases cannot work with static link.  To selectively disable such
+# tests we signal it via a definition.
+AM_CPPFLAGS += -DUSE_STATIC_LINK=1
 endif
 
 CLEANFILES = *.gcno *.gcda
 
+# Do not define global tests, use check-local so
+# environment can be set (needed for dynamic loading)
 TESTS =
 if HAVE_GTEST
 
-TESTS += run_unittests
 run_unittests_SOURCES = $(top_srcdir)/src/lib/dns/tests/unittest_util.h
 run_unittests_SOURCES += $(top_srcdir)/src/lib/dns/tests/unittest_util.cc
 run_unittests_SOURCES += ../auth_srv.h ../auth_srv.cc
@@ -28,8 +33,10 @@ run_unittests_SOURCES += ../auth_config.h ../auth_config.cc
 run_unittests_SOURCES += ../command.h ../command.cc
 run_unittests_SOURCES += ../common.h ../common.cc
 run_unittests_SOURCES += ../statistics.h ../statistics.cc
+run_unittests_SOURCES += datasrc_util.h datasrc_util.cc
 run_unittests_SOURCES += auth_srv_unittest.cc
 run_unittests_SOURCES += config_unittest.cc
+run_unittests_SOURCES += config_syntax_unittest.cc
 run_unittests_SOURCES += command_unittest.cc
 run_unittests_SOURCES += common_unittest.cc
 run_unittests_SOURCES += query_unittest.cc
@@ -64,6 +71,10 @@ run_unittests_LDADD += $(top_builddir)/src/lib/server_common/libserver_common.la
 run_unittests_LDADD += $(top_builddir)/src/lib/nsas/libnsas.la
 run_unittests_LDADD += $(top_builddir)/src/lib/util/unittests/libutil_unittests.la
 run_unittests_LDADD += $(top_builddir)/src/lib/statistics/libstatistics.la
-endif
 
-noinst_PROGRAMS = $(TESTS)
+check-local:
+	B10_FROM_BUILD=${abs_top_builddir} ./run_unittests
+
+noinst_PROGRAMS = run_unittests
+
+endif

src/bin/auth/tests/auth_srv_unittest.cc

@@ -41,6 +41,7 @@
 #include <dns/tests/unittest_util.h>
 #include <testutils/dnsmessage_test.h>
 #include <testutils/srv_test.h>
+#include <testutils/mockups.h>
 #include <testutils/portconfig.h>
 #include <testutils/socket_request.h>
 
@@ -75,7 +76,7 @@ const char* const CONFIG_INMEMORY_EXAMPLE =
 class AuthSrvTest : public SrvTestBase {
 protected:
     AuthSrvTest() :
-        dnss_(ios_, NULL, NULL, NULL),
+        dnss_(),
         server(true, xfrout),
         rrclass(RRClass::IN()),
         // The empty string is expected value of the parameter of
@@ -87,8 +88,12 @@ protected:
         server.setXfrinSession(&notify_session);
         server.setStatisticsSession(&statistics_session);
     }
+
     virtual void processMessage() {
-        server.processMessage(*io_message, parse_message, response_obuffer,
+        // If processMessage has been called before, parse_message needs
+        // to be reset. If it hasn't, there's no harm in doing so
+        parse_message->clear(Message::PARSE);
+        server.processMessage(*io_message, *parse_message, *response_obuffer,
                               &dnsserv);
     }
 
@@ -120,8 +125,18 @@ protected:
             }
         }
     }
-    IOService ios_;
-    DNSService dnss_;
+
+    // Convenience method for tests that expect to return SERVFAIL
+    // It calls processMessage, checks if there is an answer, and
+    // check the header for default SERVFAIL data
+    void processAndCheckSERVFAIL() {
+        processMessage();
+        EXPECT_TRUE(dnsserv.hasAnswer());
+        headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
+                    opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
+    }
+
+    MockDNSService dnss_;
     MockSession statistics_session;
     MockXfroutClient xfrout;
     AuthSrv server;
@@ -154,8 +169,7 @@ createBuiltinVersionResponse(const qid_t qid, vector<uint8_t>& data) {
     rrset_version_ns->addRdata(generic::NS(version_name));
     message.addRRset(Message::SECTION_AUTHORITY, rrset_version_ns);
 
-    OutputBuffer obuffer(0);
-    MessageRenderer renderer(obuffer);
+    MessageRenderer renderer;
     message.toWire(renderer);
 
     data.clear();
@@ -174,7 +188,7 @@ TEST_F(AuthSrvTest, builtInQuery) {
                                        default_qid, Name("version.bind"),
                                        RRClass::CH(), RRType::TXT());
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
     createBuiltinVersionResponse(default_qid, response_data);
     EXPECT_PRED_FORMAT4(UnitTestUtil::matchWireData,
@@ -286,7 +300,8 @@ TEST_F(AuthSrvTest, AXFRSuccess) {
     createRequestPacket(request_message, IPPROTO_TCP);
     // On success, the AXFR query has been passed to a separate process,
     // so we shouldn't have to respond.
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
     EXPECT_TRUE(xfrout.isConnected());
     checkAllRcodeCountersZero();
@@ -307,7 +322,7 @@ TEST_F(AuthSrvTest, TSIGSigned) {
     boost::shared_ptr<TSIGKeyRing> keyring(new TSIGKeyRing);
     keyring->add(key);
     server.setTSIGKeyRing(&keyring);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
 
     // What did we get?
@@ -342,7 +357,7 @@ TEST_F(AuthSrvTest, TSIGSignedBadKey) {
     // Process the message, but use a different key there
     boost::shared_ptr<TSIGKeyRing> keyring(new TSIGKeyRing);
     server.setTSIGKeyRing(&keyring);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
 
     EXPECT_TRUE(dnsserv.hasAnswer());
@@ -377,7 +392,7 @@ TEST_F(AuthSrvTest, TSIGBadSig) {
     boost::shared_ptr<TSIGKeyRing> keyring(new TSIGKeyRing);
     keyring->add(TSIGKey("key:QkFECg==:hmac-sha1"));
     server.setTSIGKeyRing(&keyring);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
 
     EXPECT_TRUE(dnsserv.hasAnswer());
@@ -415,7 +430,7 @@ TEST_F(AuthSrvTest, TSIGCheckFirst) {
     boost::shared_ptr<TSIGKeyRing> keyring(new TSIGKeyRing);
     keyring->add(TSIGKey("key:QkFECg==:hmac-sha1"));
     server.setTSIGKeyRing(&keyring);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
 
     EXPECT_TRUE(dnsserv.hasAnswer());
@@ -446,7 +461,8 @@ TEST_F(AuthSrvTest, AXFRConnectFail) {
                                        Name("example.com"), RRClass::IN(),
                                        RRType::AXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
                 opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
@@ -460,7 +476,8 @@ TEST_F(AuthSrvTest, AXFRSendFail) {
                                        Name("example.com"), RRClass::IN(),
                                        RRType::AXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(xfrout.isConnected());
 
     xfrout.disableSend();
@@ -470,7 +487,8 @@ TEST_F(AuthSrvTest, AXFRSendFail) {
                                        Name("example.com"), RRClass::IN(),
                                        RRType::AXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
                 opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
@@ -480,17 +498,17 @@ TEST_F(AuthSrvTest, AXFRSendFail) {
 }
 
 TEST_F(AuthSrvTest, AXFRDisconnectFail) {
-    // In our usage disconnect() shouldn't fail.  So we'll see the exception
-    // should it be thrown.
+    // In our usage disconnect() shouldn't fail. But even if it does,
+    // it should not disrupt service (so processMessage should have caught it)
     xfrout.disableSend();
     xfrout.disableDisconnect();
     UnitTestUtil::createRequestMessage(request_message, opcode, default_qid,
                                        Name("example.com"), RRClass::IN(),
                                        RRType::AXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    EXPECT_THROW(server.processMessage(*io_message, parse_message,
-                                       response_obuffer, &dnsserv),
-                 XfroutError);
+    EXPECT_NO_THROW(server.processMessage(*io_message, *parse_message,
+                                          *response_obuffer, &dnsserv));
+    // Since the disconnect failed, we should still be 'connected'
     EXPECT_TRUE(xfrout.isConnected());
     // XXX: we need to re-enable disconnect.  otherwise an exception would be
     // thrown via the destructor of the server.
@@ -504,7 +522,8 @@ TEST_F(AuthSrvTest, IXFRConnectFail) {
                                        Name("example.com"), RRClass::IN(),
                                        RRType::IXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
                 opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
@@ -518,7 +537,8 @@ TEST_F(AuthSrvTest, IXFRSendFail) {
                                        Name("example.com"), RRClass::IN(),
                                        RRType::IXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(xfrout.isConnected());
 
     xfrout.disableSend();
@@ -528,7 +548,8 @@ TEST_F(AuthSrvTest, IXFRSendFail) {
                                        Name("example.com"), RRClass::IN(),
                                        RRType::IXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
                 opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
@@ -538,17 +559,16 @@ TEST_F(AuthSrvTest, IXFRSendFail) {
 }
 
 TEST_F(AuthSrvTest, IXFRDisconnectFail) {
-    // In our usage disconnect() shouldn't fail.  So we'll see the exception
-    // should it be thrown.
+    // In our usage disconnect() shouldn't fail, but even if it does,
+    // procesMessage() should catch it.
     xfrout.disableSend();
     xfrout.disableDisconnect();
     UnitTestUtil::createRequestMessage(request_message, opcode, default_qid,
                                        Name("example.com"), RRClass::IN(),
                                        RRType::IXFR());
     createRequestPacket(request_message, IPPROTO_TCP);
-    EXPECT_THROW(server.processMessage(*io_message, parse_message,
-                                       response_obuffer, &dnsserv),
-                 XfroutError);
+    EXPECT_NO_THROW(server.processMessage(*io_message, *parse_message,
+                                          *response_obuffer, &dnsserv));
     EXPECT_TRUE(xfrout.isConnected());
     // XXX: we need to re-enable disconnect.  otherwise an exception would be
     // thrown via the destructor of the server.
@@ -561,7 +581,8 @@ TEST_F(AuthSrvTest, notify) {
                                        RRClass::IN(), RRType::SOA());
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
 
     // An internal command message should have been created and sent to an
@@ -596,7 +617,8 @@ TEST_F(AuthSrvTest, notifyForCHClass) {
                                        RRClass::CH(), RRType::SOA());
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
 
     // Other conditions should be the same, so simply confirm the RR class is
@@ -614,7 +636,8 @@ TEST_F(AuthSrvTest, notifyEmptyQuestion) {
     request_message.setQid(default_qid);
     request_message.toWire(request_renderer);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::FORMERR(),
                 Opcode::NOTIFY().getCode(), QR_FLAG, 0, 0, 0, 0);
@@ -629,7 +652,8 @@ TEST_F(AuthSrvTest, notifyMultiQuestions) {
                                          RRType::SOA()));
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::FORMERR(),
                 Opcode::NOTIFY().getCode(), QR_FLAG, 2, 0, 0, 0);
@@ -641,7 +665,8 @@ TEST_F(AuthSrvTest, notifyNonSOAQuestion) {
                                        RRClass::IN(), RRType::NS());
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::FORMERR(),
                 Opcode::NOTIFY().getCode(), QR_FLAG, 1, 0, 0, 0);
@@ -653,7 +678,8 @@ TEST_F(AuthSrvTest, notifyWithoutAA) {
                                        default_qid, Name("example.com"),
                                        RRClass::IN(), RRType::SOA());
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::NOERROR(),
                 Opcode::NOTIFY().getCode(), QR_FLAG | AA_FLAG, 1, 0, 0, 0);
@@ -666,7 +692,8 @@ TEST_F(AuthSrvTest, notifyWithErrorRcode) {
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     request_message.setRcode(Rcode::SERVFAIL());
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::NOERROR(),
                 Opcode::NOTIFY().getCode(), QR_FLAG | AA_FLAG, 1, 0, 0, 0);
@@ -683,7 +710,8 @@ TEST_F(AuthSrvTest, notifyWithoutSession) {
 
     // we simply ignore the notify and let it be resent if an internal error
     // happens.
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
 }
 
@@ -696,7 +724,8 @@ TEST_F(AuthSrvTest, notifySendFail) {
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
 
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
 }
 
@@ -708,7 +737,8 @@ TEST_F(AuthSrvTest, notifyReceiveFail) {
                                        RRClass::IN(), RRType::SOA());
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
 }
 
@@ -720,7 +750,8 @@ TEST_F(AuthSrvTest, notifyWithBogusSessionMessage) {
                                        RRClass::IN(), RRType::SOA());
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
 }
 
@@ -733,7 +764,8 @@ TEST_F(AuthSrvTest, notifyWithSessionMessageError) {
                                        RRClass::IN(), RRType::SOA());
     request_message.setHeaderFlag(Message::HEADERFLAG_AA);
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
 }
 
@@ -748,7 +780,8 @@ updateConfig(AuthSrv* server, const char* const config_data,
 
     ConstElementPtr result = config_answer->get("result");
     EXPECT_EQ(Element::list, result->getType());
-    EXPECT_EQ(expect_success ? 0 : 1, result->get(0)->intValue());
+    EXPECT_EQ(expect_success ? 0 : 1, result->get(0)->intValue()) <<
+        "Bad result from updateConfig: " << result->str();
 }
 
 // Install a Sqlite3 data source with testing data.
@@ -759,7 +792,8 @@ TEST_F(AuthSrvTest, updateConfig) {
     // response should have the AA flag on, and have an RR in each answer
     // and authority section.
     createDataFromFile("examplequery_fromWire.wire");
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::NOERROR(), opcode.getCode(),
                 QR_FLAG | AA_FLAG, 1, 1, 1, 0);
@@ -773,7 +807,8 @@ TEST_F(AuthSrvTest, datasourceFail) {
     // in a SERVFAIL response, and the answer and authority sections should
     // be empty.
     createDataFromFile("badExampleQuery_fromWire.wire");
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
                 opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
@@ -788,7 +823,8 @@ TEST_F(AuthSrvTest, updateConfigFail) {
 
     // The original data source should still exist.
     createDataFromFile("examplequery_fromWire.wire");
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::NOERROR(), opcode.getCode(),
                 QR_FLAG | AA_FLAG, 1, 1, 1, 0);
@@ -808,7 +844,7 @@ TEST_F(AuthSrvTest, updateWithInMemoryClient) {
 
     // The memory data source is empty, should return REFUSED rcode.
     createDataFromFile("examplequery_fromWire.wire");
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::REFUSED(),
@@ -825,7 +861,7 @@ TEST_F(AuthSrvTest, queryWithInMemoryClientNoDNSSEC) {
     EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
 
     createDataFromFile("nsec3query_nodnssec_fromWire.wire");
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
 
     EXPECT_TRUE(dnsserv.hasAnswer());
@@ -842,7 +878,7 @@ TEST_F(AuthSrvTest, queryWithInMemoryClientDNSSEC) {
     EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
 
     createDataFromFile("nsec3query_fromWire.wire");
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
 
     EXPECT_TRUE(dnsserv.hasAnswer());
@@ -860,7 +896,7 @@ TEST_F(AuthSrvTest, chQueryWithInMemoryClient) {
                                        default_qid, Name("version.bind"),
                                        RRClass::CH(), RRType::TXT());
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
     EXPECT_TRUE(dnsserv.hasAnswer());
     headerCheck(*parse_message, default_qid, Rcode::NOERROR(),
@@ -886,7 +922,7 @@ TEST_F(AuthSrvTest, queryCounterUDPNormal) {
                                        default_qid, Name("example.com"),
                                        RRClass::IN(), RRType::NS());
     createRequestPacket(request_message, IPPROTO_UDP);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
     // After processing UDP query, the counter should be 1.
     EXPECT_EQ(1, server.getCounter(AuthCounters::SERVER_UDP_QUERY));
@@ -905,7 +941,7 @@ TEST_F(AuthSrvTest, queryCounterTCPNormal) {
                                        default_qid, Name("example.com"),
                                        RRClass::IN(), RRType::NS());
     createRequestPacket(request_message, IPPROTO_TCP);
-    server.processMessage(*io_message, parse_message, response_obuffer,
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
                           &dnsserv);
     // After processing TCP query, the counter should be 1.
     EXPECT_EQ(1, server.getCounter(AuthCounters::SERVER_TCP_QUERY));
@@ -924,7 +960,8 @@ TEST_F(AuthSrvTest, queryCounterTCPAXFR) {
     createRequestPacket(request_message, IPPROTO_TCP);
     // On success, the AXFR query has been passed to a separate process,
     // so auth itself shouldn't respond.
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
     // After processing TCP AXFR query, the counter should be 1.
     EXPECT_EQ(1, server.getCounter(AuthCounters::SERVER_TCP_QUERY));
@@ -941,7 +978,8 @@ TEST_F(AuthSrvTest, queryCounterTCPIXFR) {
     createRequestPacket(request_message, IPPROTO_TCP);
     // On success, the IXFR query has been passed to a separate process,
     // so auth itself shouldn't respond.
-    server.processMessage(*io_message, parse_message, response_obuffer, &dnsserv);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
     EXPECT_FALSE(dnsserv.hasAnswer());
     // After processing TCP IXFR query, the counter should be 1.
     EXPECT_EQ(1, server.getCounter(AuthCounters::SERVER_TCP_QUERY));
@@ -962,7 +1000,8 @@ TEST_F(AuthSrvTest, queryCounterOpcodes) {
         // we intentionally use different values for each code
         for (int j = 0; j <= i; ++j) {
             parse_message->clear(Message::PARSE);
-            server.processMessage(*io_message, parse_message, response_obuffer,
+            server.processMessage(*io_message, *parse_message,
+                                  *response_obuffer,
                                   &dnsserv);
         }
 
@@ -988,11 +1027,10 @@ getDummyUnknownSocket() {
     return (socket);
 }
 
-// Submit unexpected type of query and check it throws isc::Unexpected
+// Submit unexpected type of query and check it is ignored
 TEST_F(AuthSrvTest, queryCounterUnexpected) {
     // This code isn't exception safe, but we'd rather keep the code
-    // simpler and more readable as this is only for tests and if it throws
-    // the program would immediately terminate anyway.
+    // simpler and more readable as this is only for tests
 
     // Create UDP query packet.
     UnitTestUtil::createRequestMessage(request_message, Opcode::QUERY(),
@@ -1008,9 +1046,7 @@ TEST_F(AuthSrvTest, queryCounterUnexpected) {
                                request_renderer.getLength(),
                                getDummyUnknownSocket(), *endpoint);
 
-    EXPECT_THROW(server.processMessage(*io_message, parse_message,
-                                       response_obuffer, &dnsserv),
-                 isc::Unexpected);
+    EXPECT_FALSE(dnsserv.hasAnswer());
 }
 
 TEST_F(AuthSrvTest, stop) {
@@ -1039,4 +1075,314 @@ TEST_F(AuthSrvTest, listenAddresses) {
                                 "Released tokens");
 }
 
+TEST_F(AuthSrvTest, processNormalQuery_reuseRenderer1) {
+    UnitTestUtil::createRequestMessage(request_message, Opcode::QUERY(),
+                                       default_qid, Name("example.com"),
+                                       RRClass::IN(), RRType::NS());
+
+    request_message.setHeaderFlag(Message::HEADERFLAG_AA);
+    createRequestPacket(request_message, IPPROTO_UDP);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
+    EXPECT_NE(request_message.getRcode(), parse_message->getRcode());
+}
+
+TEST_F(AuthSrvTest, processNormalQuery_reuseRenderer2) {
+    UnitTestUtil::createRequestMessage(request_message, Opcode::QUERY(),
+                                       default_qid, Name("example.com"),
+                                       RRClass::IN(), RRType::SOA());
+
+    request_message.setHeaderFlag(Message::HEADERFLAG_AA);
+    createRequestPacket(request_message, IPPROTO_UDP);
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
+    ConstQuestionPtr question = *parse_message->beginQuestion();
+    EXPECT_STRNE(question->getType().toText().c_str(),
+                 RRType::NS().toText().c_str());
+}
+//
+// Tests for catching exceptions in various stages of the query processing
+//
+// These tests work by defining two proxy classes, that act as an in-memory
+// client by default, but can throw exceptions at various points.
+//
+namespace {
+
+/// A the possible methods to throw in, either in FakeInMemoryClient or
+/// FakeZoneFinder
+enum ThrowWhen {
+    THROW_NEVER,
+    THROW_AT_FIND_ZONE,
+    THROW_AT_GET_ORIGIN,
+    THROW_AT_GET_CLASS,
+    THROW_AT_FIND,
+    THROW_AT_FIND_ALL,
+    THROW_AT_FIND_NSEC3
+};
+
+/// convenience function to check whether and what to throw
+void
+checkThrow(ThrowWhen method, ThrowWhen throw_at, bool isc_exception) {
+    if (method == throw_at) {
+        if (isc_exception) {
+            isc_throw(isc::Exception, "foo");
+        } else {
+            throw std::exception();
+        }
+    }
+}
+
+/// \brief proxy class for the ZoneFinder returned by the InMemoryClient
+///        proxied by FakeInMemoryClient
+///
+/// See the documentation for FakeInMemoryClient for more information,
+/// all methods simply check whether they should throw, and if not, call
+/// their proxied equivalent.
+class FakeZoneFinder : public isc::datasrc::ZoneFinder {
+public:
+    FakeZoneFinder(isc::datasrc::ZoneFinderPtr zone_finder,
+                   ThrowWhen throw_when, bool isc_exception,
+                   ConstRRsetPtr fake_rrset) :
+        real_zone_finder_(zone_finder),
+        throw_when_(throw_when),
+        isc_exception_(isc_exception),
+        fake_rrset_(fake_rrset)
+    {}
+
+    virtual isc::dns::Name
+    getOrigin() const {
+        checkThrow(THROW_AT_GET_ORIGIN, throw_when_, isc_exception_);
+        return (real_zone_finder_->getOrigin());
+    }
+
+    virtual isc::dns::RRClass
+    getClass() const {
+        checkThrow(THROW_AT_GET_CLASS, throw_when_, isc_exception_);
+        return (real_zone_finder_->getClass());
+    }
+
+    virtual isc::datasrc::ZoneFinderContextPtr
+    find(const isc::dns::Name& name,
+         const isc::dns::RRType& type,
+         isc::datasrc::ZoneFinder::FindOptions options)
+    {
+        using namespace isc::datasrc;
+        checkThrow(THROW_AT_FIND, throw_when_, isc_exception_);
+        // If faked RRset was specified on construction and it matches the
+        // query, return it instead of searching the real data source.
+        if (fake_rrset_ && fake_rrset_->getName() == name &&
+            fake_rrset_->getType() == type)
+        {
+            return (ZoneFinderContextPtr(new ZoneFinder::Context(
+                                             *this, options,
+                                             ResultContext(SUCCESS,
+                                                           fake_rrset_))));
+        }
+        return (real_zone_finder_->find(name, type, options));
+    }
+
+    virtual isc::datasrc::ZoneFinderContextPtr
+    findAll(const isc::dns::Name& name,
+            std::vector<isc::dns::ConstRRsetPtr> &target,
+            const FindOptions options = FIND_DEFAULT)
+    {
+        checkThrow(THROW_AT_FIND_ALL, throw_when_, isc_exception_);
+        return (real_zone_finder_->findAll(name, target, options));
+    }
+
+    virtual FindNSEC3Result
+    findNSEC3(const isc::dns::Name& name, bool recursive) {
+        checkThrow(THROW_AT_FIND_NSEC3, throw_when_, isc_exception_);
+        return (real_zone_finder_->findNSEC3(name, recursive));
+    }
+
+    virtual isc::dns::Name
+    findPreviousName(const isc::dns::Name& query) const {
+        return (real_zone_finder_->findPreviousName(query));
+    }
+
+private:
+    isc::datasrc::ZoneFinderPtr real_zone_finder_;
+    ThrowWhen throw_when_;
+    bool isc_exception_;
+    ConstRRsetPtr fake_rrset_;
+};
+
+/// \brief Proxy InMemoryClient that can throw exceptions at specified times
+///
+/// It is based on the memory client since that one is easy to override
+/// (with setInMemoryClient) with the current design of AuthSrv.
+class FakeInMemoryClient : public isc::datasrc::InMemoryClient {
+public:
+    /// \brief Create a proxy memory client
+    ///
+    /// \param real_client The real in-memory client to proxy
+    /// \param throw_when if set to any value other than never, that is
+    ///        the method that will throw an exception (either in this
+    ///        class or the related FakeZoneFinder)
+    /// \param isc_exception if true, throw isc::Exception, otherwise,
+    ///                      throw std::exception
+    /// \param fake_rrset If non NULL, it will be used as an answer to
+    /// find() for that name and type.
+    FakeInMemoryClient(AuthSrv::InMemoryClientPtr real_client,
+                       ThrowWhen throw_when, bool isc_exception,
+                       ConstRRsetPtr fake_rrset = ConstRRsetPtr()) :
+        real_client_(real_client),
+        throw_when_(throw_when),
+        isc_exception_(isc_exception),
+        fake_rrset_(fake_rrset)
+    {}
+
+    /// \brief proxy call for findZone
+    ///
+    /// if this instance was constructed with throw_when set to find_zone,
+    /// this method will throw. Otherwise, it will return a FakeZoneFinder
+    /// instance which will throw at the method specified at the
+    /// construction of this instance.
+    virtual FindResult
+    findZone(const isc::dns::Name& name) const {
+        checkThrow(THROW_AT_FIND_ZONE, throw_when_, isc_exception_);
+        const FindResult result = real_client_->findZone(name);
+        return (FindResult(result.code, isc::datasrc::ZoneFinderPtr(
+                                        new FakeZoneFinder(result.zone_finder,
+                                                           throw_when_,
+                                                           isc_exception_,
+                                                           fake_rrset_))));
+    }
+
+private:
+    AuthSrv::InMemoryClientPtr real_client_;
+    ThrowWhen throw_when_;
+    bool isc_exception_;
+    ConstRRsetPtr fake_rrset_;
+};
+
+} // end anonymous namespace for throwing proxy classes
+
+// Test for the tests
+//
+// Set the proxies to never throw, this should have the same result as
+// queryWithInMemoryClientNoDNSSEC, and serves to test the two proxy classes
+TEST_F(AuthSrvTest, queryWithInMemoryClientProxy) {
+    // Set real inmem client to proxy
+    updateConfig(&server, CONFIG_INMEMORY_EXAMPLE, true);
+
+    AuthSrv::InMemoryClientPtr fake_client(
+        new FakeInMemoryClient(server.getInMemoryClient(rrclass),
+                               THROW_NEVER, false));
+    ASSERT_NE(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
+    server.setInMemoryClient(rrclass, fake_client);
+
+    createDataFromFile("nsec3query_nodnssec_fromWire.wire");
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
+
+    EXPECT_TRUE(dnsserv.hasAnswer());
+    headerCheck(*parse_message, default_qid, Rcode::NOERROR(),
+                opcode.getCode(), QR_FLAG | AA_FLAG, 1, 1, 2, 1);
+}
+
+// Convenience function for the rest of the tests, set up a proxy
+// to throw in the given method
+// If isc_exception is true, it will throw isc::Exception, otherwise
+// it will throw std::exception
+// If non null rrset is given, it will be passed to the proxy so it can
+// return some faked response.
+void
+setupThrow(AuthSrv* server, const char *config, ThrowWhen throw_when,
+           bool isc_exception, ConstRRsetPtr rrset = ConstRRsetPtr())
+{
+    // Set real inmem client to proxy
+    updateConfig(server, config, true);
+
+    // Set it to throw on findZone(), this should result in
+    // SERVFAIL on any exception
+    AuthSrv::InMemoryClientPtr fake_client(
+        new FakeInMemoryClient(
+            server->getInMemoryClient(isc::dns::RRClass::IN()),
+            throw_when, isc_exception, rrset));
+
+    ASSERT_NE(AuthSrv::InMemoryClientPtr(),
+              server->getInMemoryClient(isc::dns::RRClass::IN()));
+    server->setInMemoryClient(isc::dns::RRClass::IN(), fake_client);
+}
+
+TEST_F(AuthSrvTest, queryWithThrowingProxyServfails) {
+    // Test the common cases, all of which should simply return SERVFAIL
+    // Use THROW_NEVER as end marker
+    ThrowWhen throws[] = { THROW_AT_FIND_ZONE,
+                           THROW_AT_GET_ORIGIN,
+                           THROW_AT_FIND,
+                           THROW_AT_FIND_NSEC3,
+                           THROW_NEVER };
+    UnitTestUtil::createDNSSECRequestMessage(request_message, opcode,
+                                             default_qid, Name("foo.example."),
+                                             RRClass::IN(), RRType::TXT());
+    for (ThrowWhen* when(throws); *when != THROW_NEVER; ++when) {
+        createRequestPacket(request_message, IPPROTO_UDP);
+        setupThrow(&server, CONFIG_INMEMORY_EXAMPLE, *when, true);
+        processAndCheckSERVFAIL();
+        // To be sure, check same for non-isc-exceptions
+        createRequestPacket(request_message, IPPROTO_UDP);
+        setupThrow(&server, CONFIG_INMEMORY_EXAMPLE, *when, false);
+        processAndCheckSERVFAIL();
+    }
+}
+
+// Throw isc::Exception in getClass(). (Currently?) getClass is not called
+// in the processMessage path, so this should result in a normal answer
+TEST_F(AuthSrvTest, queryWithInMemoryClientProxyGetClass) {
+    createDataFromFile("nsec3query_nodnssec_fromWire.wire");
+    setupThrow(&server, CONFIG_INMEMORY_EXAMPLE, THROW_AT_GET_CLASS, true);
+
+    // getClass is not called so it should just answer
+    server.processMessage(*io_message, *parse_message, *response_obuffer,
+                          &dnsserv);
+
+    EXPECT_TRUE(dnsserv.hasAnswer());
+    headerCheck(*parse_message, default_qid, Rcode::NOERROR(),
+                opcode.getCode(), QR_FLAG | AA_FLAG, 1, 1, 2, 1);
+}
+
+TEST_F(AuthSrvTest, queryWithThrowingInToWire) {
+    // Set up a faked data source.  It will return an empty RRset for the
+    // query.
+    ConstRRsetPtr empty_rrset(new RRset(Name("foo.example"),
+                                        RRClass::IN(), RRType::TXT(),
+                                        RRTTL(0)));
+    setupThrow(&server, CONFIG_INMEMORY_EXAMPLE, THROW_NEVER, true,
+               empty_rrset);
+
+    // Repeat the query processing two times.  Due to the faked RRset,
+    // toWire() should throw, and it should result in SERVFAIL.
+    OutputBufferPtr orig_buffer;
+    for (int i = 0; i < 2; ++i) {
+        UnitTestUtil::createDNSSECRequestMessage(request_message, opcode,
+                                                 default_qid,
+                                                 Name("foo.example."),
+                                                 RRClass::IN(), RRType::TXT());
+        createRequestPacket(request_message, IPPROTO_UDP);
+        server.processMessage(*io_message, *parse_message, *response_obuffer,
+                              &dnsserv);
+        headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
+                    opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
+
+        // Make a backup of the original buffer for latest tests and replace
+        // it with a new one
+        if (!orig_buffer) {
+            orig_buffer = response_obuffer;
+            response_obuffer.reset(new OutputBuffer(0));
+        }
+        request_message.clear(Message::RENDER);
+        parse_message->clear(Message::PARSE);
+    }
+
+    // Now check if the original buffer is intact
+    parse_message->clear(Message::PARSE);
+    InputBuffer ibuffer(orig_buffer->getData(), orig_buffer->getLength());
+    parse_message->fromWire(ibuffer);
+    headerCheck(*parse_message, default_qid, Rcode::SERVFAIL(),
+                opcode.getCode(), QR_FLAG, 1, 0, 0, 0);
+}
+
 }

src/bin/auth/tests/command_unittest.cc

@@ -14,6 +14,8 @@
 
 #include <config.h>
 
+#include "datasrc_util.h"
+
 #include <auth/auth_srv.h>
 #include <auth/auth_config.h>
 #include <auth/command.h>
@@ -49,8 +51,11 @@ using namespace isc::dns;
 using namespace isc::data;
 using namespace isc::datasrc;
 using namespace isc::config;
+using namespace isc::testutils;
+using namespace isc::auth::unittest;
 
 namespace {
+
 class AuthCommandTest : public ::testing::Test {
 protected:
     AuthCommandTest() :
@@ -176,23 +181,23 @@ zoneChecks(AuthSrv& server) {
     EXPECT_TRUE(server.getInMemoryClient(RRClass::IN()));
     EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test1.example")).zone_finder->
-              find(Name("ns.test1.example"), RRType::A()).code);
+              find(Name("ns.test1.example"), RRType::A())->code);
     EXPECT_EQ(ZoneFinder::NXRRSET, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test1.example")).zone_finder->
-              find(Name("ns.test1.example"), RRType::AAAA()).code);
+              find(Name("ns.test1.example"), RRType::AAAA())->code);
     EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test2.example")).zone_finder->
-              find(Name("ns.test2.example"), RRType::A()).code);
+              find(Name("ns.test2.example"), RRType::A())->code);
     EXPECT_EQ(ZoneFinder::NXRRSET, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test2.example")).zone_finder->
-              find(Name("ns.test2.example"), RRType::AAAA()).code);
+              find(Name("ns.test2.example"), RRType::AAAA())->code);
 }
 
 void
 configureZones(AuthSrv& server) {
-    ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR "/test1.zone.in "
+    ASSERT_EQ(0, system(INSTALL_PROG " -c " TEST_DATA_DIR "/test1.zone.in "
                         TEST_DATA_BUILDDIR "/test1.zone.copied"));
-    ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR "/test2.zone.in "
+    ASSERT_EQ(0, system(INSTALL_PROG " -c " TEST_DATA_DIR "/test2.zone.in "
                         TEST_DATA_BUILDDIR "/test2.zone.copied"));
     configureAuthServer(server, Element::fromJSON(
                             "{\"datasources\": "
@@ -213,28 +218,28 @@ newZoneChecks(AuthSrv& server) {
     EXPECT_TRUE(server.getInMemoryClient(RRClass::IN()));
     EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test1.example")).zone_finder->
-              find(Name("ns.test1.example"), RRType::A()).code);
+              find(Name("ns.test1.example"), RRType::A())->code);
     // now test1.example should have ns/AAAA
     EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test1.example")).zone_finder->
-              find(Name("ns.test1.example"), RRType::AAAA()).code);
+              find(Name("ns.test1.example"), RRType::AAAA())->code);
 
     // test2.example shouldn't change
     EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test2.example")).zone_finder->
-              find(Name("ns.test2.example"), RRType::A()).code);
+              find(Name("ns.test2.example"), RRType::A())->code);
     EXPECT_EQ(ZoneFinder::NXRRSET, server.getInMemoryClient(RRClass::IN())->
               findZone(Name("ns.test2.example")).zone_finder->
-              find(Name("ns.test2.example"), RRType::AAAA()).code);
+              find(Name("ns.test2.example"), RRType::AAAA())->code);
 }
 
 TEST_F(AuthCommandTest, loadZone) {
     configureZones(server_);
 
-    ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR
+    ASSERT_EQ(0, system(INSTALL_PROG " -c " TEST_DATA_DIR
                         "/test1-new.zone.in "
                         TEST_DATA_BUILDDIR "/test1.zone.copied"));
-    ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR
+    ASSERT_EQ(0, system(INSTALL_PROG " -c " TEST_DATA_DIR
                         "/test2-new.zone.in "
                         TEST_DATA_BUILDDIR "/test2.zone.copied"));
 
@@ -245,10 +250,133 @@ TEST_F(AuthCommandTest, loadZone) {
     newZoneChecks(server_);
 }
 
+// This test uses dynamic load of a data source module, and won't work when
+// statically linked.
+TEST_F(AuthCommandTest,
+#ifdef USE_STATIC_LINK
+       DISABLED_loadZoneSQLite3
+#else
+       loadZoneSQLite3
+#endif
+    )
+{
+    const char* const SPEC_FILE = AUTH_OBJ_DIR "/auth.spec";
+
+    // Prepare the database first
+    const string test_db = TEST_DATA_BUILDDIR "/auth_test.sqlite3.copied";
+    const string bad_db = TEST_DATA_BUILDDIR "/does-not-exist.sqlite3";
+    stringstream ss("example.org. 3600 IN SOA . . 0 0 0 0 0\n");
+    createSQLite3DB(RRClass::IN(), Name("example.org"), test_db.c_str(), ss);
+
+    // Then store a config of the zone to the auth server
+    // This omits many config options of the auth server, but these are
+    // not read now.
+    isc::testutils::MockSession session;
+    // The session should not take care of anything or start anything, we
+    // need it only to hold the config we're going to put into it.
+    ModuleCCSession module_session(SPEC_FILE, session, NULL, NULL, false,
+                                  false);
+    // This describes the data source in the configuration
+    const ElementPtr
+        map(Element::fromJSON("{\"datasources\": ["
+                              "  {"
+                              "    \"type\": \"memory\","
+                              "    \"zones\": ["
+                              "      {"
+                              "        \"origin\": \"example.org\","
+                              "        \"file\": \"" + test_db + "\","
+                              "        \"filetype\": \"sqlite3\""
+                              "      }"
+                              "    ]"
+                              "  }"
+                              "]}"));
+    module_session.setLocalConfig(map);
+    server_.setConfigSession(&module_session);
+
+    // The loadzone command needs the zone to be already loaded, because
+    // it is used for reloading only
+    AuthSrv::InMemoryClientPtr dsrc(new InMemoryClient());
+    dsrc->addZone(ZoneFinderPtr(new InMemoryZoneFinder(RRClass::IN(),
+                                                       Name("example.org"))));
+    server_.setInMemoryClient(RRClass::IN(), dsrc);
+
+    // Now send the command to reload it
+    result_ = execAuthServerCommand(server_, "loadzone",
+        Element::fromJSON("{\"origin\": \"example.org\"}"));
+    checkAnswer(0);
+
+    // Get the zone and look if there are data in it (the original one was
+    // empty)
+    ASSERT_TRUE(server_.getInMemoryClient(RRClass::IN()));
+    EXPECT_EQ(ZoneFinder::SUCCESS, server_.getInMemoryClient(RRClass::IN())->
+              findZone(Name("example.org")).zone_finder->
+              find(Name("example.org"), RRType::SOA())->code);
+
+    // Some error cases. First, the zone has no configuration.
+    dsrc->addZone(ZoneFinderPtr(new InMemoryZoneFinder(RRClass::IN(),
+                                                       Name("example.com"))));
+    result_ = execAuthServerCommand(server_, "loadzone",
+        Element::fromJSON("{\"origin\": \"example.com\"}"));
+    checkAnswer(1);
+
+    // The previous zone is not hurt in any way
+    EXPECT_EQ(ZoneFinder::SUCCESS, server_.getInMemoryClient(RRClass::IN())->
+              findZone(Name("example.org")).zone_finder->
+              find(Name("example.org"), RRType::SOA())->code);
+
+    module_session.setLocalConfig(Element::fromJSON("{\"datasources\": []}"));
+    result_ = execAuthServerCommand(server_, "loadzone",
+        Element::fromJSON("{\"origin\": \"example.org\"}"));
+    checkAnswer(1);
+
+    // The previous zone is not hurt in any way
+    EXPECT_EQ(ZoneFinder::SUCCESS, server_.getInMemoryClient(RRClass::IN())->
+              findZone(Name("example.org")).zone_finder->
+              find(Name("example.org"), RRType::SOA())->code);
+    // Configure an unreadable zone. Should fail, but leave the original zone
+    // data there
+    const ElementPtr
+        mapBad(Element::fromJSON("{\"datasources\": ["
+                                 "  {"
+                                 "    \"type\": \"memory\","
+                                 "    \"zones\": ["
+                                 "      {"
+                                 "        \"origin\": \"example.org\","
+                                 "        \"file\": \"" + bad_db + "\","
+                                 "        \"filetype\": \"sqlite3\""
+                                 "      }"
+                                 "    ]"
+                                 "  }"
+                                 "]}"));
+    module_session.setLocalConfig(mapBad);
+    result_ = execAuthServerCommand(server_, "loadzone",
+        Element::fromJSON("{\"origin\": \"example.com\"}"));
+    checkAnswer(1);
+    // The previous zone is not hurt in any way
+    EXPECT_EQ(ZoneFinder::SUCCESS, server_.getInMemoryClient(RRClass::IN())->
+              findZone(Name("example.org")).zone_finder->
+              find(Name("example.org"), RRType::SOA())->code);
+
+    // Broken configuration (not valid against the spec)
+    const ElementPtr
+        broken(Element::fromJSON("{\"datasources\": ["
+                                 "  {"
+                                 "    \"type\": \"memory\","
+                                 "    \"zones\": [[]]"
+                                 "  }"
+                                 "]}"));
+    module_session.setLocalConfig(broken);
+    checkAnswer(1);
+    // The previous zone is not hurt in any way
+    EXPECT_EQ(ZoneFinder::SUCCESS, server_.getInMemoryClient(RRClass::IN())->
+              findZone(Name("example.org")).zone_finder->
+              find(Name("example.org"), RRType::SOA())->code);
+}
+
 TEST_F(AuthCommandTest, loadBrokenZone) {
     configureZones(server_);
 
-    ASSERT_EQ(0, system(INSTALL_PROG " " TEST_DATA_DIR
+    ASSERT_EQ(0, system(INSTALL_PROG " -c " TEST_DATA_DIR
                         "/test1-broken.zone.in "
                         TEST_DATA_BUILDDIR "/test1.zone.copied"));
     result_ = execAuthServerCommand(server_, "loadzone",
@@ -262,7 +390,7 @@ TEST_F(AuthCommandTest, loadUnreadableZone) {
     configureZones(server_);
 
     // install the zone file as unreadable
-    ASSERT_EQ(0, system(INSTALL_PROG " -m 000 " TEST_DATA_DIR
+    ASSERT_EQ(0, system(INSTALL_PROG " -c -m 000 " TEST_DATA_DIR
                         "/test1.zone.in "
                         TEST_DATA_BUILDDIR "/test1.zone.copied"));
     result_ = execAuthServerCommand(server_, "loadzone",

src/bin/auth/tests/common_unittest.cc

@@ -36,7 +36,7 @@ public:
         BOOST_FOREACH(const Environ &env, restoreEnviron) {
             if (env.second == NULL) {
                 EXPECT_EQ(0, unsetenv(env.first.c_str())) <<
-                    "Couldn't restore environment, results of other tests"
+                    "Couldn't restore environment, results of other tests "
                     "are uncertain";
             } else {
                 EXPECT_EQ(0, setenv(env.first.c_str(), env.second->c_str(),

src/bin/auth/tests/config_syntax_unittest.cc

@@ -0,0 +1,71 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#include <cc/data.h>
+#include <config/module_spec.h>
+
+#include <gtest/gtest.h>
+
+using namespace isc::data;
+using namespace isc::config;
+
+namespace {
+
+const char* const SPEC_FILE = AUTH_OBJ_DIR "/auth.spec";
+
+class AuthConfigSyntaxTest : public ::testing::Test {
+protected:
+    AuthConfigSyntaxTest() : mspec_(moduleSpecFromFile(SPEC_FILE))
+    {}
+    ModuleSpec mspec_;
+};
+
+TEST_F(AuthConfigSyntaxTest, inmemoryDefaultFileType) {
+    // filetype is optional
+    EXPECT_TRUE(
+        mspec_.validateConfig(
+            Element::fromJSON(
+                "{\"listen_on\": [], \"datasources\": "
+                "  [{\"type\": \"memory\", \"class\": \"IN\", "
+                "    \"zones\": [{\"origin\": \"example.com\","
+                "                 \"file\": \""
+                TEST_DATA_DIR "/example.zone\"}]}]}"), true));
+}
+
+TEST_F(AuthConfigSyntaxTest, inmemorySQLite3Backend) {
+    // Specifying non-default in-memory filetype
+    EXPECT_TRUE(
+        mspec_.validateConfig(
+            Element::fromJSON(
+                "{\"datasources\": "
+                "  [{\"type\": \"memory\","
+                "    \"zones\": [{\"origin\": \"example.com\","
+                "                 \"file\": \""
+                TEST_DATA_DIR "/example.zone\","
+                "                 \"filetype\": \"sqlite3\"}]}]}"), false));
+}
+
+TEST_F(AuthConfigSyntaxTest, badInmemoryFileType) {
+    // filetype must be a string
+    EXPECT_FALSE(
+        mspec_.validateConfig(
+            Element::fromJSON(
+                "{\"datasources\": "
+                "  [{\"type\": \"memory\","
+                "    \"zones\": [{\"origin\": \"example.com\","
+                "                 \"file\": \""
+                TEST_DATA_DIR "/example.zone\","
+                "                 \"filetype\": 42}]}]}"), false));
+}
+}

src/bin/auth/tests/config_unittest.cc

@@ -21,6 +21,7 @@
 
 #include <cc/data.h>
 
+#include <datasrc/data_source.h>
 #include <datasrc/memory_datasrc.h>
 
 #include <xfr/xfrout_client.h>
@@ -29,21 +30,27 @@
 #include <auth/auth_config.h>
 #include <auth/common.h>
 
+#include "datasrc_util.h"
+
 #include <testutils/mockups.h>
 #include <testutils/portconfig.h>
 #include <testutils/socket_request.h>
 
+#include <sstream>
+
+using namespace std;
 using namespace isc::dns;
 using namespace isc::data;
 using namespace isc::datasrc;
 using namespace isc::asiodns;
-using namespace isc::asiolink;
+using namespace isc::auth::unittest;
+using namespace isc::testutils;
 
 namespace {
 class AuthConfigTest : public ::testing::Test {
 protected:
     AuthConfigTest() :
-        dnss_(ios_, NULL, NULL, NULL),
+        dnss_(),
         rrclass(RRClass::IN()),
         server(true, xfrout),
         // The empty string is expected value of the parameter of
@@ -53,8 +60,7 @@ protected:
     {
         server.setDNSService(dnss_);
     }
-    IOService ios_;
-    DNSService dnss_;
+    MockDNSService dnss_;
     const RRClass rrclass;
     MockXfroutClient xfrout;
     AuthSrv server;
@@ -146,6 +152,14 @@ TEST_F(AuthConfigTest, invalidListenAddressConfig) {
 // Try setting addresses trough config
 TEST_F(AuthConfigTest, listenAddressConfig) {
     isc::testutils::portconfig::listenAddressConfig(server);
+
+    // listenAddressConfig should have attempted to create 4 DNS server
+    // objects: two IP addresses, TCP and UDP for each.  For UDP, the "SYNC_OK"
+    // option should have been specified.
+    EXPECT_EQ(2, dnss_.getTCPFdParams().size());
+    EXPECT_EQ(2, dnss_.getUDPFdParams().size());
+    EXPECT_EQ(DNSService::SERVER_SYNC_OK, dnss_.getUDPFdParams().at(0).options);
+    EXPECT_EQ(DNSService::SERVER_SYNC_OK, dnss_.getUDPFdParams().at(1).options);
 }
 
 class MemoryDatasrcConfigTest : public AuthConfigTest {
@@ -191,7 +205,56 @@ TEST_F(MemoryDatasrcConfigTest, addOneZone) {
     // Check it actually loaded something
     EXPECT_EQ(ZoneFinder::SUCCESS, server.getInMemoryClient(rrclass)->findZone(
         Name("ns.example.com.")).zone_finder->find(Name("ns.example.com."),
-        RRType::A()).code);
+        RRType::A())->code);
+}
+
+// This test uses dynamic load of a data source module, and won't work when
+// statically linked.
+TEST_F(MemoryDatasrcConfigTest,
+#ifdef USE_STATIC_LINK
+       DISABLED_addOneWithFiletypeSQLite3
+#else
+       addOneWithFiletypeSQLite3
+#endif
+    )
+{
+    const string test_db = TEST_DATA_BUILDDIR "/auth_test.sqlite3.copied";
+    stringstream ss("example.org. 3600 IN SOA . . 0 0 0 0 0\n");
+    createSQLite3DB(rrclass, Name("example.org"), test_db.c_str(), ss);
+
+    // In-memory with an SQLite3 data source as the backend.
+    parser->build(Element::fromJSON(
+                      "[{\"type\": \"memory\","
+                      "  \"zones\": [{\"origin\": \"example.org\","
+                      "               \"file\": \""
+                      + test_db +  "\","
+                      "               \"filetype\": \"sqlite3\"}]}]"));
+    parser->commit();
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
+
+    // Failure case: the specified zone doesn't exist in the DB file.
+    delete parser;
+    parser = createAuthConfigParser(server, "datasources");
+    EXPECT_THROW(parser->build(
+                     Element::fromJSON(
+                         "[{\"type\": \"memory\","
+                         "  \"zones\": [{\"origin\": \"example.com\","
+                         "               \"file\": \""
+                         + test_db +  "\","
+                         "               \"filetype\": \"sqlite3\"}]}]")),
+                 DataSourceError);
+}
+
+TEST_F(MemoryDatasrcConfigTest, addOneWithFiletypeText) {
+    // Explicitly specifying "text" is okay.
+    parser->build(Element::fromJSON(
+                      "[{\"type\": \"memory\","
+                      "  \"zones\": [{\"origin\": \"example.com\","
+                      "               \"file\": \""
+                      TEST_DATA_DIR "/example.zone\","
+                      "               \"filetype\": \"text\"}]}]"));
+    parser->commit();
+    EXPECT_EQ(1, server.getInMemoryClient(rrclass)->getZoneCount());
 }
 
 TEST_F(MemoryDatasrcConfigTest, addMultiZones) {
@@ -292,7 +355,7 @@ TEST_F(MemoryDatasrcConfigTest, remove) {
     EXPECT_EQ(AuthSrv::InMemoryClientPtr(), server.getInMemoryClient(rrclass));
 }
 
-TEST_F(MemoryDatasrcConfigTest, adDuplicateZones) {
+TEST_F(MemoryDatasrcConfigTest, addDuplicateZones) {
     EXPECT_THROW(parser->build(
                      Element::fromJSON(
                          "[{\"type\": \"memory\","
@@ -306,6 +369,13 @@ TEST_F(MemoryDatasrcConfigTest, adDuplicateZones) {
 }
 
 TEST_F(MemoryDatasrcConfigTest, addBadZone) {
+    // origin and file are missing
+    EXPECT_THROW(parser->build(
+                     Element::fromJSON(
+                         "[{\"type\": \"memory\","
+                         "  \"zones\": [{}]}]")),
+                 AuthConfigError);
+
     // origin is missing
     EXPECT_THROW(parser->build(
                      Element::fromJSON(
@@ -313,6 +383,13 @@ TEST_F(MemoryDatasrcConfigTest, addBadZone) {
                          "  \"zones\": [{\"file\": \"example.zone\"}]}]")),
                  AuthConfigError);
 
+    // file is missing
+    EXPECT_THROW(parser->build(
+                     Element::fromJSON(
+                         "[{\"type\": \"memory\","
+                         "  \"zones\": [{\"origin\": \"example.com\"}]}]")),
+                 AuthConfigError);
+
     // missing zone file
     EXPECT_THROW(parser->build(
                      Element::fromJSON(
@@ -325,7 +402,7 @@ TEST_F(MemoryDatasrcConfigTest, addBadZone) {
                       "[{\"type\": \"memory\","
                       "  \"zones\": [{\"origin\": \"example..com\","
                       "               \"file\": \"example.zone\"}]}]")),
-                 EmptyLabel);
+                 AuthConfigError);
 
     // bogus RR class name
     EXPECT_THROW(parser->build(

src/bin/auth/tests/datasrc_util.cc

@@ -0,0 +1,77 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#include <exceptions/exceptions.h>
+
+#include <dns/masterload.h>
+#include <dns/name.h>
+#include <dns/rrclass.h>
+
+#include <cc/data.h>
+
+#include <datasrc/client.h>
+#include <datasrc/zone.h>
+#include <datasrc/factory.h>
+
+#include "datasrc_util.h"
+
+#include <boost/bind.hpp>
+
+#include <istream>
+
+#include <cstdlib>
+
+using namespace std;
+
+using namespace isc::dns;
+using namespace isc::data;
+using namespace isc::datasrc;
+
+namespace isc {
+namespace auth {
+namespace unittest {
+
+namespace {
+void
+addRRset(ZoneUpdaterPtr updater, ConstRRsetPtr rrset) {
+    updater->addRRset(*rrset);
+}
+}
+
+void
+createSQLite3DB(RRClass zclass, const Name& zname,
+                const char* const db_file, istream& rr_stream)
+{
+    // We always begin with an empty template SQLite3 DB file and install
+    // the zone data from the zone file.
+    const char* const install_cmd_prefix = INSTALL_PROG " -c " TEST_DATA_DIR
+        "/rwtest.sqlite3 ";
+    const string install_cmd = string(install_cmd_prefix) + db_file;
+    if (system(install_cmd.c_str()) != 0) {
+        isc_throw(isc::Unexpected,
+                  "Error setting up; command failed: " << install_cmd);
+    }
+
+    DataSourceClientContainer container("sqlite3",
+                                        Element::fromJSON(
+                                            "{\"database_file\": \"" +
+                                            string(db_file) + "\"}"));
+    ZoneUpdaterPtr updater = container.getInstance().getUpdater(zname, true);
+    masterLoad(rr_stream, zname, zclass, boost::bind(addRRset, updater, _1));
+    updater->commit();
+}
+
+} // end of unittest
+} // end of auth
+} // end of isc

src/bin/auth/tests/datasrc_util.h

@@ -0,0 +1,58 @@
+// Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+//
+// Permission to use, copy, modify, and/or distribute this software for any
+// purpose with or without fee is hereby granted, provided that the above
+// copyright notice and this permission notice appear in all copies.
+//
+// THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+// AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+// OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+// PERFORMANCE OF THIS SOFTWARE.
+
+#ifndef __AUTH_DATA_SOURCE_UTIL_H
+#define __AUTH_DATA_SOURCE_UTIL_H 1
+
+#include <dns/name.h>
+#include <dns/rrclass.h>
+
+#include <istream>
+
+namespace isc {
+namespace auth {
+namespace unittest {
+
+/// \brief Create an SQLite3 database file for a given zone from a stream.
+///
+/// This function creates an SQLite3 DB file for the specified zone
+/// with specified content.  The zone will be created in the given
+/// SQLite3 database file.  The database file does not have to exist;
+/// this function will automatically create a new file for the test
+/// based on a template that only contains the necessary schema. If
+/// the given file already exists this function overrides the content
+/// (so basically the file must be an ephemeral one only for that test
+/// case).
+///
+/// The input stream must produce strings as the corresponding
+/// \c dns::masterLoad() function expects.
+///
+/// \param zclass The RR class of the zone
+/// \param zname The origin name of the zone
+/// \param db_file The SQLite3 data base file in which the zone data should be
+/// installed.
+/// \param rr_stream An input stream that produces zone data.
+void
+createSQLite3DB(dns::RRClass zclass, const dns::Name& zname,
+                const char* const db_file, std::istream& rr_stream);
+
+} // end of unittest
+} // end of auth
+} // end of isc
+
+#endif  // __AUTH_DATA_SOURCE_UTIL_H
+
+// Local Variables:
+// mode: c++
+// End:

src/bin/auth/tests/statistics_unittest.cc

@@ -281,6 +281,8 @@ TEST_F(AuthCountersTest, submitStatisticsWithoutValidator) {
                          ->get(0)->stringValue());
     EXPECT_EQ("Auth", statistics_session_.sent_msg->get("command")
                          ->get(1)->get("owner")->stringValue());
+    EXPECT_EQ(statistics_session_.sent_msg->get("command")
+              ->get(1)->get("pid")->intValue(), getpid());
     ConstElementPtr statistics_data = statistics_session_.sent_msg
                                           ->get("command")->get(1)
                                           ->get("data");

src/bin/bind10/.gitignore

@@ -0,0 +1,3 @@
+/bind10
+/bind10_src.py
+/run_bind10.sh

src/bin/bind10/bind10.xml

@@ -2,7 +2,7 @@
                "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
 	       [<!ENTITY mdash "&#8212;">]>
 <!--
- - Copyright (C) 2010-2011  Internet Systems Consortium, Inc. ("ISC")
+ - Copyright (C) 2010-2012  Internet Systems Consortium, Inc. ("ISC")
  -
  - Permission to use, copy, modify, and/or distribute this software for any
  - purpose with or without fee is hereby granted, provided that the above
@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>November 23, 2011</date>
+    <date>March 1, 2012</date>
   </refentryinfo>
 
   <refmeta>
@@ -36,7 +36,7 @@
 
   <docinfo>
     <copyright>
-      <year>2011</year>
+      <year>2010-2012</year>
       <holder>Internet Systems Consortium, Inc. ("ISC")</holder>
     </copyright>
   </docinfo>
@@ -45,6 +45,7 @@
     <cmdsynopsis>
       <command>bind10</command>
       <arg><option>-c <replaceable>config-filename</replaceable></option></arg>
+      <arg><option>-i</option></arg>
       <arg><option>-m <replaceable>file</replaceable></option></arg>
       <arg><option>-n</option></arg>
       <arg><option>-p <replaceable>data_path</replaceable></option></arg>
@@ -56,6 +57,7 @@
       <arg><option>--data-path</option> <replaceable>directory</replaceable></arg>
       <arg><option>--msgq-socket-file <replaceable>file</replaceable></option></arg>
       <arg><option>--no-cache</option></arg>
+      <arg><option>--no-kill</option></arg>
       <arg><option>--pid-file</option> <replaceable>filename</replaceable></arg>
       <arg><option>--pretty-name <replaceable>name</replaceable></option></arg>
       <arg><option>--user <replaceable>user</replaceable></option></arg>
@@ -97,8 +99,8 @@
         <listitem>
           <para>The configuration filename to use. Can be either absolute or
           relative to data path. In case it is absolute, value of data path is
-          not considered.</para>
-          <para>Defaults to b10-config.db.</para>
+          not considered.
+          Defaults to <filename>b10-config.db</filename>.</para>
         </listitem>
       </varlistentry>
 
@@ -123,9 +125,11 @@
         </term>
         <listitem>
           <para>The path where BIND 10 programs look for various data files.
-          Currently only b10-cfgmgr uses it to locate the configuration file,
-          but the usage might be extended for other programs and other types
-          of files.</para>
+	  Currently only
+	  <citerefentry><refentrytitle>b10-cfgmgr</refentrytitle><manvolnum>8</manvolnum></citerefentry>
+	  uses it to locate the configuration file, but the usage
+	  might be extended for other programs and other types of
+	  files.</para>
         </listitem>
       </varlistentry>
 
@@ -154,10 +158,20 @@
       </varlistentry>
 
       <varlistentry>
+        <term><option>-i</option>, <option>--no-kill</option></term>
+        <listitem>
+	  <para>When this option is passed, <command>bind10</command>
+	  does not send SIGTERM and SIGKILL signals to modules during
+	  shutdown. (This option was introduced for use during
+	  testing.)</para>
+        </listitem>
+      </varlistentry>
+
+      <varlistentry>
         <term><option>-u</option> <replaceable>user</replaceable>, <option>--user</option> <replaceable>name</replaceable></term>
+<!-- TODO: example more detail. -->
         <listitem>
           <para>The username for <command>bind10</command> to run as.
-<!-- TODO: example more detail. -->
             <command>bind10</command> must be initially ran as the
             root user to use this option.
             The default is to run as the current user.</para>
@@ -169,7 +183,6 @@
         <listitem>
           <para>If defined, the PID of the <command>bind10</command> is stored
              in this file.
-             This is used for testing purposes.
           </para>
          </listitem>
       </varlistentry>
@@ -201,11 +214,12 @@ The default is the basename of ARG 0.
       <varlistentry>
         <term><option>-w</option> <replaceable>wait_time</replaceable>, <option>--wait</option> <replaceable>wait_time</replaceable></term>
         <listitem>
-          <para>Sets the amount of time that BIND 10 will wait for
-          the configuration manager (a key component of BIND 10) to
-          initialize itself before abandoning the start up and
-          terminating with an error.  The wait_time is specified in
-          seconds and has a default value of 10.
+	  <para>Sets the amount of time that BIND 10 will wait for
+	  the configuration manager (a key component of BIND 10)
+	  to initialize itself before abandoning the start up and
+	  terminating with an error.  The
+	  <replaceable>wait_time</replaceable> is specified in
+	  seconds and has a default value of 10.
           </para>
         </listitem>
       </varlistentry>
@@ -230,48 +244,24 @@ TODO: configuration section
     <itemizedlist>
 
       <listitem>
-        <para> <varname>/Boss/components/b10-auth</varname> </para>
-      </listitem>
-
-      <listitem>
         <para> <varname>/Boss/components/b10-cmdctl</varname> </para>
       </listitem>
 
       <listitem>
-        <para> <varname>/Boss/components/setuid</varname> </para>
-      </listitem>
-
-      <listitem>
         <para> <varname>/Boss/components/b10-stats</varname> </para>
       </listitem>
 
-      <listitem>
-        <para> <varname>/Boss/components/b10-stats-httpd</varname> </para>
-      </listitem>
-
-      <listitem>
-        <para> <varname>/Boss/components/b10-xfrin</varname> </para>
-      </listitem>
-
-      <listitem>
-        <para> <varname>/Boss/components/b10-xfrout</varname> </para>
-      </listitem>
-
-      <listitem>
-        <para> <varname>/Boss/components/b10-zonemgr</varname> </para>
-      </listitem>
-
     </itemizedlist>
 
     <para>
       (Note that the startup of <command>b10-sockcreator</command>,
       <command>b10-cfgmgr</command>, and <command>b10-msgq</command>
-      is not configurable. It is hardcoded and <command>bind10</command>
+      is not configurable. They are hardcoded and <command>bind10</command>
       will not run without them.)
     </para>
 
     <para>
-      These named sets (listed above) contain the following settings:
+      The named sets for components contain the following settings:
     </para>
 
     <variablelist>
@@ -346,7 +336,7 @@ list
           <term> <varname>special</varname> </term>
         <listitem>
           <para>
-            This defines if the component is started a special
+            This defines if the component is started a special, hardcoded
             way.
 <!--
 TODO: document this ... but maybe some of these will be removed
@@ -357,7 +347,6 @@ cfgmgr
 cmdctl
 msgq
 resolver
-setuid
 sockcreator
 xfrin
 -->
@@ -374,6 +363,22 @@ xfrin
     </para>
 <!-- TODO: let's just let bind10 be known as bind10 and not Boss -->
 
+<!-- TODO -->
+<!--
+    <para>
+      <command>drop_socket</command>
+      This is an internal command and not exposed to the administrator.
+    </para>
+-->
+
+<!-- TODO -->
+<!--
+    <para>
+      <command>get_socket</command>
+      This is an internal command and not exposed to the administrator.
+    </para>
+-->
+
     <para>
       <command>getstats</command> tells <command>bind10</command>
       to send its statistics data to the <command>b10-stats</command>
@@ -420,13 +425,13 @@ xfrin
 
     <para>
       The statistics data collected by the <command>b10-stats</command>
-      daemon include:
+      daemon for <quote>Boss</quote> include:
     </para>
 
     <variablelist>
 
       <varlistentry>
-        <term>bind10.boot_time</term>
+        <term>boot_time</term>
         <listitem><para>
           The date and time that the <command>bind10</command>
           process started.
@@ -438,13 +443,16 @@ xfrin
 
   </refsect1>
 
-<!--
   <refsect1>
     <title>FILES</title>
-    <para><filename></filename>
+    <para><filename>sockcreator-XXXXXX/sockcreator</filename>
+    &mdash;
+    the Unix Domain socket located in a temporary file directory for
+    <command>b10-sockcreator</command>
+<!--    <citerefentry><refentrytitle>b10-sockcreator</refentrytitle><manvolnum>8</manvolnum></citerefentry> -->
+    communication.
     </para>
   </refsect1>
--->
 
   <refsect1>
     <title>SEE ALSO</title>
@@ -476,6 +484,9 @@ xfrin
       <citetitle>BIND 10 Guide</citetitle>.
     </para>
   </refsect1>
+<!-- <citerefentry>
+        <refentrytitle>b10-sockcreator</refentrytitle><manvolnum>8</manvolnum>
+      </citerefentry>, -->
 
   <refsect1 id='history'><title>HISTORY</title>
     <para>The development of <command>bind10</command>

src/bin/bind10/bind10_messages.mes

@@ -24,7 +24,7 @@ needs a dedicated message bus.
 An error was encountered when the boss module specified
 statistics data which is invalid for the boss specification file.
 
-% BIND10_COMPONENT_FAILED component %1 (pid %2) failed with %3 exit status
+% BIND10_COMPONENT_FAILED component %1 (pid %2) failed: %3
 The process terminated, but the bind10 boss didn't expect it to, which means
 it must have failed.
 

src/bin/bind10/bind10_src.py.in

@@ -167,8 +167,9 @@ class BoB:
     """Boss of BIND class."""
     
     def __init__(self, msgq_socket_file=None, data_path=None,
-    config_filename=None, nocache=False, verbose=False, setuid=None,
-    username=None, cmdctl_port=None, wait_time=10):
+                 config_filename=None, clear_config=False, nocache=False,
+                 verbose=False, nokill=False, setuid=None, username=None,
+                 cmdctl_port=None, wait_time=10):
         """
             Initialize the Boss of BIND. This is a singleton (only one can run).
         
@@ -208,8 +209,10 @@ class BoB:
         self.uid = setuid
         self.username = username
         self.verbose = verbose
+        self.nokill = nokill
         self.data_path = data_path
         self.config_filename = config_filename
+        self.clear_config = clear_config
         self.cmdctl_port = cmdctl_port
         self.wait_time = wait_time
         self._component_configurator = isc.bind10.component.Configurator(self,
@@ -465,6 +468,8 @@ class BoB:
             args.append("--data-path=" + self.data_path)
         if self.config_filename is not None:
             args.append("--config-filename=" + self.config_filename)
+        if self.clear_config:
+            args.append("--clear-config")
         bind_cfgd = ProcessInfo("b10-cfgmgr", args,
                                 self.c_channel_env)
         bind_cfgd.spawn()
@@ -702,32 +707,36 @@ class BoB:
         # still not enough.
         time.sleep(1)
         self.reap_children()
-        # next try sending a SIGTERM
-        components_to_stop = list(self.components.values())
-        for component in components_to_stop:
-            logger.info(BIND10_SEND_SIGTERM, component.name(), component.pid())
-            try:
-                component.kill()
-            except OSError:
-                # ignore these (usually ESRCH because the child
-                # finally exited)
-                pass
-        # finally, send SIGKILL (unmaskable termination) until everybody dies
-        while self.components:
-            # XXX: some delay probably useful... how much is uncertain
-            time.sleep(0.1)  
-            self.reap_children()
+
+        # Send TERM and KILL signals to modules if we're not prevented
+        # from doing so
+        if not self.nokill:
+            # next try sending a SIGTERM
             components_to_stop = list(self.components.values())
             for component in components_to_stop:
-                logger.info(BIND10_SEND_SIGKILL, component.name(),
-                            component.pid())
+                logger.info(BIND10_SEND_SIGTERM, component.name(), component.pid())
                 try:
-                    component.kill(True)
+                    component.kill()
                 except OSError:
                     # ignore these (usually ESRCH because the child
                     # finally exited)
                     pass
-        logger.info(BIND10_SHUTDOWN_COMPLETE)
+            # finally, send SIGKILL (unmaskable termination) until everybody dies
+            while self.components:
+                # XXX: some delay probably useful... how much is uncertain
+                time.sleep(0.1)
+                self.reap_children()
+                components_to_stop = list(self.components.values())
+                for component in components_to_stop:
+                    logger.info(BIND10_SEND_SIGKILL, component.name(),
+                                component.pid())
+                    try:
+                        component.kill(True)
+                    except OSError:
+                        # ignore these (usually ESRCH because the child
+                        # finally exited)
+                        pass
+            logger.info(BIND10_SHUTDOWN_COMPLETE)
 
     def _get_process_exit_status(self):
         return os.waitpid(-1, os.WNOHANG)
@@ -892,7 +901,7 @@ class BoB:
         # the need to find the place ourself or bother users. Also, this
         # secures the socket on some platforms, as it creates a private
         # directory.
-        self._tmpdir = tempfile.mkdtemp()
+        self._tmpdir = tempfile.mkdtemp(prefix='sockcreator-')
         # Get the name
         self._socket_path = os.path.join(self._tmpdir, "sockcreator")
         # And bind the socket to the name
@@ -1043,6 +1052,8 @@ def parse_args(args=sys.argv[1:], Parser=OptionParser):
                       help="UNIX domain socket file the b10-msgq daemon will use")
     parser.add_option("-n", "--no-cache", action="store_true", dest="nocache",
                       default=False, help="disable hot-spot cache in authoritative DNS server")
+    parser.add_option("-i", "--no-kill", action="store_true", dest="nokill",
+                      default=False, help="do not send SIGTERM and SIGKILL signals to modules during shutdown")
     parser.add_option("-u", "--user", dest="user", type="string", default=None,
                       help="Change user after startup (must run as root)")
     parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
@@ -1053,6 +1064,10 @@ def parse_args(args=sys.argv[1:], Parser=OptionParser):
     parser.add_option("-c", "--config-file", action="store",
                       dest="config_file", default=None,
                       help="Configuration database filename")
+    parser.add_option("--clear-config", action="store_true",
+                      dest="clear_config", default=False,
+                      help="Create backup of the configuration file and " +
+                           "start with a clean configuration")
     parser.add_option("-p", "--data-path", dest="data_path",
                       help="Directory to search for configuration files",
                       default=None)
@@ -1165,9 +1180,10 @@ def main():
     try:
         # Go bob!
         boss_of_bind = BoB(options.msgq_socket_file, options.data_path,
-                           options.config_file, options.nocache,
-                           options.verbose, setuid, username,
-                           options.cmdctl_port, options.wait_time)
+                           options.config_file, options.clear_config,
+                           options.nocache, options.verbose, options.nokill,
+                           setuid, username, options.cmdctl_port,
+                           options.wait_time)
         startup_result = boss_of_bind.startup()
         if startup_result:
             logger.fatal(BIND10_STARTUP_ERROR, startup_result)

src/bin/bind10/bob.spec

@@ -8,15 +8,7 @@
         "item_type": "named_set",
         "item_optional": false,
         "item_default": {
-          "b10-auth": { "special": "auth", "kind": "needed" },
-          "b10-xfrin": { "address": "Xfrin", "kind": "dispensable" },
-          "b10-xfrout": { "address": "Xfrout", "kind": "dispensable" },
-          "b10-zonemgr": { "address": "Zonemgr", "kind": "dispensable" },
           "b10-stats": { "address": "Stats", "kind": "dispensable" },
-          "b10-stats-httpd": {
-            "address": "StatsHttpd",
-            "kind": "dispensable"
-          },
           "b10-cmdctl": { "special": "cmdctl", "kind": "needed" }
         },
         "named_set_item_spec": {

src/bin/bind10/tests/.gitignore

@@ -0,0 +1 @@
+/bind10_test.py

src/bin/bind10/tests/bind10_test.py.in

@@ -1012,6 +1012,22 @@ class TestParseArgs(unittest.TestCase):
         options = parse_args(['--config-file=config-file'], TestOptParser)
         self.assertEqual('config-file', options.config_file)
 
+    def test_clear_config(self):
+        options = parse_args([], TestOptParser)
+        self.assertEqual(False, options.clear_config)
+        options = parse_args(['--clear-config'], TestOptParser)
+        self.assertEqual(True, options.clear_config)
+
+    def test_nokill(self):
+        options = parse_args([], TestOptParser)
+        self.assertEqual(False, options.nokill)
+        options = parse_args(['--no-kill'], TestOptParser)
+        self.assertEqual(True, options.nokill)
+        options = parse_args([], TestOptParser)
+        self.assertEqual(False, options.nokill)
+        options = parse_args(['-i'], TestOptParser)
+        self.assertEqual(True, options.nokill)
+
     def test_cmdctl_port(self):
         """
         Test it can parse the command control port.
@@ -1160,11 +1176,13 @@ class TestBossComponents(unittest.TestCase):
         # We check somewhere else that the shutdown is actually called
         # from there (the test_kills).
 
-    def test_kills(self):
+    def __real_test_kill(self, nokill = False):
         """
-        Test that the boss kills components which don't want to stop.
+        Helper function that does the actual kill functionality testing.
         """
         bob = MockBob()
+        bob.nokill = nokill
+
         killed = []
         class ImmortalComponent:
             """
@@ -1194,11 +1212,33 @@ class TestBossComponents(unittest.TestCase):
         bob.shutdown()
 
         self.assertTrue(bob.ccs.stopped)
-        self.assertEqual([False, True], killed)
+
+        # Here, killed is an array where False is added if SIGTERM
+        # should be sent, or True if SIGKILL should be sent, in order in
+        # which they're sent.
+        if nokill:
+            self.assertEqual([], killed)
+        else:
+            self.assertEqual([False, True], killed)
+
         self.assertTrue(self.__called)
 
         bob._component_configurator.shutdown = orig
 
+    def test_kills(self):
+        """
+        Test that the boss kills components which don't want to stop.
+        """
+        self.__real_test_kill()
+
+    def test_nokill(self):
+        """
+        Test that the boss *doesn't* kill components which don't want to
+        stop, when asked not to (by passing the --no-kill option which
+        sets bob.nokill to True).
+        """
+        self.__real_test_kill(True)
+
     def test_component_shutdown(self):
         """
         Test the component_shutdown sets all variables accordingly.

src/bin/bindctl/.gitignore

@@ -0,0 +1,3 @@
+/bindctl
+/bindctl_main.py
+/run_bindctl.sh

src/bin/bindctl/Makefile.am

@@ -8,7 +8,7 @@ EXTRA_DIST = $(man_MANS) bindctl.xml
 noinst_SCRIPTS = run_bindctl.sh
 
 python_PYTHON = __init__.py bindcmd.py cmdparse.py exception.py moduleinfo.py \
-		mycollections.py
+		mycollections.py command_sets.py
 pythondir = $(pyexecdir)/bindctl
 
 bindctldir = $(pkgdatadir)

src/bin/bindctl/bindcmd.py

@@ -23,6 +23,7 @@ from cmd import Cmd
 from bindctl.exception import *
 from bindctl.moduleinfo import *
 from bindctl.cmdparse import BindCmdParse
+from bindctl import command_sets
 from xml.dom import minidom
 import isc
 import isc.cc.data
@@ -37,6 +38,7 @@ from hashlib import sha1
 import csv
 import pwd
 import getpass
+import copy
 
 try:
     from collections import OrderedDict
@@ -319,6 +321,8 @@ class BindCmdInterpreter(Cmd):
                                   param_spec = arg)
                 if ("item_default" in arg):
                     param.default = arg["item_default"]
+                if ("item_description" in arg):
+                    param.desc = arg["item_description"]
                 cmd.add_param(param)
             module.add_command(cmd)
         self.add_module_info(module)
@@ -361,12 +365,12 @@ class BindCmdInterpreter(Cmd):
                 if type(name) == int:
                     # lump all extraneous arguments together as one big final one
                     # todo: check if last param type is a string?
-                    if (param_count > 2):
-                        while (param_count > len(command_info.params) - 1):
-                            params[param_count - 2] += params[param_count - 1]
-                            del(params[param_count - 1])
-                            param_count = len(params)
-                            cmd.params = params.copy()
+                    while (param_count > 2 and
+                           param_count > len(command_info.params) - 1):
+                        params[param_count - 2] += " " + params[param_count - 1]
+                        del(params[param_count - 1])
+                        param_count = len(params)
+                        cmd.params = params.copy()
 
                     # (-1, help is always in the all_params list)
                     if name >= len(all_params) - 1:
@@ -391,8 +395,9 @@ class BindCmdInterpreter(Cmd):
                 param_nr += 1
 
         # Convert parameter value according parameter spec file.
-        # Ignore check for commands belongs to module 'config'
-        if cmd.module != CONFIG_MODULE_NAME:
+        # Ignore check for commands belongs to module 'config' or 'execute
+        if cmd.module != CONFIG_MODULE_NAME and\
+           cmd.module != command_sets.EXECUTE_MODULE_NAME:
             for param_name in cmd.params:
                 param_spec = command_info.get_param_with_name(param_name).param_spec
                 try:
@@ -406,16 +411,9 @@ class BindCmdInterpreter(Cmd):
         if cmd.command == "help" or ("help" in cmd.params.keys()):
             self._handle_help(cmd)
         elif cmd.module == CONFIG_MODULE_NAME:
-            try:
-                self.apply_config_cmd(cmd)
-            except isc.cc.data.DataTypeError as dte:
-                print("Error: " + str(dte))
-            except isc.cc.data.DataNotFoundError as dnfe:
-                print("Error: " + str(dnfe))
-            except isc.cc.data.DataAlreadyPresentError as dape:
-                print("Error: " + str(dape))
-            except KeyError as ke:
-                print("Error: missing " + str(ke))
+            self.apply_config_cmd(cmd)
+        elif cmd.module == command_sets.EXECUTE_MODULE_NAME:
+            self.apply_execute_cmd(cmd)
         else:
             self.apply_cmd(cmd)
 
@@ -574,6 +572,14 @@ class BindCmdInterpreter(Cmd):
             self._print_correct_usage(err)
         except isc.cc.data.DataTypeError as err:
             print("Error! ", err)
+        except isc.cc.data.DataTypeError as dte:
+            print("Error: " + str(dte))
+        except isc.cc.data.DataNotFoundError as dnfe:
+            print("Error: " + str(dnfe))
+        except isc.cc.data.DataAlreadyPresentError as dape:
+            print("Error: " + str(dape))
+        except KeyError as ke:
+            print("Error: missing " + str(ke))
 
     def _print_correct_usage(self, ept):
         if isinstance(ept, CmdUnknownModuleSyntaxError):
@@ -726,6 +732,84 @@ class BindCmdInterpreter(Cmd):
 
         self.location = new_location
 
+    def apply_execute_cmd(self, command):
+        '''Handles the 'execute' command, which executes a number of
+           (preset) statements. The command set to execute is either
+           read from a file (e.g. 'execute file <file>'.) or one
+           of the sets as defined in command_sets.py'''
+        if command.command == 'file':
+            try:
+                with open(command.params['filename']) as command_file:
+                    commands = command_file.readlines()
+            except IOError as ioe:
+                print("Error: " + str(ioe))
+                return
+        elif command_sets.has_command_set(command.command):
+            commands = command_sets.get_commands(command.command)
+        else:
+            # Should not be reachable; parser should've caught this
+            raise Exception("Unknown execute command type " + command.command)
+
+        # We have our set of commands now, depending on whether 'show' was
+        # specified, show or execute them
+        if 'show' in command.params and command.params['show'] == 'show':
+            self.__show_execute_commands(commands)
+        else:
+            self.__apply_execute_commands(commands)
+
+    def __show_execute_commands(self, commands):
+        '''Prints the command list without executing them'''
+        for line in commands:
+            print(line.strip())
+
+    def __apply_execute_commands(self, commands):
+        '''Applies the configuration commands from the given iterator.
+           This is the method that catches, comments, echo statements, and
+           other directives. All commands not filtered by this method are
+           interpreted as if they are directly entered in an active session.
+           Lines starting with any of the following characters are not
+           passed directly:
+           # - These are comments
+           ! - These are directives
+               !echo: print the rest of the line
+               !verbose on/off: print the commands themselves too
+               Unknown directives are ignored (with a warning)
+           The execution is stopped if there are any errors.
+        '''
+        verbose = False
+        try:
+            for line in commands:
+                line = line.strip()
+                if verbose:
+                    print(line)
+                if line.startswith('#') or len(line) == 0:
+                    continue
+                elif line.startswith('!'):
+                    if re.match('^!echo ', line, re.I) and len(line) > 6:
+                        print(line[6:])
+                    elif re.match('^!verbose\s+on\s*$', line, re.I):
+                        verbose = True
+                    elif re.match('^!verbose\s+off$', line, re.I):
+                        verbose = False
+                    else:
+                        print("Warning: ignoring unknown directive: " + line)
+                else:
+                    cmd = BindCmdParse(line)
+                    self._validate_cmd(cmd)
+                    self._handle_cmd(cmd)
+        except (isc.config.ModuleCCSessionError,
+                IOError, http.client.HTTPException,
+                BindCtlException, isc.cc.data.DataTypeError,
+                isc.cc.data.DataNotFoundError,
+                isc.cc.data.DataAlreadyPresentError,
+                KeyError) as err:
+            print('Error: ', err)
+            print()
+            print('Depending on the contents of the script, and which')
+            print('commands it has called, there can be committed and')
+            print('local changes. It is advised to check your settings,')
+            print('and revert local changes with "config revert".')
+
     def apply_cmd(self, cmd):
         '''Handles a general module command'''
         url = '/' + cmd.module + '/' + cmd.command

src/bin/bindctl/bindctl_main.py.in

@@ -22,6 +22,7 @@ import sys; sys.path.append ('@@PYTHONPATH@@')
 
 from bindctl.moduleinfo import *
 from bindctl.bindcmd import *
+from bindctl import command_sets
 import pprint
 from optparse import OptionParser, OptionValueError
 import isc.util.process
@@ -146,5 +147,6 @@ if __name__ == '__main__':
     tool = BindCmdInterpreter(server_addr, pem_file=options.cert_chain,
                               csv_file_dir=options.csv_file_dir)
     prepare_config_commands(tool)
+    command_sets.prepare_execute_commands(tool)
     result = tool.run()
     sys.exit(result)

src/bin/bindctl/command_sets.py

@@ -0,0 +1,95 @@
+# Copyright (C) 2012  Internet Systems Consortium.
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
+# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
+# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
+# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
+# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+# This file provides a built-in set of 'execute' commands, for common
+# functions, such as adding an initial auth server.
+# By calling the function prepare_execute_commands, the
+# commands in the command_sets map are added to the virtual
+# component called 'execute'. This is done in bindctl_main.
+
+from bindctl.moduleinfo import *
+# The name of the 'virtual' command set execution module in bindctl
+EXECUTE_MODULE_NAME = 'execute'
+
+# This is a map of command names to lists
+# Each element in the set should itself be a dict containing:
+# 'description': A string with a description of the command set
+# 'commands': A list of bindctl commands
+command_sets = {
+    'init_authoritative_server': {
+        'description':
+            'Configure and run a basic Authoritative server, with default '+
+            'SQLite3 backend, and xfrin and xfrout functionality',
+        'commands':
+            [
+            '!echo adding Authoritative server component',
+            'config add /Boss/components b10-auth',
+            'config set /Boss/components/b10-auth/kind needed',
+            'config set /Boss/components/b10-auth/special auth',
+            '!echo adding Xfrin component',
+            'config add /Boss/components b10-xfrin',
+            'config set /Boss/components/b10-xfrin/address Xfrin',
+            'config set /Boss/components/b10-xfrin/kind dispensable',
+            '!echo adding Xfrout component',
+            'config add /Boss/components b10-xfrout',
+            'config set /Boss/components/b10-xfrout/address Xfrout',
+            'config set /Boss/components/b10-xfrout/kind dispensable',
+            '!echo adding Zone Manager component',
+            'config add /Boss/components b10-zonemgr',
+            'config set /Boss/components/b10-zonemgr/address Zonemgr',
+            'config set /Boss/components/b10-zonemgr/kind dispensable',
+            '!echo Components added. Please enter "config commit" to',
+            '!echo finalize initial setup and run the components.'
+            ]
+    }
+}
+
+def has_command_set(name):
+    return name in command_sets
+
+def get_commands(name):
+    return command_sets[name]['commands']
+
+def get_description(name):
+    return command_sets[name]['description']
+
+# For each
+def prepare_execute_commands(tool):
+    """This function is called by bindctl_main, and sets up the commands
+       defined here for use in bindctl."""
+    # common parameter
+    param_show = ParamInfo(name="show", type="string", optional=True,
+        desc="Show the list of commands without executing them")
+
+    # The command module
+    module = ModuleInfo(name=EXECUTE_MODULE_NAME,
+                        desc="Execute a given set of commands")
+
+    # Command to execute a file
+    cmd = CommandInfo(name="file", desc="Read commands from file")
+    param = ParamInfo(name="filename", type="string", optional=False,
+                      desc="File to read the set of commands from.")
+    cmd.add_param(param)
+    cmd.add_param(param_show)
+    module.add_command(cmd)
+
+    # and loop through all command sets defined above
+    for name in command_sets:
+        cmd = CommandInfo(name=name, desc=get_description(name))
+        cmd.add_param(param_show)
+        module.add_command(cmd)
+
+    tool.add_module_info(module)
+

src/bin/bindctl/moduleinfo.py

@@ -57,8 +57,12 @@ class ParamInfo:
     def __str__(self):        
         return str("\t%s <type: %s> \t(%s)" % (self.name, self.type, self.desc))
 
-    def get_name(self):
-        return "%s <type: %s>" % (self.name, self.type)
+    def get_basic_info(self):
+        if self.is_optional:
+            opt_str = "optional"
+        else:
+            opt_str = "mandatory"
+        return "%s (%s, %s)" % (self.name, self.type, opt_str)
 
     def get_desc(self):
         return self.desc
@@ -155,37 +159,24 @@ class CommandInfo:
         """Prints the help info for this command to stdout"""
         print("Command ", self)
         print("\t\thelp (Get help for command)")
-                
+
         params = self.params.copy()
         del params["help"]
 
         if len(params) == 0:
-            print("No parameters for the command")
+            print("This command has no parameters")
             return
-        
-        print("\nMandatory parameters:")
-        mandatory_infos = []
-        for info in params.values():            
-            if not info.is_optional:
-                print("    %s" % info.get_name())
-                print(textwrap.fill(info.get_desc(),
-                      initial_indent="        ",
-                      subsequent_indent="        ",
-                      width=70))
-                mandatory_infos.append(info)
 
-        optional_infos = [info for info in params.values() 
-                          if info not in mandatory_infos]
-        if len(optional_infos) > 0:
-            print("\nOptional parameters:")      
-            for info in optional_infos:
-                print("    %s" % info.get_name())
-                print(textwrap.fill(info.get_desc(),
+        print("Parameters:")
+        for info in params.values():
+            print("    %s" % info.get_basic_info())
+            description = info.get_desc()
+            if description != "":
+                print(textwrap.fill(description,
                       initial_indent="        ",
                       subsequent_indent="        ",
                       width=70))
 
-
 class ModuleInfo:
     """Define the information of one module, include module name, 
     module supporting commands.

src/bin/bindctl/tests/.gitignore

@@ -0,0 +1 @@
+/bindctl_test

src/bin/bindctl/tests/bindctl_test.py

@@ -180,12 +180,10 @@ class TestCmdSyntax(unittest.TestCase):
         self.my_assert_raise(isc.cc.data.DataTypeError, "zone set zone_name ='cn', port='cn'")
         self.no_assert_raise("zone reload_all")
 
-
     def testCmdUnknownModuleSyntaxError(self):
         self.my_assert_raise(CmdUnknownModuleSyntaxError, "zoned d")
         self.my_assert_raise(CmdUnknownModuleSyntaxError, "dd dd  ")
 
-
     def testCmdUnknownCmdSyntaxError(self):
         self.my_assert_raise(CmdUnknownCmdSyntaxError, "zone dd")
 
@@ -198,6 +196,7 @@ class TestCmdSyntax(unittest.TestCase):
     def testCmdUnknownParamSyntaxError(self):
         self.my_assert_raise(CmdUnknownParamSyntaxError, "zone load zone_d='cn'")
         self.my_assert_raise(CmdUnknownParamSyntaxError, "zone reload_all zone_name = 'cn'")
+        self.my_assert_raise(CmdUnknownParamSyntaxError, "zone help a b c")
 
 class TestModuleInfo(unittest.TestCase):
 
@@ -366,10 +365,20 @@ class TestConfigCommands(unittest.TestCase):
         self.assertEqual((5, MultiConfigData.LOCAL),
                          self.tool.config_data.get_value("/foo/an_int"))
 
+        cmd = cmdparse.BindCmdParse("config unset identifier=\"foo/an_int\"")
+        self.tool.apply_config_cmd(cmd)
+
+        self.assertEqual((1, MultiConfigData.DEFAULT),
+                         self.tool.config_data.get_value("/foo/an_int"))
+
         # this should raise a NotFoundError
         cmd = cmdparse.BindCmdParse("config set identifier=\"foo/bar\" value=\"[]\"")
         self.assertRaises(isc.cc.data.DataNotFoundError, self.tool.apply_config_cmd, cmd)
 
+        cmd = cmdparse.BindCmdParse("config unset identifier=\"foo/bar\"")
+        self.assertRaises(isc.cc.data.DataNotFoundError,
+                          self.tool.apply_config_cmd, cmd)
+
         # this should raise a TypeError
         cmd = cmdparse.BindCmdParse("config set identifier=\"foo/an_int\" value=\"[]\"")
         self.assertRaises(isc.cc.data.DataTypeError, self.tool.apply_config_cmd, cmd)

src/bin/cfgmgr/.gitignore

@@ -0,0 +1,2 @@
+/b10-cfgmgr
+/b10-cfgmgr.py

src/bin/cfgmgr/b10-cfgmgr.py.in

@@ -49,6 +49,10 @@ def parse_options(args=sys.argv[1:], Parser=OptionParser):
                       help="Configuration database filename " +
                       "(default=" + DEFAULT_CONFIG_FILE + ")",
                       default=DEFAULT_CONFIG_FILE)
+    parser.add_option("--clear-config", action="store_true",
+                      dest="clear_config", default=False,
+                      help="Back up the configuration file and start with " +
+                           "a clean one")
     (options, args) = parser.parse_args(args)
     if args:
         parser.error("No non-option arguments allowed")
@@ -85,7 +89,8 @@ def main():
     options = parse_options()
     global cm
     try:
-        cm = ConfigManager(options.data_path, options.config_file)
+        cm = ConfigManager(options.data_path, options.config_file,
+                           None, options.clear_config)
         signal.signal(signal.SIGINT, signal_handler)
         signal.signal(signal.SIGTERM, signal_handler)
         cm.read_config()

src/bin/cfgmgr/tests/.gitignore

@@ -0,0 +1 @@
+/b10-cfgmgr_test.py

src/bin/cfgmgr/tests/b10-cfgmgr_test.py.in

@@ -24,12 +24,13 @@ import bind10_config
 from isc.testutils.parse_args import OptsError, TestOptParser
 
 class MyConfigManager:
-    def __init__(self, path, filename):
+    def __init__(self, path, filename, session=None, rename_config_file=False):
         self._path = path
         self.read_config_called = False
         self.notify_boss_called = False
         self.run_called = False
         self.write_config_called = False
+        self.rename_config_called = False
         self.running = True
         self.virtual_modules = []
 
@@ -45,6 +46,9 @@ class MyConfigManager:
     def write_config(self):
         self.write_config_called = True
 
+    def rename_config_file(self, ofile, nfile):
+        self.rename_config_called = True
+
     def set_virtual_module(self, spec, function):
         self.virtual_modules.append((spec, function))
 
@@ -90,6 +94,7 @@ class TestConfigManagerStartup(unittest.TestCase):
         self.assertTrue(self.loaded_plugins)
         # if there are no changes, config is not written
         self.assertFalse(b.cm.write_config_called)
+        self.assertFalse(b.cm.rename_config_called)
 
         self.assertTrue(b.cm.running)
         b.signal_handler(None, None)
@@ -187,6 +192,14 @@ class TestParseArgs(unittest.TestCase):
         self.assertRaises(OptsError, b.parse_options, ['--config-filename'],
                           TestOptParser)
 
+    def test_clear_config(self):
+        b = __import__("b10-cfgmgr")
+        parsed = b.parse_options([], TestOptParser)
+        self.assertFalse(parsed.clear_config)
+        parsed = b.parse_options(['--clear-config'], TestOptParser)
+        self.assertTrue(parsed.clear_config)
+        
+
 if __name__ == '__main__':
     unittest.main()
 

src/bin/cmdctl/.gitignore

@@ -0,0 +1,5 @@
+/b10-cmdctl
+/cmdctl.py
+/cmdctl.spec
+/cmdctl.spec.pre
+/run_b10-cmdctl.sh

src/bin/cmdctl/b10-cmdctl.8

@@ -2,12 +2,12 @@
 .\"     Title: b10-cmdctl
 .\"    Author: [see the "AUTHORS" section]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: March 9, 2010
+.\"      Date: February 28, 2012
 .\"    Manual: BIND10
 .\"    Source: BIND10
 .\"  Language: English
 .\"
-.TH "B10\-CMDCTL" "8" "March 9, 2010" "BIND10" "BIND10"
+.TH "B10\-CMDCTL" "8" "February 28, 2012" "BIND10" "BIND10"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -70,6 +70,33 @@ Enable verbose mode\&.
 .RS 4
 Display the version number and exit\&.
 .RE
+.SH "CONFIGURATION AND COMMANDS"
+.PP
+The configurable settings are:
+.PP
+
+\fIaccounts_file\fR
+defines the path to the user accounts database\&. The default is
+/usr/local/etc/bind10\-devel/cmdctl\-accounts\&.csv\&.
+.PP
+
+\fIcert_file\fR
+defines the path to the PEM certificate file\&. The default is
+/usr/local/etc/bind10\-devel/cmdctl\-certfile\&.pem\&.
+.PP
+
+\fIkey_file\fR
+defines the path to the PEM private key file\&. The default is
+/usr/local/etc/bind10\-devel/cmdctl\-keyfile\&.pem\&.
+.PP
+The configuration command is:
+.PP
+
+\fBshutdown\fR
+exits
+\fBb10\-cmdctl\fR\&. This has an optional
+\fIpid\fR
+argument to select the process ID to stop\&. (Note that the BIND 10 boss process may restart this service if configured\&.)
 .SH "FILES"
 .PP
 /usr/local/etc/bind10\-devel/cmdctl\-accounts\&.csv
@@ -93,5 +120,5 @@ The
 daemon was initially designed and coded by Zhang Likun of CNNIC\&.
 .SH "COPYRIGHT"
 .br
-Copyright \(co 2010 Internet Systems Consortium, Inc. ("ISC")
+Copyright \(co 2010-2012 Internet Systems Consortium, Inc. ("ISC")
 .br

src/bin/cmdctl/b10-cmdctl.xml

@@ -2,7 +2,7 @@
                "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
 	       [<!ENTITY mdash "&#8212;">]>
 <!--
- - Copyright (C) 2010  Internet Systems Consortium, Inc. ("ISC")
+ - Copyright (C) 2010-2012  Internet Systems Consortium, Inc. ("ISC")
  -
  - Permission to use, copy, modify, and/or distribute this software for any
  - purpose with or without fee is hereby granted, provided that the above
@@ -20,7 +20,7 @@
 <refentry>
 
   <refentryinfo>
-    <date>March 9, 2010</date>
+    <date>February 28, 2012</date>
   </refentryinfo>
 
   <refmeta>
@@ -37,7 +37,7 @@
 
   <docinfo>
     <copyright>
-      <year>2010</year>
+      <year>2010-2012</year>
       <holder>Internet Systems Consortium, Inc. ("ISC")</holder>
     </copyright>
   </docinfo>
@@ -138,6 +138,50 @@
   </refsect1>
 
   <refsect1>
+    <title>CONFIGURATION AND COMMANDS</title>
+    <para>
+      The configurable settings are:
+    </para>
+
+    <para>
+      <varname>accounts_file</varname> defines the path to the
+      user accounts database.
+      The default is
+      <filename>/usr/local/etc/bind10-devel/cmdctl-accounts.csv</filename>.
+    </para>
+
+    <para>
+      <varname>cert_file</varname> defines the path to the
+      PEM certificate file.
+      The default is
+      <filename>/usr/local/etc/bind10-devel/cmdctl-certfile.pem</filename>.
+    </para>
+
+    <para>
+      <varname>key_file</varname> defines the path to the PEM private key
+      file.
+      The default is
+      <filename>/usr/local/etc/bind10-devel/cmdctl-keyfile.pem</filename>.
+    </para>
+
+<!-- TODO: formating -->
+    <para>
+      The configuration command is:
+    </para>
+
+<!-- NOTE: print_settings is not documented since I think will be removed -->
+
+    <para>
+      <command>shutdown</command> exits <command>b10-cmdctl</command>.
+      This has an optional <varname>pid</varname> argument to
+      select the process ID to stop.
+      (Note that the BIND 10 boss process may restart this service
+      if configured.)
+    </para>
+
+  </refsect1>
+
+  <refsect1>
     <title>FILES</title>
 <!-- TODO: replace /usr/local -->
 <!-- TODO: permissions -->

src/bin/cmdctl/tests/.gitignore

@@ -0,0 +1 @@
+/cmdctl_test

src/bin/dbutil/.gitignore

@@ -0,0 +1,3 @@
+/b10-dbutil
+/dbutil.py
+/run_dbutil.sh

src/bin/dbutil/Makefile.am

@@ -0,0 +1,39 @@
+SUBDIRS = . tests
+
+bin_SCRIPTS = b10-dbutil
+man_MANS = b10-dbutil.8
+
+nodist_pylogmessage_PYTHON = $(PYTHON_LOGMSGPKG_DIR)/work/dbutil_messages.py
+pylogmessagedir = $(pyexecdir)/isc/log_messages/
+
+EXTRA_DIST = $(man_MANS) b10-dbutil.xml dbutil_messages.mes
+
+noinst_SCRIPTS = run_dbutil.sh
+
+CLEANFILES = b10-dbutil b10-dbutil.pyc
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/dbutil_messages.py
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/dbutil_messages.pyc
+CLEANFILES += $(PYTHON_LOGMSGPKG_DIR)/work/dbutil_messages.pyo
+
+if ENABLE_MAN
+
+b10-dbutil.8: b10-dbutil.xml
+	xsltproc --novalid --xinclude --nonet -o $@ http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl $(srcdir)/b10-dbutil.xml
+
+endif
+
+# Define rule to build logging source files from message file
+$(PYTHON_LOGMSGPKG_DIR)/work/dbutil_messages.py : dbutil_messages.mes
+	$(top_builddir)/src/lib/log/compiler/message \
+	-d $(PYTHON_LOGMSGPKG_DIR)/work -p $(srcdir)/dbutil_messages.mes
+
+b10-dbutil: dbutil.py $(PYTHON_LOGMSGPKG_DIR)/work/dbutil_messages.py
+	$(SED) -e "s|@@PYTHONPATH@@|@pyexecdir@|" \
+	       -e "s|@@SYSCONFDIR@@|@sysconfdir@|" \
+	       -e "s|@@LIBEXECDIR@@|$(pkglibexecdir)|" dbutil.py >$@
+	chmod a+x $@
+
+CLEANDIRS = __pycache__
+
+clean-local:
+	rm -rf $(CLEANDIRS)

src/bin/dbutil/b10-dbutil.xml

@@ -0,0 +1,192 @@
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+               "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
+	       [<!ENTITY mdash "&#8212;">]>
+<!--
+ - Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+ -
+ - Permission to use, copy, modify, and/or distribute this software for any
+ - purpose with or without fee is hereby granted, provided that the above
+ - copyright notice and this permission notice appear in all copies.
+ -
+ - THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+ - REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+ - AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+ - INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+ - LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+ - OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ - PERFORMANCE OF THIS SOFTWARE.
+-->
+
+<refentry>
+
+  <refentryinfo>
+    <date>March 20, 2012</date>
+  </refentryinfo>
+
+  <refmeta>
+    <refentrytitle>b10-dbutil</refentrytitle>
+    <manvolnum>8</manvolnum>
+    <refmiscinfo>BIND10</refmiscinfo>
+  </refmeta>
+
+  <refnamediv>
+    <refname>b10-dbutil</refname>
+    <refpurpose>Zone Database Maintenance Utility</refpurpose>
+  </refnamediv>
+
+  <docinfo>
+    <copyright>
+      <year>2012</year>
+      <holder>Internet Systems Consortium, Inc. ("ISC")</holder>
+    </copyright>
+  </docinfo>
+
+  <refsynopsisdiv>
+    <cmdsynopsis>
+      <command>b10-dbutil --check</command>
+        <arg>--verbose</arg>
+        <arg>--quiet</arg>
+        <arg><replaceable choice='req'>dbfile</replaceable></arg>
+    </cmdsynopsis>
+    <cmdsynopsis>
+      <command>b10-dbutil --upgrade</command>
+        <arg>--noconfirm</arg>
+        <arg>--verbose</arg>
+        <arg>--quiet</arg>
+        <arg><replaceable choice='req'>dbfile</replaceable></arg>
+    </cmdsynopsis>
+  </refsynopsisdiv>
+
+  <refsect1>
+    <title>DESCRIPTION</title>
+    <para>
+      The <command>b10-dbutil</command> utility is a general administration
+      utility for SQL databases. (Currently only SQLite is supported by
+      BIND 10.)  It can report the current verion of the schema, and upgrade
+      an existing database to the latest version of the schema.
+    </para>
+
+    <para>
+      <command>b10-dbutil</command> operates in one of two modes, check mode
+      or upgrade mode.
+    </para>
+
+    <para>
+      In check mode (<command>b10-dbutil --check</command>), the
+      utility reads the version of the database schema from the database
+      and prints it.  It will tell you whether the schema is at the latest
+      version supported by BIND 10. Exit status is 0 if the schema is at
+      the correct version, 1 if the schema is at an older version, 2 if
+      the schema is at a version not yet supported by this version of
+      b10-dbutil. Any higher value indicates an error during command-line
+      parsing or execution.
+    </para>
+
+    <para>
+      When the upgrade function is selected
+      (<command>b10-dbutil --upgrade</command>), the
+      utility takes a copy of the database, then upgrades it to the latest
+      version of the schema.  The contents of the database remain intact.
+      (The backup file is a file in the same directory as the database
+      file.  It has the same name, with ".backup" appended to it.  If a
+      file of that name already exists, the file will have the suffix
+      ".backup-1".  If that exists, the file will be suffixed ".backup-2",
+      and so on). Exit status is 0 if the upgrade is either succesful or
+      aborted by the user, and non-zero if there is an error.
+    </para>
+
+    <para>
+    When upgrading the database, it is <emphasis>strongly</emphasis>
+    recommended that BIND 10 not be running while the upgrade is in
+    progress.
+    </para>
+
+  </refsect1>
+
+  <refsect1>
+    <title>ARGUMENTS</title>
+
+    <para>The arguments are as follows:</para>
+
+    <variablelist>
+      <varlistentry>
+        <term>
+         <option>--check</option>
+        </term>
+        <listitem>
+          <para>Selects the version check function, which reports the
+          current version of the database.  This is incompatible
+          with the --upgrade option.
+          </para>
+        </listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>
+         <option>--noconfirm</option>
+        </term>
+        <listitem>
+          <para>Only valid with --upgrade, this disables the prompt.
+          Normally the utility will print a warning that an upgrade is
+          about to take place and request that you type "Yes" to continue.
+          If this switch is given on the command line, no prompt will
+          be issued: the utility will just perform the upgrade.
+          </para>
+        </listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>
+         <option>--upgrade</option>
+        </term>
+        <listitem>
+          <para>Selects the upgrade function, which upgrades the database
+          to the latest version of the schema.  This is incompatible
+          with the --upgrade option.
+          </para>
+          <para>
+          The upgrade function will upgrade a BIND 10 database - no matter how
+          old the schema - preserving all data.  A backup file is created
+          before the upgrade (with the same name as the database, but with
+          ".backup" suffixed to it).  If the upgrade fails, this file can
+          be copied back to restore the original database.
+          </para>
+        </listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>
+         <option>--verbose</option>
+        </term>
+        <listitem>
+          <para>Enable verbose mode.  Each SQL command issued by the
+          utility will be printed to stderr before it is executed.</para>
+        </listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>
+         <option>--quiet</option>
+        </term>
+        <listitem>
+          <para>Enable quiet mode. No output is printed, except errors during
+            command-line argument parsing, or the user confirmation dialog.
+          </para>
+        </listitem>
+      </varlistentry>
+
+      <varlistentry>
+        <term>
+        <option><replaceable choice='req'>dbfile</replaceable></option>
+        </term>
+        <listitem>
+          <para>
+          Name of the database file to check of upgrade.
+          </para>
+        </listitem>
+      </varlistentry>
+
+
+    </variablelist>
+  </refsect1>
+</refentry>

src/bin/dbutil/dbutil.py.in

@@ -0,0 +1,608 @@
+#!@PYTHON@
+
+# Copyright (C) 2012  Internet Systems Consortium.
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
+# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
+# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
+# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
+# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+"""
+@file Dabase Utilities
+
+This file holds the "dbutil" program, a general utility program for doing
+management of the BIND 10 database.  There are two modes of operation:
+
+      b10-dbutil --check [--verbose] database
+      b10-dbutil --upgrade [--noconfirm] [--verbose] database
+
+The first form checks the version of the given database.  The second form
+upgrades the database to the latest version of the schema, omitting the
+warning prompt if --noconfirm is given.
+
+For maximum safety, prior to the upgrade a backup database is created.
+The is the database name with ".backup" appended to it (or ".backup-n" if
+".backup" already exists).  This is used to restore the database if the
+upgrade fails.
+"""
+
+# Exit codes
+# These are defined here because one of them is already used before most
+# of the import statements.
+EXIT_SUCCESS = 0
+EXIT_NEED_UPDATE = 1
+EXIT_VERSION_TOO_HIGH = 2
+EXIT_COMMAND_ERROR = 3
+EXIT_READ_ERROR = 4
+EXIT_UPGRADE_ERROR = 5
+EXIT_UNCAUGHT_EXCEPTION = 6
+
+import sys; sys.path.append("@@PYTHONPATH@@")
+
+# Normally, python exits with a status code of 1 on uncaught exceptions
+# Since we reserve exit status 1 for 'database needs upgrade', we
+# override the excepthook to exit with a different status
+def my_except_hook(a, b, c):
+    sys.__excepthook__(a,b,c)
+    sys.exit(EXIT_UNCAUGHT_EXCEPTION)
+sys.excepthook = my_except_hook
+
+import os, sqlite3, shutil
+from optparse import OptionParser
+import isc.util.process
+import isc.log
+from isc.log_messages.dbutil_messages import *
+
+isc.log.init("b10-dbutil")
+logger = isc.log.Logger("dbutil")
+isc.util.process.rename()
+
+TRACE_BASIC = logger.DBGLVL_TRACE_BASIC
+
+
+# @brief Version String
+# This is the version displayed to the user.  It comprises the module name,
+# the module version number, and the overall BIND 10 version number (set in
+# configure.ac)
+VERSION = "b10-dbutil 20120319 (BIND 10 @PACKAGE_VERSION@)"
+
+# @brief Statements to Update the Database
+# These are in the form of a list of dictionaries, each of which contains the
+# information to perform an incremental upgrade from one version of the
+# database to the next.  The information is:
+#
+# a) from: (major, minor) version that the database is expected to be at
+#    to perform this upgrade.
+# b) to: (major, minor) version of the database to which this set of statements
+#    upgrades the database to.  (This is used for documentation purposes,
+#    and to update the schema_version table when the upgrade is complete.)
+# c) statements: List of SQL statments to perform the upgrade.
+#
+# The incremental upgrades are performed one after the other.  If the version
+# of the database does not exactly match that required for the incremental
+# upgrade, the upgrade is skipped.  For this reason, the list must be in
+# ascending order (e.g. upgrade 1.0 to 2.0, 2.0 to 2.1, 2.1 to 2.2 etc.).
+#
+# Note that apart from the 1.0 to 2.0 upgrade, no upgrade need alter the
+# schema_version table: that is done by the upgrade process using the
+# information in the "to" field.
+UPGRADES = [
+    {'from': (1, 0), 'to': (2, 0),
+        'statements': [
+
+            # Move to the latest "V1" state of the database if not there
+            # already.
+            "CREATE TABLE IF NOT EXISTS diffs (" +
+                "id INTEGER PRIMARY KEY, " +
+                "zone_id INTEGER NOT NULL," +
+                "version INTEGER NOT NULL, " +
+                "operation INTEGER NOT NULL, " +
+                "name STRING NOT NULL COLLATE NOCASE, " +
+                "rrtype STRING NOT NULL COLLATE NOCASE, " +
+                "ttl INTEGER NOT NULL, " +
+                "rdata STRING NOT NULL)",
+
+            # Within SQLite with can only rename tables and add columns; we
+            # can't drop columns nor can we alter column characteristics.
+            # So the strategy is to rename the table, create the new table,
+            # then copy all data across.  This means creating new indexes
+            # as well; these are created after the data has been copied.
+
+            # zones table
+            "DROP INDEX zones_byname",
+            "ALTER TABLE zones RENAME TO old_zones",
+            "CREATE TABLE zones (" +
+                "id INTEGER PRIMARY KEY, " +
+                "name TEXT NOT NULL COLLATE NOCASE, " +
+                "rdclass TEXT NOT NULL COLLATE NOCASE DEFAULT 'IN', " +
+                "dnssec BOOLEAN NOT NULL DEFAULT 0)",
+            "INSERT INTO ZONES " +
+                "SELECT id, name, rdclass, dnssec FROM old_zones",
+            "CREATE INDEX zones_byname ON zones (name)",
+            "DROP TABLE old_zones",
+
+            # records table
+            "DROP INDEX records_byname",
+            "DROP INDEX records_byrname",
+            "ALTER TABLE records RENAME TO old_records",
+            "CREATE TABLE records (" +
+                "id INTEGER PRIMARY KEY, " +
+                "zone_id INTEGER NOT NULL, " +
+                "name TEXT NOT NULL COLLATE NOCASE, " +
+                "rname TEXT NOT NULL COLLATE NOCASE, " +
+                "ttl INTEGER NOT NULL, " +
+                "rdtype TEXT NOT NULL COLLATE NOCASE, " +
+                "sigtype TEXT COLLATE NOCASE, " +
+                "rdata TEXT NOT NULL)",
+            "INSERT INTO records " +
+                "SELECT id, zone_id, name, rname, ttl, rdtype, sigtype, " +
+                    "rdata FROM old_records",
+            "CREATE INDEX records_byname ON records (name)",
+            "CREATE INDEX records_byrname ON records (rname)",
+            "CREATE INDEX records_bytype_and_rname ON records (rdtype, rname)",
+            "DROP TABLE old_records",
+
+            # nsec3 table
+            "DROP INDEX nsec3_byhash",
+            "ALTER TABLE nsec3 RENAME TO old_nsec3",
+            "CREATE TABLE nsec3 (" +
+                "id INTEGER PRIMARY KEY, " +
+                "zone_id INTEGER NOT NULL, " +
+                "hash TEXT NOT NULL COLLATE NOCASE, " +
+                "owner TEXT NOT NULL COLLATE NOCASE, " +
+                "ttl INTEGER NOT NULL, " +
+                "rdtype TEXT NOT NULL COLLATE NOCASE, " +
+                "rdata TEXT NOT NULL)",
+            "INSERT INTO nsec3 " +
+                "SELECT id, zone_id, hash, owner, ttl, rdtype, rdata " +
+                    "FROM old_nsec3",
+            "CREATE INDEX nsec3_byhash ON nsec3 (hash)",
+            "DROP TABLE old_nsec3",
+
+            # diffs table
+            "ALTER TABLE diffs RENAME TO old_diffs",
+            "CREATE TABLE diffs (" +
+                "id INTEGER PRIMARY KEY, " +
+                "zone_id INTEGER NOT NULL, " +
+                "version INTEGER NOT NULL, " +
+                "operation INTEGER NOT NULL, " +
+                "name TEXT NOT NULL COLLATE NOCASE, " +
+                "rrtype TEXT NOT NULL COLLATE NOCASE, " +
+                "ttl INTEGER NOT NULL, " +
+                "rdata TEXT NOT NULL)",
+            "INSERT INTO diffs " +
+                "SELECT id, zone_id, version, operation, name, rrtype, " +
+                    "ttl, rdata FROM old_diffs",
+            "DROP TABLE old_diffs",
+
+            # Schema table.  This is updated to include a second column for
+            # future changes.  The idea is that if a version of BIND 10 is
+            # written for schema M.N, it should be able to work for all
+            # versions of N; if not, M must be incremented.
+            #
+            # For backwards compatibility, the column holding the major
+            # version number is left named "version".
+            "ALTER TABLE schema_version " +
+                "ADD COLUMN minor INTEGER NOT NULL DEFAULT 0"
+        ]
+    }
+
+# To extend this, leave the above statements in place and add another
+# dictionary to the list.  The "from" version should be (2, 0), the "to" 
+# version whatever the version the update is to, and the SQL statements are
+# the statements required to perform the upgrade.  This way, the upgrade
+# program will be able to upgrade both a V1.0 and a V2.0 database.
+]
+
+class DbutilException(Exception):
+    """
+    @brief Exception class to indicate error exit
+    """
+    pass
+
+class Database:
+    """
+    @brief Database Encapsulation
+
+    Encapsulates the SQL database, both the connection and the cursor.  The
+    methods will cause a program exit on any error.
+    """
+    def __init__(self, db_file):
+        """
+        @brief Constructor
+
+        @param db_file Name of the database file
+        """
+        self.connection = None
+        self.cursor = None
+        self.db_file = db_file
+        self.backup_file = None
+
+    def open(self):
+        """
+        @brief Open Database
+
+        Opens the passed file as an sqlite3 database and stores a connection
+        and a cursor.
+        """
+        if not os.path.exists(self.db_file):
+            raise DbutilException("database " + self.db_file +
+                                 " does not exist");
+
+        try:
+            self.connection = sqlite3.connect(self.db_file)
+            self.connection.isolation_level = None  # set autocommit
+            self.cursor = self.connection.cursor()
+        except sqlite3.OperationalError as ex:
+            raise DbutilException("unable to open " + self.db_file +
+                                  " - " + str(ex))
+
+    def close(self):
+        """
+        @brief Closes the database
+        """
+        if self.connection is not None:
+            self.connection.close()
+
+    def execute(self, statement):
+        """
+        @brief Execute Statement
+
+        Executes the given statement, exiting the program on error.
+
+        @param statement SQL statement to execute
+        """
+        logger.debug(TRACE_BASIC, DBUTIL_EXECUTE, statement)
+
+        try:
+            self.cursor.execute(statement)
+        except Exception as ex:
+            logger.error(DBUTIL_STATEMENT_ERROR, statement, ex)
+            raise DbutilException(str(ex))
+
+    def result(self):
+        """
+        @brief Return result of last execute
+
+        Returns a single row that is the result of the last "execute".
+        """
+        return self.cursor.fetchone()
+
+    def backup(self):
+        """
+        @brief Backup Database
+
+        Attempts to copy the given database file to a backup database, the
+        backup database file being the file name with ".backup" appended.
+        If the ".backup" file exists, a new name is constructed by appending
+        ".backup-n" (n starting at 1) and the action repeated until an
+        unused filename is found.
+
+        @param db_file Database file to backup
+        """
+        if not os.path.exists(self.db_file):
+            raise DbutilException("database " + self.db_file +
+                                  " does not exist");
+
+        self.backup_file = self.db_file + ".backup"
+        count = 0
+        while os.path.exists(self.backup_file):
+            count = count + 1
+            self.backup_file = self.db_file + ".backup-" + str(count)
+
+        # Do the backup
+        shutil.copyfile(self.db_file, self.backup_file)
+        logger.info(DBUTIL_BACKUP, self.db_file, self.backup_file)
+
+def prompt_user():
+    """
+    @brief Prompt the User
+
+    Explains about the upgrade and requests authorisation to continue.
+
+    @return True if user entered 'Yes', False if 'No'
+    """
+    sys.stdout.write(
+"""You have selected the upgrade option.  This will upgrade the schema of the
+selected BIND 10 zone database to the latest version.
+
+The utility will take a copy of the zone database file before executing so, in
+the event of a problem, you will be able to restore the zone database from
+the backup.  To ensure that the integrity of this backup, please ensure that
+BIND 10 is not running before continuing.
+""")
+    yes_entered = False
+    no_entered = False
+    while (not yes_entered) and (not no_entered):
+        sys.stdout.write("Enter 'Yes' to proceed with the upgrade, " +
+                         "'No' to exit the program: \n")
+        response = sys.stdin.readline()
+        if response.lower() == "yes\n":
+            yes_entered = True
+        elif response.lower() == "no\n":
+            no_entered = True
+        else:
+            sys.stdout.write("Please enter 'Yes' or 'No'\n")
+
+    return yes_entered
+
+
+def version_string(version):
+    """
+    @brief Format Database Version
+
+    Converts a (major, minor) tuple into a 'Vn.m' string.
+
+    @param version Version tuple to convert
+
+    @return Version string
+    """
+    return "V" + str(version[0]) + "." + str(version[1])
+
+
+def compare_versions(first, second):
+    """
+    @brief Compare Versions
+
+    Compares two database version numbers.
+
+    @param first First version number to check (in the form of a
+           "(major, minor)" tuple).
+    @param second Second version number to check (in the form of a
+           "(major, minor)" tuple).
+
+    @return -1, 0, +1 if "first" is <, ==, > "second"
+    """
+    if first == second:
+        return 0
+
+    elif ((first[0] < second[0]) or
+          ((first[0] == second[0]) and (first[1] < second[1]))):
+        return -1
+
+    else:
+        return 1
+
+
+def get_latest_version():
+    """
+    @brief Returns the version to which this utility can upgrade the database
+
+    This is the 'to' version held in the last element of the upgrades list
+    """
+    return UPGRADES[-1]['to']
+
+
+def get_version(db):
+    """
+    @brief Return version of database
+
+    @return Version of database in form (major version, minor version)
+    """
+
+    # Get the version information.
+    db.execute("SELECT * FROM schema_version")
+    result = db.result()
+    if result is None:
+        raise DbutilException("nothing in schema_version table")
+
+    major = result[0]
+    if (major == 1):
+        # If the version number is 1, there will be no "minor" column, so
+        # assume a minor version number of 0.
+        minor = 0
+    else:
+        minor = result[1]
+
+    result = db.result()
+    if result is not None:
+        raise DbutilException("too many rows in schema_version table")
+
+    return (major, minor)
+
+
+def check_version(db):
+    """
+    @brief Check the version
+
+    Checks the version of the database and the latest version, and advises if
+    an upgrade is needed.
+
+    @param db Database object
+
+    returns 0 if the database is up to date
+    returns EXIT_NEED_UPDATE if the database needs updating
+    returns EXIT_VERSION_TOO_HIGH if the database is at a later version
+            than this program knows about
+    These return values are intended to be passed on to sys.exit.
+    """
+    current = get_version(db)
+    latest = get_latest_version()
+
+    match = compare_versions(current, latest)
+    if match == 0:
+        logger.info(DBUTIL_VERSION_CURRENT, version_string(current))
+        logger.info(DBUTIL_CHECK_OK)
+        return EXIT_SUCCESS
+    elif match < 0:
+        logger.info(DBUTIL_VERSION_LOW, version_string(current),
+                    version_string(latest))
+        logger.info(DBUTIL_CHECK_UPGRADE_NEEDED)
+        return EXIT_NEED_UPDATE
+    else:
+        logger.warn(DBUTIL_VERSION_HIGH, version_string(current),
+                    version_string(get_latest_version()))
+        logger.info(DBUTIL_UPGRADE_DBUTIL)
+        return EXIT_VERSION_TOO_HIGH
+
+def perform_upgrade(db, upgrade):
+    """
+    @brief Perform upgrade
+
+    Performs the upgrade.  At the end of the upgrade, updates the schema_version
+    table with the expected version.
+
+    @param db Database object
+    @param upgrade Upgrade dictionary, holding "from", "to" and "statements".
+    """
+    logger.info(DBUTIL_UPGRADING, version_string(upgrade['from']),
+         version_string(upgrade['to']))
+    for statement in upgrade['statements']:
+        db.execute(statement)
+
+    # Update the version information
+    db.execute("DELETE FROM schema_version")
+    db.execute("INSERT INTO schema_version VALUES (" +
+                    str(upgrade['to'][0]) + "," + str(upgrade['to'][1]) + ")")
+
+
+def perform_all_upgrades(db):
+    """
+    @brief Performs all the upgrades
+
+    @brief db Database object
+
+    For each upgrade, checks that the database is at the expected version.
+    If so, calls perform_upgrade to update the database.
+    """
+    match = compare_versions(get_version(db), get_latest_version())
+    if match == 0:
+        logger.info(DBUTIL_UPGRADE_NOT_NEEDED)
+
+    elif match > 0:
+        logger.warn(DBUTIL_UPGRADE_NOT_POSSIBLE)
+
+    else:
+        # Work our way through all upgrade increments
+        count = 0
+        for upgrade in UPGRADES:
+            if compare_versions(get_version(db), upgrade['from']) == 0:
+                perform_upgrade(db, upgrade)
+                count = count + 1
+
+        if count > 0:
+            logger.info(DBUTIL_UPGRADE_SUCCESFUL)
+        else:
+            # Should not get here, as we established earlier that the database
+            # was not at the latest version so we should have upgraded.
+            raise DbutilException("internal error in upgrade tool - no " +
+                                  "upgrade was performed on an old version " +
+                                  "the database")
+
+
+def parse_command():
+    """
+    @brief Parse Command
+
+    Parses the command line and sets the global command options.
+
+    @return Tuple of parser options and parser arguments
+    """
+    usage = ("usage: %prog --check [options] db_file\n" +
+             "       %prog --upgrade [--noconfirm] [options] db_file")
+    parser = OptionParser(usage = usage, version = VERSION)
+    parser.add_option("-c", "--check", action="store_true",
+                      dest="check", default=False,
+                      help="Print database version and check if it " +
+                           "needs upgrading")
+    parser.add_option("-n", "--noconfirm", action="store_true",
+                      dest="noconfirm", default=False,
+                      help="Do not prompt for confirmation before upgrading")
+    parser.add_option("-u", "--upgrade", action="store_true",
+                      dest="upgrade", default=False,
+                      help="Upgrade the database file to the latest version")
+    parser.add_option("-v", "--verbose", action="store_true",
+                      dest="verbose", default=False,
+                      help="Print SQL statements as they are executed")
+    parser.add_option("-q", "--quiet", action="store_true",
+                      dest="quiet", default=False,
+                      help="Don't print any info, warnings or errors")
+    (options, args) = parser.parse_args()
+
+    # Set the database file on which to operate
+    if (len(args) > 1):
+        logger.error(DBUTIL_TOO_MANY_ARGUMENTS)
+        parser.print_usage()
+        sys.exit(EXIT_COMMAND_ERROR)
+    elif len(args) == 0:
+        logger.error(DBUTIL_NO_FILE)
+        parser.print_usage()
+        sys.exit(EXIT_COMMAND_ERROR)
+
+    # Check for conflicting options.  If some are found, output a suitable
+    # error message and print the usage.
+    if options.check and options.upgrade:
+        logger.error(DBUTIL_COMMAND_UPGRADE_CHECK)
+    elif (not options.check) and (not options.upgrade):
+        logger.error(DBUTIL_COMMAND_NONE)
+    elif (options.check and options.noconfirm):
+        logger.error(DBUTIL_CHECK_NOCONFIRM)
+    else:
+        return (options, args)
+
+    # Only get here on conflicting options
+    parser.print_usage()
+    sys.exit(EXIT_COMMAND_ERROR)
+
+
+if __name__ == "__main__":
+    (options, args) = parse_command()
+
+    if options.verbose:
+        isc.log.init("b10-dbutil", "DEBUG", 99)
+        logger = isc.log.Logger("dbutil")
+    elif options.quiet:
+        # We don't use FATAL, so setting the logger to use
+        # it should essentially make it silent.
+        isc.log.init("b10-dbutil", "FATAL")
+        logger = isc.log.Logger("dbutil")
+
+    db = Database(args[0])
+    exit_code = EXIT_SUCCESS
+
+    logger.info(DBUTIL_FILE, args[0])
+    if options.check:
+        # Check database - open, report, and close
+        try:
+            db.open()
+            exit_code = check_version(db)
+            db.close()
+        except Exception as ex:
+            logger.error(DBUTIL_CHECK_ERROR, ex)
+            exit_code = EXIT_READ_ERROR
+
+    elif options.upgrade:
+        # Upgrade.  Check if this is what they really want to do
+        if not options.noconfirm:
+            proceed = prompt_user()
+            if not proceed:
+                logger.info(DBUTIL_UPGRADE_CANCELED)
+                sys.exit(EXIT_SUCCESS)
+
+        # It is.  Do a backup then do the upgrade.
+        in_progress = False
+        try:
+            db.backup()
+            db.open()
+            in_progress = True
+            perform_all_upgrades(db)
+            db.close()
+        except Exception as ex:
+            if in_progress:
+                logger.error(DBUTIL_UPGRADE_FAILED, ex)
+                logger.warn(DBUTIL_DATABASE_MAY_BE_CORRUPT, db.db_file,
+                            db.backup_file)
+            else:
+                logger.error(DBUTIL_UPGRADE_PREPARATION_FAILED, ex)
+                logger.info(DBUTIL_UPGRADE_NOT_ATTEMPTED)
+            exit_code = EXIT_UPGRADE_ERROR
+
+    sys.exit(exit_code)

src/bin/dbutil/dbutil_messages.mes

@@ -0,0 +1,114 @@
+# Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
+# No namespace declaration - these constants go in the global namespace
+# of the ddns messages python module.
+
+# When you add a message to this file, it is a good idea to run
+# <topsrcdir>/tools/reorder_message_file.py to make sure the
+# messages are in the correct order.
+
+% DBUTIL_BACKUP created backup of %1 in %2
+A backup for the given database file was created. Same of original file and
+backup are given in the output message.
+
+% DBUTIL_CHECK_ERROR unable to check database version: %1
+There was an error while trying to check the current version of the database
+schema. The error is shown in the message.
+
+% DBUTIL_CHECK_NOCONFIRM --noconfirm is not compatible with --check
+b10-dbutil was called with --check and --noconfirm. --noconfirm only has
+meaning with --upgrade, so this is considered an error.
+
+% DBUTIL_CHECK_OK this is the latest version of the database schema. No upgrade is required
+The database schema version has been checked, and is up to date.
+No action is required.
+
+% DBUTIL_CHECK_UPGRADE_NEEDED re-run this program with the --upgrade switch to upgrade
+The database schema version is not up to date, and an update is required.
+Please run the dbutil tool again, with the --upgrade argument.
+
+% DBUTIL_COMMAND_NONE must select one of --check or --upgrade
+b10-dbutil was called with neither --check nor --upgrade. One action must be
+provided.
+
+% DBUTIL_COMMAND_UPGRADE_CHECK --upgrade is not compatible with --check
+b10-dbutil was called with both the commands --upgrade and --check. Only one
+action can be performed at a time.
+
+% DBUTIL_DATABASE_MAY_BE_CORRUPT database file %1 may be corrupt, restore it from backup (%2)
+The upgrade failed while it was in progress; the database may now be in an
+inconsistent state, and it is advised to restore it from the backup that was
+created when b10-dbutil started.
+
+% DBUTIL_EXECUTE Executing SQL statement: %1
+Debug message; the given SQL statement is executed
+
+% DBUTIL_FILE Database file: %1
+The database file that is being checked.
+
+% DBUTIL_NO_FILE must supply name of the database file to upgrade
+b10-dbutil was called without a database file. Currently, it cannot find this
+file on its own, and it must be provided.
+
+% DBUTIL_STATEMENT_ERROR failed to execute %1: %2
+The given database statement failed to execute. The error is shown in the
+message.
+
+% DBUTIL_TOO_MANY_ARGUMENTS too many arguments to the command, maximum of one expected
+There were too many command-line arguments to b10-dbutil
+
+% DBUTIL_UPGRADE_CANCELED upgrade canceled; database has not been changed
+The user aborted the upgrade, and b10-dbutil will now exit.
+
+% DBUTIL_UPGRADE_DBUTIL please get the latest version of b10-dbutil and re-run
+A database schema was found that was newer than this version of dbutil, which
+is apparently out of date and should be upgraded itself.
+
+% DBUTIL_UPGRADE_FAILED upgrade failed: %1
+While the upgrade was in progress, an unexpected error occurred. The error
+is shown in the message.
+
+% DBUTIL_UPGRADE_NOT_ATTEMPTED database upgrade was not attempted
+Due to the earlier failure, the database schema upgrade was not attempted,
+and b10-dbutil will now exit.
+
+% DBUTIL_UPGRADE_NOT_NEEDED database already at latest version, no upgrade necessary
+b10-dbutil was told to upgrade the database schema, but it is already at the
+latest version.
+
+% DBUTIL_UPGRADE_NOT_POSSIBLE database at a later version than this utility can support
+b10-dbutil was told to upgrade the database schema, but it is at a higher
+version than this tool currently supports. Please update b10-dbutil and try
+again.
+
+% DBUTIL_UPGRADE_PREPARATION_FAILED upgrade preparation failed: %1
+An unexpected error occurred while b10-dbutil was preparing to upgrade the
+database schema. The error is shown in the message
+
+% DBUTIL_UPGRADE_SUCCESFUL database upgrade successfully completed
+The database schema update was completed successfully.
+
+% DBUTIL_UPGRADING upgrading database from %1 to %2
+An upgrade is in progress, the versions of the current upgrade action are shown.
+
+% DBUTIL_VERSION_CURRENT database version %1
+The current version of the database schema.
+
+% DBUTIL_VERSION_HIGH database is at a later version (%1) than this program can cope with (%2)
+The database schema is at a higher version than b10-dbutil knows about.
+
+% DBUTIL_VERSION_LOW database version %1, latest version is %2.
+The database schema is not up to date, the current version and the latest
+version are in the message.

src/bin/dbutil/run_dbutil.sh.in

@@ -0,0 +1,40 @@
+#! /bin/sh
+
+# Copyright (C) 2010  Internet Systems Consortium.
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SYSTEMS CONSORTIUM
+# DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
+# INTERNET SYSTEMS CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
+# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
+# WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+PYTHON_EXEC=${PYTHON_EXEC:-@PYTHON@}
+export PYTHON_EXEC
+
+DBUTIL_PATH=@abs_top_builddir@/src/bin/dbutil
+
+PYTHONPATH=@abs_top_srcdir@/src/bin:@abs_top_builddir@/src/lib/python/isc/log_messages:@abs_top_builddir@/src/lib/python:@abs_top_builddir@/src/bin:@abs_top_srcdir@/src/lib/python
+export PYTHONPATH
+
+# If necessary (rare cases), explicitly specify paths to dynamic libraries
+# required by loadable python modules.
+SET_ENV_LIBRARY_PATH=@SET_ENV_LIBRARY_PATH@
+if test $SET_ENV_LIBRARY_PATH = yes; then
+	@ENV_LIBRARY_PATH@=@abs_top_builddir@/src/lib/dns/.libs:@abs_top_builddir@/src/lib/dns/python/.libs:@abs_top_builddir@/src/lib/cryptolink/.libs:@abs_top_builddir@/src/lib/cc/.libs:@abs_top_builddir@/src/lib/config/.libs:@abs_top_builddir@/src/lib/log/.libs:@abs_top_builddir@/src/lib/util/.libs:@abs_top_builddir@/src/lib/util/io/.libs:@abs_top_builddir@/src/lib/exceptions/.libs:@abs_top_builddir@/src/lib/datasrc/.libs:$@ENV_LIBRARY_PATH@
+	export @ENV_LIBRARY_PATH@
+fi
+
+B10_FROM_SOURCE=@abs_top_srcdir@
+export B10_FROM_SOURCE
+
+BIND10_MSGQ_SOCKET_FILE=@abs_top_builddir@/msgq_socket
+export BIND10_MSGQ_SOCKET_FILE
+
+exec ${PYTHON_EXEC} -O ${DBUTIL_PATH}/b10-dbutil "$@"

src/bin/dbutil/tests/.gitignore

@@ -0,0 +1,2 @@
+/dbutil_test.sh
+/dbutil_test_verfile_*

src/bin/dbutil/tests/Makefile.am

@@ -0,0 +1,6 @@
+SUBDIRS = . testdata
+
+# Tests of the update script.
+
+check-local:
+	$(SHELL) $(abs_builddir)/dbutil_test.sh

src/bin/dbutil/tests/dbutil_test.sh.in

@@ -0,0 +1,481 @@
+#!/bin/sh
+# Copyright (C) 2012  Internet Systems Consortium, Inc. ("ISC")
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
+# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
+# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+# PERFORMANCE OF THIS SOFTWARE.
+
+# Checks that the logger will limit the output of messages less severe than
+# the severity/debug setting.
+
+testname="Database Upgrade Test"
+echo $testname
+
+failcount=0
+tempfile=@abs_builddir@/dbutil_test_tempfile_$$
+backupfile=${tempfile}.backup
+testdata=@abs_srcdir@/testdata
+verfile=@abs_builddir@/dbutil_test_verfile_$$
+
+# @brief Record a success
+succeed() {
+    echo "--- PASS"
+}
+
+
+# @brief Record a fail
+#
+# @param $1 Optional additional reason to output
+fail() {
+    if [ "$1" != "" ]
+    then
+        echo "ERROR: $1"
+    fi
+    echo "*** FAIL"
+    failcount=`expr $failcount + 1`
+}
+
+
+# @brief Record a pass if the argument is zero
+#
+# @param $1 Value to test
+passzero() {
+    if [ $1 -eq 0 ]; then
+        succeed
+    else
+        fail
+    fi
+}
+
+
+# @brief Record a fail if the argument is non-zero
+#
+# @param $1 Value to test
+failzero() {
+    if [ $1 -ne 0 ]; then
+        succeed
+    else
+        fail
+    fi
+}
+
+
+# @brief Copy File
+#
+# Executes a "cp" operation followed by a "chmod" to make the target writeable.
+#
+# @param $1 Source file
+# @param $2 Target file
+copy_file () {
+    cp $1 $2
+    chmod a+w $2
+}
+
+
+
+# @brief Check backup file
+#
+# Record a failure if the backup file does not exist or if it is different
+# to the data file. (N.B. No success is recorded if they are the same.)
+#
+# @param $1 Source database file
+# @param $2 Backup file
+check_backup() {
+    if [ ! -e $1 ]
+    then
+        fail "database file $1 not found"
+
+    elif [ ! -e $2 ]
+    then
+        fail "backup file $2 not found"
+
+    else
+        diff $1 $2 > /dev/null
+        if [ $? -ne 0 ]
+        then
+            fail "database file $1 different to backup file $2"
+        fi
+    fi
+}
+
+
+# @brief Check No Backup File
+#
+# Record a failure if the backup file exists.  (N.B. No success is recorded if
+# it does not.)
+#
+# @param $1 Source database file (unused, present for symmetry)
+# @param $2 Backup file
+check_no_backup() {
+    if [ -e $2 ]
+    then
+        fail "backup of database $2 exists when it should not"
+    fi
+}
+
+
+# @brief Get Database Schema
+#
+# As the schema stored in the database is format-dependent - how it is printed
+# depends on how the commands were entered (on one line, split across two
+# lines etc.) - comparing schema is awkward.
+#
+# The function sets the local variable db_schema to the output of the
+# .schema command, with spaces removed and upper converted to lowercase.
+#
+# The database is copied before the schema is taken (and removed after)
+# as SQLite3 assummes a writeable database, which may not be the case if
+# getting the schema from a reference copy.
+#
+# @param $1 Database for which the schema is required
+get_schema() {
+    db1=@abs_builddir@/dbutil_test_schema_$$
+    copy_file $1 $db1
+
+    db_schema=`sqlite3 $db1 '.schema' | \
+               awk '{line = line $0} END {print line}' | \
+               sed -e 's/ //g' | \
+               tr [:upper:] [:lower:]`
+    rm -f $db1
+}
+
+
+# @brief Successful Schema Upgrade Test
+#
+# This test is done where the upgrade is expected to be successful - when
+# the end result of the test is that the test database is upgraded to a
+# database of the expected schema.
+#
+# Note: the caller must ensure that $tempfile and $backupfile do not exist
+#       on entry, and is responsible for removing them afterwards.
+#
+# @param $1 Database to upgrade
+# @param $2 Expected backup file
+upgrade_ok_test() {
+    copy_file $1 $tempfile
+    ../run_dbutil.sh --upgrade --noconfirm $tempfile
+    if [ $? -eq 0 ]
+    then
+        # Compare schema with the reference
+        get_schema $testdata/v2_0.sqlite3
+        expected_schema=$db_schema
+        get_schema $tempfile
+        actual_schema=$db_schema
+        if [ "$expected_schema" = "$actual_schema" ]
+        then
+            succeed
+        else
+            fail "upgraded schema not as expected"
+        fi
+
+        # Check the version is set correctly
+        check_version $tempfile "V2.0"
+
+        # Check that a backup was made
+        check_backup $1 $2
+    else
+        # Error should have been output already
+        fail
+    fi
+}
+
+
+# @brief Unsuccessful Upgrade Test
+#
+# Checks that an upgrade of the specified database fails.
+#
+# Note: the caller must ensure that $tempfile and $backupfile do not exist
+#       on entry, and is responsible for removing them afterwards.
+#
+# @param $1 Database to upgrade
+# @param $2 Expected backup file
+upgrade_fail_test() {
+    copy_file $1 $tempfile
+    ../run_dbutil.sh --upgrade --noconfirm $tempfile
+    failzero $?
+    check_backup $1 $backupfile
+}
+
+
+# @brief Record Count Test
+#
+# Checks that the count of records in each table is preserved in the upgrade.
+#
+# Note 1: This test assumes that the "diffs" table is present.
+# Note 2: The caller must ensure that $tempfile and $backupfile do not exist
+#         on entry, and is responsible for removing them afterwards.
+#
+# @brief $1 Database to upgrade
+record_count_test() {
+    copy_file $1 $tempfile
+
+    diffs_count=`sqlite3 $tempfile 'select count(*) from diffs'`
+    nsec3_count=`sqlite3 $tempfile 'select count(*) from nsec3'`
+    records_count=`sqlite3 $tempfile 'select count(*) from records'`
+    zones_count=`sqlite3 $tempfile 'select count(*) from zones'`
+
+    ../run_dbutil.sh --upgrade --noconfirm $tempfile
+    if [ $? -ne 0 ]
+    then
+        # Reason for failure should already have been output
+        fail
+    else
+        new_diffs_count=`sqlite3 $tempfile 'select count(*) from diffs'`
+        new_nsec3_count=`sqlite3 $tempfile 'select count(*) from nsec3'`
+        new_records_count=`sqlite3 $tempfile 'select count(*) from records'`
+        new_zones_count=`sqlite3 $tempfile 'select count(*) from zones'`
+
+        if [ $diffs_count -ne $new_diffs_count ]
+        then
+            fail "diffs table was not completely copied"
+        fi
+
+        if [ $nsec3_count -ne $new_nsec3_count ]
+        then
+            fail "nsec3 table was not completely copied"
+        fi
+
+        if [ $records_count -ne $new_records_count ]
+        then
+            fail "records table was not completely copied"
+        fi
+
+        if [ $zones_count -ne $new_zones_count ]
+        then
+            fail "zones table was not completely copied"
+        fi
+
+        # As an extra check, test that the backup was successful
+        check_backup $1 $backupfile
+    fi
+}
+
+
+# @brief Version Check
+#
+# Checks that the database is at the specified version (and so checks the
+# --check function).  On success, a pass is recorded.
+#
+# @param $1 Database to check
+# @param $2 Expected version string
+check_version() {
+    copy_file $1 $verfile
+    ../run_dbutil.sh --check $verfile
+    if [ $? -gt 2 ]
+    then
+        fail "version check failed on database $1; return code $?"
+    else
+        ../run_dbutil.sh --check $verfile 2>&1 | grep "$2" > /dev/null
+        if [ $? -ne 0 ]
+        then
+            fail "database $1 not at expected version $2 (output: $?)"
+        else
+            succeed
+        fi
+    fi
+    rm -f $verfile
+}
+
+
+# @brief Version Check Fail
+#
+# Does a version check but expected the check to fail
+#
+# @param $1 Database to check
+# @param $2 Backup file
+check_version_fail() {
+    copy_file $1 $verfile
+    ../run_dbutil.sh --check $verfile
+    failzero $?
+    check_no_backup $tempfile $backupfile
+}
+
+
+# Main test sequence
+
+rm -f $tempfile $backupfile
+
+# Test 1 - check that the utility fails if the database does not exist
+echo "1.1. Non-existent database - check"
+../run_dbutil.sh --check $tempfile
+failzero $?
+check_no_backup $tempfile $backupfile
+
+echo "1.2. Non-existent database - upgrade"
+../run_dbutil.sh --upgrade --noconfirm $tempfile
+failzero $?
+check_no_backup $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+
+# Test 2 - should fail to check an empty file and fail to upgrade it
+echo "2.1. Database is an empty file - check"
+touch $tempfile
+check_version_fail $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+echo "2.2. Database is an empty file - upgrade"
+touch $tempfile
+../run_dbutil.sh --upgrade --noconfirm $tempfile
+failzero $?
+# A backup is performed before anything else, so the backup should exist.
+check_backup $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "3.1. Database is not an SQLite file - check"
+echo "This is not an sqlite3 database" > $tempfile
+check_version_fail $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+echo "3.2. Database is not an SQLite file - upgrade"
+echo "This is not an sqlite3 database" > $tempfile
+../run_dbutil.sh --upgrade --noconfirm $tempfile
+failzero $?
+# ...and as before, a backup should have been created
+check_backup $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "4.1. Database is an SQLite3 file without the schema table - check"
+check_version_fail $testdata/no_schema.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+echo "4.1. Database is an SQLite3 file without the schema table - upgrade"
+upgrade_fail_test $testdata/no_schema.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "5.1. Database is an old V1 database - check"
+check_version $testdata/old_v1.sqlite3 "V1.0"
+check_no_backup $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+echo "5.2. Database is an old V1 database - upgrade"
+upgrade_ok_test $testdata/old_v1.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "6.1. Database is new V1 database - check"
+check_version $testdata/new_v1.sqlite3 "V1.0"
+check_no_backup $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+echo "6.2. Database is a new V1 database - upgrade"
+upgrade_ok_test $testdata/new_v1.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "7.1. Database is V2.0 database - check"
+check_version $testdata/v2_0.sqlite3 "V2.0"
+check_no_backup $tempfile $backupfile
+rm -f $tempfile $backupfile
+
+echo "7.2. Database is a V2.0 database - upgrade"
+upgrade_ok_test $testdata/v2_0.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "8.1. Database is V2.0 database with empty schema table - check"
+check_version_fail $testdata/empty_version.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+echo "8.2. Database is V2.0 database with empty schema table - upgrade"
+upgrade_fail_test $testdata/empty_version.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "9.1. Database is V2.0 database with over-full schema table - check"
+check_version_fail $testdata/too_many_version.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+echo "9.2. Database is V2.0 database with over-full schema table - upgrade"
+upgrade_fail_test $testdata/too_many_version.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "10.0. Upgrade corrupt database"
+upgrade_fail_test $testdata/corrupt.sqlite3 $backupfile
+rm -f $tempfile $backupfile
+
+
+echo "11. Record count test"
+record_count_test $testdata/new_v1.sqlite3
+rm -f $tempfile $backupfile
+
+
+echo "12. Backup file already exists"
+touch $backupfile
+touch ${backupfile}-1
+upgrade_ok_test $testdata/v2_0.sqlite3 ${backupfile}-2
+rm -f $tempfile $backupfile ${backupfile}-1 ${backupfile}-2
+
+
+echo "13.1 Command-line errors"
+copy_file $testdata/old_v1.sqlite3 $tempfile
+../run_dbutil.sh $tempfile
+failzero $?
+../run_dbutil.sh --upgrade --check $tempfile
+failzero $?
+../run_dbutil.sh --noconfirm --check $tempfile
+failzero $?
+../run_dbutil.sh --check
+failzero $?
+../run_dbutil.sh --upgrade --noconfirm
+failzero $?
+../run_dbutil.sh --check $tempfile $backupfile
+failzero $?
+../run_dbutil.sh --upgrade --noconfirm $tempfile $backupfile
+failzero $?
+rm -f $tempfile $backupfile
+
+echo "13.2 verbose flag"
+copy_file $testdata/old_v1.sqlite3 $tempfile
+../run_dbutil.sh --upgrade --noconfirm --verbose $tempfile
+passzero $?
+rm -f $tempfile $backupfile
+
+echo "13.3 Interactive prompt - yes"
+copy_file $testdata/old_v1.sqlite3 $tempfile
+../run_dbutil.sh --upgrade $tempfile << .
+Yes
+.
+passzero $?
+check_version $tempfile "V2.0"
+rm -f $tempfile $backupfile
+
+echo "13.4 Interactive prompt - no"
+copy_file $testdata/old_v1.sqlite3 $tempfile
+../run_dbutil.sh --upgrade $tempfile << .
+no
+.
+passzero $?
+diff $testdata/old_v1.sqlite3 $tempfile > /dev/null
+passzero $?
+rm -f $tempfile $backupfile
+
+echo "13.5 quiet flag"
+copy_file $testdata/old_v1.sqlite3 $tempfile
+../run_dbutil.sh --check --quiet $tempfile 2>&1 | grep .
+failzero $?
+rm -f $tempfile $backupfile
+
+# Report the result
+if [ $failcount -eq 0 ]; then
+    echo "PASS: $testname"
+elif [ $failcount -eq 1 ]; then
+    echo "FAIL: $testname - 1 test failed"
+else
+    echo "FAIL: $testname - $failcount tests failed"
+fi
+
+# Exit with appropriate error status
+exit $failcount

src/bin/dbutil/tests/testdata/Makefile.am

@@ -0,0 +1,12 @@
+EXTRA_DIST =
+EXTRA_DIST += corrupt.sqlite3
+EXTRA_DIST += empty_schema.sqlite3
+EXTRA_DIST += empty_v1.sqlite3
+EXTRA_DIST += empty_version.sqlite3
+EXTRA_DIST += invalid_v1.sqlite3
+EXTRA_DIST += new_v1.sqlite3
+EXTRA_DIST += no_schema.sqlite3
+EXTRA_DIST += old_v1.sqlite3
+EXTRA_DIST += README
+EXTRA_DIST += too_many_version.sqlite3
+EXTRA_DIST += v2_0.sqlite3

src/bin/dbutil/tests/testdata/README

@@ -0,0 +1,41 @@
+The versioning of BIND 10 databases to date has not been the best:
+
+The original database is known here as the "old V1" schema.  It had a
+schema_version table, with the single "version" value set to 1.
+
+The schema was then updated with a "diffs" table.  This is referred to
+here as the "new V1" schema.
+
+The Spring 2012 release of BIND 10 modified the schema.  The
+schema_version table was updated to include a "minor" column, holding the
+minor version number. Other changes to the database included redefining
+"STRING" columns as "TEXT" columns.  This is referred to as the "V2.0
+schema".
+
+The following test data files are present:
+
+empty_schema.sqlite3: A database conforming to the new V1 schema.
+However, there is nothing in the schema_version table.
+
+empty_v1.sqlite3: A database conforming to the new V1 schema.
+The database is empty, except for the schema_version table, where the
+"version" column is set to 1.
+
+empty_version.sqlite3: A database conforming to the V2.0 schema but without
+anything in the schema_version table.
+
+no_schema.sqlite3: A valid SQLite3 database, but without a schema_version
+table.
+
+old_v1.sqlite3: A valid SQLite3 database conforming to the old V1 schema.
+It does not have a diffs table.
+
+invalid_v1.sqlite3: A valid SQLite3 database that, although the schema
+is marked as V1, does not have the nsec3 table.
+
+new_v1.sqlite3: A valid SQLite3 database with data in all the tables
+(although the single rows in both the nsec3 and diffs table make no
+sense, but are valid).
+
+too_many_version.sqlite3: A database conforming to the V2.0 schema but with
+too many rows of data.