dhcp-perf-guide.html 49 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600
  1. <html><head>
  2. <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
  3. <title>DHCP Performance Guide</title><link rel="stylesheet" type="text/css" href="bind10-guide.css"><meta name="generator" content="DocBook XSL Stylesheets V1.76.1"><meta name="description" content="BIND 10 is a framework that features Domain Name System (DNS) suite and Dynamic Host Configuration Protocol (DHCP) servers with development managed by Internet Systems Consortium (ISC). This document describes various aspects of DHCP performance, measurements and tuning. It covers BIND 10 DHCP (codename Kea), existing ISC DHCP4 software, perfdhcp (a DHCP performance measurement tool) and other related topics."></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="book" title="DHCP Performance Guide"><div class="titlepage"><div><div><h1 class="title"><a name="d0e3"></a>DHCP Performance Guide</h1></div><div><div class="author"><h3 class="author"><span class="firstname">Tomasz</span> <span class="surname">Mrugalski</span></h3></div></div><div><div class="author"><h3 class="author"><span class="firstname">Marcin</span> <span class="surname">Siodelski</span></h3></div></div><div><p class="releaseinfo">This is a companion document for BIND 10 version
  4. 20120712.</p></div><div><p class="copyright">Copyright &copy; 2012 Internet Systems Consortium, Inc. ("ISC")</p></div><div><div class="abstract" title="Abstract"><p class="title"><b>Abstract</b></p><p>BIND 10 is a framework that features Domain Name System
  5. (DNS) suite and Dynamic Host Configuration Protocol (DHCP)
  6. servers with development managed by Internet Systems Consortium (ISC).
  7. This document describes various aspects of DHCP performance,
  8. measurements and tuning. It covers BIND 10 DHCP (codename Kea),
  9. existing ISC DHCP4 software, perfdhcp (a DHCP performance
  10. measurement tool) and other related topics.</p></div></div></div><hr></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="preface"><a href="#d0e28">Preface</a></span></dt><dd><dl><dt><span class="section"><a href="#acknowledgements">Acknowledgements</a></span></dt></dl></dd><dt><span class="chapter"><a href="#intro">1. Introduction</a></span></dt><dt><span class="chapter"><a href="#dhcp4">2. ISC DHCP 4.x</a></span></dt><dt><span class="chapter"><a href="#kea">3. Kea</a></span></dt><dd><dl><dt><span class="section"><a href="#d0e54">Backend performance evaluation</a></span></dt><dt><span class="section"><a href="#mysql-backend">MySQL backend</a></span></dt><dd><dl><dt><span class="section"><a href="#d0e156">MySQL tweaks</a></span></dt></dl></dd><dt><span class="section"><a href="#sqlite-ubench">SQLite-ubench</a></span></dt><dd><dl><dt><span class="section"><a href="#sqlite-tweaks">SQLite tweaks</a></span></dt></dl></dd><dt><span class="section"><a href="#memfile-ubench">memfile-ubench</a></span></dt><dd><dl><dt><span class="section"><a href="#memfile-tweaks">memfile tweaks</a></span></dt></dl></dd><dt><span class="section"><a href="#d0e245">Performance measurements</a></span></dt><dt><span class="section"><a href="#d0e521">Possible further optimizations</a></span></dt></dl></dd><dt><span class="chapter"><a href="#perfdhcp">4. perfdhcp</a></span></dt><dd><dl><dt><span class="section"><a href="#d0e533">Purpose</a></span></dt><dt><span class="section"><a href="#perfdhcp-key-features">Key features</a></span></dt><dt><span class="section"><a href="#perfdhcp-command-line">Command line options</a></span></dt><dt><span class="section"><a href="#starting-perfdhcp">Starting perfdhcp</a></span></dt><dt><span class="section"><a href="#perfdhcp-commandline-examples">perfdhcp command line examples</a></span></dt><dd><dl><dt><span class="section"><a href="#perfdhcp-basic-usage">Example: basic usage</a></span></dt><dt><span class="section"><a href="#perfdhcp-rate-control">Example: rate control</a></span></dt><dt><span class="section"><a href="#perfdhcp-templates">Example: templates</a></span></dt></dl></dd></dl></dd></dl></div><div class="list-of-tables"><p><b>List of Tables</b></p><dl><dt>3.1. <a href="#d0e256">Synchronous results</a></dt><dt>3.2. <a href="#d0e332">Asynchronous results</a></dt><dt>3.3. <a href="#tbl-perf-results">Estimated performance</a></dt></dl></div><div class="preface" title="Preface"><div class="titlepage"><div><div><h2 class="title"><a name="d0e28"></a>Preface</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#acknowledgements">Acknowledgements</a></span></dt></dl></div><div class="section" title="Acknowledgements"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="acknowledgements"></a>Acknowledgements</h2></div></div></div><p>ISC would like to acknowledge generous support for
  11. BIND 10 development of DHCPv4 and DHCPv6 components provided
  12. by <a class="ulink" href="http://www.comcast.com/" target="_top">Comcast</a>.</p></div></div><div class="chapter" title="Chapter&nbsp;1.&nbsp;Introduction"><div class="titlepage"><div><div><h2 class="title"><a name="intro"></a>Chapter&nbsp;1.&nbsp;Introduction</h2></div></div></div><p>
  13. This document is in its early stages of development. It is
  14. expected to grow significantly in a near future. It will
  15. cover topics like database backend perfomance measurements,
  16. pros an cons of various optimization techniques and
  17. tools.
  18. </p></div><div class="chapter" title="Chapter&nbsp;2.&nbsp;ISC DHCP 4.x"><div class="titlepage"><div><div><h2 class="title"><a name="dhcp4"></a>Chapter&nbsp;2.&nbsp;ISC DHCP 4.x</h2></div></div></div><p>
  19. TODO: Write something about ISC DHCP4 here.
  20. </p></div><div class="chapter" title="Chapter&nbsp;3.&nbsp;Kea"><div class="titlepage"><div><div><h2 class="title"><a name="kea"></a>Chapter&nbsp;3.&nbsp;Kea</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#d0e54">Backend performance evaluation</a></span></dt><dt><span class="section"><a href="#mysql-backend">MySQL backend</a></span></dt><dd><dl><dt><span class="section"><a href="#d0e156">MySQL tweaks</a></span></dt></dl></dd><dt><span class="section"><a href="#sqlite-ubench">SQLite-ubench</a></span></dt><dd><dl><dt><span class="section"><a href="#sqlite-tweaks">SQLite tweaks</a></span></dt></dl></dd><dt><span class="section"><a href="#memfile-ubench">memfile-ubench</a></span></dt><dd><dl><dt><span class="section"><a href="#memfile-tweaks">memfile tweaks</a></span></dt></dl></dd><dt><span class="section"><a href="#d0e245">Performance measurements</a></span></dt><dt><span class="section"><a href="#d0e521">Possible further optimizations</a></span></dt></dl></div><p>
  21. </p><div class="section" title="Backend performance evaluation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d0e54"></a>Backend performance evaluation</h2></div></div></div><p>
  22. Kea will support several different database backends, using
  23. both popular databases (like MySQL or SQLite) and
  24. custom-developed solutions (like in-memory database). BIND 10
  25. source code features set of performance microbenchmarks.
  26. These are small tools written in C/C++ that simulate expected
  27. DHCP server behaviour and evaluate the performance of
  28. considered databases. As implemented benchmarks are not really
  29. simulating DHCP operation, but rather use set of primitives
  30. that can be used by a real server, they are called
  31. micro-benchmarks.
  32. </p><p>Although there are many operations and data types that
  33. server could store in a database, the most frequently used data
  34. type is lease information. Although lease information for IPv4
  35. and IPv6 differs slightly, it is expected that the performance
  36. differences will be minimal between IPv4 and IPv6 lease operations.
  37. Therefore each test uses lease4 table for performance measurements.
  38. </p><p>All benchmarks are implemented as single threaded applications
  39. that take advantage of a single database connection.</p><p>
  40. Those benchmarks are stored in tests/tools/dhcp-ubench
  41. directory. This directory contains simplified prototypes for
  42. various DB back-ends that are planned or considered as a
  43. backend engine for BIND10 DHCP. Athough trivial now, they are
  44. expected to evolve into useful tools that will allow users to
  45. measure performance in their specific environment.
  46. </p><p>
  47. Currently the following benchmarks are implemented:
  48. </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>in memory+flat file</p></li><li class="listitem"><p>SQLite</p></li><li class="listitem"><p>MySQL</p></li></ul></div><p>
  49. </p><p>
  50. As they require additional (sometimes heavy) dependencies, they are not
  51. built by default. Actually, their build system is completely separated.
  52. It will be eventually merged with the main BIND10 makefile system, but
  53. that is a low priority for now.
  54. </p><p>
  55. All benchmarks will follow the same pattern:
  56. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>prepare operation (connect to a database, create a file etc.)</p></li><li class="listitem"><p>Measure timestamp 0</p></li><li class="listitem"><p>Commit new lease4 (repeated X times)</p></li><li class="listitem"><p>Measure timestamp 1</p></li><li class="listitem"><p>Search for random lease4 (repeated X times)</p></li><li class="listitem"><p>Measure timestamp 2</p></li><li class="listitem"><p>Update existing lease4 (repeated X times)</p></li><li class="listitem"><p>Measure timestamp 3</p></li><li class="listitem"><p>Delete existing lease4 (repeated X times)</p></li><li class="listitem"><p>Measure timestamp 4</p></li><li class="listitem"><p>Print out statistics, based on X and measured timestamps.</p></li></ol></div><p>
  57. Although this approach does not attempt to simulate actual DHCP server
  58. operation that has mix of all steps intervening, it answers the
  59. questions about basic database strenghts and weak points. In particular
  60. it can show what is the impact of specific DB optimizations, like
  61. changing engine, optimizing for writes/reads etc.
  62. </p><p>
  63. The framework attempts to do the same amount of operations for every
  64. backend thus allowing fair complarison between them.
  65. </p></div><div class="section" title="MySQL backend"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="mysql-backend"></a>MySQL backend</h2></div></div></div><p>MySQL backend requires MySQL client development libraries. It uses
  66. mysql_config tool (that works similar to pkg-config) to discover required
  67. compilation and linking options. To install required packages on Ubuntu,
  68. use the following command:
  69. </p><pre class="screen">$ <strong class="userinput"><code>sudo apt-get install mysql-client mysql-server libmysqlclient-dev</code></strong></pre><p>
  70. Make sure that MySQL server is running. Make sure that you have your setup
  71. configured so there is a user that is able to modify used database.</p><p>Before running tests, you need to initialize your database. You can
  72. use mysql.schema script for that purpose. WARNING: It will drop existing
  73. Kea database. Do not run this on your production server. Assuming your
  74. MySQL user is kea, you can initialize your test database by:
  75. </p><pre class="screen">$ <strong class="userinput"><code>mysql -u kea -p &lt; mysql.schema</code></strong></pre><p>
  76. </p><p>After database is initialized, you are ready to run the test:
  77. </p><pre class="screen">$ <strong class="userinput"><code>./mysql_ubench</code></strong></pre><p>
  78. or
  79. </p><pre class="screen">$ <strong class="userinput"><code>./mysql_ubench &gt; results-&gt;mysql.txt</code></strong></pre><p>
  80. Redirecting output to a file is important, because for each operation
  81. there is a single character printed to show progress. If you have a slow
  82. terminal, this may considerably affect test perfromance. On the other hand,
  83. printing something after each operation is required, as poor DB setting
  84. may slow down operations to around 20 per second. Observant user is expected
  85. to note that initial dots are printed too slowly and abort the test.</p><p>Currently all default parameters are hardcoded. Default values can be
  86. overwritten using command line switches. Although all benchmarks take
  87. the same list of parameters, some of them are specific to a given backend
  88. type. To get a list of supported parameters, run your benchmark with -h option:
  89. </p><pre class="screen">$ <strong class="userinput"><code>./mysql_ubench -h</code></strong>
  90. This is a benchmark designed to measure expected performance
  91. of several backends. This particular version identifies itself
  92. as following:
  93. MySQL client version is 5.5.24
  94. Possible command-line parameters:
  95. -h - help (you are reading this)
  96. -m hostname - specifies MySQL server to connect (MySQL backend only)
  97. -u username - specifies MySQL user name (MySQL backend only)
  98. -p password - specifies MySQL passwod (MySQL backend only)
  99. -f name - database or filename (MySQL, SQLite and memfile)
  100. -n integer - number of test repetitions (MySQL, SQLite and memfile)
  101. -s yes|no - synchronous/asynchronous operation (MySQL, SQLite and memfile)
  102. -v yes|no - verbose mode (MySQL, SQLite and memfile)
  103. -c yes|no - should compiled statements be used (MySQL only)
  104. </pre><p>
  105. </p><div class="section" title="MySQL tweaks"><div class="titlepage"><div><div><h3 class="title"><a name="d0e156"></a>MySQL tweaks</h3></div></div></div><p>One parameter that has huge impact on performance is a a backend engine.
  106. You can get a list of engines of your MySQL implementation by using
  107. </p><pre class="screen">&gt; <strong class="userinput"><code>show engines;</code></strong></pre><p>
  108. in your mysql client. Two notable engines are MyISAM and InnoDB. mysql_ubench will
  109. use MyISAM for synchronous mode and InnoDB for asynchronous.</p></div></div><div class="section" title="SQLite-ubench"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="sqlite-ubench"></a>SQLite-ubench</h2></div></div></div><p>SQLite backend requires both sqlite3 development and run-time package. Their
  110. names may vary from system to system, but on Ubuntu 12.04 they are called
  111. sqlite3 libsqlite3-dev. To install them, use the following command:
  112. </p><pre class="screen">&gt; <strong class="userinput"><code>sudo apt-get install sqlite3 libsqlite3-dev</code></strong></pre><p>
  113. Before running the test the database has to be created. Use the following command for that:
  114. </p><pre class="screen">&gt; <strong class="userinput"><code>cat sqlite.schema | sqlite3 sqlite.db</code></strong></pre><p>
  115. A new database called sqlite.db will be created. That is the default name used
  116. by sqlite_ubench test. If you prefer other name, make sure you update
  117. sqlite_ubench.cc accordingly.</p><p>Once the database is created, you can run tests:
  118. </p><pre class="screen">&gt; <strong class="userinput"><code>./sqlite_ubench</code></strong></pre><p>
  119. or
  120. </p><pre class="screen">&gt; <strong class="userinput"><code>./sqlite_ubench &gt; results-sqlite.txt</code></strong></pre><p>
  121. </p><div class="section" title="SQLite tweaks"><div class="titlepage"><div><div><h3 class="title"><a name="sqlite-tweaks"></a>SQLite tweaks</h3></div></div></div><p>To modify default sqlite_ubench parameters, command line
  122. switches can be used. Currently supported parameters are
  123. (default values specified in brackets):
  124. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>-f filename - name of the database file ("sqlite.db")</p></li><li class="listitem"><p>-n num - number of iterations (100)</p></li><li class="listitem"><p>-s yes|no - should the operations be performend in synchronous (yes)
  125. or asynchronous (no) manner (yes)</p></li><li class="listitem"><p>-v yes|no - verbose mode. Should the test print out progress? (yes)</p></li><li class="listitem"><p>-c yes|no - compiled statements. Should the SQL statements be precompiled?</p></li></ol></div><p>
  126. </p><p>SQLite can run in asynchronous or synchronous mode. This
  127. mode can be controlled by using sync parameter. It is set
  128. using (PRAGMA synchronous = ON or OFF).</p><p>Another tweakable feature is journal mode. It can be
  129. turned to several modes of operation. Its value can be
  130. modified in SQLite_uBenchmark::connect(). See
  131. http://www.sqlite.org/pragma.html#pragma_journal_mode for
  132. detailed explanantion.</p></div></div><div class="section" title="memfile-ubench"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="memfile-ubench"></a>memfile-ubench</h2></div></div></div><p>Memfile backend is custom developed prototype backend that
  133. somewhat mimics operation of ISC DHCP4. It uses in-memory
  134. storage using standard C++ and boost mechanisms (std::map and
  135. boost::shared_ptr&lt;&gt;). All database changes are also
  136. written to a lease file. That file is strictly write-only. This
  137. approach takes advantage of the fact that simple append is faster
  138. than edition with potential whole file relocation.</p><div class="section" title="memfile tweaks"><div class="titlepage"><div><div><h3 class="title"><a name="memfile-tweaks"></a>memfile tweaks</h3></div></div></div><p>To modify default memfile_ubench parameters, command line
  139. switches can be used. Currently supported parameters are
  140. (default values specified in brackets):
  141. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>-f filename - name of the database file ("dhcpd.leases")</p></li><li class="listitem"><p>-n num - number of iterations (100)</p></li><li class="listitem"><p>-s yes|no - should the operations be performend in synchronous (yes)
  142. or asynchronous (no) manner (yes)</p></li><li class="listitem"><p>-v yes|no - verbose mode. Should the test print out progress? (yes)</p></li></ol></div><p>
  143. </p><p>memfile can run in asynchronous or synchronous mode. This
  144. mode can be controlled by using sync parameter. It uses
  145. fflush() and fsync() in synchronous mode to make sure that
  146. data is not buffered and physically stored on disk.</p></div></div><div class="section" title="Performance measurements"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d0e245"></a>Performance measurements</h2></div></div></div><p>This section contains sample results for backend performance measurements,
  147. taken using microbenchmarks. Tests were conducted on reasonably powerful machine:
  148. </p><pre class="screen">
  149. CPU: Quad-core Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 logical cores)
  150. HDD: 1,5TB Seagate Barracuda ST31500341AS 7200rpm (used only one of them), ext4 partition
  151. OS: Ubuntu 12.04, running kernel 3.2.0-26-generic SMP x86_64
  152. compiler: g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
  153. MySQL version: 5.5.24
  154. SQLite version: 3.7.9sourceid version is 2011-11-01 00:52:41 c7c6050ef060877ebe77b41d959e9df13f8c9b5e</pre><p>
  155. </p><p>Benchmarks were run in two series: synchronous and
  156. asynchronous. As those modes offer radically different
  157. performances, synchronous mode was conducted for 1000 (one
  158. thousand) repetitions and asynchronous mode was conducted for
  159. 100000 (hundred thousand) repetitions.</p><div class="table"><a name="d0e256"></a><p class="title"><b>Table&nbsp;3.1.&nbsp;Synchronous results</b></p><div class="table-contents"><table summary="Synchronous results" border="1"><colgroup><col align="center" class="Backend"><col align="center" class="Num"><col align="center" class="Create"><col align="center" class="Search"><col align="center" class="Update"><col align="center" class="Delete"></colgroup><thead><tr><th align="center">Backend</th><th align="center">Operations</th><th align="center">Create</th><th align="center">Search</th><th align="center">Update</th><th align="center">Delete</th><th align="center">Average</th></tr></thead><tbody><tr><td align="center">MySQL</td><td align="center">1000</td><td align="center">31.603978s</td><td align="center"> 0.116612s</td><td align="center">27.964191s</td><td align="center">27.695209s</td><td align="center">21.844998s</td></tr><tr><td align="center">SQLite</td><td align="center">1000</td><td align="center">61.421356s</td><td align="center"> 0.033283s</td><td align="center">59.476638s</td><td align="center">56.034150s</td><td align="center">44.241357s</td></tr><tr><td align="center">memfile</td><td align="center">1000</td><td align="center">41.711886s</td><td align="center"> 0.000724s</td><td align="center">42.267578s</td><td align="center">42.169679s</td><td align="center">31.537467s</td></tr></tbody></table></div></div><br class="table-break"><p>Following parameters were measured for asynchronous mode.
  160. MySQL and SQLite were run with 100 thousand repetitions. Memfile
  161. was run for 1 million repetitions due to much larger performance.</p><div class="table"><a name="d0e332"></a><p class="title"><b>Table&nbsp;3.2.&nbsp;Asynchronous results</b></p><div class="table-contents"><table summary="Asynchronous results" border="1"><colgroup><col align="center" class="Backend"><col align="center" class="Num"><col align="center" class="Create"><col align="center" class="Search"><col align="center" class="Update"><col align="center" class="Delete"></colgroup><thead><tr><th align="center">Backend</th><th align="center">Operations</th><th align="center">Create [s]</th><th align="center">Search [s]</th><th align="center">Update [s]</th><th align="center">Delete [s]</th><th align="center">Average [s]</th></tr></thead><tbody><tr><td align="center">MySQL</td><td align="center">100000</td><td align="center">10.584842s</td><td align="center">10.386402s</td><td align="center">10.062384s</td><td align="center"> 8.890197s</td><td align="center"> 9.980956s</td></tr><tr><td align="center">SQLite</td><td align="center">100000</td><td align="center"> 3.710356s</td><td align="center"> 3.159129s</td><td align="center"> 2.865354s</td><td align="center"> 2.439406s</td><td align="center"> 3.043561s</td></tr><tr><td align="center">memfile</td><td align="center">1000000 (sic!)</td><td align="center"> 6.084131s</td><td align="center"> 0.862667s</td><td align="center"> 6.018585s</td><td align="center"> 5.146704s</td><td align="center"> 4.528022s</td></tr></tbody></table></div></div><br class="table-break"><p>Presented performance results can be computed into operations per second metrics.
  162. It should be noted that due to large differences between various operations (sometime
  163. over 3 orders of magnitude), it is difficult to create a simple, readable chart with
  164. that data.</p><div class="table"><a name="tbl-perf-results"></a><p class="title"><b>Table&nbsp;3.3.&nbsp;Estimated performance</b></p><div class="table-contents"><table summary="Estimated performance" border="1"><colgroup><col align="center" class="Backend"><col align="center" class="Create"><col align="center" class="Search"><col align="center" class="Update"><col align="center" class="Delete"><col align="center" class="Average"></colgroup><thead><tr><th align="center">Backend</th><th align="center">Create [oper/s]</th><th align="center">Search [oper/s]</th><th align="center">Update [oper/s]</th><th align="center">Delete [oper/s]</th><th align="center">Average [oper/s]</th></tr></thead><tbody><tr><td align="center">MySQL (async)</td><td align="center">9447.47</td><td align="center">9627.97</td><td align="center">9938.00</td><td align="center">11248.34</td><td align="center">10065.45</td></tr><tr><td align="center">SQLite (async)</td><td align="center">26951.59</td><td align="center">31654.29</td><td align="center">34899.70</td><td align="center">40993.59</td><td align="center">33624.79</td></tr><tr><td align="center">memfile (async)</td><td align="center">164362.01</td><td align="center">1159195.84</td><td align="center">166152.01</td><td align="center">194299.11</td><td align="center">421002.24</td></tr><tr><td align="center">MySQL (sync)</td><td align="center">31.64</td><td align="center">8575.45</td><td align="center">35.76</td><td align="center">36.11</td><td align="center">2169.74</td></tr><tr><td align="center">SQLite (sync)</td><td align="center">16.28</td><td align="center">20045.37</td><td align="center">16.81</td><td align="center">17.85</td><td align="center">7524.08</td></tr><tr><td align="center">memfile (sync)</td><td align="center">23.97</td><td align="center">1381215.47</td><td align="center">23.66</td><td align="center">23.71</td><td align="center">345321.70</td></tr></tbody></table></div></div><br class="table-break"><div class="mediaobject"><img src="performance-results-graph1.png" alt="Performance measurements"><div class="caption"><p>Graphical representation of the performance results
  165. presented in table <a class="xref" href="#tbl-perf-results" title="Table&nbsp;3.3.&nbsp;Estimated performance">Table&nbsp;3.3, &#8220;Estimated performance&#8221;</a>.</p></div></div></div><div class="section" title="Possible further optimizations"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d0e521"></a>Possible further optimizations</h2></div></div></div><p>
  166. For debugging purposes the code was compiled with -g -O0
  167. flags. While majority of the time was spent in backend
  168. functions (that was probably compiled with -O2 flags), the
  169. benchmark code could perform faster, when compiled with -O2,
  170. rather than -O0. That is expected to affect memfile benchmark.
  171. </p><p>
  172. Currently all operations were conducted on one by one
  173. basis. Each operation was treated as a separate
  174. transaction. Grouping X operations together will potentially
  175. bring almost X fold increase in synchronous operations.
  176. Extension for this benchmark in this regard should be considered.
  177. That affects only write operations (insert, update and delete). Read
  178. operations (search) are expected to be barely affected.
  179. </p><p>
  180. Multi-threaded or multi-process benchmark may be considered in
  181. the future. It may be somewhat difficult as only some backends
  182. support concurrent access.
  183. </p></div></div><div class="chapter" title="Chapter&nbsp;4.&nbsp;perfdhcp"><div class="titlepage"><div><div><h2 class="title"><a name="perfdhcp"></a>Chapter&nbsp;4.&nbsp;perfdhcp</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#d0e533">Purpose</a></span></dt><dt><span class="section"><a href="#perfdhcp-key-features">Key features</a></span></dt><dt><span class="section"><a href="#perfdhcp-command-line">Command line options</a></span></dt><dt><span class="section"><a href="#starting-perfdhcp">Starting perfdhcp</a></span></dt><dt><span class="section"><a href="#perfdhcp-commandline-examples">perfdhcp command line examples</a></span></dt><dd><dl><dt><span class="section"><a href="#perfdhcp-basic-usage">Example: basic usage</a></span></dt><dt><span class="section"><a href="#perfdhcp-rate-control">Example: rate control</a></span></dt><dt><span class="section"><a href="#perfdhcp-templates">Example: templates</a></span></dt></dl></dd></dl></div><div class="section" title="Purpose"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d0e533"></a>Purpose</h2></div></div></div><p>
  184. There is a growing need to evaluate performance of DHCP servers in
  185. different traffic conditions to understand their bottle necks.
  186. This helps to elimante bugs in existing DHCP software as well
  187. as make informed decisions regarding new DHCP software designs
  188. to significantly improve its performance. The perfdhcp tool has
  189. been created to fill the gap in performance measurement capabilities
  190. mostly. However, the number of implemented features and parameters
  191. exposed to the user make this tool useful for functional testing as
  192. well.
  193. </p></div><div class="section" title="Key features"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perfdhcp-key-features"></a>Key features</h2></div></div></div><p>
  194. The perfdhcp exposes the number of command line parameters to
  195. control DHCP message exchanges. Currently they fall back to
  196. the following categories:
  197. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>
  198. Rate control - control how many DHCP exchanges
  199. are initiated within a period of time. Tool can also simulate
  200. best effort conditions attempting to initiate as many DHCP
  201. packet exchanges within a unit of time as possible.
  202. </p></li><li class="listitem"><p>
  203. Test exit specifiers - control the conditions when test
  204. completes including number of initiated exchanges, test period or
  205. maximum number of dropped packets.
  206. </p></li><li class="listitem"><p>
  207. Packet templates - specify files containing packet templates that
  208. are used by perfdhcp to create custom DHCP messages instead of
  209. default. Tool also allows to specify number of values indicating
  210. offsets of variable values within a packet that are modified in
  211. flight by the tool.
  212. </p></li><li class="listitem"><p>
  213. Reporting - for each test produce the set of performance data
  214. including achieved packet exchange rate (server performance).
  215. There is also a number of diagnostic selectors available that
  216. enable periodic (intermediate) reporting, packet timestamps
  217. printing and detailed information about perfdhcp internal
  218. states (for debugging).
  219. </p></li><li class="listitem"><p>
  220. Different mode of operations - specify DHCP version used
  221. (v4 or v6), 2-way or 4-way exchanges, use Rapid Commit option
  222. for DHCPv6.
  223. </p></li><li class="listitem"><p>
  224. IP layer options - specify local/remote address, local interface
  225. and local port to be used for communication with DHCP server.
  226. </p></li></ol></div><p>
  227. </p></div><div class="section" title="Command line options"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perfdhcp-command-line"></a>Command line options</h2></div></div></div><p>
  228. The following command line options may be used with perfdhcp tool.
  229. This summary also presents its possible exit codes as well as
  230. error counters printed along when test is complete:
  231. </p><pre class="screen">$ <strong class="userinput"><code>./perfdhcp -h</code></strong>
  232. perfdhcp [-hv] [-4|-6] [-r&lt;rate&gt;] [-t&lt;report&gt;] [-R&lt;range&gt;] [-b&lt;base&gt;]
  233. [-n&lt;num-request&gt;] [-p&lt;test-period&gt;] [-d&lt;drop-time&gt;] [-D&lt;max-drop&gt;]
  234. [-l&lt;local-addr|interface&gt;] [-P&lt;preload&gt;] [-a&lt;aggressivity&gt;]
  235. [-L&lt;local-port&gt;] [-s&lt;seed&gt;] [-i] [-B] [-c] [-1]
  236. [-T&lt;template-file&gt;] [-X&lt;xid-offset&gt;] [-O&lt;random-offset]
  237. [-E&lt;time-offset&gt;] [-S&lt;srvid-offset&gt;] [-I&lt;ip-offset&gt;]
  238. [-x&lt;diagnostic-selector&gt;] [-w&lt;wrapped&gt;] [server]
  239. The [server] argument is the name/address of the DHCP server to
  240. contact. For DHCPv4 operation, exchanges are initiated by
  241. transmitting a DHCP DISCOVER to this address.
  242. For DHCPv6 operation, exchanges are initiated by transmitting a DHCP
  243. SOLICIT to this address. In the DHCPv6 case, the special name 'all'
  244. can be used to refer to All_DHCP_Relay_Agents_and_Servers (the
  245. multicast address FF02::1:2), or the special name 'servers' to refer
  246. to All_DHCP_Servers (the multicast address FF05::1:3). The [server]
  247. argument is optional only in the case that -l is used to specify an
  248. interface, in which case [server] defaults to 'all'.
  249. The default is to perform a single 4-way exchange, effectively pinging
  250. the server.
  251. The -r option is used to set up a performance test, without
  252. it exchanges are initiated as fast as possible.
  253. Options:
  254. -1: Take the server-ID option from the first received message.
  255. -4: DHCPv4 operation (default). This is incompatible with the -6 option.
  256. -6: DHCPv6 operation. This is incompatible with the -4 option.
  257. -a&lt;aggressivity&gt;: When the target sending rate is not yet reached,
  258. control how many exchanges are initiated before the next pause.
  259. -b&lt;base&gt;: The base mac, duid, IP, etc, used to simulate different
  260. clients. This can be specified multiple times, each instance is
  261. in the &lt;type&gt;=&lt;value&gt; form, for instance:
  262. (and default) mac=00:0c:01:02:03:04.
  263. -d&lt;drop-time&gt;: Specify the time after which a request is treated as
  264. having been lost. The value is given in seconds and may contain a
  265. fractional component. The default is 1 second.
  266. -E&lt;time-offset&gt;: Offset of the (DHCPv4) secs field / (DHCPv6)
  267. elapsed-time option in the (second/request) template.
  268. The value 0 disables it.
  269. -h: Print this help.
  270. -i: Do only the initial part of an exchange: DO or SA, depending on
  271. whether -6 is given.
  272. -I&lt;ip-offset&gt;: Offset of the (DHCPv4) IP address in the requested-IP
  273. option / (DHCPv6) IA_NA option in the (second/request) template.
  274. -l&lt;local-addr|interface&gt;: For DHCPv4 operation, specify the local
  275. hostname/address to use when communicating with the server. By
  276. default, the interface address through which traffic would
  277. normally be routed to the server is used.
  278. For DHCPv6 operation, specify the name of the network interface
  279. via which exchanges are initiated.
  280. -L&lt;local-port&gt;: Specify the local port to use
  281. (the value 0 means to use the default).
  282. -O&lt;random-offset&gt;: Offset of the last octet to randomize in the template.
  283. -P&lt;preload&gt;: Initiate first &lt;preload&gt; exchanges back to back at startup.
  284. -r&lt;rate&gt;: Initiate &lt;rate&gt; DORA/SARR (or if -i is given, DO/SA)
  285. exchanges per second. A periodic report is generated showing the
  286. number of exchanges which were not completed, as well as the
  287. average response latency. The program continues until
  288. interrupted, at which point a final report is generated.
  289. -R&lt;range&gt;: Specify how many different clients are used. With 1
  290. (the default), all requests seem to come from the same client.
  291. -s&lt;seed&gt;: Specify the seed for randomization, making it repeatable.
  292. -S&lt;srvid-offset&gt;: Offset of the server-ID option in the
  293. (second/request) template.
  294. -T&lt;template-file&gt;: The name of a file containing the template to use
  295. as a stream of hexadecimal digits.
  296. -v: Report the version number of this program.
  297. -w&lt;wrapped&gt;: Command to call with start/stop at the beginning/end of
  298. the program.
  299. -x&lt;diagnostic-selector&gt;: Include extended diagnostics in the output.
  300. &lt;diagnostic-selector&gt; is a string of single-keywords specifying
  301. the operations for which verbose output is desired. The selector
  302. keyletters are:
  303. * 'a': print the decoded command line arguments
  304. * 'e': print the exit reason
  305. * 'i': print rate processing details
  306. * 'r': print randomization details
  307. * 's': print first server-id
  308. * 't': when finished, print timers of all successful exchanges
  309. * 'T': when finished, print templates
  310. -X&lt;xid-offset&gt;: Transaction ID (aka. xid) offset in the template.
  311. DHCPv4 only options:
  312. -B: Force broadcast handling.
  313. DHCPv6 only options:
  314. -c: Add a rapid commit option (exchanges will be SA).
  315. The remaining options are used only in conjunction with -r:
  316. -D&lt;max-drop&gt;: Abort the test if more than &lt;max-drop&gt; requests have
  317. been dropped. Use -D0 to abort if even a single request has been
  318. dropped. If &lt;max-drop&gt; includes the suffix '%', it specifies a
  319. maximum percentage of requests that may be dropped before abort.
  320. In this case, testing of the threshold begins after 10 requests
  321. have been expected to be received.
  322. -n&lt;num-request&gt;: Initiate &lt;num-request&gt; transactions. No report is
  323. generated until all transactions have been initiated/waited-for,
  324. after which a report is generated and the program terminates.
  325. -p&lt;test-period&gt;: Send requests for the given test period, which is
  326. specified in the same manner as -d. This can be used as an
  327. alternative to -n, or both options can be given, in which case the
  328. testing is completed when either limit is reached.
  329. -t&lt;report&gt;: Delay in seconds between two periodic reports.
  330. Errors:
  331. - tooshort: received a too short message
  332. - orphans: received a message which doesn't match an exchange
  333. (duplicate, late or not related)
  334. - locallimit: reached to local system limits when sending a message.
  335. Exit status:
  336. The exit status is:
  337. 0 on complete success.
  338. 1 for a general error.
  339. 2 if an error is found in the command line arguments.
  340. 3 if there are no general failures in operation, but one or more
  341. exchanges are not successfully completed.
  342. </pre><p>
  343. </p></div><div class="section" title="Starting perfdhcp"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="starting-perfdhcp"></a>Starting perfdhcp</h2></div></div></div><p>
  344. In order to run performance test at least two separate systems
  345. have to be installed: client and server. The first one has to have
  346. perfdhcp tool installed, the latter has to have DHCP server
  347. running (v4 or v6). If only single system is available the client
  348. and server can be run on virtual machines (running on the same
  349. phisical system) but in this case performance data may be heavily
  350. impacted by the performance of VMs.
  351. </p><p>
  352. The DHCP operates on low port numbers (67 for DHCPv4 relays and
  353. 547 for DHCPv6). Running perfdhcp with non-root priviliges will
  354. usually result in the error message similar to this:
  355. </p><pre class="screen"><strong class="userinput"><code>$./perfdhcp -4 -l eth3 -r 100 all</code></strong>
  356. Error running perfdhcp: Failed to bind socket 3 to 172.16.1.2/port=67
  357. </pre><p>
  358. perfdhcp has the '-L' command line option that
  359. imposes use of custom local port. Thus the following command
  360. line will work:
  361. </p><pre class="screen"><strong class="userinput"><code>$./perfdhcp -4 -l eth3 -r 100 -L 10067 all</code></strong></pre><p>
  362. but in the standard configuration no responses will be received
  363. from the ISC DHCP server because server responds to default relay
  364. port 67.
  365. Alternative way to overcome this issue is to run perfdhcp as root.
  366. </p><p>
  367. Currently, perfdhcp is seen from the server perspective as relay agent.
  368. This simplifies its implementation (specifically there is no need to
  369. receive traffic sent to braodcast addresses). This imposes that IPv4
  370. address has to be set manually on the interface that will be used to
  371. communicate with the server. For example, if DHCPv4 server is listening
  372. on the interface connected to 172.16.1.0 subnet, interface on client
  373. machine has to have network address assigned from the same subnet
  374. on one of its interfaces connected to this subnet:
  375. </p><pre class="screen"><strong class="userinput"><code>#ifconfig eth3 172.16.1.2. netmask 255.255.255.0 up</code></strong></pre><p>
  376. </p></div><div class="section" title="perfdhcp command line examples"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perfdhcp-commandline-examples"></a>perfdhcp command line examples</h2></div></div></div><p>
  377. In this section the perfdhcp command line examples
  378. are presented as a quick start guide for new users. For the
  379. detailed list of command line options refer to
  380. <a class="xref" href="#perfdhcp-command-line" title="Command line options">the section called &#8220;Command line options&#8221;</a>.
  381. </p><div class="section" title="Example: basic usage"><div class="titlepage"><div><div><h3 class="title"><a name="perfdhcp-basic-usage"></a>Example: basic usage</h3></div></div></div><p>
  382. If server is listening on interface with IPv4 address 172.16.1.1
  383. the simpliest perfdhcp command line will look like this:
  384. </p><pre class="screen"><strong class="userinput"><code>#./perfdhcp 172.16.1.1</code></strong>
  385. ***Rate statistics***
  386. Rate: 206.345
  387. ***Statistics for: DISCOVER-OFFER***
  388. sent packets: 21641
  389. received packets: 350
  390. drops: 21291
  391. orphans: 0
  392. min delay: 9.022 ms
  393. avg delay: 143.100 ms
  394. max delay: 259.303 ms
  395. std deviation: 56.074 ms
  396. collected packets: 30
  397. ***Statistics for: REQUEST-ACK***
  398. sent packets: 350
  399. received packets: 268
  400. drops: 82
  401. orphans: 0
  402. min delay: 3.010 ms
  403. avg delay: 152.470 ms
  404. max delay: 258.634 ms
  405. std deviation: 56.936 ms
  406. collected packets: 0
  407. </pre><p>
  408. In this case perfdhcp will use remote address 172.16.1.1 as a
  409. destination address and will use suitable local interface for
  410. communication. Since, no rate control parameters have been specified
  411. it will be initiating DHCP exchanges with the maximum possible
  412. rate (it will try to initiate maximum number of exchanges per
  413. second and count number of completed exchanged). Due to server's
  414. performance constraints, many DHCP packets sent to server are likely
  415. to be dropped. The performance test will be running until it is
  416. not interrupted by the user (with ^C).
  417. </p><p>
  418. The default performance statistics reported by perfdhcp have the
  419. following meaning:
  420. </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>Rate - number of packet exchanges (packet sent
  421. to the server and matching response received from the server)
  422. completed within a second.</p></li><li class="listitem"><p>sent packets - total number of DHCP packets of
  423. a specific type sent to the server.</p></li><li class="listitem"><p>received packets - total number of DHCP packets
  424. of specific type received from the server.</p></li><li class="listitem"><p>drops - number of dropped packets for the
  425. particular exchange. Number of dropped packets is calculated as
  426. a difference between number of sent packets and number of
  427. response packets received from the server. It is likely that
  428. server sent the reponse but perfdhcp execution had ended before
  429. reponse arrived. In such case this packet will be assumed
  430. dropped.</p></li><li class="listitem"><p>orphans - number of packets that have been
  431. received from the server and did not match any packet sent by
  432. perfdhcp. This may occur if received packet has been sent
  433. to some other host or if exchange time out has occured and
  434. has been been garbage collected.</p></li><li class="listitem"><p>min delay - minimum delay that occured between
  435. sending the packet to the server and receiving reponse from
  436. it.</p></li><li class="listitem"><p>avg delay - average delay between sending the
  437. packet of the specific type the server and receiving response
  438. from it.</p></li><li class="listitem"><p>max delay - maximum delat that occured between
  439. sedning the packet to the server and receiveing response from
  440. it.</p></li><li class="listitem"><p>std deviation - standard deviation of delay
  441. between sending the packet of a specific type to the server and
  442. receiving response from it.</p></li><li class="listitem"><p>collected packets - number of garbage collected
  443. sent packets. Packets may get garbage collected when waiting time
  444. for server response exceeds value set with
  445. -d&lt;drop-time&gt;.</p></li></ul></div><p>
  446. </p><p>
  447. perfdhcp allows to run the test using specified interface:
  448. </p><pre class="screen"><strong class="userinput"><code>#./perfdhcp -l eth3</code></strong></pre><p>
  449. or local address assigned to it:
  450. </p><pre class="screen"><strong class="userinput"><code>#./perfdhcp -l 172.16.1.2</code></strong></pre><p>
  451. </p></div><div class="section" title="Example: rate control"><div class="titlepage"><div><div><h3 class="title"><a name="perfdhcp-rate-control"></a>Example: rate control</h3></div></div></div><p>
  452. In the examples above perfdhcp initiates new exchanges with best
  453. effort rate. In this case many packets is expected to be dropped by the
  454. server due to performance limitations. Many times it is desired to set
  455. the expected (reasonable) rate and verify if generated traffic is
  456. handled without packet dropes by DHCP server. The following command will
  457. make perfdhcp to initiate 300 4-way exchanges per second and test will
  458. last for 60 seconds:
  459. </p><pre class="screen"><strong class="userinput"><code>#./perfdhcp -l eth3 -p 60 -r 300</code></strong>
  460. ***Rate statistics***
  461. Rate: 256.683 exchanges/second, expected rate: 300 exchanges/second
  462. ***Statistics for: DISCOVER-OFFER***
  463. sent packets: 17783
  464. received packets: 15401
  465. drops: 2382
  466. orphans: 0
  467. min delay: 0.109 ms
  468. avg delay: 75.756 ms
  469. max delay: 575.614 ms
  470. std deviation: 60.513 ms
  471. collected packets: 11
  472. ***Statistics for: REQUEST-ACK***
  473. sent packets: 15401
  474. received packets: 15317
  475. drops: 84
  476. orphans: 0
  477. min delay: 0.536 ms
  478. avg delay: 72.072 ms
  479. max delay: 576.749 ms
  480. std deviation: 58.189 ms
  481. collected packets: 0
  482. </pre><p>
  483. Note that in this example the packet drops have been significantly
  484. reduced thanks to setting reasonable rate. The non-zero number of
  485. packet drops and achived rate (256/s) below expected rate (300/s)
  486. indicate that server's measured performance is lower than 300 leases
  487. per second. Further rate decrease should eliminate most of the packet
  488. drops and bring achived rate close to expected rate:
  489. </p><pre class="screen"><strong class="userinput"><code>#./perfdhcp -l eth3 -p 60 -r 100 -R 30</code></strong>
  490. ***Rate statistics***
  491. Rate: 99.8164 exchanges/second, expected rate: 100 exchanges/second
  492. ***Statistics for: DISCOVER-OFFER***
  493. sent packets: 5989
  494. received packets: 5989
  495. drops: 0
  496. orphans: 0
  497. min delay: 0.023 ms
  498. avg delay: 2.198 ms
  499. max delay: 181.760 ms
  500. std deviation: 9.429 ms
  501. collected packets: 0
  502. ***Statistics for: REQUEST-ACK***
  503. sent packets: 5989
  504. received packets: 5989
  505. drops: 0
  506. orphans: 0
  507. min delay: 0.473 ms
  508. avg delay: 2.355 ms
  509. max delay: 189.658 ms
  510. std deviation: 5.876 ms
  511. collected packets: 0
  512. </pre><p>
  513. Note that the last parameter (-R 30) configures perfdhcp to simulate
  514. traffic from distinct 30 clients.
  515. </p></div><div class="section" title="Example: templates"><div class="titlepage"><div><div><h3 class="title"><a name="perfdhcp-templates"></a>Example: templates</h3></div></div></div><p>
  516. By default the DHCP messages are formed in-flight with default options.
  517. If desired, there is a way to define custom packet format with template
  518. files. Content in template files is encoded in hexadecimal format. The
  519. perfdhcp forms the packet by replacing parts of the binary stream read
  520. from the file with variable data such as elapsed time, HW address, DUID
  521. etc. The offsets where such variable data is placed is specific to the
  522. template file and have to be specified from the command line. Refer to
  523. <a class="xref" href="#perfdhcp-command-line" title="Command line options">the section called &#8220;Command line options&#8221;</a> to find out how to
  524. specify offsets for particular options and fields. With the following
  525. command line the DHCPv6 SOLICIT and REQUEST packets will be formed from
  526. solicit-example.hex and request6-example.hex packets:
  527. </p><pre class="screen"><strong class="userinput"><code>#./perfdhcp -6 -l eth3 -r 100 -R 20 -T templates/solicit-example.hex -T templates/request6-example.hex -O 21 -E 84 -S 22 -I 40 servers</code></strong>
  528. ***Rate statistics***
  529. Rate: 99.5398 exchanges/second, expected rate: 100 exchanges/second
  530. ***Statistics for: SOLICIT-ADVERTISE***
  531. sent packets: 570
  532. received packets: 569
  533. drops: 1
  534. orphans: 0
  535. min delay: 0.259 ms
  536. avg delay: 0.912 ms
  537. max delay: 6.979 ms
  538. std deviation: 0.709 ms
  539. collected packets: 0
  540. ***Statistics for: REQUEST-REPLY***
  541. sent packets: 569
  542. received packets: 569
  543. drops: 0
  544. orphans: 0
  545. min delay: 0.084 ms
  546. avg delay: 0.607 ms
  547. max delay: 6.490 ms
  548. std deviation: 0.518 ms
  549. collected packets: 0
  550. </pre><p>
  551. where:
  552. </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>two occurences of -O 21 - DUID's last octet
  553. positions in SOLICIT and REQUEST respectively.</p></li><li class="listitem"><p>-E 84 - elapsed time option position in
  554. REQUEST template</p></li><li class="listitem"><p>-S 22 - server id position in REQUEST
  555. template</p></li><li class="listitem"><p>-I 40 - IA_NA option position in REQUEST
  556. template</p></li></ul></div><p>
  557. </p></div></div></div></div></body></html>