kea/tests/lettuce/README.tutorial

Quick tutorial and overview
---------------------------

Lettuce is a framework for doing Behaviour Driven Development (BDD).

The idea behind BDD is that you first write down your requirements in
the form of scenarios, then implement their behaviour.

We do not plan on doing full BDD, but such a system should also help
us make system tests. And, hopefully, being able to better identify
what exactly is going wrong when a test fails.

Lettuce is a python implementation of the Cucumber framework, which is
a ruby system. So far we chose lettuce because we already need python
anyway, so chances are higher that any system we want to run it on
supports it. It only supports a subset of cucumber, but more cucumber
features are planned. As I do not know much details of cucumber, I
can't really say what is there and what is not.

A slight letdown is that the current version does not support python 3.
However, as long as the tool-calling glue is python2, this should not
cause any problems, since these aren't unit tests; We do not plan to use
our libraries directly, but only through the runnable scripts and
executables.

-----

Features, Scenarios, Steps.

Lettuce makes a distinction between features, scenarios, and steps.

Features are general, well, features. Each 'feature' has its own file
ending in .feature. A feature file contains a description and a number
of scenarios. Each scenario tests one or more particular parts of the
feature. Each scenario consists of a number of steps.

So let's open up a simple one.

-- example.feature
Feature: showing off BIND 10
    This is to show BIND 10 running and that it answer queries

    Scenario: Starting bind10
        # steps go here
--

I have predefined a number of steps we can use, as we build test we
will need to expand these, but we will look at them shortly.

This file defines a feature, just under the feature name we can
provide a description of the feature.

The one scenario we have no has no steps, so if we run it we should
see something like:

-- output
> lettuce
Feature: showing off BIND 10
  This is to show BIND 10 running and that it answer queries

  Scenario: Starting bind10

1 feature (1 passed)
1 scenario (1 passed)
0 step (0 passed)
--

Let's first add some steps that send queries.

--
        A query for www.example.com should have rcode REFUSED
        A query for www.example.org should have rcode NOERROR
--

Since we didn't start any bind10, dig will time out and the result
should be an error saying it got no answer. Errors are in the
form of stack traces (trigger by failed assertions), so we can find
out easily where in the tests they occurred. Especially when the total
set of steps gets bigger we might need that.

So let's add a step that starts bind10.

--
        When I start bind10 with configuration example.org.config
--

This is not good enough; it will fire of the process, but setting up
b10-auth may take a few moments, so we need to add a step to wait for
it to be started before we continue.

--
        Then wait for bind10 auth to start
--

And let's run the tests again.

--
> lettuce

Feature: showing off BIND 10
  This is to show BIND 10 running and that it answer queries

  Scenario: Starting bind10
    When I start bind10 with configuration example.org.config
    Then wait for bind10 auth to start
    A query for www.example.com should have rcode REFUSED
    A query for www.example.org should have rcode NOERROR

1 feature (1 passed)
1 scenario (1 passed)
4 steps (4 passed)
(finished within 2 seconds)
--

So take a look at one of those steps, let's pick the first one.

A step is defined through a python decorator, which in essence is a
regular expression; each captured group will be passed as an argument
to the function we define. For bind10, i defined a configuration file,
a cmdctl port, and a process name. The first two should be
self-evident, and the process name is an optional name we give it,
should we want to address it in the rest of the tests. This is most
useful if we want to start multiple instances. In the next step (the
wait for auth to start), I added a 'of <instance>'. So if we define
the bind10 'as my_bind10', we can specify that one here as 'of my
bind10'.

But notice how we needed two steps, which we probably always need (but
not entirely always)? We can also combine steps; for instance:

--
@step('have bind10 running(?: with configuration ([\w.]+))?')
def have_bind10_running(step, config_file):
    step.given('start bind10 with configuration ' + config_file)
    step.given('wait for bind10 auth to start')
--

Now we can replace the two steps with one:

--
    Given I have bind10 running
--

That's it for the quick overview. For some more examples, with comments, 
take a look at features/example.feature. You can read more about lettuce and
its features on http://www.lettuce.it, and if you plan on adding tests and
scenarios, please consult the last section of the main README first.