ASCII Tables For Clearer Testing

I often find it difficult to understand a test. This can happen when the test was written by someone else, or even when I wrote it myself mere months ago. Usually the heart of the test is simple and each line of code is easily understood, but the context and setup can be extensive and located far away.

I’ve started using ASCII tables to define the initial test data. These tables are both concise and easy to understand, and that leads to simpler tests that are easier to maintain.

For context, at Indiegogo we use rspec, jasmine, and other testing frameworks in the Behavior-Driven Development category. Our tests will often specify nested contexts that enumerate possible states before adding our expectations. A typical test looks something like this:


describe '#my_method' do
  context 'when foo is true' do
    let(:foo) { true }
    context 'and bar is true' do
      let (:bar) { true }
      it 'returns W' do
        expect(my_obj.my_method).to eq('W')
      end
    end
    context 'and bar is false' do
      let(:bar) { false }
      it 'returns X' do
        expect(my_obj.my_method).to eq('X')
      end
    end
  end
  context 'when foo is false' do
    let (:foo) { false }
    context 'and bar is true' do
      let (:bar) { true }
      it 'returns Y' do
        expect(my_obj.my_method).to eq('Y')
      end
    end
    context 'and bar is false' do
      let (:bar) { false }
      it 'returns Z' do
        expect(my_obj.my_method).to eq('Z')
      end
    end
  end
end

Unfortunately, these structures often explode into huge files with thousands of lines of code. It’s pretty difficult to know the expected state for a line of code when the test data is initialized in multiple places hundreds of lines away.

Now, consider an alternative organization for the example above using ASCII tables:


|-------+-------+----------|
| foo   | bar   | expected |
|-------+-------+----------|
| true  | true  | W        |
|-------+-------+----------|
| true  | false | X        |
|-------+-------+----------|
| false | true  | Y        |
|-------+-------+----------|
| false | false | Z        |
|-------+-------+----------|

The table provides a concise and easily-understood presentation of the various states and the expected result.

This is not a new idea. Ward Cunningham’s Fit: Framework for Integrated Test and Bob Martin’s FitNesse both show state and expectations in tables, but making tables with ASCII characters means that you can embed these in existing test files, yielding many of the benefits of Fit and FitNesse without adopting the whole framework.

Introducing The ATV Gem

We initially added tables as here documents to some spec files and parsed them with locally-defined methods. Now we have extracted these table-parsing methods and created an open source project on GitHub, as well as the ATV (ASCII Table Values) ruby gem.

Building Tests With ATV

ATV returns your data as strings. Your code can use those strings however you want:

You can insert dynamic values using string interpolation:


data = <<EOD
|------+---------------|
| foo  | date          |
|------+---------------|
| true | #{Date.today} |
|------+---------------|
EOD

With a little meta-programming you can include methods that are rspec assertions:


describe '#new?' do
  cases_as_table = <<TEXT
|--------------------------+-----------|
| email                    | assertion |
|--------------------------+-----------|
| neighbors@example.com    | to        |
|--------------------------+-----------|
| message_8302@example.com | not_to    |
|--------------------------+-----------|
TEXT
  it 'handles these cases' do
    ATV.from_string(cases_as_table).each do |row|
      attributes = row.to_hash
      assertion = attributes.delete('assertion').to_sym

      my_obj = MyClass.new(attributes)
      expect(my_obj).send(assertion, be_new, '#{assertion} be_new failed for #{attributes.inspect}')
      # for example:
      # expect(my_obj).to be_new, 'to be_new failed for {'email'=&gt;'neighbors@example.com'}
    end
  end
end

One challenge with expressing rspec assertions in tables is keeping the rspec failure message meaningful. Normally that message is generated from the describe, contexts and example descriptions.

A possible remedy uses the table to dynamically create the complete rspec example, including the description. Another approach is to add a failure message to your rspec assertion, as we did above.

Here a method is used to load the data (via ATV) and assign the resulting instances to instance variables so that specific, individual object instances can be accessed in the example:


def read_and_assign_from_ascii_table(ascii_table)
  ATV.from_string(ascii_table).each do |row|
    attributes = row.to_hash
    name = attributes.delete('name')
    instance_variable_set(name.to_sym, MyClass.new(attributes))
  end
end

it 'handles these cases' do
  states_as_table = <<TEXT
|------+-------|
| name | state |
|------+-------|
| @new | new   |
|------+-------|
| @old | old   |
|------+-------|
TEXT
  read_and_assign_from_ascii_table(states_as_table)
  expect(@new).to be_new
  expect(@old).not_to be_new
end

Tables That Are Not ASCII Tables

Some of my coworkers were inspired by the tables concept but wanted something different. Their approach uses white space to organize a hash for easy reading:


header =
  {:whn =>   [:foo , :bar ], :expt => [:meth]}

examples =
  [
    {:whn => [true , true ], :expt => ['W'  ]},
    {:whn => [true , false], :expt => ['X'  ]},
    {:whn => [false, true ], :expt => ['Y'  ]},
    {:whn => [false, false], :expt => ['Z'  ]}
  ]

Their solution is concise, easy to understand and easy to create. It’s also actual ruby code making it easier to customize. Obviously ASCII tables are not the only way to organize your data.

How do you create these ASCII tables?

The benefit of these tables is easy to see but they may be difficult to create. I suspect most modern editors have modes that ease the creation and maintenance of these tables. I prefer to use the built-in table editor that comes with emacs Org Mode, or you can use the terminal-table gem to create your tables.

RubyMine’s column selection mode helps but I’m hoping this offering will inspire you to create a RubyMine plugin that is column aware like the table editor in org mode. If you do, please share.

Are ASCII tables right for your test?

Scale is important when deciding if you should use an ASCII table or some other approach for establishing test data. An ASCII table is probably overkill if you are initializing just a few variables. Likewise, if your test is initializing 2 or more attributes with two or more states then consider what the initialization would be like if summarized in a table. Also consider organizing your data using arrays and hashes with white space to show structure.

Concatenate State and Case

Nested logic is difficult to get right the first time and really difficult to maintain. Here’s a simple technique that flattens nested logic into a single level case statement that is easier to get right and far easier to maintain. The technique has 2 parts: concatenate state, and case.

The goal is to transform a nested “if” into a flattened case statement.

    if a == 'x' then
      if b == 'y' then
        print 'a=x, b=y'
      else
        print 'a=x, b!=y'
      end
    else
      if b == 'y' then
        print 'a!=x, b=y'
      else
        print 'a!=x, b!=y'
      end
    end

becomes

    case
      when a == 'x' && b == 'y'
        print 'a=x, b=y'
      when a == 'x' && b != 'y'
        print 'a=x, b!=y'
      when a != 'x' && b == 'y'
        print 'a!=x, b=y'
      when a != 'x' && b != 'y'
        print 'a!=x, b!=y'
    end

Why?

This is easier to understand. It is very easy to enumerate all permutations of the possible state components.

When you do this the initial implementation is easier. But the really big win is when you return to the code weeks, months, or years later. By then you’ve forgotten all of the interconnected logic that led you to the original implementation. You are forced to stare at the implementation and attempt to rebuild your mental model. With concatenate state, and case all of this is laid out in simple order.

How do you do it? First you bundle all of the state into a single variable. Then you drop into a case statement with all posible permutations of those states to find the code to execute.

Some languages let you have multiple value case expressions, with syntax something like:

case
when a>b && c<d
when a>b && c=>d
when a<=b && c<d
when a<=b && c=>d
end

If your language supports this and your state can be expressed cleanly then you’re done — you’ve achieved the simple enumeration of all possible states.

Often the state cannot be expressed so succinctly, so the expressions alone bloat and obfuscate the logic.

To get around this, simply reduce the expressions to easy to read and understand strings, then concatenate them together for a single composite state. For example:

ab_state = a>b ? 'a>b' : 'a<=b'
cd_state = c<d ? 'c<d' : 'c>=d'

case ab_state + ' and ' + cd_state
when 'a>b and c<d'
when 'a>b and c>=d'
when 'a<=b and c<d'
when 'a<=b and c>=d'

In this contrived example the strings are just as verbose as their non-string expressions, so there is no win. But often the expressions are not so succinct but the strings can be as succinct as you make them, while hopefully communicating the status clearly.

Here’s a real life example pulled from some code I wrote a few years ago. The problem is to validate passwords in a user object on save. There are 3 attributes that are of interest: a password hash, a new password, and a new password confirmation.

The logic goes like this: We need either an existing password hash or a new password and matching password confirmation. If we have a new password or a new password confirmation, then we are attempting to change or set the password, and both items must be present and they must match. If neither are present, but we have an existing password hash, then we are valid, meaning we are not trying to change our password. If neither the new password or new password confirmation are present, and the password hash is blank, then we are invalid since all user records must have a password.

A first implementation might look something like this:

if password_hash.blank?
  if new_password.blank?
    errors.add('new_password', 'is required')
  else
    if new_password_again.blank?
      errors.add('new_password_again', 'new password must be entered twice')
    else
      if new_password != new_password_again
        errors.add('new_password_again', 'does not match your new password')
      end
    end
  end
else
  if new_password.blank?
    if new_password_again.present?
      errors.add('new_password', 'new password must be entered twice')
      #else
      # ok
    end
  else
    if new_password_again.blank?
      errors.add('new_password_again', 'new password must be entered twice')
    else
      if new_password != new_password_again
        errors.add('new_password_again', 'does not match your new password')
      end
    end
  end
end

That’s 27 lines of code (and 2 comment lines). And it is treacherous code. I think it works because I have a test suite and it passed. Without that test suite my confidence in this code would be near zero.

Let’s try cleaning it up a bit.

if password_hash.blank? && new_password.blank? && new_password_again.blank?
  errors.add('new_password', 'is required')
end
if new_password_again.present? && new_password.blank?
  errors.add('new_password', 'new password must be entered twice')
end
if new_password.present? && new_password_again.blank?
  errors.add('new_password_again', 'new password must be entered twice')
end
if new_password.present? && new_password_again.present?
  if new_password != new_password_again
    errors.add('new_password_again', 'does not match your new password')
  end
end

This is a little easier to read but no easier to ensure that it is correct.

Here is the code with concatenated state and case:

newpw_key = new_password.blank? ? 'nonew' : 'new'
newpw_again_key = new_password_again.blank? ? 'noagain' : 'again'
newpw_equal_key = new_password == new_password_again ? 'eql' : 'noteql'
pw_key = password_hash.blank? ? 'nopw' : 'pw'

case '#{newpw_key}-#{newpw_again_key}-#{newpw_equal_key}-#{pw_key}'
  when 'nonew-noagain-eql-pw'
  when 'nonew-noagain-eql-nopw'
    errors.add('new_password', 'is required')
  # when 'nonew-noagain-noteql-pw'
  #   not possible
  # when 'nonew-noagain-noteql-nopw'
  #   not possible
  # when 'nonew-again-eql-pw'
  #   not possible
  # when 'nonew-again-eql-nopw'
  #   not possible
  when 'nonew-again-noteql-pw'
    errors.add('new_password', 'new password must be entered twice')
  when 'nonew-again-noteql-nopw'
    errors.add('new_password', 'new password must be entered twice')
  # when 'new-noagain-eql-pw'
  #   not possible
  # when 'new-noagain-eql-nopw'
  #   not possible
  when 'new-noagain-noteql-pw'
    errors.add('new_password_again', 'new password must be entered twice')
  when 'new-noagain-noteql-nopw'
    errors.add('new_password_again', 'new password must be entered twice')
  when 'new-again-eql-pw'
  when 'new-again-eql-nopw'
  when 'new-again-noteql-pw'
    errors.add('new_password_again', 'does not match your new password')
  when 'new-again-noteql-nopw'
    errors.add('new_password_again', 'does not match your new password')
  else
    raise 'unexpected case #{newpw_key}-#{newpw_equal_key}-#{pw_key}'
end

This weighs in at 26 lines and 12 comments — still a pretty heavy method. However, this one carefully enumerates all the possible permutations of the state values — 4 state values, each with 2 possible values, for 16 possible states.

Of the 16 possible states, some are logically impossible. For example, you can’t say that the new password was specified and the new password confirmation was not specified but that the new password and the new password confirmation are the same — it can’t happen, so you can safely ignore that case.

The remaining states are either valid or invalid. When they are invalid some action is taken.

This code is as clear to me today as it was when I wrote it over 3 years ago.

I hope this helps you write cleaner and easier to maintain code.

-Kelly

Rails Migrations and AR Models — be careful my friend

Be careful when using AR models in migrations. Many of us know this and some know the workaround of defining your model in the migration, as documented in the aging Rails Recipes book by Chad Fowler in the recipe “Safely Use Models in Migrations.”

As the recipe specifies, you may need to refresh the model’s column data, with the “reset_column_information” class method.

I was caught by surprise when I did this on models that extended a base class using single table inheritance. My first try was to reset the column information of the base class.

Nothing complained. The data was simply not what I expected. After a while I decided to reset the column information for every sub class — voilà, success.

Happy Path Testing With Selenium RC Fu

Note: this was originally posted on the Pivotal Labs blog.

Selenium RC Fu is a fantastic system for testing Ruby On Rails applications. It is the blending of xUnit testing with selenium. Selenium is an amazing system that operates your browser as if a human were sitting there moving the mouse, pressing buttons and keys.

Selenium RC Fu is also a remarkable example of the power of open source. It’s Selenium remotely controlled by rails and ruby. You can learn more about it by viewing the slides for Full-stack webapp testing with Selenium and Rails presented by my colleagues Alex Chaffee and Brian Takita at the SDForum Silicon Valley Ruby Conference.

Now that you are excited about Selenium RC Fu, by law I must inform you that this wonderful testing tool comes with some costs. First, this is the daisy cutter of testing — problems will be detected, but it won’t be too specific about those problems. A failed selenium test will likely only tell you some expected text was not present on the page — you have to do some digging to discover the real problem.

It’s also slow. To be fair, a lot of software is running to do this testing.

So use selenium testing sparingly. A good strategy is to restrict selenium testing to “happy path” testing. These happy path tests become a compliment to other more focused and faster unit and integration tests.

Getting Started With Selenium RC Fu

Selenium RC Fu is hosted at rubyforge.org in the “pivotal.rb” project. It’s not well documented and a little hard to find. The primary documentation is the README File.

Step 1 is to add it to your project:


    script/plugin install svn://rubyforge.org/var/svn/pivotalrb/seleniumrc_fu/trunk

Building Your Tests

Create your tests in app/tests/selenium. The basic structure mirrors the other test types:



    require File.dirname(__FILE__) + "/selenium_helper"
    class HappyPathsTest < MyProject::SeleniumTestCase
    
      def test_nav_bar
          ...
      end
    
      def test_about
          ...
      end
    
      # more tests
    end

Note that HappyPathsTest extends something called MyProject::SeleniumTestCase. Selenium RC Fu provides a sample selenium helper file that suggests this convention for name spacing your tests.

For the happy paths tests I simply wanted to go to a page and know that the page loads. A pattern I learned from my colleague Shifra is:

  1. Pick something you know is on the target page, but not on the current page. Let’s call it ‘evidence’. Assert that evidence is not present on the current page.
  2. go to the target page
  3. assert the evidence is present on the target page.

This process insures that you are moving about as you expect. Here’s an example from my project:



      def test_tasks
        ...

        assert_element_not_present "xpath=//h1&91;text()='The Daily Planet']"
        click_and_wait "link=The Daily Planet"
        assert_element_present "xpath=//h1[text()='The Daily Planet']"

        ...    
      end

I repeated this simple pattern for every path.

Selenium Functions/Assertions/Commands

Here are the functions and assertions I used in my tests:

  • click_and_wait locator
  • go_back
  • wait_for_page_to_load
  • type locator, text
  • assert_text_present text
  • assert_text_not_present text
  • assert_equal value1, value2

Many of the functions require a ‘locator’ argument. This needs to identify a single element on the page. Often the name or id of an element is sufficient, but you may need to use an XPath. Check the docs for more information on element locators.

These functions were sufficient for me. Look for more in vendor/plugins/seleniumrc_fu/lib/seleniumrc_fu/selenium_dsl.rb.

Selenium RC Fu comes with some rake commands too (from rake -T selenium):



    rake selenium:restart_servant           # Stop and start the selenium servant (the server that launches browsers) on localhost
    rake selenium:run_server                # Run the selenium servant in the foreground
    rake selenium:server                    # Run the selenium remote-control server
    rake selenium:start_servant             # Start the selenium servant (the server that launches browsers) on localhost
    rake selenium:stop_servant              # Stop the selenium servant (the server that launches browsers) on localhost
    rake selenium:test                      # Run a selenium test file (default test/selenium/selenium_suite)
    rake selenium:test_with_server_started  # Run the selenium tests in Firefox

Running Your Tests

You can run your selenium tests individually or as part of a suite. Selenium RC Fu comes with a suite script in the vendor/plugins/seleniumnrc_fu/sample directory. Drop that in your test/selenium directory.

The selenium server must be running before running individual tests. You can start the server with the rake command rake selenium:server. Once the server is running you run your tests from the command line or via the suite.

You can also run the suite with the command rake selenium:test. This command is smart enough to start the server if it’s not already running. However, when the server is started by this mechanism its a little more work to stop it. You can stop the server with a simple ^c if it was started via the rake selenium:server command.

Make It So

That’s really all you need to build your tests but here are few more things that may help.

Site Diagram

A site diagram is handy for building your happy paths tests. There will be a lot of paths through the site and marking the paths on the diagram as you visit them is an easy way to track your progress. On the diagram you need to indicate both pages and the paths to those pages.

The later is really important. It’s the paths that you will be testing with your happy path selenium testing. Often there are buttons, links, or other controls such as check boxes that may touch the server and reload the current page, or just update the current page via ajax. Be sure to add these to your navigation diagram.

Here’s the diagram I used for my site.

site diagram

Selenium IDE

You can code your tests by hand using Selenium RC Fu functions but it’s easier if you use Selenium IDE. This is a Firefox plugin that records your activity. That activity is available in various selenium dialects and can be pasted into your tests. Unfortunately, it does not have output for Selenium RC Fu. Nonetheless I still found it useful. I simply clicked through my application while Selenium IDE was recording, pasted the resulting selenium commands into my tests, then converted those commands to equivalent Selenium RC Fu commands. For example:

Output from Selenium IDE:



    @selenium.click "link=My Account"
    @selenium.wait_for_page_to_load "30000"
    assert @selenium.is_text_present("Update Account")

Becomes:



    click_and_wait "link=My Account"
    assert_text_present "Update Account"

Test Data

Your selenium tests will be exercising your application just like standard unit or integration tests. These tests require data and there are a variety of techniques to make data available to your tests. Building your happy path tests is easier if you have fixture data and that fixture data is loaded in your development environment via rake db:fixtures:load. If you use fixture scenarious, it is handy to have a scenario for selenium testing, which you would load with something like rake db:scenario:load SCENARIO=selenium.

With your fixtures loaded in your development environment you can see exactly what is present during testing and the selenium IDE will record exactly what is played back by selenium.

XPath Checker

Usually Selenium IDE can assign a locator that simply works but there are times when it can’t or won’t. Using XPath Checker you can right click on an element and it will display an XPath to that element. You can also experiment with varitions and XPath Checker will list all of the elements on the page that can be identified by that XPath.

That’s it. Let me know how it goes.

Teaching Your Tests To Report Unused Parameters

Note: a Pivotal Labs version of this article is available here.

Recently I was about to check in some changes and did a last minute click through of the application. All of a sudden I’m staring at a stack trace. My tests were green and I had functional tests for the failing
controller/action.

Tests are like pants — they cover your backside while you focus on other things like adding features to your application. Suddenly I felt a breeze on my cheeks. Something was amiss.

I soon discovered the action and its associated tests had diverged over time. Some of the parameters were renamed in the action but not in the functional test. Since some of the work of the action was
conditional on the presence of certain parameters, that work was no longer being tested.

This exposed weaknesses in the tests and code, such as expected side effects in the tests that are never checked. If they had been checked the tests would have failed and the parameter name mismatch
would have been discovered.

Most functional tests provide specific parameters that should at least be examined during the processing of the action. Reporting unread parameters would strengthen those tests. It was conceivable to me that
some of the other functional tests had similar unused parameters. I wanted all of my functional tests to report all unused parameters.

The first step was to instrument the params hash. I wanted to track access to the params hash and report parameters that were not read during the processing of the action. I don’t know what all is done to params during the life cycle of a test. I’m only interested in access
from the time the action starts till it returns so I need to be able
to turn the tracking on and off at specific times.

It turns out that Rails uses a subclass of Hash called HashWithIndifferentAccess. I added my changes to HashWithIndifferentAccess in test/test_helper.rb:

    class HashWithIndifferentAccess
      def [](key)
        @accessed_keys ||= {}
        @accessed_keys[key] = true
        super
      end
    
      def start_logging
        @accessed_keys = {}
      end
    
      def end_logging
        @accessed_keys['action'] = true
        @accessed_keys['controller'] = true
        never_accessed = []
        self.each_key do |key|
          never_accessed << key unless @accessed_keys.include?(key)
        end
        raise "Some keys never accessed: #{never_accessed.join(', ')}" unless never_accessed.empty?
      end
    end

With these changes an exception will be raised if any first level keys are not read between start_logging and end_logging.

In each of my functional tests I added code similar to this (from account_controller_test.rb):

    class AccountController
      around_filter :check_params
      private
      def check_params
        params.start_logging
        yield
        params.end_logging
      end
    end

The around filter starts and ends the logging in the context of the action.

With these changes in place my tests no longer passed and my backside was warm and protected again.

bread and butter capistrano

29.pngThe East Bay Ruby Meetup for February was all about Capistrano.

Capistrano’s creator, Jamis Buck, says capistrano

…is a utility for executing commands in parallel on multiple machines, such as for automating the deployment of applications

Three of us talked about capistrano, deployment and scaling. My talk was titled “Bread and Butter Capistrano” because I wanted to emphasize that I use it as a tool to get my work done, find it very helpful, and use it all the time, but I haven’t spent a lot of time studying the tool itself.

Here’s the talk. Enjoy!

Bread and Butter Capistrano

If you’ve read this far, please consider rating me at Working With Rails.

denim – simply sketch your user interface

I’ve been using a tool for user interface prototyping and testing called “DENIM“. DENIM is an application that allows you to sketch a user interface using a tablet connected to your pc or mac. It allows you to zoom in and out to create varying levels of detail as needed.Example UI sketch of DENIM

For example, often you will first sketch the general flow of the site. This will look like boxes with a textual name or title. You can then draw lines between these boxes indicating a flow from one box to the other. With DENIM, you can then select a box and zoom in. When you do this the box becomes a page. You can then add details to the page, such as (sketched) forms, buttons and links. Now you can draw a line from one of these sketched buttons to another page and the sketched button becomes a live link. What does this mean? Well, DENIM allows you to save your prototype UI as html pages with clickable images–your sketches embedded in html. When you click on one of the hot areas, which are blue (non hot areas are black ink on white, hot areas are blue ink on white), you jump to the target page.

Suddenly you have a live prototype. I’ve shown these live prototypes to customers and colleagues. At first people seem perplexed. They seem to think I’ve scribbled on a tablet, taken screen a shot and this is how I’m going to show them the proposed UI. Then they click on a link or two and suddenly they get it… and off they go exploring the whole site, spewing out suggested changes. After a few minutes you go back to your office and spend a few hours coming up with a new version. A couple iterations of this and you and the customer have a pretty good understanding of what is to be built.

But It’s Not All Rosy

This program feels abandoned. It has some bugs. The web site has a video of a future version, but the video is a few years old and the features in the video have not been released. I’ve learned to save my work often, and to save it as different versions. Occasionally your DENIM model will become corrupted and sometimes you cannot fix it, so you have to revert to an earlier state. DENIM creates snapshots for you but I lost trust in them early on and haven’t tried again. DENIM also seems to have a size or complexity limit. When the complexity of your UI exceeds some level the model can no longer be opened. I’ve also noticed that some machines can open some models while others cannot — perhaps it is a memory issue.Even with the bugs DENIM is a worthwhile tool to have in your bag. Please check it out.