Testing Complex Behavior with JUnit for Highly Parallel Platforms

At Attivio we strive to create software of the highest quality. One technique we've adopted heavily inside our organization is the concept of 'unit' testing. I put unit in quotes because a unit test here might mean validating that MyClass.add2plus2() == 4 however it might also mean starting up an instance of our full application, feeding a few test documents through a workflow and validating via search that the entire system works end to end. Further, while we try to design code in order to be easily testable we try to avoid modifying production code specifically for a given test case.

As you can imagine, the full integration tests are often times very difficult to write and verify. We have a highly parallel system that runs multiple processes, with multiple threads and a message based architecture to perform the business logic in a workflow. We wanted to share some of the testing challenges we've encountered and the strategies we've come up with to solve those issues.

Testing log output

The Problem

We need to verify that a certain block of code is being executed or that a certain log message is being sent to the user.

Our Solution

First a little background. We use logging heavily throughout our system to convey information, error messages and trace level debugging. Our logging system is based on Slf4j with a Log4j backing implementation. All of our normal unit tests run with our default log4j configuration file. In order to address the logging problem we've created a CapturingLog4jAppender class to unsurprisingly, capture log output. It does this by dynamically inserting a new Appender into the log4j system. It works in a test as follows:

// first we need to install the logger and tell it to listen on a given class / package / etc
CapturingLog4jAppender log = CapturingLog4jAppender.install(MyClassBeingTested.class);

// perform test that exercises MyClassBeingTested

// check to see if there were any unexpected warnings
Assert.assertEquals(0, log.getWarningCount());

// we can also check for the existence of certain log messages
// the last flag determines if it is an exact match or a regex match.  both of these samples use regex
Assert.assertEquals(1, captured.count(".*Some log message I expect to see.*", false));
Assert.assertEquals(0, captured.count(".*Some log message I didn't expect to see.*", false));

// uninstall the log
log.uninstall();

The nice thing about this approach is that we can effectively monitor any number of classes for very intricate behaviors assuming we have the logging in place. For example, we can make sure that certain error cases were handled correctly if their side effects are hard to evaluate. We've also found that the log messages we add for testing are almost always useful in the field for debugging problems with customer configurations.

Checking for exceptions in other threads

The Problem

Since we have such a highly parallel system we often times encounter exceptions in other threads that are unhandled. While this is usually not a problem in production, it does come up while doing development of new features.*

Our Solution

We use Java's built-in uncaught exception handler to trap these exceptions.

public class TestExceptionHandler {
  
  private static ConcurrentLinkedQueue<Throwable> exceptions = new ConcurrentLinkedQueue<Throwable>();

  public static final init() {
    Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
      @Override
      public void uncaughtException(Thread thread, Throwable throwable) {
        exceptions.add(throwable);
      }
    }
  });

  public static Collection<Throwable> getExceptions() {
    return exceptions;
  }

  public static void clear() {
    exceptions.clear();
  }
}

Then in your test you can check for exceptions with the following logic:// initialize at the beginning of your test.

// initialize at the beginning of your test
TestExceptionHandler.init();

// run a test that may have lots of exceptions in many threads

// check to make sure there were no exceptions thrown
Assert.assertEquals(0, TestExceptionHandler.getExceptions().size()); 

// clear it out for the next test
TestExceptionHandler.clear();

Detecting state in an asynchronous workflow engine

The Problem

We need to know when something is done and/or cause work to be paused in the middle of our asynchronous workflow processing engine.

Our Solution

One of the big improvements made in Java 5 and 6 was the inclusion of the concurrent package. We use these classes heavily in our code however we've also found use for some of these in our tests as well, specifically the CountDownLatch. The Javadoc for the CountDownlatch has very good examples of how to use it however we've used the class as a test control mechanism in addition to the normal multi-threaded production code management. If you're only doing simple yes/no type operations you can use a single volatile boolean variable.

CountDownLatch latch = new CountDownLatch(5);

startSomeAsyncProcess();
// wait for my mock async processor to run 5 times.
latch.await();
// check some other affected by the asynchronous processing since we know it's "done"

public static final class MockAsyncProcessor() {
  public void run() {
    // do 'stuff'
    // each time this processor is called the counter decreases by 1.
    latch.countDown();
  }
}

Each of these mechanisms helps us ensure that our code does what we say it's going to do and more importantly, it enables us to add new functionality at any level of the product and be sure that we're not going to break some small intricate behavior of some other part of the system.

TrackBack URI for this entry

Comments (1)