When we embarked on the journey of implementing a new performance testing framework, I thought it would be a useful tool, I just did not realize how useful and how much I would enjoy solving problems with it.

Although we do a combination of manual and automated testing here (sometimes you just don't have a choice) we all firmly believe in automation and all the wonderful things it buys you: reusability, reproducibility, consistency, etc. That's why we did not think twice about investing in implementing a framework that would help to get us to 100% automation of all our performance tests.

We started out thinking we would be writing a lot of Java code around a performance tool. We had some experience with JMeter and decided to keep using that. What we did not realize was how much we would be using Ant which we also had been using for compiling our code and running our JUnit tests. Instead of writing the Java code we thought we needed, we ended up using the very simple and convenient Ant tasks. We quickly realized that a lot of the functionality we needed was already provided by Ant! We gained a lot of time right there!

Using Ant tasks, we are now able to fetch a specific build (nightly build or previous releases), install it on a specified host machine (no matter the OS), start our server with a specific configuration and start running the actual tests which themselves may consist of a collection of Ant tasks. All the steps are self contained in a single test configuration file in XML. Plain and simple, but it works! To launch the test, all you need to do is call Ant with the default target and specify the test you want to run.

In order to run more than one test, as we do for our nightly system test/performance test runs, I created a smokeTest target in our main project build file and there, I aggregate the calls to all the tests I want to run in a particular Bamboo plan.

What happens behind the scene? When a user of the framework writes the install part of a test, the operations that happen behind the scene are all transparent. The user only needs to specify the name of the machine, the operating system, the user name and password on the machine and a set of configurations files (if any) that are needed to run the AIE server.

There is a task that fetches the build to be installed. Another task handles the actual intallation process. The build, the configuration files and properties files specified are copied onto the specified remote host machine, and the appropriate commands are sent to actually start the installation process. Once that has completed successfully, the same process is used to issue the commands that actually start the server remotely. Now we are ready to start testing.

Here is a snippet of an Ant target that does a fetch and install on 2 nodes. In this case, a different version of the system is installed on each node: the current nightly build is installed on "node1" and our released 1.3.1 version on "node2".

<target name="install">
 <fetchInstaller property="installerFile1" repository="${repository}" os="linux"   arch="x64" />
 <fetchInstaller property="installerFile2" repository="${repository}" release="1.3.1" os="linux" arch="x64" />
 <attivioSystem id="test_system" localDir="${basedir}/test/indexCompatibilityTest">
 <hosts>
 <host id="host" hostname="${host}" os="LINUX" arch="x64" username="${user}" password="${password}" />
 </hosts>
 <installs>
 <install id="install_1" licenseFile="${licenseFile2}" installerFile="${installerFile1}" installPath="${installPath1}"        
        hostRefid="host" installerConfigFile="${installConfigFile1}" />
 <install id="install_2" licenseFile="${licenseFile}" installerFile="${installerFile2}" installPath="${installPath2}"        
        hostRefid="host" installerConfigFile="${installConfigFile2}" />
 </installs>
 <nodes>
 <node id="node1" projectPath="indexCompatibility13" installRefid="install_1"  port="${port1}" startupArgs="-Xmx1g"        
        startupTimeout="450"/>
 <node id="node2" projectPath="indexCompatibility12" installRefid="install_2" overlayPaths="indexCompatibility12" port="${port2}"        
        startupArgs="-Xmx2g"/>
 </nodes>
 </attivioSystem>
</target>

We had to implement some specific tasks to handle the operations we need. For example, we have a task wrapped in a macro that is our interface to JMeter. We have a task to start our connectors (file, XML, CSV, database) for the ingestion process. We have tasks to run our connectors via the Java client interface and also to interface with our search client.

Nowadays, I use this infrastructure for most system level tests that I want to write. It's just so easy and reliable. When I need a test, my first thought is can I use the new framework? If I'm missing a function, can I implement it there? The answer so far has consistently been yes and yes, but not only that, the actual implementation of missing functions has been a breeze. All that is needed is knowledge of Java (we are a java shop) and Ant of course. Usually adding a missing function just involves implementing an Ant task in Java, hooking it up to the library, and you're ready to go. Sometimes it makes sense to wrap the task in a macro, but that is very trivial as well. The one flag I have to raise is that I had to be aware of the inheritRefs and inheritAll properties, but once you understand how they work, use them appropriately and you're all set.

I even use this infrastructure for tests that I know will be temporary and that I probably will not re-use. Even then it is worth it because it is so easy and quick to set the test up. For example, I was doing some last minute acceptance testing of our product and going over some documentation. There were some examples of configuration settings in the documentation that I wanted to test. Normally I would have used the command line. I would have configured the servers, started them up, ingested content, started a browser and issued some queries from there to make sure that everything worked properly.

In this case however, very quickly, I set up the install task for the servers, set up the ingestion and query tasks (I actually had an existing test file that I copied from), copied and pasted the text from the documentation file into my configuration file for each node, started the test up, done! And guess what, turns out I had to go through a couple iterations of that acceptance test, so I was able to re-use my temporary test files after all!

The other thing that this infrastructure offers us is flexibility. We can run very complex configuration tests as well as other more simple tests where we just want to issue a set of queries against an already running server, where ingestion has already been done. Here is an example of a complex configuration that ran very smoothly:

• The goal was to test a configuration with 8 nodes, each one with its own configuration file, each node on an individual host machine or sharing the same host.

• First, we had to start up the 8 servers

• Then ingest data on 2 nodes that dispatch the ingestion on 4 other nodes

• Query on 2 other nodes that also dispatch to the same 4 previous nodes.

Once a quick smokeTest proved that the basic configuration worked, the real test was to simulate error conditions by shutting down some particular nodes and gather some performance information. All of these error conditions were implemented in a single test file, the appropriate nodes were shutdown, restarted, we issued queries, timed the operations (commit, queries), and generated graphs and reports based on the timing numbers -- truly an end to end test.

Writing tests in this environment is a pleasure. This is a real example of the importance of working with the right testing framework. It's easy, quick and reliable.

Author Bio

Myriam Midy has worked in software development for 17 years before joining Attivio's engineering team. She has used multiple technologies and languages from Basic Assembler to C, C++ & Java on both UNIX and Windows and has been a strong contributor to XML related technologies, especially XSLT. She was one of the original developers on the open source Apache Xalan team. Since joining Attivio, Myriam has been working on automating the testing framework and test development. She's also a really nice person!

TrackBack URI for this entry

Comments (0)