days
-4
-5
hours
-1
-1
minutes
-1
-7
seconds
-3
-6
search
JAX Developers Puzzle deadline approaching: Play for a chance to win a complimentary JAX London 2018 ticket!
Test different!

Introduction to Functional Testing with SWTBot

Lorenzo Bettini
@shutterstock/Oleksiy Mark

In this article we will give an introduction to functional testing with SWTBot and we will also show how to integrate such tests in a Continuous Integration system. The article assumes that you are already familiar with Eclipse plug-in development, SWT concepts and JUnit tests.

SWTBot is an open-source Eclipse project for functional testing of SWT applications. SWTBot provides a Java API for specifying the behavior of the UI application to test, thus it is a purely programmatic approach to UI testing. Differently from other functional testing frameworks, like Jubula and RCP Testing Tool, you do not specify the expected behavior using a visual editor and a model representing the actions to perform on the application under test: you write a standard JUnit test using SWTBot fluent Java API (such API hides most of the complexities of SWT and Eclipse).

However, starting from version 2.1.0, SWTBot also provides a Test Recorder and Generator. This is really useful to get quickly a first sketch of your SWTBot scenario and reduce costs of writing tests. We refer to https://wiki.eclipse.org/SWTBot/Generator for further details about the SWTBot generator, which we will not describe in this article.

SWTBot integrates with Eclipse PDE and also with all the other typical build tools like Maven/Tycho, Ant and Buckminster, so that it is easy to run SWTBot tests both in Eclipse and CI servers.

SWTBot can be installed using this update site.

You should install at least these features:

  • “SWTBot for Eclipse Testing”

  • “SWTBot IDE Features”

  • “SWTBot Recorder and Test Generator”

In this article we will give an introduction to functional testing with SWTBot and we will also show how to integrate such tests in a Continuous Integration system. The article assumes that you are already familiar with Eclipse plug-in development, SWT concepts and JUnit tests. We will develop two Eclipse plug-ins that we will test with SWTBot. SWTBot can also be used to test e4 applications.

The complete source code of the examples shown in this article are available at Github. You may want to implement the examples yourself while following the article.

Main concepts

The SWTBot API is based on the concept of bot. The bot is the object that can perform typical user interaction tasks, like find an SWT widget, click on that, click on a button, select a menu, insert content into input controls, etc.

Since all the accesses to the SWT UI will be performed through the bot, the first thing to do in your test class is to instantiate such bot and store it in a field of the test class. The bot can be initialized once for all the test methods, so the bot can be a static field which will be instantiated in a @BeforeClass method:

private static SWTWorkbenchBot bot;

@BeforeClass
public static void beforeClass() throws Exception {
	bot = new SWTWorkbenchBot();
}

Now in the test methods the bot can be used to access SWT elements, like menus, views, input widgets, buttons, etc. Of course, if any of such SWT elements is not present the bot will throw an exception, making the test fail. Thus, the test logic is implied by the actions we perform on the bot. Remember that what you want to test is the interaction of the user with the interface of your UI contributions.

The SWTBot Java API is meant to be readable, fluent and easy to use. For this reason, instead of looking at SWTBot Javadoc, the easiest way to start using the API is to use the content assist on the bot object. Furthermore, the method calls on the bot are meant to be chained. This is possible since the API is statically typed, so the methods available during the method call chain will reflect only the actions that can be performed on the specific widget returned by a method.

Even if you’re not yet familiar with SWTBot, the following line should be easily understood:

bot.menu("File").menu("New").menu("Project...").click();

We are trying to access the menu “Project…” in the submenu “New” in the menu “File”, and to click it.
Although not strictly required, we suggest you annotate your test classes with:

@RunWith(SWTBotJunit4ClassRunner.class)

This way, if anything fails during the test execution, SWTBot will automatically take a screenshot of the running application that can be examined to try to understand what is going wrong. Screenshots are saved in the folder screenshots of the project with SWTBot tests.

A first example

Let’s start using SWTBot with a very simple example.

First of all, we create a plug-in project, say, swtbot.example.menu, that makes contribution to the UI, and we choose the PDE template “Hello world command”, using all the defaults. This project will add a menu “Sample Menu”, with a submenu “Sample Command”, which, once selected, will show a message dialog with title “Example Menu” and message “Hello, Eclipse world”. You may want to start another Eclipse instance to manually verify that.

Now, let’s test this behavior with SWTBot.

We create another plug-in project, say, swtbot.example.tests, where we will write all our SWTBot tests. In this project, we specify as dependency the bundle org.eclipse.swtbot.go, which contains all the requirements to run SWTBot tests, including JUnit itself.

We write the following JUnit test:

package swtbot.example.tests;

import static org.eclipse.swtbot.swt.finder.waits.Conditions.shellCloses;

import org.eclipse.swtbot.eclipse.finder.SWTWorkbenchBot;
import org.eclipse.swtbot.swt.finder.junit.SWTBotJunit4ClassRunner;
import org.eclipse.swtbot.swt.finder.widgets.SWTBotShell;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(SWTBotJunit4ClassRunner.class)
public class SampleMenuTest {

	private static SWTWorkbenchBot bot;

	@BeforeClass
	public static void initBot() {
		bot = new SWTWorkbenchBot();
		bot.viewByTitle("Welcome").close();
	}

	@AfterClass
	public static void afterClass() {
		bot.resetWorkbench();
	}

	@Test
	public void testSampleMenu() {
		bot.menu("Sample Menu").menu("Sample Command").click();
		SWTBotShell dialog = bot.shell("Example Menu");
		dialog.activate();
		bot.label("Hello, Eclipse world");
		bot.button("OK").click();
		bot.waitUntil(shellCloses(dialog));
	}
}

As said in the previous section we use the specific SWTBot test runner because of the automatic screenshots in case of failures. We initialize the bot in the @BeforeClass static method (we also close the standard “Welcome” view so that it does not disturb the tests when they are running) and we also reset the workbench, through the bot, in the @AfterClass method.

In the test method we use the SWTBot fluent API to test the expected behavior. Remember that SWTBot aims at testing that the application behaves as expected when the user interacts with its UI. So, in this test, we simulate the user actions and we check that our menu implementation behaves as expected.

Let’s examine the contents of the test method:

  • we select the menu “Sample Menu”, and we click on its submenu “Sample Command”

  • we expect that a dialog appears on the screen (i.e., an SWT shell) with the title “Example Menu”

  • we activate that dialog, i.e., we put the focus on that

  • we verify that it contains a label “Hello, Eclipse world”

  • we expect the dialog contains a button “OK” and we click on it

  • we expect that after clicking the button the dialog disappears

The test will fail if any of the interactions above cannot be performed, e.g., because there’s no such menu with the specified string, a button cannot be clicked, etc.

We are ready to run the test. SWTBot provides a specific run configuration, “SWTBot Test”, so you can create a new one as shown in the following screenshot.

launch-configuration1

Note that this custom SWTBot run configuration is available only if you installed SWTBot in your IDE: if you installed SWTBot only in your target platform such run configuration will not be available. However, you can still run SWTBot tests as standard “JUnit Plug-in Test” as long as the checkbox “Run in UI thread” is NOT checked. In fact, SWTBot tests are meant to be run in a non-UI thread.

This also means that, after you run the SWTBot tests, you should keep your hands off the keyboard: interacting with your computer during the SWTBot test execution will surely make them fail, since the bot will not be able to interact with the application. Even taking the focus away from the running application will make the tests fail. If you’re using Linux we refer to this hint to run SWTBot tests on another DISPLAY, so that you can keep using your computer while tests are running.

You can now run the test. You should see another Eclipse instance running, and you should see the dialog of our example menu plug-in appear and soon disappear. The test should be green, meaning that we successfully tested the expected behavior.

Note that SWTBot enables Test Driven Development, that is, writing tests (as specifications) before the actual production code. In fact, the above test method can be easily read as the specification of the expected behavior of our plug-in.

Let’s make an experiment with the automatic screenshot mechanism in case of failures: let’s modify the above method by changing one of the specifications. For example, let’s change the line:

bot.label("Hello, Eclipse world");

with something different, e.g.,

bot.label("Hello world");

If we run the test again, it will fail with the following exception (we will talk about SWTBot timeouts in the concluding section):

org.eclipse.swtbot.swt.finder.widgets.TimeoutException: Timeout after: 5000 ms.: Could not find widget matching: (of type 'Label' and with mnemonic 'Hello world')

In the eclipse project swtbot.example.tests you will find a folder screenshots with a jpeg file (named after the name of the failed test). If you open it, you will find the application in the state where the test failed: you will see the dialog opened, and you can verify the cause of the failure (the label is different).

Testing Views

We now create another plug-in project, swtbot.example.view, choosing the PDE template, “Plug-in with a View”.

sample-view-pde-template

This plug-in will contribute a “Sample View”, with a tree and some example contents (it also implements some toolbars, double click on elements of the tree). The view will be visible in the “Java perspective”.

If you run another Eclipse instance and switch to the Java perspective, you can have a look at the features of our Sample View. As shown in the following screenshot, it features a tree with some contents, context menus on tree nodes and some toolbar buttons with a tooltip:

sampleview

Let’s start testing the expected behavior in a test class, SampleViewTest. The structure of the test class is similar to the previous one.
First of all, we want to check that our view is visible in the Java perspective. To do that we write:

@Test
public void testSampleViewInJavaPerspective() {
        bot.viewByTitle("Sample View").show();
}

This test will fail, since the Java perspective is not enabled by default when the Eclipse application is launched by the test. Thus, switching to the Java perspective is part of the @BeforeClass method, since it is part of the setup of our SWTBot tests for our view. We could switch to the Java perspective by simulating through the bot the actions that a user usually do to switch to a Perspective in Eclipse. However, we think that this would not make much sense, since we do not want to test such behavior (we trust the Eclipse platform implements that correctly). Thus, we switch to the Java perspective programmatically. To do that, we need to use the method PlatformUI.getWorkbench().showPerspective(…). Recall that SWTBot runs on a non-UI thread, thus PlatformUI.getWorkbench() will result with a NullPointerException. In order to execute code that needs the UI-thread we can use Display.getDefault().syncExec() to wrap the piece of code that requires UI-thread:

private static void openJavaPerspective() throws InterruptedException {
  Display.getDefault().syncExec(new Runnable() {
    public void run() {
      try {
         IWorkbench workbench = PlatformUI.getWorkbench();
         workbench.showPerspective("org.eclipse.jdt.ui.JavaPerspective",
                workbench.getActiveWorkbenchWindow());
      } catch (WorkbenchException e) {
         e.printStackTrace();
      }
    }
  });
}

This method will be called in the @BeforeClass method, and the previous test will now succeed.

In the following test, we check the structure of the tree of our view. Note that, since we need to access the tree of our view, once we get access to our view, we need to use the view’s bot to access the tree:

@Test
public void testViewTree() {
  SWTBotView view = bot.viewByTitle("Sample View");
  view.bot().tree().getTreeItem("Root").expand();
  view.bot().tree().getTreeItem("Root").getNode("Parent 1").expand();
  view.bot().tree().getTreeItem("Root").getNode("Parent 1").
     getNode("Leaf 1").select();
}

Again, we used the SWTBot fluent Java API for accessing the node of an SWT tree.

Here are some other tests that check that our view behaves as expected (you may want to have a look at the view’s code and to the screenshot above to understand the behavior we are testing):

@Test
public void testViewTreeDoubleClick() {
        SWTBotView view = bot.viewByTitle("Sample View");
        view.bot().tree().getTreeItem("Root").doubleClick();
        assertDialog("Double-click detected on Root");
}

@Test
public void testViewToolbar() {
        bot.toolbarButtonWithTooltip("Action 1 tooltip").click();
        assertDialog("Action 1 executed");
        bot.toolbarButtonWithTooltip("Action 2 tooltip").click();
        assertDialog("Action 2 executed");
}

@Test
public void testViewTreeContextMenu() {
        SWTBotView view = bot.viewByTitle("Sample View");
        view.bot().tree().getTreeItem("Root").contextMenu("Action 1").click();
        assertDialog("Action 1 executed");
}

private void assertDialog(String labelInDialog) {
        SWTBotShell dialog = bot.shell("Sample View");
        dialog.activate();
        bot.label(labelInDialog);
        bot.button("OK").click();
        bot.waitUntil(shellCloses(dialog));
}

Continuous Integration

Since SWTBot tests are JUnit tests it is easy to run them in a Continuous Integration system. In particular, being Eclipse projects, the default choice concerning the build system is Maven/Tycho.

Configuring Maven/Tycho

It is out of the scope of the present article to go into the details on how to configure your projects to be built with Maven/Tycho (please have a look at the sources of the example). Here, we will concentrate on how to configure the test project.

We run SWTBot tests using the tycho-surefire-plugin, after specifying that the packaging type is eclipse-test-plugin. We need to configure the tycho-surefire-plugin specifying the product to run, e.g., org.eclipse.platform.ide, that our tests need the UI harness, and that tests must NOT be run using the UI thread (as already discussed in the previous sections).

This is the typical configuration for running SWTBot tests using tycho-surefire-plugin.


        org.eclipse.tycho
        tycho-surefire-plugin
        ${tycho-version}
        
                true
                false
                org.eclipse.platform.ide
        

The tycho-surefire-plugin will run the specified product with the SWTBot plug-in project and all its dependencies. Thus, you need to make sure that also the plug-ins you want to test are part of the run product. This can be achieved by adding to the swtbot.example.tests‘ MANIFEST a dependency on our menu and view plug-ins. In this example, you also need to add a dependency on org.eclipse.jdt.ui bundle, since we need the Java perspective. Finally, starting from Luna, you also need to add an explicit dependency in the tycho-surefire-plugin on the feature org.eclipse.rcp, otherwise the fragment org.eclipse.osgi.compatibility.state will not be part of the run product, and the workbench will not start correctly.

We suggest to follow a cleaner and more maintainable approach, by specifying all such dependencies in an Eclipse feature project, instead of cluttering the test plug-in’s MANIFEST with dependencies that are not needed for compilation, and specify such Eclipse feature as an explicit dependency when running tests with the tycho-surefire-plugin.

For example, we create an Eclipse feature project, swtbot.example.tests.feature, and we specify this dependency as follows (Since Tycho 0.21.0, this is the suggested way to specify additional dependencies when running tests, instead of specifying them in the tycho-surefire-plugin configuration section):



	org.eclipse.tycho
	target-platform-configuration
	${tycho-version}
	
		
			
				
					eclipse-feature
					swtbot.example.tests.feature
					0.0.0
				
			
		
	


Since our plug-ins are meant to be installed in Eclipse as an installable feature, we also create an Eclipse project, swtbot.example.feature, which includes our two plug-ins (the menu and view projects). Then, our testing feature, swtbot.example.tests.feature, includes swtbot.example.feature and depends on the feature org.eclipse.rcp and on the bundle org.eclipse.jdt.ui:




   
      This feature defines all the additional dependencies that are needed to execute our SWTBot tests
   

   

   
      
      
   


Differences

When running JUnit tests involving the Eclipse workbench with the tycho-surefire-plugin you must be ready to deal with differences with respect to the behavior you have when running the same tests from Eclipse. In our case, the main difference is that the “Welcome” view will not be present when running the SWTBot tests we wrote. This would make all our tests fail when run during the Maven build. In particular, since we close the “Welcome” view in the @BeforeClass static method, the failure in such method will prevent from executing the actual test methods.

In order to deal with such a different behavior, we change our strategy for closing the “Welcome” view as follows:

@BeforeClass
public static void initBot() throws InterruptedException {
        bot = new SWTWorkbenchBot();
        closeWelcomePage();
}

private static void closeWelcomePage() {
        for (SWTBotView view : bot.views()) {
                if (view.getTitle().equals("Welcome")) {
                        view.close();
                }
        }
}

That is, we iterate over the views detected by SWTBot and, if one of such views is the “Welcome” view then we close it. This will make our tests work correctly both when run from Eclipse and when run from Maven/Tycho.

Use Target Platforms

Although not strictly required, we suggest to create a Target Platform definition file and use that target platform both when developing in Eclipse and in the Maven/Tycho build (we refer to this presentation about Eclipse target platforms: http://www.slideshare.net/mickaelistria/a-journey-with-target-platforms).

The use of a target platform has at least two benefits:

  • We will use the same set of dependencies in Eclipse and in the build;

  • By keeping the set of dependencies minimal in the target platform definition file, we will run our tests in Eclipse with a reduced set of plug-ins. This will make such tests a little bit faster and will avoid that other plug-ins, which we use only as tooling in the IDE, interfere with our tests (for example, think about Mylyn or Egit, which need some start up time and are not required to test our plug-ins in most cases).

CI Servers

We now show how to configure CI servers to run SWTBot tests.

First of all, we consider Jenkins (though the same steps can be used in Hudson as well), assuming that it is installed on a Linux system.

In order to run tests that need a graphical server, you need to install Xvnc and the Xvnc Jenkins plug-in. In particular, your jobs need to be configured to start Xvnc. SWTBot tests need a Window Manager. We recommend metacity. Thus, you need to make sure that metacity is installed in your Jenkins Linux machine. Then, your jobs that run SWTBot tests need to be configured with an initial Build step “Execute Shell”, specifying this command line that starts metacity:

metacity --sm-disable --replace 2> metacity.err &

The following screenshot summarizes the above procedure:

jenkins-configuration

Then, the Maven build step can be configured as usual.

Another popular CI system is Travis , which provides a free cloud-based CI solution for open source projects hosted on Github. Connecting Travis with your Github repositories is straightforward and it is out of the scope of the article.

In order to build with Travis, a .travis.yml file must be present in the root of your git repository. This file contains all the setup steps required by your build.

This is a minimal travis file to run SWTBot tests (it is part of the source code of this example):

sudo: false

language: java

jdk: oraclejdk7

cache:
  directories:
  - $HOME/.m2

env: DISPLAY=:99.0

install: true

addons:
  apt:
    packages:
    - metacity

before_script:
 - sh -e /etc/init.d/xvfb start
 - metacity --sm-disable --replace 2> metacity.err &

script:
 - export
 - mvn -f swtbot.example.parent/pom.xml clean verify

This reflects the configuration we had to perform for Jenkins: we need to install the metacity package and start metacity before the actual Maven build. Note that we are using the caching infrastructure of Travis. In particular, we cache the whole .m2 Maven directory, where Maven caches all its dependencies. This means that further builds will be faster since all the dependencies have already been downloaded.

Concluding Remarks

It is worth mentioning that SWTBot relies on timeouts when searching for SWT widgets. This is a strict requirement, otherwise the absence of a widget will make a test be stuck forever.

Unfortunately, timeouts introduce a non-determinism factor in tests, meaning that SWTBot tests might fail due to timeouts in slower machines or in overloaded Continuous Integration servers. However, SWTBot timeouts can be fully configured.

SWTBot uses this public static field for timeout SWTBotPreferences.TIMEOUT, which defaults to 5 seconds (formally, 5000 milliseconds). This can be changed programmatically from the SWTBot test classes themselves, or passed as the Java property org.eclipse.swtbot.search.timeout.

We hope we managed to give an introduction to the main concepts and features of SWTBot.

We conclude by suggesting not to abuse SWTBot tests (nor functional tests in general). If you tried the example, you noticed that SWTBot tests take lot of time to run, just like JUnit Plug-in tests, because of the start of the Eclipse application. You should really use SWTBot to test the functional aspects of your applications and Eclipse plug-ins. Instead, you should write simple JUnit tests, which are known to be really fast, for testing aspects that do not concern user interactions (i.e., the functional aspects).

Even UI features can be tested using plain JUnit if they are not related to user interactions or they do not need a running Eclipse workbench. For example, to test the content provider of a Jface tree viewer there is no need to use SWTBot, since it can be tested with plain JUnit. On the contrary, you want to use SWTBot to test, for example, the context menus of a tree viewer. We also suggest to setup the application for your SWTBot tests using the standard Eclipse platform API, for aspects which are not implemented by your own plug-ins. We showed an example in this article when we open the Java perspective programmatically, instead of simulating the user actions with SWTBot.

Note that in this article we used SWTWorkbenchBot, which is specific of Eclipse 3.x applications. In order to test pure Eclipse 4.x RCP applications you need to use SWTBot, which is the base class of SWTWorkbenchBot.

Finally, we just mention that SWTBot can be used also to test GEF/GMF/Sirius diagrams, editors and editParts. In that respect, you need to install the specific feature of SWTBot for GEF support. Then, you need to use a specific bot, SWTGefBot, and its API. SWTBot also supports advanced Nebula widgets like NatTable.

We refer the interested reader to the SWTBot wiki for further documentation. In particular, SWTBot welcomes contributions from the community and in that respect, we refer to the contribution guide.

Author
Lorenzo Bettini
Lorenzo Bettini is an Assistant Professor (Researcher) in Computer Science at Dipartimento di Informatica, University of Torino, Italy. His research interests are Design, Theory and Implementation of Statically Typed Object Oriented Languages, in particular their type systems. He is the author of more than 80 research papers, published in international conferences and international journals. He is also the author of the book “Implementing Domain-Specific Languages with Xtext and Xtend”, Packt Publishing (August 2013). He is one of the project leads of the Eclipse project EMF Parsley, https://www.eclipse.org/emf-parsley, and he is a committer of SWTBot. He is a big fan and sustainer of Open Source Software.

Leave a Reply

Be the First to Comment!

avatar
400
  Subscribe  
Notify of