Test methods will be failed because of many reasons like uncertain behavior by application, network issues..etc. To overcome the false failures we may want to re execute failed test methods. How to re run failed test methods with RetryAnalyzer automatically is explained in this post.

Following are the steps:

  1. Implement IRetryAnalyzer
  2. Define the retryAnalyzer in @Test annotation
1. Implement IRetryAnalyzer
  • IRetryAnalyzer is an interface, it has a method “retry”.
  • Method retry returns true if the test method has to be retried, false otherwise.
  • We can write logic to re run the test methods any number of times as we wish.

Below is the code, implemented retry method.

  • package com.exploreselenium.listeners;
    
    import org.testng.IRetryAnalyzer;
    import org.testng.ITestResult;
    
    public class CRetryAnalyzer implements IRetryAnalyzer {
    
        int count=0;
        private int iterationTimes = 1; // These many number of times Test methods will be re executed
    
        @Override
        public boolean retry(ITestResult result) {
            System.out.println("running retry logic for  '"
                    + result.getName()
                    + "' in class " + result.getTestClass().getRealClass().getSimpleName() );
            if(count < iterationTimes) {
                count++;
                return true;
            }
            return false;
        }
    
    
    }
    
  • Given #1 for iterationTimes instance variable, so that it will re run failed tests one time.
2. Define the retryAnalyzer in @Test annotation
  • In @Test annotation, we should define retryAnalyzer. When ever this test method fails, it will be executed again.
  • @Test(retryAnalyzer = CRetryAnalyzer.class)
        public void openFacebook(){
            driver.get("http://facebook.com");
            System.out.println("Facebook Opened ..");
            Assert.fail("Test was failed");
    
    }
    

But defining  retryAnalyzer at every test method is not an easy way, if the class has more number of test methods.

Instead of defining retryAnalyzer at every @Test annotation, we can use  “IAnnotationTransformer” listener.

IAnnotationTransformer implementation:

Annotation Transformer is used to modify the content of all the annotations at run time. This is useful when we want to override the annotation values.

  • “IAnnotationTransformer” is an interface and it’s one of the TestNG listeners. It has a method “transform”.
  • IAnnotationTransformer allows you to modify a @Test annotation.
  • To modify another annotations like @DataFactory, use an IAnnotationTransformer2

Below is the implementation of transform method.

  • This method has a logic to set the “CretryAnalyzer” class (written logic to re run tests ) to the @Test Annotation if not set
  • package com.exploreselenium.listeners;
    
    import org.testng.IAnnotationTransformer;
    import org.testng.annotations.ITestAnnotation;
    import java.lang.reflect.Constructor;
    import java.lang.reflect.Method;
    
    public class CRetryListener implements IAnnotationTransformer {
    
    	@Override
    	public void transform(ITestAnnotation testAnnotation, Class testClass, Constructor testConstructor,
    			Method testMethod) {
    
            // This will set retryAnalyzer to the test methods if not set in class level
    		if (testAnnotation.getRetryAnalyzer() == null) {
    
                testAnnotation.setRetryAnalyzer(CRetryAnalyzer.class);
    
    		}
    
    	}
    
    }
    
  • Define this listener in testng.xml file as below.
  • <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <suite name="Retry Test Suite" verbose="2">
      <listeners>
        <listener class-name='com.exploreselenium.listeners.CRetryListener' />
      </listeners>
      
      <test name="ReRun-Test">
       <classes>
        <class name="com.exploreselenium.ScreenshotInResultsTest"></class>
       </classes>
      </test>
    
    </suite>
    

In testng.xml file, used a class “ScreenshotInResultsTest”. It has two test methods

  • Test method “openFacebook”, this will be failed as Assert.fail() is used. Hence this method should be re run.
  • Test method “openGoogle”, this will be passed.
  • Below is the complete TestNG class code
  • package com.exploreselenium;
    
    import com.exploreselenium.listeners.CRetryAnalyzer;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.firefox.FirefoxDriver;
    import org.testng.Assert;
    import org.testng.annotations.AfterClass;
    import org.testng.annotations.BeforeClass;
    import org.testng.annotations.Test;
    
    import java.util.HashMap;
    
    /**
     * Created by maheshv on 12/1/2016.
     */
    public class ScreenshotInResultsTest {
    
        private static HashMap<String, WebDriver> driverObjMap=new HashMap<String, WebDriver>();
        WebDriver driver;
        @BeforeClass
        public void browserInstantiate(){
            System.setProperty("webdriver.chrome.driver", System.getProperty("user.dir") + "\\geckodriver.exe");
            driver=new FirefoxDriver();
            driverObjMap.put(getClass().getName(),driver);
            driver.manage().window().maximize();
            System.out.println("Browser Instantiated..");
        }
    
        @Test
        public void openFacebook(){
            driver.get("http://facebook.com");
            System.out.println("Facebook Opened ..");
            Assert.fail("Test was failed");
    
        }
    
        @Test
        public void openGoogle(){
            driver.get("http://google.com");
            System.out.println("Google Opened ..");
            //Assert.fail("Test was failed");
        }
    
        public static WebDriver gerDriverDetails(String className){
            return driverObjMap.get(className);
        }
    
        public static HashMap<String, WebDriver> gerDriverObjMap(){
            return driverObjMap;
        }
    
        @AfterClass
        public void quitDriver(){
            if(gerDriverObjMap().get(getClass().getName())!=null){
                gerDriverObjMap().get(getClass().getName()).quit();
                gerDriverObjMap().remove(getClass().getName());
                System.out.println("driver for class : "+ getClass().getName() + "is closed");
            }
        }
    }
    

When executed the above testng.xml file,  two test methods were executed and re executed “openFacebook” method as it was failed.

When re run, test method openFacebook was failed again. Hence in TestNG results, this test method was shown twice, first time the method was marked as “failed” and second time the same method was marked as skipped.

Here is the results for reference.

Re run failed test methods with RetryAnalyzer

Summary:

In this post, we learned how to re run the failed test methods with retry analyzer automatically.

TestNG emailable report customization
Execute TestNG XML Programmatically
0