How to Test Functions that are NOT Predictable?

Recently, I wrote a Java Servlet for sending SMS messages. It had a method that communicated with a GSM modem connected to my PC. It was working fine – well, most of the time! Whenever you interact with a piece of hardware, or for that matter any external system, things can go wrong. A good program should handle the various exceptions gracefully.

So here is what I did: Since the modem is capable of sending only one SMS at a time, I created a queue where the send requests will be put. Requests will be serviced by the modem in a FIFO manner. If there is any problem, the request will be marked for retry and put at the back of the queue. If the modem cannot send it after 3 tries, it will be taken out of the queue and marked unreachable.

Simple logic right? But there is a problem if I want to test my exception handling logic! I cannot afford to send hundreds of messages to encounter a failure to see my logic working properly. It will take lot of time and money. So how do we test these kind of methods?

You will probably think unit tests, but that might not help. In unit tests, we try to give all different kinds of inputs to a function, expecting a stated behavior. Here, the problem is other way round. For the same inputs, the function might behave differently. One way how I tackle this problem is to write a stub method that would simulate the return values of the real function.

1. Binary Functions

Suppose we are only interested in knowing if a service failed or not, and take actions accordingly. We can model such a function using a discrete random variable which can take two values. Say for instance I suspect it to fail 10% of the time, I can write a stub method like this:

public void sendSMSStub(String message, String dest) {
    if (Math.random() < 0.1) return false;
    return true;
}

The function Math.random() returns a uniformly distributed random number between 0 and 1. What we did was to alter the distribution such that it passes (returns true) 90% of the time and fails rest of the time. Having this function in hand, I can call this as much as I want to check if my queue/retry strategy works properly. I can also increase the failure probability to say, 30% if I need to check many occurrences of failures.

2. Multi-valued Functions

The function we saw in the last section returned binary values, true or false. However, there may be situations where your function could return multiple error codes. For instance, if you try to get a connection to web site URL to fetch its contents, you can get status codes such as these.

200 OK
404 Not Found
500 Internal Server Error
503 Service Unavailable

We can reasonably assume these codes will be returned by HTTP servers with varying probability. Say the probabilities are { 200: 0.6, 404: 0.3, 500: 0.07, 503: 0.03 }

Generating the results using the above distribution is quite easy. We can generate a random number between 0 and 1, and looking at the intervals in the cumulative distribution, we can generate the appropriate status code.

/**
 * Example showing how to simulate a function returning a random 
 * variable that can take more than 2 values
 * @author Raj
 */
public class MultiValueTest {
    
    public static int fetchContentsStub(URL url) {
        double[] probs = new double[] {0.6,0.3,0.07,0.03};
        int[] codes = new int[] {200, 404, 500, 503 };
        double cumulative = 0;
        double rnd = Math.random();
        for (int i=0; i<probs.length; i++) {
            cumulative += probs[i];
            if (rnd < cumulative)
                return codes[i];
        }
        return codes[codes.length-1];
    }

    public static void main(String[] args) {
        for (int i=0; i<100; i++) {
            System.out.println(fetchContents(null));
        }   
    }

}

One Reply to “How to Test Functions that are NOT Predictable?”

  1. Brilliant. If programmers are as good as you are, i see the need for testing team decreasing, but then the probablity of me loosing my job increases as well :) Never mind i can do something else using my artist skills, “Bravo Rajkumar”.

Leave a Reply

Your email address will not be published. Required fields are marked *