C# Regex: Get random URL from HTTP response in Web Perfomance and Load Test Project

Question

Could you help me to solve the problem?

I have the webrequest

 WebTestRequest request1 = new WebTestRequest("http://www.theverge.com/");
        request1.ParseDependentRequests = false;
        yield return request1;

and the response on it:

      <li><a href="https://www.facebook.com/verge" tabindex="2">
        <svg class="p-svg-icon"><use xlink:href="#icon-facebook"></use></svg>
      </a></li>

      <li><a href="https://twitter.com/verge" tabindex="3">
        <svg class="p-svg-icon"><use xlink:href="#icon-twitter"></use></svg>
      </a></li>

      <li><a href="/rss/index.xml" tabindex="4">
        <svg class="p-svg-icon"><use xlink:href="#icon-rss"></use></svg>
      </a></li>

I want to find all URLs on this page and random use them in the next step. As I understand, I should use

string response = LastResponse.BodyString;

than regex, to make an ArrayList Class and take a random url.

Could somebody help me with it?

score 1 · Accepted Answer · edited May 23 '17 at 12:01

There are two main ways I have used. First by using a regular expression and getting all the matches. Create a regular expression that matches all the wanted URLs but does not match unwanted URLs. One web site I tested had many URLs where the href= was preceeded by a class= having several numeric levels. I only wanted levels 2 and 3, leading to the expression in the code below.

This approach work when you are confident that the HTML will be in a fixed format for the total duration of the test development and test execution. If the HTML of (or near) the href=s is likely to change then you need a more complex regular expression, or another method.

I have used the method GetARandomNumber because the .NET random number generators are not thread safe. See here for example.

public class GetAUrl : WebTestRequestPlugin
{
    public string ContextParameter { get; set; }

    public override void PostRequest(object sender, PostRequestEventArgs e)
    {
        string body = e.Response.BodyString;

        // Looking for requests in a specific class.

        string pattern = "\\bclass=\"nav__link-[23]\" +href=\"(/[^\"]*)\"";
        // The URL is in this capture:                        ^^^^^^^^^

        MatchCollection matches = Regex.Matches(body, pattern);

        int randomIndex = GetARandomNumber(matches.Count);
        //  Above will get a value 0 <= randomIndex < matches.Count

        e.WebTest.Context[ContextParameter] = matches[randomIndex].Groups[1].Value;
    }
}

Another method uses the HtmlAgilityPack where the essence of the method uses:

HtmlAgilityPack.HtmlNodeCollection nodes = input_doc.DocumentNode.SelectNodes("//a");

and then you screen the collection of nodes to select those of interest and then choose one at random.

Hi @AdrianHHH, Thank you for the help. I really appreciate. I'll try to figure out and use it in my project. — Bob McTester, Dec 08 '16 at 13:24

score 0 · Answer 2 · answered Dec 09 '16 at 07:44

0

The result:

string response = LastResponse.BodyString;
            string pattern = @"regexp";

            MatchCollection results = Regex.Matches(response, pattern);
            Random rnd = new Random();
            string randomUrl = results[rnd.Next(results.Count)].Groups[1].Value;

answered Dec 09 '16 at 07:44

Bob McTester

13
2

As I said in my answer, `Random` is not thread safe. Additionally, calling `new Random()` uses the current time as a seed for the random numbers. In load testing we can often get several test cases starting with the same value of "current time" and so they will all generate the same set of random numbers. Please do some research on thread safe random numbers. – AdrianHHH Dec 09 '16 at 08:57

C# Regex: Get random URL from HTTP response in Web Perfomance and Load Test Project

2 Answers2