16

I have a system to test, which is a video ads distribution technology. I need to load every video like 1-2 mins to serve the ads. The videos are played in a Flash client and streamed as FLV streams like in YouTube.

The reason why I need to test it only via browsers -- and every other method won't work -- is to stress test both the video streaming servers and the ads servers simultaneously and displaying ads in real-time.

I have used Selenium, WatiN, Automation Anywhere and many other automation tools. However, when I am trying to start like 10000 browsers on my machine (32GB RAM, 16-core CPU), none of them are able to do the job.

With Selenium, I am able to start the maximum FireFox instances so far, but that's still too low: half of the instances don't run the test.

Any suggestions to do with Selenium?

Braiam
  • 1
  • 11
  • 47
  • 78
Sumit Ghosh
  • 3,264
  • 4
  • 40
  • 59
  • If you need to simulate a load you can use open source tool called pylot (http://pylot.org/) this tool can simulate concurrent connections and you can provide a test scenario that written in XML (read the getting started on the site). The tool is not using real browsers. – alonp Mar 26 '12 at 17:11

6 Answers6

5

You aren't going to run 10,000 browsers on your machine. That would give 3.2MB of physical memory per browser instance and I'm pretty sure FireFox just won't like that.

You could create a JMeter script that hits your server with many threads. It won't interact with the UI but would simulate the load of many clients hitting whatever URLs you tell it. I believe it also includes the ability to record a session and play it back for easy setup of your sessions.

Andrew T.
  • 4,701
  • 8
  • 43
  • 62
digitaljoel
  • 26,265
  • 15
  • 89
  • 115
  • 1
    10000 was a hypothetical number, For the current server config even 1000-5000 would be fine. Jmeter is not useful in my case as I have a rich media application which needs to execute for a minute for the test to succeed. – Sumit Ghosh Mar 26 '12 at 17:10
  • 2
    3.2MB or 32MB of ram, I suspect firefox will choke equally well on either. Arguably you are testing the server if you are looking to throw 1000 threads at it so who cares what the client is doing? Whether the client is full fledged firefox using the media and doing something with it, or just a curl or wget to the appropriate URLs with the same timing as the rich media app shouldn't matter to the server, should it? – digitaljoel Mar 26 '12 at 17:15
  • 2
    The testing has more to do with the how the rich media app behaves under stress, not just the server under load. I am trying to test the possible nearest real world user scenario here. A curl request wont execute the rich media client app, so it wont be a real test of the entire application. – Sumit Ghosh Mar 27 '12 at 07:08
  • 2
    If you are looking to test the client app, then what about creating a severely under-powered virtual machine and starting a few instance on that? You could still hit the server with a jmeter script to get the load up there, and you could then run your selenium script on the weak vm. – digitaljoel Mar 27 '12 at 14:42
4

Selenium isn't really optimized for load/stress testing, especially if you're running your browsers locally. Running 1000+ browsers is going to choke even the beefiest server. Though RAM is an obvious bottleneck, you also have limited CPU resources and bandwidth. The latter being a primary concern if you are loading videos.

Not to mention you'd be testing from a single IP with 10k browsers, so load balancing may not kick in properly, as well as the actual distribution of video ads to specific virtual users.

If you want to stick with existing Selenium tests, I've had good experiences with BrowserMob. They basically have a huge grid to do real browser load-testing, distributed across AWS.

Another recommendation would be an actual performance testing tool. I'd recommend Soasta CloudTest. They have a free version that runs 100 users so you can see if it will be a good fit for you. I have found that scripting for CloudTest is relatively simple.

Disclaimer: My experiences with both companies have been as a paying customer and I have never worked for either.

Andrew T.
  • 4,701
  • 8
  • 43
  • 62
kenki
  • 538
  • 3
  • 4
1

Your post smells of some form of ad-based fraud to me, but either way: have you considered using different web browsers besides Firefox? PhantomJS is a headless webkit-based browser that is compatible with Selenium. It supports all the core browser features like DOM handling, CSS selectors, Javascript and Canvas. I do not know if it supports Flash.

This post has a decent list of other headless and automatable web-browsers that you might consider.

Also, if each browser instance is instantiating a Flash plugin, don't neglect the possibility that the issue could be with Flash and not Firefox. Alternatively, why instantiate several different Firefox processes? Can you accomplish what you want through the use of tabs instead?

Community
  • 1
  • 1
Ponkadoodle
  • 5,777
  • 5
  • 38
  • 62
1

If you are using Windows machine then as per my experience there is a limit on number of browser window instances to be opened. As per my test last time, it does restrict between 100-150 browser windows.

I would recommend you using headless robot, which doesn't require opening browser window. I think latest version of Selenium has that capability. But it seems to be more like a load test as you are trying to simulate 10,000+ user instances, I would recommend you using load testing tool like JMeter or LoadRunner.

Sandip Dhummad
  • 993
  • 1
  • 7
  • 10
  • Iam able to run 500+ firefox instances with selenium but the problem is half of them wont load the recorded test script, they just show the blank browser. This test is not possible without a real browser, I will update my original post with some more details about the test scenario. – Sumit Ghosh Mar 27 '12 at 08:14
  • Sumit, if you must run your test with active browser window then you have following options now: – Sandip Dhummad Mar 27 '12 at 15:09
  • Sumit, if you must run your test with active browser window and you have already invested your lots of time and money writing selenium scripts then I would suggest following options: 1) Employ multiple instances at Amazon EC2. You can setup Selenium Grid at your EC2 instance. Find details at: http://selenium-grid.seleniumhq.org/setting_up_selenium_grid_on_ec2.html. 2) If due to data security or any other reason you can not use cloud then you can setup selenium grid using multiple test machine at your lab. Personally I would prefer using EC2 as its fast, easy and cheap option. – Sandip Dhummad Mar 27 '12 at 15:18
  • Hey Sandip, I have no problems doing that, but right now I am not even able to use up barely 30% of the ram and cpu combined in my monster test server. Had I crossed 80% on there, I would have thought for horizontal scaling. The problem is not about scaling outside, its about efficiency firing up 1000 instances of firefox and make them do an automated job driven by a selenium script. – Sumit Ghosh Mar 27 '12 at 18:54
  • 2
    Sumit, do you have proper ramp up time in place while firing browser windows up. We had around 10 milliseconds sleep time between starting of 2 browser windows. We had seen that setting proper sleep time had increased capacity of number of browser instances. 10 ms seconds may not work as is in your case, but you can start with that number. I know that it complicates your test data requirement as now you will have to make each user run its test scenario multiple times to achieve ultimate scenario where all users are up and running. We had to set up multiple test data for same test script. – Sandip Dhummad Mar 27 '12 at 21:58
1

It looks to me that you are trying to verify what the client will see based on high traffic, no?

In that case, Joel is quite correct. If you absolutely have to see what the client sees, you could use threaded hits and just dump the results in a database. That'll show you anything the client would see anyway, and it's a lot easier to sort through than thousands of browser instances.

Either way, your client will not see errors if there are no errors present on the server side. If you're testing functionality in bandwidth restricted environments, CPU-intensive environments, or memory-intensive environments, those are much easier achieved than running thousands of browser instances.

Andrew T.
  • 4,701
  • 8
  • 43
  • 62
Nate I
  • 946
  • 4
  • 10
0

The in-house way to this wiht selenium is using browsermob proxy and multiple broswser agents to recreate the experience of different users, changing the ip is more difficult because it requires changing your home network.

Here is a good example

Rudolf Fanchini
  • 103
  • 1
  • 7