10

We've been using protractor for end-to-end testing for a while.

Now we are trying to cover several corner cases, which involve modifying the response from the API endpoint requests - for this we are using protractor-http-mock which provides an easy-to-use way to replace HTTP responses with pre-defined mocks.

But, what if we want to test a situation when a sudden connection loss happens? What are our options in this case?

In other words, we want to achieve a case when requests to specific endpoints would produce a network connection error and see how our application would react.

I'm open to any suggestions, I am currently thinking of the following strategies:

  • see if there are third-party nodejs libraries similar to protractor-http-mock
  • mock $http angularjs service
  • fire up a proxy and somehow control it during the tests (grunt-connect-proxy looks pretty mature though I'm not sure if it is possible to dynamically change the behavior of the proxy from spec to spec)
  • control it on a browser level - e.g. with a Network Throttling google chrome feature (though I'm pretty sure it is something that selenium cannot control, Network throttling with chrome and selenium) (browser addon/extension?)
Community
  • 1
  • 1
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • Unit testing is so chic that developers often over test. I think you are reaching too far in your test effort. We could definitely work through ways to simulate a lost connection, but a lost connection should be an integration test. The best way to test lost connections is with to manually cut your connection or stop your server and observe the results. It's not a test you want repeated automatically, but rather with manual effort. The give-away that you are over-reaching for a unit test is when your mock / scenario is so complex it is in need of unit testing and debugging. – Dave Alperovich Apr 04 '15 at 03:32
  • @DaveAlperovich well, this is really an end-to-end integration test (not a unit test) we are asking about (using `protractor`). Though, I accept the point that it could be not worth the effort. Thank you! – alecxe Apr 04 '15 at 04:01
  • Good observation. My language exposed my bias. I think of end-to-end testing as an extension of unit testing (multiple units - lego style). I assume that all components in the chain to use categorical data that can be neatly mocked. The moment we delve into non-deterministic data or environment problems, I see it as manual effort. – Dave Alperovich Apr 04 '15 at 04:08
  • @DaveAlperovich I understand your opinion, it makes sense - I've been actually told I overtest sometimes. I understand that I won't cover every possible mix of environmental + connection + performance + etc problem that an end-user could probably have, but at least I would like to know if I have a way to check it in an automated manner - at least I'll cover critical parts of the application. This conversation is probably getting us nowhere though I like to discuss these kinds of philosophical topics about where does this end in end-to-end testing ends :) – alecxe Apr 04 '15 at 04:17
  • 1
    take a look at sinon FakeXMLHttpRequest. You would have to implement a mock server response, which is an investment, and it's transactional, so it falls short in the sense that you ideally would want a random number of transactions or random time range for ideal testing. But, this is the best I can imagine to simulating a connection. – Dave Alperovich Apr 04 '15 at 15:45
  • 1
    Another thought, you could connect to blocked port. I suspect you would prefer to create a robust Server mocking pattern instead. PS, while I criticize your perfectionism, I admire it too. This kind of open-ended issue is fun once you start wrapping your mind around it. – Dave Alperovich Apr 04 '15 at 21:16
  • @DaveAlperovich I'll explore this option too. Thank you, this is a good exercise and an interesting challenge to solve. Hope I'll have time to concentrate on the problem this week. – alecxe Apr 06 '15 at 00:09
  • 2
    If you are going to mock a http service not to work then why mock it at all? Just make your tests run on invalid uri's, for that test. – Pedro Silva Apr 07 '15 at 22:08
  • @PedroSilva sounds logical, I would somehow need to let the application know when to make an API requests to invalid URLs and when to switch back to valid URLs. It is in a way similar to what Dave suggested (connect to a blocked port). Thank you (see if you can elaborate it into an answer - probably some more information or an example would be needed..). – alecxe Apr 07 '15 at 22:25
  • But i dont see why this is useful, if this is a web app then how can he (the user) use it wihtout internet? – Pedro Silva Apr 07 '15 at 23:00
  • @PedroSilva we all experience connectivity issues from time to time depending on the quality of the connection and other factors. This is a real world use case - we have users on an unstable internet. It is important for the application to adequately react on network problems and connection errors. This is what I'm trying to find a reliable way to simulate and check. – alecxe Apr 07 '15 at 23:02

4 Answers4

4

Seems like the Comcast tool would provide the majority of the functionality you need:

Comcast is a tool designed to simulate common network problems like latency, bandwidth restrictions, and dropped/reordered/corrupted packets.

It works by wrapping up some system tools in a portable(ish) way. On BSD-derived systems such as OSX, we use tools like ipfw and pfctl to inject failure. On Linux, we use iptables and tc. Comcast is merely a thin wrapper around these controls.

https://github.com/tylertreat/Comcast

Example on Linux:

comcast --device=eth0 --latency=250 --target-bw=1000 --default-bw=1000000 --packet-loss=10% --target-addr=8.8.8.8,10.0.0.0/24 --target-proto=tcp,udp,icmp --target-port=80,22,1000:2000

You can change the settings on the fly and put packet loss up to 100% as needed.

13rac1
  • 1,057
  • 11
  • 14
2

I hope this helps you to define your best way to implement the mocks.

The project explains the general idea, there are semantics and translation problems with the readme.md, i'll working on that soon please be patient. Please forget my English/misspelling let me known of any correction

I was working on this Demo project of how to implement Protractor + CucumberJS + sugar-step . Right now it contains a very very simple Angular APP and 3 E2E Test.

The demo is working, but is unfinished yet, still need some features which i think are useful for E2E Testing.

  • The first test check a service against a default service (Dev environment mock)
  • The Second test check that the service request returns an error with status code 404
  • The Third test check that the service request returns an error with status code 500

How can i set different Mock responses for each Scenario?

  • Using Cucumber's Around Feature, which can be used to inject a angular.module('mock-Service-response-x',fn...) before any feature or scenario, and remove it after the feature or scenario runs

What can be tested this way?

  • Any kind of scenarios, responses with extreme delays, response any Status Code, response special data for test very particular scenarios which otherwise will be impossible/hard to obtain from a normal back-end service, like data inconsistency on the back-end

Why implement your UI-Mocks in AngularJS and not with a proxy?

  • UI Control, (of course depending of your particular needs) using Angular you can do things like: disable ngAnimations or jQuery animations entirely, Mock your local browser Date, to emulate behaviors which depends on date in a automatically.

  • Cost, (depending of your project) the cost of implement a local proxy for all the web services on every dev environment can be expensive (in hardware or effort or hours of implementation).

  • Dependency, if your dev teams are working in parallel, it's more likely to be depending from each-other, if the UI team is working in a feature that depends on a X Web Service, knowing the implementation specs the UI team can create simple mocks to continue with his work making more probable the deliver on time of the hole project

All those arguments depends of which project are you working on, and doesn't mean at all that this is the best solution, but i'm more in favor of keep the project layers independent from each other, and be able to release a new feature in one layer no matter if other layer is delayed in his own new features.

To anyone who reads this, any comment, correction or suggestion will be more than welcome

Community
  • 1
  • 1
Matias Fernandez Martinez
  • 1,877
  • 13
  • 15
  • Thank you for the answer,. We currently simulate `404` and `500` server errors with `protractor-http-mock`, but need to simulate connection errors - haven't found an example in the project you've mentioned.. – alecxe Apr 08 '15 at 14:08
  • The project i posted doesn't implement that scenario yet, but please check [this file](https://github.com/thematho/cucumberjs-and-mock-e2e-demo/blob/master/test/e2e/features/mocks/authentication-service-mock-e2e.js). (Can you use `protractor-http-mock` for protractor for automation test on the UI ? I think only is useful on karma test) – Matias Fernandez Martinez Apr 08 '15 at 14:25
  • By "connection error" what do you mean? **444 No Response** or **499 Client Closed Request** Or you are talking of a service that never respond? If is the first case, the only thing to do is change the status code returned. If is the second case, is ver simple only have to do is something like: `$httpBackend.whenPOST('/services/login').respond(function() { // never return, so the client will wait until detects any timeout fallback });` – Matias Fernandez Martinez Apr 08 '15 at 14:25
  • It is the second case, I'll try what you've suggested and come back to you, thanks. – alecxe Apr 08 '15 at 14:35
1

That's also depends on how are going to do that? Should it be programmatically?

If so the best solution is proxy depending, and the best tool could be used with all selenium/webdriver tools is browserMobProxy,

If you want to do it from the outside, I would recommend to use any external Traffic shapers/proxies also. Eg: http://vaurien.readthedocs.org/en/1.8/ and

vaurien --protocol http --proxy nonexistingproxy.com:8000 --backend website.com:80

Why shouldn't you use 9n0browser capabilities? First of all, you will need to reload the page to see them in action, second one is you will not redefine the proxy settings in protractor with them. As for the plugins, it's not so easy to interact with them programmatically and I see no any benefits, while you should append a lot of the for different browsers.

Stan E
  • 3,396
  • 20
  • 31
1

There is a tool from Facebook that called ATC (A tool to simulate network conditions).

It allows you to control bandwith, latency, packet loss and packet corruption rates in your network.

Since it's open source, I think you can extend the functionality in order to use different "error" profiles, or set up integration with Webdriver.

You can find general information about ATC here. ATCs github repo is here.

Viktor Malyi
  • 2,298
  • 2
  • 23
  • 40