9

I am working on a performance test which involves several clients bombarding the server with 150 requests each, as fast as they can.

The server is constructed from 3 WCF services, one is opened to outside with httpbinding, it talks to 2 other services via net.pipe (IPC). one of the services is in charge of DB connections (SQL server 2008 R2).

This DB connection service uses the following connection string enhancments:

Min Pool Size=20; Max Pool Size=1000; Connection Timeout=20;

and is WCF throttled(like all other WCF services).

I noticed that when I activate 1 client it may take 3 seconds, but when I activate 3 clients it may take 8-9 or more.

I checked with the SQL server profiler to see how many concurrent processes are used, and I saw that only about 8 processes are being used.

So I realized that somewhere in the server the requests get queued instead of concurrently processed.

In order to get to the bottom of it I've used a performance profiler (ANTS to be exact) which showed me that about 70% of the time was wasted on "Waiting for synchronization

When I open the call graph I find two things that look strange but I'm not sure what they mean:

  1. System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke is being used in the top of the tree, is that ok for concurrent processing?
  2. All synchronized problems involve some kind of SQL Server activity, like ExecuteNonQuery, ExecuteReader and so on (When I get to the bottom of the call tree)

I noticed that the DB connection service uses a DAL project(some legacy code, unfortunately) which is totally static.

After reading this I'm not sure if DAL's code is problematic or not, here is a sample of a stored procedure call.

    public static int PerformStoredProcedure(string storedP,string ext,out string msg)
    {
        msg = "";
        SqlCommand command = GetSqlCommand(storedP,ext);
        command.Connection.Open();
        int result = (int)PerformStoredProcedure(command,out msg);
        command.Connection.Close();
        return result;
    }

This Method is usualy called from the DB connection service:

    public static int PerformStoredProcedureWithParams(string storedP,string ext,out string msg, params object[] pars)
    {
        msg = "";
        SqlCommand command = GetSqlCommand(storedP,ext);
        UpdateCommandParams(command, pars);
        command.Connection.Open();
        int result = (int)PerformStoredProcedure(command,out msg);
        command.Connection.Close();
        return result;
    }

So, is there anything wrong here?

Or maybe I should look somewhere else?

Edit:

After Brijesh's comment I realized I hadn't change the default InstanceContextMode and ConcurrencyMode of the WCF services... Kind of beginner's mistake I guess.

I'm still not certain if I should use PerSession/Multiple or PerCall/ Single. As I see it each service should handle each request as on object, regardless of the client.

What should I use?

2nd EDIT:

After using PerCall and PerSession/Multiple, I noticed that there is still no change (at least in the DB service). what I see is that the main entry point service may open up alot of threads, but only a few (still around 8-10 threads) are opened at the DB connection service.

is there any other reason why this can happen? I ruled out DAL being a problem because not enough requests go in the DB service so I figure its something in the service or something in the clients...

3rd EDIT:

Here are the config files:

Manager's config wcf service section:

<services>
  <service behaviorConfiguration="ServicesBehavior" name="Verifone.GenericPP.GPPManagerService.GPPManagerServiceImpl">
    <host>
      <baseAddresses>
        <add baseAddress="http://localhost:9090/GPPManagerService/"/>
      </baseAddresses>
    </host>
    <endpoint contract="Verifone.GenericPP.GPPManagerService.IGPPManagerService"  binding="basicHttpBinding" address="GPPManagerService"></endpoint>
  </service>
</services>
<behaviors>
  <serviceBehaviors>
    <behavior name="ServicesBehavior">
      <!--amith 13-05-2012-->
      <serviceThrottling
        maxConcurrentCalls="1000"
        maxConcurrentSessions="1000"
        maxConcurrentInstances="1000"
      />
      <serviceMetadata httpGetEnabled="true" />
      <serviceDebug includeExceptionDetailInFaults="true" />
    </behavior>
  </serviceBehaviors>
</behaviors>
<bindings>
  <basicHttpBinding>
    <binding name="basicHttpBinding" maxBufferSize="10000000" maxReceivedMessageSize="10000000">
      <readerQuotas maxStringContentLength="10000000" maxArrayLength="10000000"/>
      <security mode="None">
        <transport clientCredentialType="None"/>
      </security>
    </binding>

Manager's clients:

      <endpoint name="endpoint1" contract="IDBConnectionContract"  bindingConfiguration="basicHttpBinding"  binding="basicHttpBinding" address="http://localhost:9010/DBConnectionService/DBService"></endpoint>
  <endpoint name="endpoint2" contract="IGPPService"  bindingConfiguration="basicHttpBinding"  binding="basicHttpBinding" address="http://localhost:9095/GenericPPService/GenericPPService"></endpoint>

DB Connection Service:

<service behaviorConfiguration="ServicesBehavior" name="Verifone.DBConnectionService.DBConnectionContracImpl">
    <host>
      <baseAddresses>
        <add baseAddress="http://localhost:9010/DBConnectionService/"/>
        <add baseAddress="net.pipe://localhost/DBConnectionService/"/>
      </baseAddresses>
    </host>
    <endpoint contract="Verifone.DBConnectionService.IDBConnectionContract"  binding="basicHttpBinding" address="DBService"></endpoint>

    <endpoint contract="Verifone.DBConnectionService.IDBConnectionContract"  binding="netNamedPipeBinding"  bindingConfiguration="NetNamedPipeBinding_Configuration"  address="" name="pipeEndpoint"/>
  </service>

The business logic service's client is pretty much the same like the Manager's.

All services are self hosted and I have a DBConnectionProxy class at Manager's and Business code which they activate like this:

 DBConnectionContractClient _dbConnectionContractClient = null;
        try
        {
            objDBConnectionContractClient = new DBConnectionContractClient();
            objDBConnectionContractClient.ExecuteStoredProcedure(input, out result);
        }
Community
  • 1
  • 1
Mithir
  • 2,355
  • 2
  • 25
  • 37
  • what type of instance and concurency mode are you using? – Brijesh Mishra Jun 27 '12 at 07:30
  • It looks like I didn't set any mode, so default settings apply... I remember I wanted to add Per Session/ Multiple, but probably missed it somehow, should I add it? – Mithir Jun 27 '12 at 07:40
  • 4
    persession/single is default, try with concurency mode multiple – Brijesh Mishra Jun 27 '12 at 08:13
  • Wouldn't it be better to use PerCall/Single ? I need every request to be unique, and it doesnt matter who is the client. – Mithir Jun 27 '12 at 09:45
  • I don't know what I'm doing wrong, but I put the attribute with Multiple/PerSession and it doesn't seem to do any difference – Mithir Jun 27 '12 at 10:33
  • Are you hosting this in iis or is it windows service application? – Akash Kava Jul 01 '12 at 07:55
  • Your DAL code's command.Connection.Close should be inside finally block in try finally block, in case of exception connection won't close and won't be reused. – Akash Kava Jul 01 '12 at 07:58
  • @AkashKava it is self-hosted windows service application. You are absolutely right, in the case of an exception the connection will be hanging and wasted, but its not why I'm experiencing the problem described... – Mithir Jul 01 '12 at 08:04
  • Try hosting it in IIS, there is also IIS Hostable Core available, we did see some improvement when we moved away from self hosted application to IIS. To my guess, WCF might be using some thread pool, and if might depend on your Thread Pool of your Service Application. Are you reusing Service Client instance? Synchronization context will not be used if you are creating new Service Client instance for each call. – Akash Kava Jul 01 '12 at 09:41

3 Answers3

4

PerCall You may consider this instancing mode in these circumstances.

  • If your service is stateless

  • If your service has light-weight initialization code (or none at
    all).

  • If your service is single threaded.

Some good tutorials. See the third link on tuning.

WCF Instancing, Concurrency, and Throttling – Part 1

WCF Instancing, Concurrency, and Throttling – Part 2

WCF Instancing, Concurrency, and Throttling – Part 3

kenny
  • 21,522
  • 8
  • 49
  • 87
paparazzo
  • 44,497
  • 23
  • 105
  • 176
  • Thanks for the suggestion, I've tried both but still requests get queued while send to the db service – Mithir Jun 28 '12 at 15:13
  • Just a reach maybe look at at how you are activating the clients to test. Are you possibly taking an exclusive lock on the same table? – paparazzo Jun 28 '12 at 15:27
  • The problem does not occur on the actual DB but on a service which activates the db. The requests are queued before entering that service, before tables are even an issue. – Mithir Jun 28 '12 at 15:33
  • Wow, could you post the test code where you queue up the requests? – paparazzo Jun 28 '12 at 15:39
  • Just to add a bit, when I run the performance test multiple threads are opened in the main service and in the second service but only a few in the db connection service. Both services are sending requests to that db service, so I think that the problem is not in the test code but maybe in those services code. I think that maybe because both services are clients of the same service from the same server they are competing on the same connections, makes sense? – Mithir Jun 28 '12 at 16:49
  • If you would post you test code then I could try and reproduce. I don't experience the problem you describe. – paparazzo Jun 28 '12 at 17:59
  • Having those services hosted on same server should not cause any issues...unless if you might reach the limit of allowed tcp connections on your test machine. What OS are you using? – Mihai H Jun 28 '12 at 18:25
  • @Blam I've added some config files hope it helps – Mithir Jul 01 '12 at 07:34
  • By test code I mean the post the code where you spin up multiple calls to the service. – paparazzo Jul 01 '12 at 12:21
2

Would prefer per call. Session would be usefull if you want to minimize initializing resource intensive objects or running some preprocessing routing for each request or want to maintain state. Session can give you better performance but per call always scales better. In per call you dont have to worry about concurency. Whereas disadvantage with persession/multiple is that you have to always think about thread safety in your code.

Brijesh Mishra
  • 2,738
  • 1
  • 21
  • 36
  • I really don't know what is the matter... I set it to PerCall or PerSession multiple on all services but it still doesnt seem to matter, The main http wcf service opens alot of threads, but only a few get to the DB connection service... – Mithir Jun 27 '12 at 13:46
  • is it problem with concureent request, or it slows doen by just opening multiple instance? – Brijesh Mishra Jun 28 '12 at 04:06
  • I've added some config files of the WCF services, hope it helps a little – Mithir Jul 01 '12 at 07:35
1

Best option would be per call if you don't care about service state and if no heavy lifting is done on initialization, I assume that describes your case. Try and remove those DAL calls and see how your service behaves, in this way you will be able to identify if that is your bottle neck.

Mihai H
  • 3,291
  • 4
  • 25
  • 34