Need expert advice for WCF Service programming

Question

!--- I'm sorry if this post is redundant with some others, but the problem with my project seems to be resulting of my architecture, so I need general help. ---!

I'm trying to configure a WCF Service connected to a Silverlight 5 client-side and a C# class library doing CRUD requests to a database. The service will have to pass GIGANTIC amount of data. For example, I have a public IList< RainRecord> GetAllRain() method in my service class that can retrieve hundreds of thousands of records in the database, will probably be even more than that someday. So I thought I could get all these records by smaller batches (I think it's a pretty good approach).

Here's my web.config file :

<?xml version="1.0" encoding="utf-8"?>

<!--This file contains the web configuration used by the WCF service. -->

<configuration>

  <system.web>
    <compilation debug="true" targetFramework="4.0" maxBatchGeneratedFileSize="2147483647" maxBatchSize="2147483647" />
    <httpRuntime maxRequestLength="2147483647" />
  </system.web>

  <system.serviceModel>

   <serviceHostingEnvironment aspNetCompatibilityEnabled="true">
     <serviceActivations>
       <add service="PoseidonServiceNamespace.PoseidonService" relativeAddress="~/PoseidonService.svc"/>
     </serviceActivations>
   </serviceHostingEnvironment>

    <bindings>
      <basicHttpBinding>
        <binding  closeTimeout="00:01:00"
                  openTimeout="00:01:00"
                  receiveTimeout="00:10:00"
                  sendTimeout="00:01:00"
                  allowCookies="false"
                  bypassProxyOnLocal="false"
                  hostNameComparisonMode="StrongWildcard"
                  maxBufferSize="2147483647"
                  maxBufferPoolSize="2147483647"
                  maxReceivedMessageSize="2147483647"
                  messageEncoding="Text"
                  textEncoding="utf-8"
                  transferMode="Buffered"
                  useDefaultWebProxy="true">
          <readerQuotas maxDepth="2147483647"
                        maxStringContentLength="2147483647"
                        maxArrayLength="2147483647"
                        maxBytesPerRead="2147483647"
                        maxNameTableCharCount="2147483647" />
          <security mode="None">
            <transport  clientCredentialType="None"
                        proxyCredentialType="None"
                        realm="" />
            <message clientCredentialType="UserName"
                     algorithmSuite="Default" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>

    <behaviors>
      <serviceBehaviors>
        <behavior name="PoseidonServiceBehavior">
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="true"/>
          <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
        </behavior>
      </serviceBehaviors>
    </behaviors>



    <services>
      <service  behaviorConfiguration="PoseidonServiceBehavior"
                name="PoseidonServiceNamespace.PoseidonService">
        <endpoint address="http://localhost:49455/PoseidonService.svc"
                  binding="basicHttpBinding"
                  contract="PoseidonServiceNamespace.IPoseidonService"/>

        <endpoint address="mex"
                  binding="mexHttpBinding"
                  contract="IMetadataExchange"/>
      </service>
    </services>
  </system.serviceModel>

 <system.webServer>
    <modules runAllManagedModulesForAllRequests="true"/>
  </system.webServer>

</configuration>

What are the changes I should do in this file (and why?), and how should I implement my public IList GetAllRain() method in my service?

Thank you really much, I've been banging my head on this for a while

Answer 1

Are you sure you need to transfer hundreds of thousands of records to client application? How client would view these records? Would these records be pre-processed / aggregated somehow, or just displayed in some grid? If preprocessed - consider doing preprocessing on server side. If just displayed - consider redesigning your interface to include paging:

IList<RainRecord> GetList(int startPage, int pageSize);

If paging is not an option, but you would still want to retrieve data in batches - you can implement manual batching by introducing state to your service (You would need to know who requested list, at which position you currently at, and how much data left to transfer):

public interface IService
{
  Guid BeginGetList();
  IList<RainRecord> GetNextBatch(Guid id);
}

Empty list or null returned by GetNextBatch would indicate end of transfer. In other words, it is the same paging, but implemented using stateful service.

Now, if transfering hundreds of thousands of records is absolutely neccessary, and must be done as one call - consider using streaming to transfer your data. But you would need to change your contract, to include Stream with serialized RainRecords.

Another alternative would be to implement duplex service, which would pass callback to client, and when client has finished processing previous batch, it can request next. Downside of this is that as far as I know, WSDualHttpBinding is not supported by silverlight.

With binding you attached, you would be limited to 2Gb per batch, which I believe is enough. Also you would probably need to tweak timeouts, this post describe all possible timeout values in great detail.