Transferring 500,000 (half a million) records in one go is too large for your system to handle. I'd also say that it was too many for your users to handle.
You should break this down into pages of data and only return a couple of pages at a time. The Silverlight/WCF (RIAServices) DomainDataService can handle all this for you:
<riaControls:DomainDataSource QueryName="GetResults"
LoadSize="200"
PageSize="100"
AutoLoad="True"/>
You add a pager control to your page to move through the pages of data under user control.
This makes your application more responsive as you are only returning a small amount of data each time. Returning 500,000 records in one go will also more than likely cause timeouts for people with slow connections.
I'd also suggest you look into filtering your data so that you only return the data the user is interested in.