0

I am hitting a URL that returns a long JSON set (REALLY long, 20 million characters). Just pasting the URL into Chrome, it takes about 3 minutes to return the full result set. Whatever the default settings in Chrome are, it prompts me several times to either Kill the page or Wait. But the page will return after several minutes.

I'm running this from SSIS with a script task. I'm not very familiar with C#. I copied/pasted this code from a sample:

{

    HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl);
    httpWReq.Method = "GET";
    httpWReq.ContentType = "application/json";
    httpWReq.Timeout = 300000;
    HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();
    RootObject jsonResponse = null;

    try
    {
        //Get the stream of JSON
        Stream responseStream = httpWResp.GetResponseStream();

        //Deserialize the JSON stream
        using (StreamReader reader = new StreamReader(responseStream))
        {
            //Deserialize our JSON
            DataContractJsonSerializer sr = new DataContractJsonSerializer(typeof(RootObject));
            jsonResponse = (RootObject)sr.ReadObject(responseStream);
        }
    }
    //Output JSON parsing error
    catch (Exception e)
    {
        FailComponent(e.ToString());
    }
    return jsonResponse;

I am 110% positive that the wURL string is a valid JSON endpoint. When I step through my code, it waits maybe 15 seconds on this line:

HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();

... and then returns without error... but it doesn't populate what I'd expect into httpWResp (ContentLength = -1). When it gets to:

jsonResponse = (RootObject)sr.ReadObject(responseStream);

... jsonResponse holds my pre-defined json container object, set to null. There are thousands of json arrays returned from my URL.

I don't see any interesting attributes in responseStream that would indicate that it actually contains anything?

What am I missing here?

I can't post the actual URL because it's a private company URL.

=================================

EDIT: I tried a URL with a much shorter string, and it returned. So it appears to be something about the length. I ran the return value through a validator and it succeeded... so possibly a special character, but I'm thinking likely the length.

Quicksilver
  • 295
  • 4
  • 16
  • 1
    put try catch around your code and see what exception it is throwing if any. try { Your Code here } catch(WebException ex) { // catch web exception } catch(Exception ex) { // Any other exception } – Hakunamatata Nov 29 '16 at 01:29
  • what version of .net/sql server are you using? The HttpWebRequest library has undergone quite a lot of changes over the last few years – AllMadHare Nov 29 '16 at 01:34
  • @AllMadHare, I'm using .NET Framework v 4.6.1 with SSIS VS 2015, pointed to SQL Server 2012. I don't think the SQL Server should have anything to do with it, as I haven't added any SQL Server components to my package yet? I added a try/catch block. It's not hitting an exception. Is it possible that the response is too long for a get...? – Quicksilver Nov 29 '16 at 01:46
  • 1
    Not sure what you mean "doesn't populate anything." If it returns a non-null HttpWebResponse object, it is definitely populated with *something* (even if all the values are 0 or `null` or empty strings). Can you be more specific? Is httpWResp showing up as `undefined` in your watch window? – John Wu Nov 29 '16 at 01:48
  • Paste the exact url you are using in code in the browser and see what that returns – CodingYoshi Nov 29 '16 at 01:53
  • @JohnWu, my httpWResp object is populated with the appropriate object, but the ContentLength attribute is -1. – Quicksilver Nov 29 '16 at 02:26
  • What are the values of `StatusCode` and `ContentType` on the response object? – David Tansey Nov 29 '16 at 03:34
  • 1
    Take a look at the accepted answer in this post: http://stackoverflow.com/questions/15375499/c-sharp-httpwebresponse-contentlength-1-when-file-too-large for an explanation of why `ContentLength` may reflect a value of -1 but still be a valid/usable response (and how to handle it). – David Tansey Nov 29 '16 at 03:37
  • @DavidTansey, StatusCode = OK, ContentType = "application/json;charset=UTF-8". So I guess this indicates that the data is chunked? Is there an easy example of how to iterate through a stream reader to get the full set? I'll post my full code. – Quicksilver Nov 29 '16 at 04:22

1 Answers1

1

From the comments we now know that the response object you get back from GetResponse() has a StatusCode of OK and a ContentType of application/json;charset=UTF-8 -- indicating that the server has returned the data 'chunked' which is why the ContentLength = -1.

You should be able to use the ReadToEnd() method on the StreamReader, something like this:

//Deserialize the JSON stream
using (StreamReader reader = new StreamReader(responseStream))
    {
        string r = reader.ReadToEnd();

        //Deserialize our JSON
        DataContractJsonSerializer sr = new DataContractJsonSerializer(typeof(RootObject));
        MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(r));
        jsonResponse = (RootObject)sr.ReadObject(ms);
}
Quicksilver
  • 295
  • 4
  • 16
David Tansey
  • 5,813
  • 4
  • 35
  • 51
  • I tried something like this first, and it didn't work, which is why I tried to avoid using a string. This does appear to be a different error. I get: '{"Expecting element 'root' from namespace ''.. Encountered 'None' with name '', namespace ''. "} System.Exception {System.Runtime.Serialization.SerializationException}' ... which to my inexperienced eye, looks like perhaps this JSON does not contain the expected starter nodes?? – Quicksilver Nov 29 '16 at 20:29
  • Can you get the entire response string without error? If that's handled ok now you can try to resolve any serialization issues. Is there a way for you to contrive a test with a much smaller response just so you can validate the approach to serialization? – David Tansey Nov 29 '16 at 20:41
  • I don't think I can get the entire response string yet, since I get the error above when I use the ReadToEnd attribute. I saw another link that said that the error I posted above means that the Stream needs to be moved back to position 0, but when I tried that, I got the error that the stream does not support seek operations. I'm kind of puzzled as to why this is so difficult, as it appears to be a very standard approach to pulling JSON. Oh well. – Quicksilver Nov 29 '16 at 20:57
  • Actually, ignore my last comment, I think it's something else. FYI, the last line in your code won't compile, wrong object type: jsonResponse = (RootObject)sr.ReadObject(r);. Looking if changing that will address my issue. – Quicksilver Nov 29 '16 at 21:16
  • Yeah, so I think the issue is that I need a way to serialize that string now. – Quicksilver Nov 29 '16 at 21:25
  • I added a MemoryStream object to handle the string, and I seem now to have gotten myself into the same place I was before: my code all processes, and then I get back an empty object for jsonResponse. – Quicksilver Nov 29 '16 at 21:45
  • You are placing the string contents in the `MemoryStream` and then passing that `MemoryStream` reference to `ReadObject()` -- yes? – David Tansey Nov 29 '16 at 21:48
  • Yes. So it would seem that my problem isn't with the WebRequest at all (blush). That seems to be working, and it's the attempt to serialize to my RootObject that bombs. I think it's a problem with my JSON object array declaration. I'm kind of shooting in the dark here, but I noticed that my attributes are not cased the way the URL return values are, so perhaps that's it.Since we are wandering away from my original question, if you'd like to correct your answer to reflect a memory stream, I'll mark it as the answer. I might spawn a new question. – Quicksilver Nov 29 '16 at 22:00
  • FYI: http://stackoverflow.com/questions/40878350/c-i-cant-serialize-my-json-array-from-my-web-request/40879048#40879048. – Quicksilver Nov 30 '16 at 02:05