.Net Parser JSON from a open API

Question

I am working on a little project to process some big data from a game, that will enable me to view some advanced analytics, However, I have come up against a wall .... The following URL (Click Here) is an open API that returns a JSON result and I am trying to process this as my own data.

I have made a class that should process this data into my model, however, every time line 5 is called client.DownloadString i receive an error 403, is there any way around this? I do not know the owner of the api.

public IActionResult Index(object sender, EventArgs e)
{
    var model = new FresnoVm();
    WebClient client = new WebClient();
    string strPageCode = client.DownloadString("https://api.upx.world/bigdata/query?neighborhood=210&neighborhood=359&neighborhood=367&neighborhood=366&neighborhood=356&neighborhood=364&city=0&status=All&mintMin=0&mintMax=100000000&saleMin=0&saleMax=100000000&skip=0&fsa=All&sort=mint_price&ascOrDesc=1");
    dynamic dobj = JsonConvert.DeserializeObject<dynamic>(strPageCode);

    price = dobj["data"]["properties"]["sale_price_upx"].ToString();

    model.test = price;

    return View("~/Features/Fresno/Index.cshtml", model);
}

Answer 1

While adding "User-Agent: Other" to wb.Headers as suggested in this answer by Borg8 to WebClient 403 Forbidden is sufficient to download the response string, since the returned string is roughly 1.6MB it would be better to deserialize directly from the response stream, as recommended in Newtonsoft's Performance Tips: Optimize Memory Usage.

First, define the following helper method:

public static class JsonExtensions
{
    public static T GetFromJson<T>(string url, JsonSerializerSettings settings = default)
    {
        var request = (HttpWebRequest)HttpWebRequest.Create(url);
        // User-Agent as suggested by this answer https://stackoverflow.com/a/6905471/3744182
        // by https://stackoverflow.com/users/873595/borg8
        // To https://stackoverflow.com/questions/3272067/webclient-403-forbidden
        request.UserAgent = "Other";

        using (var response = (HttpWebResponse)request.GetResponse())
        {
            if (response.StatusCode == HttpStatusCode.OK)
            {
                using (var stream = response.GetResponseStream())
                using (var textReader = new StreamReader(stream))
                {
                    settings = settings ?? new JsonSerializerSettings { CheckAdditionalContent = true };
                    return (T)JsonSerializer.CreateDefault(settings).Deserialize(textReader, typeof(T));
                }
            }
            else
            {
                throw new ApplicationException(); // Throw some exception with a message of your choice
            }
        }
    }
}

And then you can do:

var dobj = JsonExtensions.GetFromJson<JToken>(url);
var prices = dobj["data"]["properties"].Select(t => (decimal?)t["last_paid_price_upx"]).ToList(); // Or cast to (string) if you prefer

Notes:

• That there is no property named "sale_price_upx" in the returned JSON. Some, but not all, of the data.properties[*] objects contain a "last_paid_price_upx" property, so the code above shows an example of how to extract those as nullable decimals.

• the LINQ-to-JSON document object model can use significant amounts of memory for property names. You may get better performance deserializing large amounts of data to directly an explicit data model that contains only the properties you need.

Demo fiddle here.

Answer 2

Add the simple line like below

wb.Headers.Add("User-Agent: Other");   //that is the simple line!