C# - How do I extract a CSV file from a JSON API using HttpClient object?

Question

My company uses a C# .net application to programmatically retrieve data from a vendor website in the form of a CSV file. However, the vendor changed their API recently to be a JSON API. I'm a bit of a novice programmer, and don't have any experience with JSON API calls. When the vendor registered my new API userID, they provided an example of how to download the necessary csv data file using curl. Here is the sample:

echo 'Going to log in to sso'
data=$(curl -c cookies.txt -X POST https://sso.theice.com/api/authenticateTfa -H "Content-Type: application/json" \
--data '{"userId":"yourApiUserId","password":"yourpassword","appKey":"yourAppKey"}')

echo 'Going to download icecleared_oil_2020_06_26.dat file from Settlement_Reports_CSV/Oil'
curl -b cookies.txt -O ' https://downloads2.theice.com/Settlement_Reports_CSV/Oil/icecleared_oil_2020_06_26.dat

Since I have no experience with Curl, I used https://curl.olsh.me/ to convert the sample above to C#. Doing so generated the following code:

//Authenticate userId (and get cookie?)
using (var httpClient = new HttpClient())
{
    using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://sso.theice.com/api/authenticateTfa"))
    {
        request.Content = new StringContent("{\"userId\":\"yourApiUserId\",\"password\":\"yourpassword\",\"appKey\":\"yourAppKey\"}");
        request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json"); 

        var response = await httpClient.SendAsync(request);
    }
}

//Using authentication cookie, from previous code block, retrieve dat file
var handler = new HttpClientHandler();
handler.UseCookies = false;

using (var httpClient = new HttpClient(handler))
{
    using (var request = new HttpRequestMessage(new HttpMethod("GET"), "https://downloads2.theice.com/Settlement_Reports_CSV/Oil/icecleared_oil_2020_06_26.dat"))
    {
        request.Headers.TryAddWithoutValidation("Cookie", "cookies.txt");

        var response = await httpClient.SendAsync(request);
    }
}

I integrated this code into my project and was able to get it to authenticate my userID (the POST request), but cannot for the life of me figure out how to retrieve the DAT file from the second code block (the GET request). I have observed that the response from the GET request is the same no matter whether I successfully login or not with the POST request. At this point I am assuming the validation cookie from the POST request is not being picked up by the GET request, but even if it was I'd have no idea how to extract the file from the response object in the GET request.

Can anyone advise on what I need to do? Below is the code that's been integrated into my project:

public async Task DownloadAsync(List<string> iceReportFileNamesFullUrl)
{
    await AuthenticateUserCredentialsOnICE();
    await RetrieveIceRportFiles(iceReportFileNamesFullUrl);

}

private async Task AuthenticateUserCredentialsOnICE()
{
    using (HttpClient httpClient = new HttpClient())
    {
        using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("POST"), "https://sso.theice.com/api/authenticateTfa"))
        {               
            request.Content = new StringContent("{\"userId\":\"xxxxxxxxx\",\"password\":\"xxxxxxxxxx\",\"appKey\":\"xxxxxxxxxxx\"}");
            request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");

            HttpResponseMessage response = await httpClient.SendAsync(request);
            var value = response.Content.ReadAsStringAsync();
        }
    }
}
private static async Task RetrieveIceRportFiles(List<string> iceReportFileNamesFullUrl)
{
    foreach (string iceReportUrl in iceReportFileNamesFullUrl)
    {
        HttpClientHandler handler = new HttpClientHandler();
        handler.UseCookies = false;

        using (HttpClient httpClient = new HttpClient(handler))
        {
            using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("GET"), iceReportUrl))
            {
                request.Headers.TryAddWithoutValidation("Cookie", "cookies.txt");

                HttpResponseMessage response = await httpClient.SendAsync(request);
                var value = response.Content.ReadAsStringAsync();

            }
        }
    }
}

-UPDATE: 25 AUG 2020-

Thanks for your comments y'all. You made it clear that I wasn't capturing the cookies from the POST request. After some research (this, this, and this), I've updated the code so that it now successfully downloads the CSV/DAT file. I've posted what I have below, though it still requires error handling, checks for failed login, etc. I'm also sure there will be critical feedback.

Here it is:

private CookieContainer authenticationCookieContainer;
//main method for downloading file
public async Task DownloadAsync(string iceReportDownloadUrl)
{
    await AuthenticateUserCredentialsOnIceAsync();
    await SendDownloadReportRequestAsync(iceReportDownloadUrl);
}

Authenticate the credientials via JSON API and store authentication cookies in authenticationCookieContainter

private async Task AuthenticateUserCredentialsOnIceAsync()
{
    //This will contain the ICE authentication cookies
    //if the login is successful
    authenticationCookieContainer = new CookieContainer();

    //Create and assign handler for proxy config and cookie storage
    using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = authenticationCookieContainer })
    {
        using (HttpClient httpClient = new HttpClient(handler))
        {
            using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("POST"), "https://sso.theice.com/api/authenticateTfa"))
            {
                request.Content = new StringContent("{\"userId\":\"xxxxxxxxxxx\",\"password\":\"xxxxxxxxxxxxxx\",\"appKey\":\"xxxxxxxxxxxxx\"}");
                request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
                HttpResponseMessage response = await httpClient.SendAsync(request);
            }
        }
    }
}

If the login was successful, the authenticationCookieContainer will contain the login cookies and allow you to request the data at the specified url

private async Task SendDownloadReportRequestAsync(string iceReportDownloadUrl)
{
    if (authenticationCookieContainer != null)
    {
        //Create and assign handler for proxy config and cookie storage
        using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = authenticationCookieContainer })
        {
            //Set to true to use the cookies obtained during authentication call
            handler.UseCookies = true;

            using (HttpClient httpClient = new HttpClient(handler))
            {
                Uri iceReportUri = new Uri(iceReportDownloadUrl);
                //Request the ICE data file using the url passed through
                using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("GET"), iceReportUri.AbsoluteUri))
                {
                    HttpResponseMessage response = await httpClient.SendAsync(request);
                    using (Stream responseStream = await response.Content.ReadAsStreamAsync())
                    {
                        //Write the extracted file to the file path
                        byte[] content = StreamToByteArray(responseStream);
                        File.WriteAllBytes(@"C:\Users\my.name\Desktop\" + iceReportUri.Segments.Last(), content);
                    }
                }
            }
        } 
    }
}
private static byte[] StreamToByteArray(Stream stream)
{
    using (MemoryStream memoryStream = new MemoryStream())
    {
        stream.CopyTo(memoryStream);
        return memoryStream.ToArray();
    }
}