0

I have an SSIS package which sends data as a string to an external company's Web API via a Post request, and the string is a pipe delimited one derived from thousands of rows retrieved by a SQL query - the data from the SQL table is them passed as a string to a User variable called User::RequestBody, which is then referenced in the Script Task for the webclient.UploadString call. The problem I am facing is that, if there's a single bit of bad data amongst the thousands rows, the API request is rejected. I then have to try and identify which of the data is bad and as you can imagine, it is pretty time-consuming. Below is the code I have in the Script Task of the SSIS package;

public void Main()
        {

            string username = Convert.ToString(Dts.Variables["$Project::DestinationUserName"].Value);
            string password = Convert.ToString(Dts.Variables["$Project::DestinationPassword"].GetSensitiveValue());
            Uri DestinationURI = new Uri((string)Dts.Variables["$Project::DestinationSite"].Value);
            string stringToSend = (string)Dts.Variables["User::RequestBody"].Value;
            if (stringToSend != "")
            {
                string result;
                using (WebClient webClient = new WebClient())
                {

                    var bytes = Encoding.UTF8.GetBytes(username + ":" + password);
                    var auth = "Basic " + Convert.ToBase64String(bytes);

                    //NetworkCredential myCreds = new NetworkCredential(username, password);
                    //webClient.Credentials = myCreds;
                    webClient.Headers[HttpRequestHeader.Authorization] = auth; 
                    webClient.Headers[HttpRequestHeader.ContentType] = "text/plain"; 
                    ServicePointManager.SecurityProtocol = ServicePointManager.SecurityProtocol | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; beyond the standard SSL3 and TLS
                    try
                    {
                      result = webClient.UploadString(DestinationURI, stringToSend); 

                    }
                    catch (WebException ex)
                    {
                        Dts.Events.FireError(0, "", "UnableToSendData: " + ex.Message.ToString() + ex.StackTrace, string.Empty, 0);
                        return;
                    }
                }

            }
            Dts.TaskResult = (int)ScriptResults.Success;
        }

Below is a sample of 2 records for reference (\n is the start and ending of each data row), but of course we usually send about 9,000 records daily;

"123456|123456|TestUser1|Adam|Mphil/PhD History (PT) Year|3|Department of History||P||||XK|Information not sought|||123456|123456@gmail.com|||||V1ZR-Y||HISs|H|M|1998-05-21||PGR|||0777XXXXXX|||2021-09-19|2022-09-10| |Test NOK: 58 Wiltshire Ave, HA8 5DG: 0208XXXXXX|||||\n678910|678910|TestUser2|Smith|Mphil/PhD Near and Middle East (PT) Year|12|Other Middle East||P||||XK|No known disability|||678910|678910@hotmail.com|||||T6ZT/D-Y||OMEs|H|M|1990-03-15||PGR|||079XXXXXXXX|||2022-09-26|2023-09-25| |Test NOK2: 45 Britannia Street, MK11 2ZQ: 0207XXXXXX|||||\n"

I must add, when we send the Post Request to the Web API, we log the response to table in SSMS - the table simple logs the request response, and in this case it's a '400 Bad Request' response. Is there any way to identify where exactly the Request falls over in a case where thousands of records are being sent, as in, can the offending records be identified in a simple way? I'm thinking maybe attempting to send the data one-by-one and if the Post Request encounters a record with bad data, it can be flagged.

GBM
  • 99
  • 6
  • There are log files. I would log into database with SQL Server Management Studio and the logs are in the explorer under Management. There also may be log files on machine with server under Window Start Button : Microsoft SQL. – jdweng Mar 27 '23 at 16:12
  • @jdweng thank you for the reply and input - I have looked on SSMS >> Management but there aren't any log files associated with the SSIS Package. I was hoping for perhaps a solution which can be put in place in SSIS within the Package itself. – GBM Mar 27 '23 at 16:25
  • Did you enable logs? https://learn.microsoft.com/en-us/sql/integration-services/performance/integration-services-ssis-logging?force_isolation=true&view=sql-server-ver16 – jdweng Mar 27 '23 at 16:59

0 Answers0