0

The problem is, using ssis, I do an ado source to ado destination. This method only writes over 88 rows per second and is very slow.

using Oracle.ManagedDataAccess.Client;
using System;
using System.Data;
using System.Data.SqlClient;

namespace SQLconnection
{
internal static class Program
{
    private static void Main(string[] args)
    {
        SqlConnection conn = new SqlConnection("Data Source=;Database=;Integrated Security=yes");
        conn.Open();
        SqlCommand cmd = new SqlCommand("SELECT * FROM TABLE", conn);
        SqlDataReader reader = cmd.ExecuteReader();
        while (reader.Read())
        {
            Console.WriteLine(reader.GetString(0) + ", " + reader.GetString(19));
        }

        conn.Close();
        conn.Dispose();

        Console.ReadLine();

        OracleConnection con = new OracleConnection("User Id=;Password=;Data Source=;");
        con.Open();
        OracleCommand cmd2 = con.CreateCommand();
        cmd2
            .CommandText = "SELECT \'Hello World!\' FROM dual";

        OracleDataReader reader2 = cmd2.ExecuteReader();
        reader2.Read();
        Console.WriteLine(reader2.GetString(0));

        Console.WriteLine(con.ServiceName);
        Console.WriteLine(con.ServerVersion);
        Console.WriteLine(con.HostName);

        con.Close();

        Console.ReadLine();

    }
}
}

Is there any way I can do a connection and pass the data via a console application? I feel that would be faster than 88 rows per sec.

Chad Lomax
  • 93
  • 2
  • 9
  • 1
    That's not SSIS code. That's plain-old ADO.NET. In SSIS you can simply connect an SQL Server source with an Oracle destination and get thousands of rows per second. 88/sec means you have a problem either in the source or target. Using raw ADO.NET won't fix that problem – Panagiotis Kanavos Feb 14 '19 at 15:00
  • Ok, Thanks. Ill keep looking into solutions. – Chad Lomax Feb 14 '19 at 15:09
  • I would avoid using `select * from ...` in your code sample you are only using 2 columns (0 and 19) so just select what you need otherwise you are asking Oracle to send you a lot of unnecessary data. – Dominic Cotton Feb 18 '19 at 16:21

2 Answers2

0

Yes, you can write a console app that uses a native Oracle provider to pass data to Oracle.

https://www.oracle.com/webfolder/technetwork/tutorials/obe/db/dotnet/GettingStartedNETVersion/GettingStartedNETVersion.htm

Tab Alleman
  • 31,483
  • 7
  • 36
  • 52
0

I have found that file based operations are much quicker when doing bulk data transfers.

I would investigate using the BCP out utility to generate delimited text files from SQL server. Have a read: https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-2017

Getting it into Oracle could possibly be a little harder (I have very limited Oracle experience). As per the following question, you can investigate using SQL Loader scripts:

Oracle: Import CSV file

There are a couple gotchas when using BCP, though:

  • Depending on the makeup of the data (do you have commas and carriage returns in text fields in your data?), consider using custom delimiters for both fields and records. This can be easily specified in your BCP command by using the -t and -r options
  • Obviously, make sure the fields and the data matches (or is at least comparable) between formats. You can use the QUERYOUT option in BCP to create custom queries, which should give you the ability to cast if you need to and order the columns the way you want.

It might not be the sexiest solution, but it can work, can be very repeatable and can see a high throughput of data. We have been doing this for a Sybase ASE to SQL Server ETL process and saw processing times drop to 10% of what they were using other database to database methods.

Obviously, though, YMMV, so test first.

Ash
  • 390
  • 4
  • 18
  • 1
    I appreciate the the answer. After some thought, I have come to think that a file based approach would be the most effective way to process this data between systems. I also understand that I am new to the ETL process and it will take time. I will be thoughtful and develop a reliable testable solution. I already have the first table casted. Anyway, thanks a bunch! – Chad Lomax Feb 15 '19 at 02:20
  • No problems at all. From my point of view, the biggest hurdle is going to be getting data into Oracle, but that is because I am not overly familiar with the tools available. But, I am sure that this can be worked through. Cheers. – Ash Feb 15 '19 at 03:48