0

I need to import huge amount of data from CSV into SQL Server.

I found this amazing code (it works extremely fast, better than anything else I tried, that's why I use it despite I don't understand it completely) in the Internet and adapt it a bit for my tasks, but the problem is: what can I do if my CSV file has empty field that must be INT (nullable int)? It's ok if we can convert all NULLS to 0 and all empty values to 0, but now I just got this error:

Input string was not in a correct format.Couldn't store <> in column_name Column. Expected type is Single.

And I don't know how to convert data types in PowerShell at all (I want to improve code so it can convert dtypes like SQL Server Import Wizard does, hope it's possible). For example, I want to have opportunity to convert date from "dd.mm.yyyy" in CSV to "yyyy-mm-dd", or float from "70.0" in CSV to "70" in SQL Server.

My code:

# Database variables
$sqlserver = "servername"
$database = "DBname"
$table = "tablename"

# CSV variables
$csvfile = "test.csv"
$csvdelimiter = ","
$firstRowColumnNames = $true

################### No need to modify anything below ###################
Write-Host "Script started..."
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
[void][Reflection.Assembly]::LoadWithPartialName("System.Data")
[void][Reflection.Assembly]::LoadWithPartialName("System.Data.SqlClient")

# 50k worked fastest and kept memory usage to a minimum
$batchsize = 50000

# Build the sqlbulkcopy connection, and set the timeout to infinite
$connectionstring = "Data Source=$sqlserver;Integrated Security=true;Initial Catalog=$database;"
$bulkcopy = New-Object Data.SqlClient.SqlBulkCopy($connectionstring, [System.Data.SqlClient.SqlBulkCopyOptions]::TableLock)
$bulkcopy.DestinationTableName = $table
$bulkcopy.bulkcopyTimeout = 0
$bulkcopy.batchsize = $batchsize

# Create the datatable, and autogenerate the columns.
$datatable = New-Object System.Data.DataTable

# Open the text file from disk
$Default = [System.Text.Encoding]::GetEncoding("utf8")
$reader = New-Object System.IO.StreamReader($csvfile,$Default)
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
if ($firstRowColumnNames -eq $true) { $null = $reader.readLine() }

#foreach ($column in $columns) {
#    $null = $datatable.Columns.Add()
#}

$col1 = New-Object system.Data.DataColumn nullable_int_test,([int])
$col2 = New-Object system.Data.DataColumn bad_date_test,([datetime])
$col3 = New-Object system.Data.DataColumn string_test,([string])
$col4 = New-Object system.Data.DataColumn float_test,([float])

$datatable.columns.add($col1)
$datatable.columns.add($col2)
$datatable.columns.add($col3)
$datatable.columns.add($col4)

#$line = $reader.ReadLine()
#write-host $line

# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null) {
    $null = $datatable.Rows.Add($line.Split($csvdelimiter))
    $i++; 

    if (($i % $batchsize) -eq 0) {
        $bulkcopy.WriteToServer($datatable)
        Write-Host "$i rows have been inserted in $($elapsed.Elapsed.ToString())."
        $datatable.Clear()
    }
}

# Add in all the remaining rows since the last clear
if($datatable.Rows.Count -gt 0) {
    $bulkcopy.WriteToServer($datatable)
    $datatable.Clear()
}

# Clean Up
$reader.Close(); $reader.Dispose()
$bulkcopy.Close(); $bulkcopy.Dispose()
$datatable.Dispose()

Write-Host "Script complete. $i rows have been inserted into the database."
Write-Host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"
# Sometimes the Garbage Collector takes too long to clear the huge datatable.
[System.GC]::Collect()

CSV example:

nullable_int_test,bad_date_test,string_test,float_test
70,01.25.2021,testtext,70.0
,01.26.2021,testtext,85.0

P.S I'm really bad at PowerShell syntax so if you have solution for my I would be extremely graceful if you show me exactly what lines in the code and how to modify. Thanks!

jarlh
  • 42,561
  • 8
  • 45
  • 63
Faenno
  • 27
  • 8
  • The Power Script is using the BCP.EXE command line utility which comes with SQL Server. See following : https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-ver15 – jdweng Jan 27 '22 at 17:28
  • It's not powershell, but you could check this answer https://stackoverflow.com/a/70655927/1745616 – BdR Jan 27 '22 at 17:42
  • Thank you for your comments but links didn't help – Faenno Jan 27 '22 at 18:29

0 Answers0