Huge spreadhseet - slow calculation speed for formulas

Question

I have an excel spreadsheet with about 300,000 records. I am applying a Vlookup formula on the first row and then trying to copy and paste special as formulas to the remaining rows. But, the calculation tends to consume a lot of time and shows progress in multi-threads which is really slow.

Could someone suggest a way to do this to save some time.

Thank you!

Answer 1

Assuming you need to do an exact match lookup then the fastest formula way to do this is to sort your data and then use the double VLOOKUP trick.

see my blog post for details: https://fastexcel.wordpress.com/2012/03/29/vlookup-tricks-why-2-vlookups-are-better-than-1-vlookup/

Answer 2

If you're stuck with it, your best bet is to optimize your formulas by avoiding common pitfalls. You've mentioned that you're using VLOOKUP, why not try INDEX/MATCH combo, it is more versatile and is faster than VLOOKUP. Other optimizing techniques include:

• avoiding volatile formulas
• using helper columns
• arrange the columns in such a way that the sequence of your calculation is from left to right, top to bottom, very much like reading English text
• Use structured and named references
• Enable manual calculation so that formulas are not recalculated every time you make even the slightest change

You may also want to use a timer to measure Excel calculation time, for you to compare your different approaches, such as this one by Charles Williams, from an article in Microsoft Developer Network. Paste this code into the VBA editor:

#If VBA7 Then
    Private Declare PtrSafe Function getFrequency Lib "kernel32" Alias _
        "QueryPerformanceFrequency" (cyFrequency As Currency) As Long
    Private Declare PtrSafe Function getTickCount Lib "kernel32" Alias _
         "QueryPerformanceCounter" (cyTickCount As Currency) As Long
#Else
    Private Declare Function getFrequency Lib "kernel32" Alias _                                            "QueryPerformanceFrequency" (cyFrequency As Currency) As Long
    Private Declare Function getTickCount Lib "kernel32" Alias _
        "QueryPerformanceCounter" (cyTickCount As Currency) As Long
#End If
Function MicroTimer() As Double
'

' Returns seconds.
    Dim cyTicks1 As Currency
    Static cyFrequency As Currency
    '
    MicroTimer = 0

' Get frequency.
    If cyFrequency = 0 Then getFrequency cyFrequency

' Get ticks.
    getTickCount cyTicks1                            

' Seconds
    If cyFrequency Then MicroTimer = cyTicks1 / cyFrequency 
End Function

Sub RangeTimer()
    DoCalcTimer 1
End Sub
Sub SheetTimer()
    DoCalcTimer 2
End Sub
Sub RecalcTimer()
    DoCalcTimer 3
End Sub
Sub FullcalcTimer()
    DoCalcTimer 4
End Sub

Sub DoCalcTimer(jMethod As Long)
    Dim dTime As Double
    Dim dOvhd As Double
    Dim oRng As Range
    Dim oCell As Range
    Dim oArrRange As Range
    Dim sCalcType As String
    Dim lCalcSave As Long
    Dim bIterSave As Boolean
    '
    On Error GoTo Errhandl

' Initialize
    dTime = MicroTimer              

    ' Save calculation settings.
    lCalcSave = Application.Calculation
    bIterSave = Application.Iteration
    If Application.Calculation <> xlCalculationManual Then
        Application.Calculation = xlCalculationManual
    End If
    Select Case jMethod
    Case 1

        ' Switch off iteration.

        If Application.Iteration <> False Then
            Application.Iteration = False
        End if

        ' Max is used range.

        If Selection.Count > 1000 Then
            Set oRng = Intersect(Selection, Selection.Parent.UsedRange)
        Else
            Set oRng = Selection
        End If

        ' Include array cells outside selection.

        For Each oCell In oRng
            If oCell.HasArray Then
                If oArrRange Is Nothing Then 
                    Set oArrRange = oCell.CurrentArray
                End If
                If Intersect(oCell, oArrRange) Is Nothing Then
                    Set oArrRange = oCell.CurrentArray
                    Set oRng = Union(oRng, oArrRange)
                End If
            End If
        Next oCell

        sCalcType = "Calculate " & CStr(oRng.Count) & _
            " Cell(s) in Selected Range: "
    Case 2
        sCalcType = "Recalculate Sheet " & ActiveSheet.Name & ": "
    Case 3
        sCalcType = "Recalculate open workbooks: "
    Case 4
        sCalcType = "Full Calculate open workbooks: "
    End Select

' Get start time.
    dTime = MicroTimer
    Select Case jMethod
    Case 1
        If Val(Application.Version) >= 12 Then
            oRng.CalculateRowMajorOrder
        Else
            oRng.Calculate
        End If
    Case 2
        ActiveSheet.Calculate
    Case 3
        Application.Calculate
    Case 4
        Application.CalculateFull
    End Select

' Calculate duration.
    dTime = MicroTimer - dTime
    On Error GoTo 0

    dTime = Round(dTime, 5)
    MsgBox sCalcType & " " & CStr(dTime) & " Seconds", _
        vbOKOnly + vbInformation, "CalcTimer"

Finish:

    ' Restore calculation settings.
    If Application.Calculation <> lCalcSave Then
         Application.Calculation = lCalcSave
    End If
    If Application.Iteration <> bIterSave Then
         Application.Iteration = bIterSave
    End If
    Exit Sub
Errhandl:
    On Error GoTo 0
    MsgBox "Unable to Calculate " & sCalcType, _
        vbOKOnly + vbCritical, "CalcTimer"
    GoTo Finish
End Sub

You may also check the links under References for more suggestions and techniques in optimizing Excel performance. Although in the second link, it said:

"It is usually faster to use the Excel formula calculations and worksheet functions than to use VBA user-defined functions."2

Although this may be true if we're talking about the process of calculation itself but if you do your calculations in VBA, your output will be static. So if you're doing let's say financial modelling, what-if scenarios, and such, it'll be painful to wait for 15 minutes for each minute changes if automatic calculation is enabled because the formulas are actively referencing the cells, monitoring for any changes.

Hope this helps..

---

References:

Excel performance: Improving calculation performance

Excel performance: Tips for optimizing performance obstructions

Excel performance: Performance and limit improvements