2

I'm doing a lookup using the below code, where I have 100+ headers files from that I need to lookup for Product ID and get the size, base & variation IDs at the end of the headers.

Option Explicit
Private Sub dellkp()


Application.ScreenUpdating = False


Dim sht As Worksheet, lsht As Worksheet
Dim wbk As Workbook, lwbk As Workbook
Dim rng As Range, rng1 As Range, cell As Range
Dim h1 As Integer, h2 As Integer, h3 As Integer, p As Integer, s As Integer, b As Integer, v As Integer, n As Integer
Dim lr As Long, lr1 As Long, fn As Long
Dim f As String, fname As String, a As String

f = Sheet1.Range("B1").Value & "\"

Set wbk = ActiveWorkbook
Set sht = Sheet2

h1 = 106
h2 = 107
h3 = 108

'product id
p = 9
'sku variant
s = 93
'base id
b = 71
'variant id
v = 87

sht.Cells(1, h1) = "Size"
sht.Cells(1, h2) = "Base"
sht.Cells(1, h3) = "Variation"


lr = sht.Cells(sht.Rows.Count, "A").End(xlUp).Row

fname = Dir(f & "*.xlsx")

sht.Activate
Set rng = sht.Range(Cells(2, h1), Cells(lr, h1))

n = 0

Do While fname <> ""

a = Replace(fname, ".xlsx", "")
    
     Set lwbk = Workbooks.Open(f & fname)
     Set lsht = lwbk.ActiveSheet
    
    lr1 = lsht.Cells(lsht.Rows.Count, "A").End(xlUp).Row
    
    
    Set rng1 = lsht.Range(Cells(1, p), Cells(lr1, p))
        
    For Each cell In rng
    If IsEmpty(cell) Then
        If Not IsError(Application.Match(sht.Cells(cell.Row, p), rng1, 0)) Then
            fn = Application.Match(sht.Cells(cell.Row, p), rng1, 0)
            cell = lsht.Cells(fn, s)
            cell.Offset(0, 1) = lsht.Cells(fn, b)
            cell.Offset(0, 2) = lsht.Cells(fn, v)
            
        End If
    End If
    
    Next
              
    lwbk.Close
    wbk.Save
    
    n = n + 1
    
   fname = Dir
          
Loop

Application.ScreenUpdating = True

MsgBox n & " File(s) are Lookedup!!!", , "Linga"

End Sub

Column numbers are as follows for Product ID, size, base & variation in the files to be looked up

enter image description here

The output of the above code at the end of header columns will be as shown below

enter image description here

The above code is working fine, however it's taking time if the volume is high. Also files to be looked up will be increasing every week. Hence, I have to make it quicker. Can anyone guide me?

Can we show progress bar how much completed in % or number files?

Linga
  • 945
  • 1
  • 14
  • 31

2 Answers2

0

There are two things you can do to speed up the code significantly:

First: avoid Application.Match inside the loop

I suggest creating a Scripting.Dictionary before the While loop storing Key/Item pairs. Code might look something like this:

'Note: early binding requires that you add `Microsoft Scripting Runtime` reference to your project
Dim Col2Dict as Scripting.Dictionary 
Set Col2Dict = New Scripting.Dictionary 

For Each cell in rng
    ' If duplicates exist in rng, this will raise an error, but then again
    ' I guess it's a key column, so there should be no duplicates
    Col2Dict.Add Key:=cell.Value, Item:=cell.row
Next cell

Now you have the dictionary containing all the pairs (Value of sht.Cells(cell.Row, 2) as Keys and cell.row as Items) the lines

If Not IsError(Application.Match(sht.Cells(cell.Row, p), rng1, 0)) Then
    fn = Application.Match(sht.Cells(cell.Row, p), rng1, 0)

can become

If Col2Dict.Exists(sht.Cells(cell.Row, p)) Then
    fn = Col2Dict(sht.Cells(cell.Row, p))

which will run many orders of magnitude faster, especially for large ranges... You can find more information on how to use dictionaries here.

Second: Do the editing in arrays stored in memory

The next optimization would be to load the entire range into an array, perform the edits there and then write the entire array back to the worksheet at once. In your case that would be the range sht.Range(Cells(2, h1), Cells(lr, h1+2)). A replacement for your While loop might look like this:

'These two additionay variables need to be declared
Dim wArr As Variant
Dim i As Long

'Putting the content of the range to edit into an array
wArr = rng.Resize(, 3).Value

Do While fname <> ""
    a = replace(fname, ".xlsx", "")
    
    Set lwbk = Workbooks.Open(f & fname)
    Set lsht = lwbk.ActiveSheet
    
    lr1 = lsht.Cells(lsht.Rows.Count, "A").End(xlUp).row
    
    Set rng1 = lsht.Range(Cells(1, p), Cells(lr1, p))
    
    i = 0
    For Each cell In rng
        i = i + 1
        If IsEmpty(cell) Then
            If Col2Dict.Exists(sht.Cells(cell.row, p)) Then
                fn = Col2Dict(sht.Cells(cell.row, p))
                wArr(i, 1) = lsht.Cells(fn, s)
                wArr(i, 2) = lsht.Cells(fn, b)
                wArr(i, 3) = lsht.Cells(fn, v)
            End If
        End If
    Next
 
    lwbk.Close
    wbk.Save
    
    n = n + 1
    fname = Dir
Loop
    
'Putting the edited array back to the range
rng.Resize(, 3).Value = wArr

You might notice, that I only converted the range that is actually being edited to an array. This is because converting the ranges, from which only read operations are performed, does not provide a performance benefit.

You can find more information on how to use arrays to speed up your code here.

Progress bar

It is relatively easy to implement this using userforms, the general idea is described here.

A finished and in my opinion slightly over-engineered version can be found here.

Personally, I have been using a simplified version of Mathieu Guindon's solution (second link), which I will describe here:

You need a Userform of approximately the following layout:

Progress Indicator Userform

It needs to contain at least:

  1. A label called "ProgressLabel"
  2. A frame
  3. A label called "ProgressBar" inside the frame. This label should have the "BackColor" property set to a noticeable color, like blue in my example.

The UserForm is called ProgressView, and contains the following code:

Option Explicit

Private Const PROGRESSBAR_MAXWIDTH As Integer = 264
Private Const PROGRESSBAR_DEFAULT_CAPTION As String = "Progress..."

'source for sub SystemButtonSettings: https://exceloffthegrid.com/hide-or-disable-a-vba-userform-x-close-button/
Private Const GWL_STYLE = -16
Private Const WS_CAPTION = &HC00000
Private Const WS_SYSMENU = &H80000

#If VBA7 Then
    Private Declare PtrSafe Function GetWindowLong Lib "user32" Alias "GetWindowLongA" (ByVal hwnd As Long, ByVal nIndex As Long) As Long
    Private Declare PtrSafe Function SetWindowLong Lib "user32" Alias "SetWindowLongA" (ByVal hwnd As Long, ByVal nIndex As Long, ByVal dwNewLong As Long) As Long
    Private Declare PtrSafe Function FindWindowA Lib "user32" (ByVal lpClassName As String, ByVal lpWindowName As String) As Long
    Private Declare PtrSafe Function DrawMenuBar Lib "user32" (ByVal hwnd As Long) As Long
#Else
    Private Declare Function GetWindowLong Lib "user32" Alias "GetWindowLongA" (ByVal hWnd As Long, ByVal nIndex As Long) As Long
    Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" (ByVal hWnd As Long, ByVal nIndex As Long, ByVal dwNewLong As Long) As Long
    Private Declare Function FindWindowA Lib "user32" (ByVal lpClassName As String, ByVal lpWindowName As String) As Long
    Private Declare Function DrawMenuBar Lib "user32" (ByVal hWnd As Long) As Long
#End If

Public Sub SystemButtonSettings(frm As Object, show As Boolean)
    Dim windowStyle As Long
    Dim windowHandle As Long
    
    windowHandle = FindWindowA(vbNullString, frm.Caption)
    windowStyle = GetWindowLong(windowHandle, GWL_STYLE)
    
    If show = False Then
        SetWindowLong windowHandle, GWL_STYLE, (windowStyle And Not WS_SYSMENU)
    Else
        SetWindowLong windowHandle, GWL_STYLE, (windowStyle + WS_SYSMENU)
    End If
    
    DrawMenuBar (windowHandle)
End Sub

Private Sub UserForm_Activate()
    ProgressBar.Width = 0 ' it's set to 10 to be visible at design-time
    Me.StartupPosition = 0
    Me.Width = 288
    Me.Height = 112

    Me.Caption = PROGRESSBAR_DEFAULT_CAPTION
    
    SystemButtonSettings Me, False
End Sub

Public Sub Update(ByVal percentValue As Single, Optional ByVal labelValue As String, Optional ByVal captionValue As String)
    If labelValue <> vbNullString Then
        ProgressLabel.Caption = labelValue
    End If

    If captionValue <> vbNullString Then
        Me.Caption = captionValue
    End If

    ProgressBar.Width = percentValue * PROGRESSBAR_MAXWIDTH
    
    DoEvents
End Sub

Now you are ready to use the progress indicator! Usage works as follows:

Before your While loop, create a new instance of the UserForm:

Dim progress As ProgressView
Set progress = New ProgressView
progress.show vbModeless

Dim fileCount as Long
fileCount = 0

And inside the While loop specify how the progress is updated:

fileCount = fileCount + 1
progress.Update (1. / (totalNumberOfFiles)) * fileCount, "Working on File " & name

After the While loop, make sure you unload the UserForm:

Unload progress
Set progress = Nothing

Everything implemented, your final code might look like this:

Option Explicit

Private Sub dellkp()
    Application.ScreenUpdating = False

    Dim sht As Worksheet, lsht As Worksheet
    Dim wbk As Workbook, lwbk As Workbook
    Dim rng As Range, rng1 As Range, cell As Range
    Dim h1 As Integer, h2 As Integer, h3 As Integer, p As Integer, s As Integer, b As 
    Integer, v As Integer, n As Integer
    Dim lr As Long, lr1 As Long, fn As Long
    Dim f As String, fname As String, a As String

    f = Sheet1.Range("B1").Value & "\"

    Set wbk = ActiveWorkbook
    Set sht = Sheet2

    h1 = 106
    h2 = 107
    h3 = 108

    'product id
    p = 9
    'sku variant
    s = 93
    'base id
    b = 71
    'variant id
    v = 87

    sht.Cells(1, h1) = "Size"
    sht.Cells(1, h2) = "Base"
    sht.Cells(1, h3) = "Variation"

    lr = sht.Cells(sht.Rows.Count, "A").End(xlUp).Row

    fname = Dir(f & "*.xlsx")

    sht.Activate
    Set rng = sht.Range(Cells(2, h1), Cells(lr, h1))

    ' Creating the Dictionary
    Dim Col2Dict as Object 
    'Using late binding so you don't have to worry about the reference
    Set Col2Dict = CreateObject("Scripting.Dictionary") 

    For Each cell in rng
        ' If duplicates exist in rng, this will raise an error, but then again
        ' I guess it's a key column, so there should be no duplicates
        Col2Dict.Add Key:=cell.Value, Item:=cell.row
    Next cell

    n = 0

    'These two additional variables need to be declared
    Dim wArr As Variant
    Dim i As Long

    'Putting the content of the range to edit into an array
    wArr = rng.Resize(, 3).Value

    Do While fname <> ""
        a = replace(fname, ".xlsx", "")
    
        Set lwbk = Workbooks.Open(f & name)
        Set lsht = lwbk.ActiveSheet
    
        lr1 = lsht.Cells(lsht.Rows.Count, "A").End(xlUp).row
    
        Set rng1 = lsht.Range(Cells(1, p), Cells(lr1, p))
    
        i = 0
        For Each cell In rng
            i = i + 1
            If IsEmpty(cell) Then
                If Col2Dict.Exists(sht.Cells(cell.row, p)) Then
                    fn = Col2Dict(sht.Cells(cell.row, p))
                    wArr(i, 1) = lsht.Cells(fn, s)
                    wArr(i, 2) = lsht.Cells(fn, b)
                    wArr(i, 3) = lsht.Cells(fn, v)
                End If
            End If
        Next
 
        lwbk.Close
        wbk.Save
    
        n = n + 1
        fname = Dir
    Loop
    
    'Putting the edited array back to the range
    rng.Resize(, 3).Value = wArr

    Application.ScreenUpdating = True

    MsgBox n & " File(s) are Lookedup!!!", , "Linga"

End Sub
GWD
  • 3,081
  • 14
  • 30
  • 1
    Thank you GWD i will check and update you asap – Linga Dec 07 '20 at 13:52
  • bit confusing as we have the code in the answer is scattered can i have the code collated? – Linga Dec 08 '20 at 13:36
  • @Linga I added the full optimized sub at the bottom. The progress indicator is not implemented though. – GWD Dec 08 '20 at 15:01
  • Thank you, for the above code im not getting any output!! Will look into this and try modifying lets see – Linga Dec 09 '20 at 09:10
0

I wouldn't necessarily avoid Application.Match. As shown here, it may be the most efficient option. It also shows that dictionaries may not be the most efficient way (but it depends on the amount of manipulations done with them). However, in this case, since more manipulations is needed apart from matching, arrays is the way to go.

This is the rewritten code using arrays instead of ranges. Hopefully it works. :)

Private Sub dellkp() 

Application.ScreenUpdating = False

Dim sht As Worksheet, lsht As Worksheet
Dim wbk As Workbook, lwbk As Workbook
'Dim rng As Range, rng1 As Range, cell As Range
Dim h1 As Integer, h2 As Integer, h3 As Integer, p As Integer, s As Integer, b As Integer, v As Integer, n As Integer
Dim lr As Long, lr1 As Long, fn As Long, i As Long, j As Long
Dim f As String, fname As String, a As String
Dim arr() As Variant, arrP() As Variant, arrP1() As Variant, arrS1() As Variant, arrB1() As Variant, arrV1() As Variant

f = Sheet1.Range("B1").Value & "\"
Set wbk = ActiveWorkbook
Set sht = Sheet2
h1 = 106
h2 = 107
h3 = 108

'product id
p = 9
'sku variant
s = 93
'base id
b = 71
'variant id
v = 87

sht.Cells(1, h1) = "Size"
sht.Cells(1, h2) = "Base"
sht.Cells(1, h3) = "Variation"
lr = sht.Cells(sht.Rows.Count, "A").End(xlUp).Row
fname = Dir(f & "*.xlsx")
'sht.Activate
'Set rng = sht.Range(Cells(2, h1), Cells(lr, h1))
arrP = sht.Range(Cells(2, p), Cells(lr, p)).Value
arr = sht.Range(Cells(2, h1), Cells(lr, h3)).Value

n = 0
Do While fname <> ""
  a = Replace(fname, ".xlsx", "")
  Set lwbk = Workbooks.Open(f & fname)
  Set lsht = lwbk.ActiveSheet
  lr1 = lsht.Cells(lsht.Rows.Count, "A").End(xlUp).Row
'  Set rng1 = lsht.Range(Cells(1, p), Cells(lr1, p))
'  For Each cell In rng
'    If IsEmpty(cell) Then
'      If Not IsError(Application.Match(sht.Cells(cell.Row, p), rng1, 0)) Then
'        fn = Application.Match(sht.Cells(cell.Row, p), rng1, 0)
'        cell = lsht.Cells(fn, s)
'        cell.Offset(0, 1) = lsht.Cells(fn, b)
'        cell.Offset(0, 2) = lsht.Cells(fn, v)
'      End If
'    End If
'  Next
  arrP1 = lsht.Range(Cells(1, p), Cells(lr1, p)).Value
  arrS1 = lsht.Range(Cells(1, s), Cells(lr1, s)).Value
  arrB1 = lsht.Range(Cells(1, b), Cells(lr1, b)).Value
  arrV1 = lsht.Range(Cells(1, v), Cells(lr1, v)).Value
  For i = 1 To UBound(arr, 1)
    For j = 1 To UBound(arrP1, 1)
      If arrP(i, 1) = arrP1(j, 1) Then
        arr(i, 1) = arrS1(j, 1)
        arr(i, 2) = arrB1(j, 1)
        arr(i, 3) = arrV1(j, 1)
      End If
    Next
  Next
  lwbk.Close
  wbk.Save
  n = n + 1
  fname = Dir
Loop
sht.Cells(2, h1).Resize(UBound(arr(), 1), UBound(arr(), 2)).Value = arr

Application.ScreenUpdating = True
MsgBox n & " File(s) are Lookedup!!!", , "Linga"

End Sub

As for progress bar, I would count all the files and use your n to show n/all in the status bar using Application.StatusBar. This may require Application.ScreenUpdating turned on. I'm sure you can easily test it on your own.

ZygD
  • 22,092
  • 39
  • 79
  • 102
  • You are misunderstanding the post you linked. In that post, the time to write the entire dictionary + perform a lookup in the dictionary is compared to the time it takes to perform a single `Match`. I propose writing the dictionary **once** and performing the potentially 1000's of lookups in the dictionary, which will be 100% much, much, **much** faster. +1 for the array conversion! – GWD Dec 07 '20 at 11:16
  • @ZygD Amazing thank you, i will check and update you asap. – Linga Dec 07 '20 at 13:53
  • I think this array method also time consuming once again i will check and get back – Linga Dec 08 '20 at 13:37
  • This is faster than your original. You mostly consume time opening and closing the files, not on calculations. I know, because I have done something like this. FYI, there are faster ways, without actually opening the file, for cases where you only read data. I have only done it once and had to retweak that approach quite much. If you feel like you desperately need more speed, then try searching on how to get data from excel without opening file. Do it only if you feel competent in tweaking such approach, have time for it and if you are in true need of faster application. – ZygD Dec 09 '20 at 13:04
  • Yes i need speed, hope without opening the files if we are getting the data and processing accuracy wont get affected? – Linga Dec 10 '20 at 09:30
  • Have you tried this array method? What are the results compared to dictionary method in your case? – ZygD Dec 10 '20 at 09:41
  • same result and array is take more time. For dictionary i need to make few changes in the code. Will do it as im stuck with other works.. – Linga Dec 10 '20 at 16:40