0

I need to runs a thousand of excel files (.xlsx) to get data. Each file has many sheets, and in each sheet, cell A1 contains a function of an Excel add-in (Morningstar Excel add-in). Right now, I have to manually open each file. When the add-in loads, the function in cell A1 gets executed and cell A1 displays "Processing...". I would have to wait a few seconds or minutes for the data to return. Once the sheet is filled with data, I would save it as a csv file.

How do I automate this process?

I have written a macro to open the excel files and save the sheets as CSV files. However, it bypasses the data requesting and downloading process. I added the option to wait a few seconds but the Excel files opens in a frozen status, i.e. the add-in is not loaded and the function in cell A1 does not run. How can I:

  1. Open the file
  2. Make sure the add-in is loaded
  3. Make sure the function in cell A1 of each sheet runs
  4. Check if there is any data. One way is to check, say cell A10, is not empty
  5. Save the sheet as a CSV file

Here is my code so far:

Sub morningstar_VBA()
'PURPOSE: To loop through all Excel files in a user specified folder and perform a set task on them

Dim wb As Workbook
Dim myPath As String
Dim myFile As String
Dim myExtension As String
Dim filename As String
Dim path_to_save As String
Dim FldrPicker As FileDialog
Dim w As Long

'Retrieve Target Folder Path From User
  Set FldrPicker = Application.FileDialog(msoFileDialogFolderPicker)

    With FldrPicker
      .Title = "Select A Target Folder"
      .AllowMultiSelect = False
        If .Show <> -1 Then GoTo NextCode
        myPath = .SelectedItems(1) & "\"
    End With

'In Case of Cancel
NextCode:
  myPath = myPath
  If myPath = "" Then GoTo ResetSettings

'Target File Extension (must include wildcard "*")
  myExtension = "*.xlsx*"

'Target Path with Ending Extention
  myFile = Dir(myPath & myExtension)

'Loop through each Excel file in folder
  Do While myFile <> ""
    'Set variable equal to opened workbook
    Set wb = Workbooks.Open(filename:=myPath & myFile)

    'Ensure Workbook has opened before moving on to next line of code
    For w = 1 To Worksheets.Count
        With Worksheets(w).Copy
            'the ActiveWorkbook is now the new workbook populated with a copy of the current worksheet
            With ActiveWorkbook
                filename = .Worksheets(1).Name
                path_to_save = "E:\Morningstar_download\test\" & filename
                .SaveAs filename:=path_to_save, FileFormat:=xlCSV
                DoEvents
                .Close savechanges:=False
            End With
        End With
    Next w

    wb.Close savechanges:=True

    'Ensure Workbook has closed before moving on to next line of code
    DoEvents

    'Get next file name
    myFile = Dir
  Loop

'Message Box when tasks are completed
  MsgBox "Task Complete!"

ResetSettings:
  'Reset Macro Optimization Settings
    Application.EnableEvents = True
    Application.Calculation = xlCalculationAutomatic
    Application.ScreenUpdating = True

End Sub
Rubén
  • 34,714
  • 9
  • 70
  • 166
duckman
  • 687
  • 1
  • 15
  • 30
  • Whyever would you open the file before making sure the add-in is loaded? You can load the add-in manually. Doing it using code is a cherry on the cake. Why do you let it stop you? Your code can't be better than the logic that underpins it. Your points 1 to 4 are all cherries. They are all useless if your #5 doesn't work. That's the place to start. What's your question about that? – Variatus May 13 '20 at 06:30
  • "I added the option to wait a few seconds" - exactly how did you implement that? – Tim Williams May 13 '20 at 07:07
  • @Variatus How can you open the file before making sure the add-in runs? whenever I open Excel, I see the add-in runs a bit before I can do anything with Excel. I'm not clear about what you mean by my #5 does not work. I think it works. It does not save the xlsx file but it can save the sheets to csv files. My current issue is 2,3, and 4. I am an amateur with VBA. and what do you mean by "What's your question about that?"? – duckman May 13 '20 at 07:08
  • @TimWilliams I added "Application.Wait Now + TimeValue("00:00:10")" after openning the Workbook – duckman May 13 '20 at 07:36
  • The "add-in" (I begin to doubt that you are talking about an add-in) is either loaded or it isn't. If it isn't loaded then what runs? Jump to the end: If the your code can save a file and close it your question should be how to do that in a loop. I don't see such a question. Your question is "how to automate that process?" but your list provides the answer to that. Then what do you need help with if you have both working code and a working plan? – Variatus May 13 '20 at 08:49
  • @Variatus Regarding the add-in, I think a better way to describe that is: that add-in starts whenever Excel is open. The way that add-in works is that there is a function in cell A1. I can change some parameters, press enter and wait for a few seconds or minutes. I think when I press enter, the add-in would start working and returns data. (this add-in helps downloading data btw). I have more than a thousand of value which i need to change so I created excel files with that function in cell A1. The moment I open the file, that function would run. – duckman May 13 '20 at 08:56
  • (continued) There are many sheets in one file. Each contains that function in cell A1. My current code can open the xlsx files and save the sheets to CSV files. However, it does that before the data is returned. So my current problem is at step 2 or 3. I would need to wait for the data is returned before saving the sheets as CSV files. I think the list provides a workable plan. But I dont know how to implement that in the codes – duckman May 13 '20 at 09:01
  • Your "function in A1" is a bit of a mystery. Perhaps it's a UDF, perhaps residing in or calling an add-in. It's not a given that a UDF can do all you want but given that it does, it would seem that the UDF can't wait for the download to finish before allowing your code to save the file. That might be a problem of the UDF, not of your code. Waiting for another process to catch up could be done with `DoEvents` in a loop, like `Do While [whatever condition]: DoEvents: Loop`. I still think your problem is finding the question. The answer may be easy if you do. – Variatus May 13 '20 at 09:09
  • when the function is running, cell A1 displays "Processing". Does VBA have anything that check: if A1 displays Processcing then wait. Another option is to check if another cell has value (this means data has been returned). How do I implement that in your suggested Do loop? – duckman May 13 '20 at 11:30
  • @duckman Maybe try to use `DoEvent` Can you update your issue post with the code we made together on an other thread ? I think its the same ^^ – TourEiffel May 13 '20 at 12:49
  • Also you do not need a thousand Excel files if the only purpose of each file is to contain a different value in A1 - you can loop over a list of values, populate A1, wait for the add-in to complete, then repeat with the next value. Are there other differences between these files? – Tim Williams May 13 '20 at 14:59
  • @TimWilliams there are no differences between the files, except for the values of the parameters in the function in cell A1 that is used to trigger the data downloading – duckman May 14 '20 at 01:46
  • So you only really need one file and just update that in a loop. – Tim Williams May 14 '20 at 03:42
  • yes, that part is done. the hard part is to tell Excel to wait for the data to load and save – duckman May 14 '20 at 06:10
  • Maybe something here which might help https://stackoverflow.com/questions/46908493/how-to-ensure-excel-calculation-is-completed-from-within-a-vba-procedure. Without the add-in to test with it's difficult to say what might work. – Tim Williams May 14 '20 at 06:25
  • This is solved. I added "Set cmd = Application.CommandBars("Cell").Controls("Refresh All") cmd.Execute" and it refreshes the add-in – duckman May 16 '20 at 00:27

1 Answers1

0

include the code below to refresh add-ins, i.e. making sure all functions from add-ins run. Not sure about other add-ins

Set cmd = Application.CommandBars("Cell").Controls("Refresh All")
    cmd.Execute
duckman
  • 687
  • 1
  • 15
  • 30