This is completely unsupported through any documentation but I think I found something interesting. I'm curious for other opinions on this.
Hypothesis
So, each time I went through all my file properties, there was one property that seemingly changed when a file was password protected, this was property 42 (being the "Program name"), part of the extended file properties. See screenshot below (by @Tom), where the left is an unprotected file and the right is protected.

Everytime I unprotected a workbook, a value showed up, e.g "Microsoft Excel" or even sometimes "Microsoft Excel Online". However, on all cases I protected the workbook, the value was empty. Hence, that left me thinking that looking at this specific property is telling us in some way that the file is protected when the property is empty. Might this because the property can't be read because of the protection?
With the help of @Tom we found that this property's index can differ. While on my system this property has got index 42, it appeared that at Tom's system it would sit under 8. Therefor he kindly implemented a smart loop to return the right index before looping the files. Noteworthy: The property's name is language dependent! For Dutch, I would look for "Programmanaam" for example.
Code
Using the following code we can go through a specific folder and loop files to return the value of this specific property:
Sub MySub()
Dim sFile As Variant
Dim oShell: Set oShell = CreateObject("Shell.Application")
Dim oDir: Set oDir = oShell.Namespace("C:\Users\...\")
Dim i as long, x as long
For i = 0 To 288
If oDir.GetDetailsOf(oDir.Items, i) = "Program name" Then
x = i
Exit For
End If
Next i
For Each sFile In oDir.Items
If oDir.GetDetailsOf(sFile, x) = "" Then
Debug.Print sFile.Name & " is protected"
Else
Debug.Print sFile.Name & " is unprotected and can be openened"
End If
Next
End Sub
To adapt that a bit more to loop a range and check a bunch of workbook names that could look like:

Working code looks like:
Sub MySub()
Dim MainPath As String: MainPath = "C:\Users\...\"
Dim i As Long, x As Long
Dim oDir As Object: Set oDir = CreateObject("Shell.Application").Namespace(CStr(MainPath))
'Get the right index for property "Program Name"
For i = 0 To 288
If oDir.GetDetailsOf(oDir.Items, i) = "Program Name" Then
x = i
Exit For
End If
Next i
'Loop the range of workbooks and check whether or not they are protected
With ThisWorkbook.Sheets("Sheet1") 'Change accordingly
For i = 1 To .Cells(.Rows.Count, 1).End(xlUp).Row
If oDir.GetDetailsOf(oDir.Items.Item(CStr(.Cells(i, 1))), x) = "" Then
Debug.Print .Cells(i, 1) & " is protected"
Else
Debug.Print .Cells(i, 1) & " is unprotected and can be openened"
'Open your workbook here?
End If
Next i
End With
End Sub
Note: Please notice the use of Cstr()
on both the MainPath and the cell's value. It's as far as I know not very clear why, but
without it, the code will return an 'Error 445: Object doesn't support
this action' Update: Check
this
question for some more insight on this specific issue.
Example
For example, I have the following workbooks, with Map2 and Map5 protected:

Immediate window after running the first macro:

Next I only protected map1 and map3 with the following result:

Conclusion
Hypothesis proven? I don't know, but on my end there has not been a single time the hypothesis has been proven wrong. Again, there is no documentation on this. But this might just be your way into knowing very quickly if a workbook is protected or not.
Btw, I borrowed some code form here