Async and LINQ Queries: Retrieve all lines in all files in a directory tree

Question

I have some UWP code that I am trying to maintain. Its goal is to coalesce some plaintext files kept in a directory structure into a single XML file.

In the interest of doing this, I previously grab all distinct lines in all files in the dirrectory structure using LINQ:

Imports System.IO

Async Function getAllLines() As IEnumerable(Of String)
    Dim _folderPicker As New Windows.Storage.Pickers.FolderPicker()
    _folderPicker.FileTypeFilter.Add("*") 'We must add this; it is a known bug
    Dim _rootDir As Windows.Storage.StorageFolder = Await _folderPicker.PickSingleFolderAsync()

    Dim allDistinctLines As IEnumerable(Of String) = From _file In Directory.GetFiles(_rootDir.Name, "*.txt", SearchOption.AllDirectories) From line In File.ReadAllLines(_file) Where line.Trim().Length > 0 Select line.Trim() Distinct

    Return allDistinctLines
End Function

Since late October, however, it seems some things have changed regarding file perrmissions (Source). I came back to the code after about a month of it working to find it throwing an exception saying that the folder that I was previously able to write to was not accessable due to permissions. The folder existed and I grabbed the folder from a picker so I thought that I was good. Previously, as long as I had grabbed a File or Folder using a picker, I could still use a System.IO function to enumerate, read from or write to said folder or directory.

Now, it seems as though you must go though the proper UWP faculties, which is fine, but I am unsure how to rewrite my LINQ query as the UWP methods are asynchronous.

Here is what I have tried:

Async Function getAllLines() As IEnumerable(Of String)
    Dim _folderPicker As New Windows.Storage.Pickers.FolderPicker()
    _folderPicker.FileTypeFilter.Add("*") 'We must add this; it is a known bug
    Dim _rootDir As Windows.Storage.StorageFolder = Await _folderPicker.PickSingleFolderAsync()

    Dim queryOptions As New Windows.Storage.Search.QueryOptions()
    queryOptions.FileTypeFilter.Add(".txt")
    queryOptions.FolderDepth = Windows.Storage.Search.FolderDepth.Deep

    Dim allDistinctLines As IEnumerable(Of String) = From _file In Await _rootDir.CreateFileQueryWithOptions(queryOptions).GetFilesAsync() From line In Await Windows.Storage.FileIO.ReadLinesAsync(_file) Where line.Trim().Length > 0 Select line.Trim() Distinct

    Return allDistinctLines
End Function

In the above code, I am at least able to enumerate the folder if I comment out the second await in the LINQ. The issue is that with the second Await in the LINQ query, I cannot compile as

'Await' may only be used in a query expression within the first collection expression of the initial 'From' clause or within the collection expression of a 'Join' clause.

So I read up on Join clauses, which seem to equate disparate data, which isn't really what I am looking for.

I read up on the error in general and it seems like the LINQ queries and Async Functions only have limited functionality. Honestly, I don't know enough about LINQ to really make that claim, but that was my feeling.

I found this answer, which uses Task.WhenAll() to facilitate the LINQ and Async Functions usage, but I can't wrap my head around exactly how it would be put into practice.

How to await a method in a Linq query

So I tried:

Async Function getAllLines() As IEnumerable(Of String)
    Dim _folderPicker As New Windows.Storage.Pickers.FolderPicker()
    _folderPicker.FileTypeFilter.Add("*") 'We must add this; it is a known bug
    Dim _rootDir As Windows.Storage.StorageFolder = Await _folderPicker.PickSingleFolderAsync()

    Dim queryOptions As New Windows.Storage.Search.QueryOptions()
    queryOptions.FileTypeFilter.Add(".txt")
    queryOptions.FolderDepth = Windows.Storage.Search.FolderDepth.Deep

    Dim allFiles As IEnumerable(Of Windows.Storage.StorageFile) = From _file In Await _rootDir.CreateFileQueryWithOptions(queryOptions).GetFilesAsync() Select _file

    Dim allDistinctLines As IEnumerable(Of String) = Task.WhenAll(allFiles.Select(Function(_file) Windows.Storage.FileIO.ReadLines.Async(_file).Where(Function(_line) _line.Trim().Length > 0).Distinct()))

    Return allDistinctLines
End Function

But the return type isn't the same, it ends up being some strange IList(Of String)(), which doesn't work for me strangely. Maybe there is just a critical misunderstanding there.

Regardless, any help understanding Async file operations is appriciated!

Please check this case [reply](https://stackoverflow.com/questions/16624648/how-to-await-a-method-in-a-linq-query). — CoCaIceDew, Nov 09 '18 at 09:35
@CoCalceDew I have seen that post, and it was included in my original question. I need some clarifcation though on how it should be utilized. Any help would be appriciated. — , Nov 12 '18 at 17:21

score 1 · Accepted Answer · answered Nov 12 '18 at 02:30

As Stephen said in this thread, LINQ has very limited support for async/await. I recommend you use For Each in conjunction with Windows.Storage api instead like the follow:

Async Function getAllLines() As Task(Of IEnumerable(Of String))
    Dim _folderPicker As New Windows.Storage.Pickers.FolderPicker()
    _folderPicker.FileTypeFilter.Add("*") 'We must add this; it is a known bug
    Dim _rootDir As Windows.Storage.StorageFolder = Await _folderPicker.PickSingleFolderAsync()

    Dim queryOptions As New Windows.Storage.Search.QueryOptions()
    queryOptions.FileTypeFilter.Add(".txt")
    queryOptions.FolderDepth = Windows.Storage.Search.FolderDepth.Deep

    Dim allDistinctLines As New List(Of String)

    For Each file As StorageFile In Await _rootDir.CreateFileQueryWithOptions(queryOptions).GetFilesAsync()
        For Each line In Await Windows.Storage.FileIO.ReadLinesAsync(file)
            If line.Trim().Length > 0 Then
                allDistinctLines.Add(line.Trim())
            End If
        Next
    Next

    Return allDistinctLines.Distinct()
End Function

Thanks for the answer. Interestingly, this is almost identical to the currently working code that I ended up producing. I really enjoyed being able to produce a very powerful, singular line of code that enumerated the files to the depth that I specified and retrieved the distinct lines at the same time. Sure, I could write an extension, but this seems like a bit of a loss of expressiveness if that isn't too melodramatic. Still, the enumeration in my testing completed much faster, not that the application is all the better for it; the files I need to enumerate produce an xml of total size < 1MB — , Nov 12 '18 at 17:18

Async and LINQ Queries: Retrieve all lines in all files in a directory tree

1 Answers1