1

In a personal project of mine, I have created an API caller to retrieve a user's saved tracks from the Spotify API. The Spotify endpoint which I am using has a limit (maximum of 50 tracks per request) as well as an offset (starting index of first track in request), which is why I decided to use a FOR loop to get a series of track pages (each 50 tracks) and append them to a global array. The data is loaded from the main thread, and while the data is being requested, I display a child view controller with a spinner view. Once the data request has completed, I remove the spinner view, and transition to another view controller (passing the data as a property).

I have tried many things, but the array of tracks is always empty following the API request. I have a feeling it has to do with the synchronicity of my request, or maybe its possible that I'm not handling it correctly. Ideally, I would like to wait until the request from my API finishes, then append the result to the array. Do you have any suggestions on how I could solve this? Any help is much appreciated!

func createSpinnerView() {

    let loadViewController = LoadViewController.instantiateFromAppStoryboard(appStoryboard: .OrganizeScreen)
    add(asChildViewController: loadViewController)

    DispatchQueue.main.async { [weak self] in
        if (self?.dropdownButton.dropdownLabel.text == "My saved music") {
            self?.fetchSavedMusic() { tracksArray in
                self?.tracksArray = tracksArray
            }
        }
        ...
        self?.remove(asChildViewController: loadViewController)
        self?.navigateToFilterScreen(tracksArray: self!.tracksArray)
    }
}

private func fetchSavedMusic(completion: @escaping ([Tracks]) -> ()) {
    let limit = 50
    var offset = 0
    var total = 200
    for _ in stride(from: 0, to: total, by: limit) {
        getSavedTracks(limit: limit, offset: offset) { tracks in
            //total = tracks.total
            self.tracksArray.append(tracks)
        }
        print(offset, limit)
        offset = offset + 50
    }
    completion(tracksArray)
}

private func getSavedTracks(limit: Int, offset: Int, completion: @escaping (Tracks) -> ()) {
    APICaller.shared.getUsersSavedTracks(limit: limit, offset: offset) { (result) in
        switch result {
        case .success(let model):
            completion(model)
            print("success")
        case .failure(let error):
            print("Error retrieving saved tracks: \(error.localizedDescription)")
            print(error)
        }
    }
}

private func navigateToFilterScreen(tracksArray: [Tracks]) {
    let vc = FilterViewController.instantiateFromAppStoryboard(appStoryboard: .OrganizeScreen)
    vc.paginatedTracks = tracksArray
    show(vc, sender: self)
}
Chris
  • 13
  • 4
  • In fetchSavedMusic, you need to call the completion when getSavedTracks is complete with all iterations, not at the end of the method before getSavedTracks completes. Also, you should use a DispatchGroup to ensure that getSavedTracks has completed for all iterations. – Rob C Feb 03 '22 at 03:29
  • Just to clarify, I could use a DispatchGroup to signal when each iteration has completed (by calling .enter() before the request, .leave() after the request), and then .notify when all of the iterations have been completed? The order of the requests may be out of order, but this doesn't really matter, since I will be sorting the data later. – Chris Feb 03 '22 at 03:43
  • You are exactly right – Rob C Feb 03 '22 at 03:50
  • You're going to want to make sure to call the completion from the main queue – Rob C Feb 03 '22 at 04:02
  • Ok thanks. I was able to get the array of tracks to return from the completion of the fetchSavedMusic() method, but when I call the fetchSavedMusic() method from createSpinnerView(), the array is empty, since the program stores the array before it is overwritten by the requested data. Is there anything I can do in createSpinnerView() to wait for this request to complete before continuing? – Chris Feb 03 '22 at 04:02
  • Sorry, I don't follow. Can you update your post with your latest code? I have two questions, what is the "..." doing in the createSpinnerView() method? Also, why are you dispatching to main in createSpinnerView? – Rob C Feb 03 '22 at 04:07
  • Anything you want done after your music is fetched you need to put that code inside of the completion block: self?.fetchSavedMusic() { tracksArray in PUT YOUR CODE HERE } – Rob C Feb 03 '22 at 04:12
  • The ... refers to a few other IF statements, which check for input from a dropdown button, and depending on the input, it will execute different API requests. I just used ... to simplify the question. I'm not sure why I dispatched to main in CreateSpinnerView(), but my intent was to run the process in the background while the spinner was displayed. – Chris Feb 03 '22 at 13:58

1 Answers1

0

First you need to call completion when all of your data is loaded. In your case you call completion(tracksArray) before any of the getSavedTracks return.

For this part I suggest you to recursively accumulate tracks by going through all pages. There are multiple better tools to do so but I will give a raw example of it:

class TracksModel {
    
    static func fetchAllPages(completion: @escaping ((_ tracks: [Track]?) -> Void)) {
        var offset: Int = 0
        let limit: Int = 50
        var allTracks: [Track] = []
        
        func appendPage() {
            fetchSavedMusicPage(offset: offset, limit: limit) { tracks in
                guard let tracks = tracks else {
                    completion(allTracks) // Most likely an error should be handled here
                    return
                }
                if tracks.count < limit {
                    // This was the last page because we got less than limit (50) tracks
                    completion(allTracks+tracks)
                } else {
                    // Expecting another page to be loaded
                    offset += limit // Next page
                    allTracks += tracks
                    appendPage() // Recursively call for next page
                }
            }
        }

        appendPage() // Load first page
        
    }
    
    private static func fetchSavedMusicPage(offset: Int, limit: Int, completion: @escaping ((_ tracks: [Track]?) -> Void)) {
        APICaller.shared.getUsersSavedTracks(limit: limit, offset: offset) { result in
            switch result {
            case .success(let model):
                completion(model)
            case .failure(let error):
                print(error)
                completion(nil) // Error also needs to call a completion
            }
        }
    }
    
}

I hope comments will clear some things out. But the point being is that I nested an appendPage function which is called recursively until server stops sending data. In the end either an error occurs or the last page returns fewer tracks than provided limit. Naturally it would be nicer to also forward an error but I did not include it for simplicity.

In any case you can now anywhere TracksModel.fetchAllPages { } and receive all tracks.

When you load and show your data (createSpinnerView) you also need to wait for data to be received before continuing. For instance:

func createSpinnerView() {

    let loadViewController = LoadViewController.instantiateFromAppStoryboard(appStoryboard: .OrganizeScreen)
    add(asChildViewController: loadViewController)

    TracksModel.fetchAllPages { tracks in
        DispatchQueue.main.async {
            self.tracksArray = tracks
            self.remove(asChildViewController: loadViewController)
            self.navigateToFilterScreen(tracksArray: tracks)
        }
    }
    
}

A few components may have been removed but I hope you see the point. The method should be called on main thread already. But you are unsure what thread the API call returned on. So you need to use DispatchQueue.main.async within the completion closure, not outside of it. And also call to navigate within this closure because this is when things are actually complete.

Adding situation for fixed number of requests

For fixed number of requests you can do all your requests in parallel. You already did that in your code. The biggest problem is that you can not guarantee that responses will come back in same order than your requests started. For instance if you perform two request A and B it can easily happen due to networking or any other reason that B will return before A. So you need to be a bit more sneaky. Look at the following code:

private func loadPage(pageIndex: Int, perPage: Int, completion: @escaping ((_ items: [Any]?, _ error: Error?) -> Void)) {
    // TODO: logic here to return a page from server
    completion(nil, nil)
}

func load(maximumNumberOfItems: Int, perPage: Int, completion: @escaping ((_ items: [Any], _ error: Error?) -> Void)) {
    let pageStartIndicesToRetrieve: [Int] = {
        var startIndex = 0
        var toReturn: [Int] = []
        while startIndex < maximumNumberOfItems {
            toReturn.append(startIndex)
            startIndex += perPage
        }
        return toReturn
    }()
    
    guard pageStartIndicesToRetrieve.isEmpty == false else {
        // This happens if maximumNumberOfItems == 0
        completion([], nil)
        return
    }
    
    enum Response {
        case success(items: [Any])
        case failure(error: Error)
    }
    
    // Doing requests in parallel
    // Note that responses may return in any order time-wise (we can not say that first page will come first, maybe the order will be [2, 1, 5, 3...])
    
    var responses: [Response?] = .init(repeating: nil, count: pageStartIndicesToRetrieve.count) { // Start with all nil
        didSet {
            // Yes, Swift can do this :D How amazing!
            guard responses.contains(where: { $0 == nil }) == false else {
                // Still waiting for others to complete
                return
            }
            
            let aggregatedResponse: (items: [Any], errors: [Error]) = responses.reduce((items: [], errors: [])) { partialResult, response in
                switch response {
                case .failure(let error): return (partialResult.items, partialResult.errors + [error])
                case .success(let items): return (partialResult.items + [items], partialResult.errors)
                case .none: return (partialResult.items, partialResult.errors)
                }
            }
            
            let error: Error? = {
                let errors = aggregatedResponse.errors
                if errors.isEmpty {
                    return nil // No error
                } else {
                    // There was an error.
                    return NSError(domain: "Something more meaningful", code: 500, userInfo: ["all_errors": errors]) // Or whatever you wish. Perhaps just "errors.first!"
                }
            }()
            
            completion(aggregatedResponse.items, error)
        }
    }
    
    pageStartIndicesToRetrieve.enumerated().forEach { requestIndex, startIndex in
        loadPage(pageIndex: requestIndex, perPage: perPage) { items, error in
            responses[requestIndex] = {
                if let error = error {
                    return .failure(error: error)
                } else {
                    return .success(items: items ?? [])
                }
            }()
        }
    }
    
}

The first method is not interesting. It just loads a single page. The second method now collects all the data.

First thing that happens is we calculate all possible requests. We need a start index and per-page. So the pageStartIndicesToRetrieve for case of 145 items using 50 per page will return [0, 50, 100]. (I later found out we only need count 3 in this case but that depends on the API, so let's stick with it). We expect 3 requests starting with item indices [0, 50, 100].

Next we create placeholders for our responses using

var responses: [Response?] = .init(repeating: nil, count: pageStartIndicesToRetrieve.count)

for our example of 145 items and using 50 per page this means it creates an array as [nil, nil, nil]. And when all of the values in this array turn to not-nil then all requests have returned and we can process all of the data. This is done by overriding the setter didSet for a local variable. I hope the content of it speaks for itself.

Now all that is left is to execute all requests at once and fill the array. Everything else should just resolve by itself.

The code is not the easiest and again; there are tools that can make things much easier. But for academical purposes I hope this approach explains what needs to be done to accomplish your task correctly.

Matic Oblak
  • 16,318
  • 3
  • 24
  • 43
  • This is an interesting solution, using a recursive function to request tracks until there are none remaining. The only thing which may not work is using the limit as a condition to check for the last page, since each page has 50 tracks per request, and I want to append each page to an array until it has stored all of the tracks. Perhaps I could modify your code to use a total, instead of a limit, since each request contains a total number of tracks. – Chris Feb 03 '22 at 13:52
  • @Chris In what case would this limit not work? It basically says "keep loading pages until a page with less than 50 items is returned". So for instance if there are 145 items total then this will be split into [50, 50, 45] where operation ends with 45 because obviously that was the last page, returning 45 items. – Matic Oblak Feb 03 '22 at 15:36
  • @Chris I also added another section explaining another approach. Mostly the difference is that first approach will execute one request after another (first request must receive response before the second one even starts). Second approach will execute all requests at once which requires to manually order responses. – Matic Oblak Feb 03 '22 at 16:14
  • Sorry, I was in a bit of a rush this morning, so I apologize if I misunderstood your initial response. Algorithmically, I think it works exactly as I had specified in the question. Also, the order of the requests does not matter to me, since I will be sorting the data eventually, so the first solution is better in my case. The only problem I am having is that the request returns a blank array, so synchronicity may be the issue there. – Chris Feb 04 '22 at 01:20
  • @Chris you still have the issue that you call your completion before all requests have returned. You will need some form of locking to wait for all request to complete and then call the completion. Check for instance this response https://stackoverflow.com/a/35906703/526828 using `DispatchGroup`. – Matic Oblak Feb 04 '22 at 07:05
  • Thanks for the help, I got it now. – Chris Feb 04 '22 at 22:48