3

I am trying to figure out what is the best way to handle the upload of a lot of data ( up to 1000s of files/images, around 500kb each) with retrofit2 and kotlin.

I have tried to use coroutines withContext() method, but when using enqueue, it just iterates through my entire list of files enqueuing them all at once, which sometimes leads to problems that some of the files are not uploaded correctly.

If i use retrofits execute(), the file upload is handled synchronous, which is also a pain and very slow to execute, but definetly far more reliable, since in testing it managed to upload all files every time.

How could I manage to get the best of both worlds, to run this asynchronous with multiple files uploading at once, but not as slow as the synchronous method?

My current implementation was something like this, please also mind that I am still a beginner in Kotlin and do not leverage all of the functional programming advantages for working with lists yet:

for(item in list) { ...
    launch {
        withContext(Dispatchers.Main) {
            restAPIService.enqueue(object : RetryCallback<JsonObject>() {
                override fun onResponse(
                    call: Call<JsonObject>,
                    response: Response<JsonObject>) {
                        if (response.isSuccessful) {
                            // handle response etc.

------------- EDIT -------------

After the response and hint how to use the .flatMap(), I have now implemented something like this and it seems to be working:

            myList
            .asFlow()
            .flatMapMerge(5) { frame ->
                flow<Pair<Frame, Response<JsonObject>>> {
              //init parameters to pass to the retrofit 2 call 
                          val restAPIService = RestAPIService.RetrofitApi.retrofitService.uploadImages(
                                "Bearer $accessToken",
                                projectPart,
                                filePart,
                                folderNamePart,
                                metaDataPart
                            )

                     retrofitSuspendCall(restAPIService, frame, recording)
                    }
                }
            }





private suspend fun retrofitSuspendCall(
    request: Call<JsonObject>, frame: Frame, recording: RecordingWithFrames): Response<JsonObject> = suspendCoroutine { continuation ->
    request.enqueue(object : Callback<JsonObject> {
        override fun onResponse(call: Call<JsonObject>, response: Response<JsonObject>) {
            if (response.isSuccessful) {
               continuation.resume(response)
               }
            //also handling onfailure
    })
Igor Bozin
  • 116
  • 6
  • Using `Flow.flatMapMerge` you can achieve bounded concurrency. See [this answer](https://stackoverflow.com/questions/58658630/parallel-request-with-retrofit-coroutines-and-suspend-functions). – Marko Topolnik Jan 27 '20 at 11:01
  • However, I see many other problems with your code. You call non-suspendable functions with explicit callbacks. If you remove all coroutines from that code, it will still behave the same way. If you want to make use of coroutines, then write a separate suspend fun that does the enqueue and resumes the coroutine from `onResponse`. – Marko Topolnik Jan 27 '20 at 11:43
  • @MarkoTopolnik Hey Marko, thank you very much for your response - with the approach from the link you have sent me I was also able to remove the coroutines from the code, I will create an edit to the OP and see if it is better now the way i implemented it. – Igor Bozin Jan 27 '20 at 11:54
  • [This doc](https://github.com/Kotlin/KEEP/blob/master/proposals/coroutines.md#wrapping-callbacks) shows how to write a suspend fun starting from an async, callback-based API. – Marko Topolnik Jan 27 '20 at 11:57
  • Is the way I did it in the edit the correct way to approach the issue that i had? – Igor Bozin Jan 27 '20 at 12:03
  • It probably isn't because you use non-suspending, async functions for upload so the Flow has no idea when an upload is done. You should first implement a suspendable function that uploads one file and returns when the upload is done. – Marko Topolnik Jan 27 '20 at 12:06
  • I think I understand, I have edited the snippet again - is this the way to go? – Igor Bozin Jan 27 '20 at 21:30
  • The suspend fun looks good, but i don't understand the `uploadImages` call above it. It doesn't upload images? If it's just init then maybe it doesn't have to run before every upload request? – Marko Topolnik Jan 28 '20 at 09:14
  • The uploadImages is a function which is creating a POST request for each image I want to upload and returns a call which i can enqueue, I dont put all images in one request since it is a lot of data, rather I create a separate request for each image I am uploading, but maybe that is also something I could change. – Igor Bozin Jan 28 '20 at 09:28
  • I was expecting to see the return value of the first call going into the second (suspendable) call. So it sets up the request in the `restAPIService` variable somehow? – Marko Topolnik Jan 28 '20 at 09:32
  • Oh yes, of course - I just forgot to update that part when I was adding the dummy code for the suspend function, the return value of that function is passed to the suspend function as a call ( the first parameter passed to the function). Thank you very much, hvala ti ! :D – Igor Bozin Jan 28 '20 at 12:42
  • I am still terrible at formatting the code here on stackoverflow – Igor Bozin Jan 28 '20 at 12:44

0 Answers0