0

I struggle with large file upload to Autodesk Forge.

This is what im doing: I properly authorise access using 2legged auth, create bucket, and i'm able to upload revit file and in return i receive URN (object ID). This is all spot on.

The upload works fine for files with size say up to ~100MB. I also implemented resume upload regards to instructions taken from https://github.com/Autodesk-Forge/forge-php-client

This resume upload also works great for files relatively small - say 50-100MB. I tested with chunks 2 and 5MB.

but i have a real problem with bigger files. Say 200MB, 500MB, or larger. I noticed that if i upload 200MB file using non-resume option then it goes properly and i receive json with data. With larger files - its random. Once they pass through, once error. The largest file i managed to upload usin non-resume method was ~500MB.

However, if i use resume upload for the same 200MB file, then i'm receiving 502 error. I also noticed that this happen randomly. I managed to upload 200 MB files twice. but in most cases (i wld say 9 of 10 cases) it gives nasty 502 error. And im not sure what to do with this.

I'm sending my files from server in google cloud. we have also load balancer.

Can you help what can cause this 502 error, or give some info if there are any limitation etc? I'm sure my code is ok because it works for smaller files <= 100MB. I also tested curl commands directyly from CLI rather than from my source code but no any difference. So i assume its someting related to server overload, load balancer or such limitations at forge end.

alex

alexz
  • 1
  • 1
  • Hi @alexz, I think it would not require max timeout if with a much small chunk (2M is recommended). Could you make a record on the session ID of the resumable uploading? I can ask our engineer team to check at backend if there is more information. I heard about such issue from another customer who uses C#. While it is not always reproducible. I also asked him to make the note with session ID when he hits next time. – Xiaodong Liang May 08 '17 at 06:22

2 Answers2

0

You may need to increase the configured allowed execution time in your backend. For example, in PHP, you may need to increase the max_execution_time value in your php.ini as discussed in this post. This could account for the randomness in your failure...one moment your upload speed might be slightly faster than in another moment.

Community
  • 1
  • 1
Luke Hollenback
  • 769
  • 6
  • 15
  • thx for response, but it doesnt explain why it also happens when i run curl command directly in bash - not in php. – alexz May 08 '17 at 06:02
  • @alexz if the backend is written in PHP, it doesn't matter how you are getting the data to it. A PHP program is still receiving it, and is thus subject to the limitations specified in the `php.ini` file. – Luke Hollenback May 08 '17 at 15:51
  • but what i mean is the situation where i send command directylu from bash curl -v ... not from PHP. – alexz May 08 '17 at 18:15
0

thx for your feedback - i appreciate it.

@Treebasher, @Xiaodong - guys, some more info (maybe it will give some more light...)

I spent lot of time today to investigate this issue and final verdict is that its kind of limitation on google cloud server. There is such timeout 60s that breaks connection. I tested various scenarios,upgraded curl, etc.

in every scenario: upload file using bash/curl upload from php if file is relatively small - say 50megs - it goes ok. if its larger >= 200mb and using CHUNK method - after 60seconds i got timeout. ( i checeked with timer :)

I also tested my code from local pc and on different server (digital ocean) and it works fine, even 900 mega files are properly send (it was sending for around 7 minutes) but finally i got proper json code. So - for sure -its not wrong php code, not any default settings in php.ini - as i have 2 standard configuratuiosn on 2 other servers, but for 100% something related to specific configuration on google cloud server and related to timeouts.

I also excluded potential explainations: its not an issue at forge's end. its not curl version its not net.ipv4.tcp_fin_timeout

the only "suspected" is google cloud so the next step is ot trace source of problem at google's cloud and find solution.

alexz
  • 1
  • 1