Ruby : File upload to Google drive using google-api-ruby-client SDK failing with Timeout::Error

Question

I am using the google-api-ruby-client in my Rails application to interact with Google Drive. All the basic functions like listing files in a folder, fetching, moving, copying and deleting a file, create new folder, etc are working fine.

However uploading a new file is always failing with Timeout::Error

I receive the file to be uploaded as a regular file upload from my website (multipart/form-data) and this is how I upload it to Google drive

result = nil
new_file_obj = google_drive.files.insert.request_schema.new({
  'title' => file_name,
  'mimeType' => file_mime_type,
  'parents' => [{'id' => current_folder_id}]
})

file_content = Google::APIClient::UploadIO.new(new_file.tempfile, file_mime_type, file_name)

result = google_client.execute(
    api_method: google_drive.files.insert,
    body_object: new_file_obj,
    media: file_content,
    parameters: {
      'uploadType' => 'multipart',
      'alt' => 'json'
    }
  )

Here new_file is the file that was uploaded by the client. new_file.tempfile gives an object of type Tempfile.

The execute method never returns and ultimately I get a Timeout::Error exception. This is the relevant stack trace :

/lib/ruby/1.9.1/net/protocol.rb:140:in `rescue in rbuf_fill'
/lib/ruby/1.9.1/net/protocol.rb:134:in `rbuf_fill'
/lib/ruby/1.9.1/net/protocol.rb:116:in `readuntil'
/lib/ruby/1.9.1/net/protocol.rb:126:in `readline'
/lib/ruby/1.9.1/net/http.rb:2211:in `read_status_line'
/lib/ruby/1.9.1/net/http.rb:2200:in `read_new'
/lib/ruby/1.9.1/net/http.rb:1183:in `transport_request'
/lib/ruby/1.9.1/net/http.rb:1169:in `request'
/lib/ruby/1.9.1/net/http.rb:1162:in `block in request'
/lib/ruby/1.9.1/net/http.rb:627:in `start'
/lib/ruby/1.9.1/net/http.rb:1160:in `request'
/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:74:in `perform_request'
/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:37:in `call'
/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/request/url_encoded.rb:14:in `call'
/lib/ruby/gems/1.9.1/gems/google-api-client-0.5.0/lib/google/api_client/request.rb:154:in `send'
/lib/ruby/gems/1.9.1/gems/google-api-client-0.5.0/lib/google/api_client.rb:546:in `execute'

I wrote this code following the example here : https://developers.google.com/drive/examples/ruby#saving_new_files What am I missing here?

The file I am trying to upload is a small png image. The file is coming properly to my web application, as I can view the file if I write it to disk. The Google servers are definitely not down, as I can upload a file from drive.google.com. This also means my network is good enough.

So what exactly is causing the timeout?

A solution to this same exception is mentioned as increasing the read_timeout (http://stackoverflow.com/questions/10011387/rescue-in-rbuf-fill-timeouterror-timeouterror). Is that what I should do? If so how do I do it here using the google-api-ruby-client sdk?

Answer 1

From some quick experimentation, it might just be that the tempfile wasn't rewound for reading. I modified the quickstart example to read from a tempfile and was able to reproduce the error. For example:

tmp = Tempfile.new('foo')
tmp.write("hello world")

media = Google::APIClient::UploadIO.new(tmp, 'text/plain', 'foo.txt')
file = drive.files.insert.request_schema.new({
  'title' => 'My document',
  'description' => 'A test document',
  'mimeType' => 'text/plain'
})
result = client.execute(
  :api_method => drive.files.insert,
  :body_object => file,
  :media => media,
  :parameters => {
    'uploadType' => 'multipart',
    'alt' => 'json'})

This will produce the same stack trace after hanging for ~30 seconds or so. However, adding a call to rewind to reset the tempfile makes it work just fine

tmp = Tempfile.new('foo')
tmp.write("hello world")
tmp.rewind # Reset for reading...

Give that a try and see if it fixes your problem.

Answer 2

Looking at https://github.com/google/google-api-ruby-client/blob/master/lib/google/api_client.rb#L532-560

You can pass a Faraday connection to the execute method. Something like

conn = Faraday.default_connection
conn.options[:timeout] = 500 

result = google_client.execute(
api_method: google_drive.files.insert,
body_object: new_file_obj,
media: file_content,
parameters: {
  'uploadType' => 'multipart',
  'alt' => 'json'
 },
connection: conn
)

This will increase the Net:HTTP timeout.

Answer 3

Apparently Google's Documentation on uploading file to the folder using Ruby is yet to documented. It is meant to be implemented as following.

class DriveUpload
    def self.process
            drive_service = Google::Apis::DriveV3::DriveService.new

            drive_service.authorization = Google::Auth::ServiceAccountCredentials.make_creds(
            json_key_io: File.open('client_secret.json'),
            scope: Google::Apis::DriveV3::AUTH_DRIVE)
            meta_data = Google::Apis::DriveV3::File.new(name:'README_ONDRIVE.md',parents:["1gUxa7_9kbaBX_iofwUYKU_3BFXbBu6Ip"])

            file = drive_service.create_file(meta_data, upload_source: 'README.md',content_type: 'text/plain')

    end
end

Please note that parent mean folder inside the Drive(folder that you shared with service account)

Answer 4

This answer is wrong. drive.install permission is NOT needed to upload files.

Today I tried the upload without drive.install as suggested @Steve Bazyl and it just worked (even without the tmp_file.rewind call).

Looks like the problem was ephemeral as nothing else in the environment has changed.

Turns out the problem was that of insufficient privileges. Previously these were the scopes for which I was asking user's permission :

https://www.googleapis.com/auth/userinfo.email 
https://www.googleapis.com/auth/userinfo.profile 
https://www.googleapis.com/auth/drive

Where as Google drive requires the application to be installed for it to upload files on behalf of the user. And for the app to be installed permission to the following scope should also be requested : https://www.googleapis.com/auth/drive.install

Sources :

1. How do I add/create/insert files to Google Drive through the API? (You do not have to list your app in chrome store as mentioned there. Just asking for the drive.install permission is sufficient. Be sure to re-authenticate/re-authorize all the users to get new tokens)

2. https://developers.google.com/drive/scopes#google_drive_scopes

However the Timeout::Error is still somewhat of an anomaly. Probably the Google servers did not respond with an error message either when they saw insufficient privileges in the token provided.

P.S : If anyone explains the Timeout::Error, it will be accepted as the right answer.