I need to read and process my Gmail from an Amazon linux server without user intervention. I am following the documentation at https://developers.google.com/identity/protocols/OAuth2ServiceAccount and can successfully login and retrieve my emails from my test server once I have allowed access by filling in the browser form.
I now need to move the working application to the server and authenticate the keys for the first time. Using either client mode or domain mode authentication, the application on the server sends a message to stdout (or is it stderr?) with an URL to visit.
Using command line browsers on the server (I have tried elinks, lynx, w3get and curl) I am unable to authenticate.
I have also tried ssh tunneling with both normal tunnels and -D option
ssh -L 80:remotehost:80 user@myserver
ssh -D 5000 user@myserver
Here is the source code I use to test connectivity. From what I can see, Domain authentication is the preferred and only way to get this to work.
/*
* To change this license header, choose License Headers in Project Properties.
* To change this template file, choose Tools | Templates
* and open the template in the editor.
*/
package tester;
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.extensions.java6.auth.oauth2.AuthorizationCodeInstalledApp;
import com.google.api.client.extensions.jetty.auth.oauth2.LocalServerReceiver;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.util.store.FileDataStoreFactory;
import com.google.api.services.gmail.Gmail;
import com.google.api.services.gmail.GmailScopes;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
import com.google.api.services.gmail.model.ListThreadsResponse;
import java.security.GeneralSecurityException;
import java.util.Collections;
/**
*
* @author chris
*/
public class TestGmail {
/**
* Set to same name as Google project as in Google auth setup
*
*/
private static final String APPLICATION_NAME = "<Company>Java";
private static java.io.File DATA_STORE_DIR;
private static FileDataStoreFactory DATA_STORE_FACTORY;
private static final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
private static HttpTransport HTTP_TRANSPORT;
private static final List<String> SCOPES
= Arrays.asList(GmailScopes.GMAIL_SEND, GmailScopes.GMAIL_LABELS, GmailScopes.GMAIL_COMPOSE,
GmailScopes.GMAIL_READONLY);
/**
* THis is the email name generated by Google for my server account
*
*
*/
static final String serverFakeEmail = "<Company>@<Company>java.iam.gserviceaccount.com";
/**
* This is the everyday Gmail account that is used from the web browser. It
* is the mail address that receives incoming emails
*
*/
static final String realEmailName = "<Company>@gmail.com";
/**
* The following two methods from the following URL
* https://developers.google.com/identity/protocols/OAuth2ServiceAccount
* Can't use Domain authentication since I know how to create .p12
* https://console.developers.google.com/apis/credentials?project=<Company>java
* create credentials <big blue button>
* select type .p12
* I also probably don't need this since I have a single user scenario
*
* @return
* @throws IOException
* @throws GeneralSecurityException
*/
static public Credential authenticateDomain() throws IOException, GeneralSecurityException {
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(serverFakeEmail)
.setServiceAccountPrivateKeyFromP12File(new File(System.getProperty("user.home"),".credentials/<Company>Java-xxxxx.p12")) // password =notasecret
.setServiceAccountScopes(SCOPES)
.setServiceAccountUser(realEmailName)
.build();
return credential;
}
/**
* This is probably the authentication I need to use.
* I do not have access to a http browser frommy server which must read incoming emails to my account
*
* It is not possible to setServiceAccountUser(realEamilName) as might be implied by the documentation
* (no such method)
* Nor is there any build() or exec() method
* @return
* @throws IOException
*/
static public Credential authenticateAsService() throws IOException {
File f = new File(System.getProperty("user.home"), ".credentials/serviceIamKey.json");
//File f = new File(System.getProperty("user.home"), ".credentials/domainwide_client_secret_105798575021034224165.json");
GoogleCredential credential = GoogleCredential.fromStream(new FileInputStream(f)).createScoped(SCOPES);
return credential;
}
/**
* Creates an authenticated Credential object.
*
* This works, however it needs a browser in order to do the first-time authentication
* I do not have a browser installed, and company policy wont allow it
* I therefore cannot get this to work in my server environment
* I need a way to read emails on a GMail account and process them unattended
*
* @return an authenticated Credential object.
* @throws IOException
*/
static public Credential authenticateAsClient() throws IOException {
File f = new File(System.getProperty("user.home"), ".credentials/client_secret.json");
InputStream in = new FileInputStream(f);
GoogleClientSecrets clientSecrets
= GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));
// Build flow and trigger user authorization request.
GoogleAuthorizationCodeFlow flow
= new GoogleAuthorizationCodeFlow.Builder(
HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, SCOPES)
.setDataStoreFactory(DATA_STORE_FACTORY)
.setAccessType("offline")
.build();
Credential credential = new AuthorizationCodeInstalledApp(
flow, new LocalServerReceiver()).authorize("user");
System.out.println(
"Credentials saved to " + DATA_STORE_DIR.getAbsolutePath());
return credential;
}
/**
* Build and return an authenticated Gmail client service.
*
* @return an authenticated Gmail client service
* @throws IOException
*/
static public Gmail getGmailService() throws IOException {
//Credential credential = authenticateAsService();
Credential credential = authenticateAsClient();
Gmail service = new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
.setApplicationName(APPLICATION_NAME)
.build();
return service;
}
public void doit(String testName, Credential credential, String username) throws Exception {
System.out.println(String.format("Test: %s User:%s Credential is:%s",
testName, username, ((credential == null) ? "Null" : "Ok")));
Gmail service = new Gmail.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
.setApplicationName(APPLICATION_NAME)
.build();
System.out.println(String.format("Test: %s GmailService is:%s",
testName, ((service == null) ? "Null" : "Ok")));
Gmail.Users.Threads threads = service.users().threads();
System.out.println(String.format("Test: %s Threads is:%s",
testName, ((threads == null) ? "Null" : "Ok")));
Gmail.Users.Threads.List tList = threads.list(username);
System.out.println(String.format("Test: %s Thread List Size =:%d",
testName, ((tList == null) ? -1 : tList.size())));
ListThreadsResponse ltr = tList.execute();
System.out.println(String.format("Test: %s List Threads Response is:%s",
testName, ((ltr == null) ? "Null" : "Ok")));
List<com.google.api.services.gmail.model.Thread> lt = ltr.getThreads();
System.out.println(String.format("Test: %s Thread count =:%d",
testName, ((lt == null) ? -1 : lt.size())));
}
public void testAsClientEmailname() {
try {
System.out.println("******************************************** Test Client authorisation using real Emailname " + realEmailName + " expected to work");
HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
DATA_STORE_DIR = new java.io.File("testAsClientEmailname");
DATA_STORE_FACTORY = new FileDataStoreFactory(DATA_STORE_DIR);
Credential credential = authenticateAsClient();
doit("runAsClientEmailname", credential, realEmailName);
} catch (Exception exc) {
System.out.println(exc.getMessage());
// exc.printStackTrace();
}
}
/**
* This one fials because google only allow domain wide usage
*/
public void testAsServiceUsername() {
System.out.println("******************************************** Test Server authentication using server iam key- " + serverFakeEmail + " expected to work");
try {
HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
DATA_STORE_DIR = new java.io.File("testAsServiceUsername");
DATA_STORE_FACTORY = new FileDataStoreFactory(DATA_STORE_DIR);
Credential c = authenticateAsService();
doit("runAsService:" + serverFakeEmail, c, serverFakeEmail);
} catch (Exception exc) {
System.out.println(exc.getMessage());
// exc.printStackTrace();
}
}
public void testAsServiceDomainEmailname() {
System.out.println("******************************************** Test SDomain authentication for real email " + realEmailName + " expected to fail");
try {
HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
DATA_STORE_DIR = new java.io.File("testAsServerEmailname");
DATA_STORE_FACTORY = new FileDataStoreFactory(DATA_STORE_DIR);
Credential c = authenticateDomain();
doit("RunAsService:" + realEmailName, c, realEmailName);
} catch (Exception exc) {
System.out.println(exc.getMessage());
// exc.printStackTrace();
}
}
}
Here is the output when I run it on my test machine. From this I infer that my processes are correct, but I am still required to fill in an annoying browser form. I need the supplied key to be sufficient to authenticate my application.
Running tester.TestGmail
******************************************** Test Client authorisation using real Emailname <Company>@gmail.com expected to work
Credentials saved to /home/chris/NetBeansProjects/<Company>Loader/testAsClientEmailname
Test: runAsClientEmailname User:<Company>@gmail.com Credential is:Ok
Test: runAsClientEmailname GmailService is:Ok
Test: runAsClientEmailname Threads is:Ok
Test: runAsClientEmailname Thread List Size =:1
Test: runAsClientEmailname List Threads Response is:Ok
Test: runAsClientEmailname Thread count =:100
******************************************** Test Server authentication using server iam key- <Company>@<Company>java.iam.gserviceaccount.com expected to work
Test: runAsService:<Company>@<Company>java.iam.gserviceaccount.com User:<Company>@<Company>java.iam.gserviceaccount.com Credential is:Ok
Test: runAsService:<Company>@<Company>java.iam.gserviceaccount.com GmailService is:Ok
Test: runAsService:<Company>@<Company>java.iam.gserviceaccount.com Threads is:Ok
Test: runAsService:<Company>a@<Company>java.iam.gserviceaccount.com Thread List Size =:1
400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Bad Request",
"reason" : "failedPrecondition"
} ],
"message" : "Bad Request"
}
******************************************** Test SDomain authentication for real email <Company>@gmail.com expected to fail
Test: RunAsService:<Company>@gmail.com User:<Company>@gmail.com Credential is:Ok
Test: RunAsService:<Company>@gmail.com GmailService is:Ok
Test: RunAsService:<Company>@gmail.com Threads is:Ok
Test: RunAsService:<Company>@gmail.com Thread List Size =:1
Test: RunAsService:<Company>@gmail.com List Threads Response is:Ok
Test: RunAsService:<Company>@gmail.com Thread count =:100
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.938 sec
Here is the output on my server using domain authentication.
[ec2-user@ip-nnn_nnn_nnn_nnn]$ ~/jre1.8.0_111/bin/java -jar MailProgram-1.0-SNAPSHOT.jar
Runtime failed:ip-nnn-nnnn-nnn-: ip-nnn-nnn-nnn-nn: Name or service not known
Hostname:ip-nn-nn-nn-n
MySQL Connection Succeeded
2016-12-19 11:50:14.928:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
2016-12-19 11:50:14.929:INFO::jetty-6.1.x
2016-12-19 11:50:14.952:INFO::Started SocketConnector@localhost:35280
Please open the following address in your browser:
https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=nnnnnnnnn-xxxxxxxxxxxxxx.apps.googleusercontent.com&redirect_uri=http://localhost:35280/Callback&response_type=code&scope=https://www.googleapis.com/auth/gmail.send%20https://www.googleapis.com/auth/gmail.labels%20https://www.googleapis.com/auth/gmail.compose%20https://www.googleapis.com/auth/gmail.readonly
As far as I am aware, I should not be getting this message, but should instead be immediately authenticated and able to process emails.
The command line browsers tried may have special keys or functions I am unaware of that display the popup and allow me to respond appropriately - that too would be an answer.
[StackOverflow Gmail REST API : 400 Bad Request + Failed Precondition] requires G-suite and is thus not a solution2
Any ideas on what to try next?
