ETrade API unattended authentication

Question

Background
The ETrade authentication system has me creating a RequestToken, then executing an Authorization URL, which opens an ETrade page. The user logs in to authorize the activity on their account. They receive a pin, which they enter in my app. I call ExchangeRequestTokenForAccessToken with the RequestToken and the Pin. Then we are off and running.

Question
The problem is I'm creating a service that runs continuously in the background. There won't be any user to log in. Conversely, I won't be making any trades. Just crunching numbers, looking for stocks that meet certain criteria. I can't figure how to get this to work unattended.

Thanks, Brad.

Answer 1

Previously, I have used a series of WebRequests and manually added headers to simulate the authorization pages. This worked until about a year ago when ETrade complicated their headers with something that appears to be tracking information. I now use http://watin.org/ to log in, and to strip the Auth Code.

Sloppy code looks like this:

            using WatiN.Core; // IE Automation
...
            // verify current thread in STA.

            Settings.Instance.MakeNewIeInstanceVisible = false;

            var ieStaticInstanceHelper = new IEStaticInstanceHelper();
            Settings.AutoStartDialogWatcher = false;

            using (ieStaticInstanceHelper.IE = new IE())
            {
                string authCode = "";
                ieStaticInstanceHelper.IE.GoTo(GetAuthorizationLink());

                if (ieStaticInstanceHelper.IE.ContainsText("Scheduled System Maintenance"))
                {
                    throw new ApplicationException("eTrade down for maintenance.");
                }

                TextField user = ieStaticInstanceHelper.IE.TextField(Find.ByName("USER"));

                TextField pass = ieStaticInstanceHelper.IE.TextField(Find.ById("txtPassword"));

                TextField pass2 = ieStaticInstanceHelper.IE.TextField(Find.ByName("PASSWORD"));

                Button btn = ieStaticInstanceHelper.IE.Button(Find.ByClass("log-on-btn"));
                Button btnAccept = ieStaticInstanceHelper.IE.Button(Find.ByValue("Accept"));


                TextField authCodeBox = ieStaticInstanceHelper.IE.TextField(Find.First());

                if (user != null && pass != null && btn != null &&
                    user.Exists && pass2.Exists && btn.Exists)
                {
                    user.Value = username;
                    pass2.Value = password;
                    btn.Click();
                }

                btnAccept.WaitUntilExists(30);
                btnAccept.Click();

                authCodeBox.WaitUntilExists(30);
                authCode = authCodeBox.Value;

                SavePin(authCode);
            }

Answer 2

Current version of Brad Melton's code. WatiN has changed and no longer contains the IE.AttachToIE function. So, IEStaticInstanceHelper is now called StaticBrowserInstanceHelper, but that code is hard to find, so I've included it here.

class StaticBrowserInstanceHelper<T> where T : Browser {
    private Browser _browser;
    private int _browserThread;
    private string _browserHwnd;

    public Browser Browser {
        get {
            int currentThreadId = GetCurrentThreadId();
            if (currentThreadId != _browserThread) {
                _browser = Browser.AttachTo<T>(Find.By("hwnd", _browserHwnd));
                _browserThread = currentThreadId;
            }
            return _browser;
        }
        set {
            _browser = value;
            _browserHwnd = _browser.hWnd.ToString();
            _browserThread = GetCurrentThreadId();
        }
    }

    private int GetCurrentThreadId() {
        return Thread.CurrentThread.GetHashCode();
    }
}

ETrade's login pages have changed as well. They have several. All the login pages I checked consistently had a USER field and a PASSWORD field, but the login buttons had various names that look fragile. So if this doesn't work, that's the first thing I'd check. Second, if I go directly to the auth page, it prompts to log in, but then it frequently doesn't take you to the auth page. I got more consistent results by going to the home page to log in, then going to the auth page.

    static public string GetPin(string username, string password, string logonLink, string authLink) {
        // Settings.Instance.MakeNewIeInstanceVisible = false;

        var StaticInstanceHelper = new StaticBrowserInstanceHelper<IE>();
        Settings.AutoStartDialogWatcher = false;

        // This code doesn't always handle it well when IE is already running, but it won't be in my case. You may need to attach to existing, depending on your context. 
        using (StaticInstanceHelper.Browser = new IE(logonLink)) {
            string authCode = "";

            // Browser reference was failing because IE hadn't started up yet.
            // I'm in the background, so I don't care how long it takes.
            // You may want to do a WaitFor to make it snappier.
            Thread.Sleep(5000);
            if (StaticInstanceHelper.Browser.ContainsText("Scheduled System Maintenance")) {
                throw new ApplicationException("eTrade down for maintenance.");
            }

            TextField user = StaticInstanceHelper.Browser.TextField(Find.ByName("USER"));

            TextField pass2 = StaticInstanceHelper.Browser.TextField(Find.ByName("PASSWORD"));

            // Class names of the Logon and Logoff buttons vary by page, so I find by text. Seems likely to be more stable.
            Button btnLogOn = StaticInstanceHelper.Browser.Button(Find.ByText("Log On"));
            Element btnLogOff = StaticInstanceHelper.Browser.Element(Find.ByText("Log Off"));
            Button btnAccept = StaticInstanceHelper.Browser.Button(Find.ByValue("Accept"));

            TextField authCodeBox = StaticInstanceHelper.Browser.TextField(Find.First());

            if (user != null && btnLogOn != null &&
                user.Exists && pass2.Exists && btnLogOn.Exists) {
                user.Value = username;
                pass2.Value = password;
                btnLogOn.Click();
            }

            Thread.Sleep(1000);
            if (StaticInstanceHelper.Browser.ContainsText("Scheduled System Maintenance")) {
                Element btnContinue = StaticInstanceHelper.Browser.Element(Find.ByName("continueButton"));
                if (btnContinue.Exists)
                    btnContinue.Click();
            }
            btnLogOff.WaitUntilExists(30);

            // Here we go, finally.
            StaticInstanceHelper.Browser.GoTo(authLink);
            btnAccept.WaitUntilExists(30);
            btnAccept.Click();

            authCodeBox.WaitUntilExists(30);
            authCode = authCodeBox.Value;
            StaticInstanceHelper.Browser.Close();

            return authCode;
        }
    }

Being able to automate it like this means that I no longer care about how long the token is valid. Thanks BradM!

Answer 3

This was amazingly helpful. I used your code plus what was posted here to automate this (because tokens expire daily): E*Trade API frequently returns HTTP 401 Unauthorized when fetching an access token but not always

I made two edits:

1. Changed the authorize URL to what was posted here: https://seansoper.com/blog/connecting_etrade.html

2. For the log on button, changed it to search by ID: Button btnLogOn = StaticInstanceHelper.Browser.Button(Find.ById("logon_button"));

I ran into issues with Watin and setting up the Apartmentstate. So did this:

    static void Main(string[] args)
    {
        System.Threading.Thread th = new Thread(new ThreadStart(TestAuth));
        th.SetApartmentState(ApartmentState.STA);
        th.Start();
        th.Join();
    }

Then put the code in the TestAuth method.