17

I need to pull bank account statements from a number of different sources. I'd love to be able to get this data in some sort of standardized fashion, be it CSV or OFX or XML or whatever. I just need to be able to get this data. I'll be using my own account to grab transactions from, so there shouldn't be too many 'legal' problems there. How do sites like Mint.com get bank statements? They simply ask for your account information, the URL, and then they're off and running. How can I do this?

Naftuli Kay
  • 87,710
  • 93
  • 269
  • 411

5 Answers5

8

Write a screen scraper in your favorite language to do it. Automate it away.

Normally banks give you access up to three months of transactions via their online services.

Don't forget to make it push the panic button when the screen scraping fails.

The main point here is to automate it so that the pain (of running it manually) goes away so that all you see are "constantly" updated transactions of your accounts.

holygeek
  • 15,653
  • 1
  • 40
  • 50
  • I'm completely new to screen scraping. I'm writing my application in Python, any idea on where to start, module-wise? I found Bankjob (a Ruby scraper), but I really need it in Python. My bank does support exporting transactions to OFX, but there's not specifically a way to automate it other than via scraping. Thanks. – Naftuli Kay Apr 07 '11 at 19:05
  • 2
    Have a look at beautiful soup: http://www.crummy.com/software/BeautifulSoup/. It's pretty decent. – holygeek Apr 08 '11 at 00:24
  • 3
    How do you write a screen scraper that will deal with the fact that some banks have complicated javascript that runs during the login process and the need to run the scraper from the command line without a GUI? – Michael May 25 '12 at 21:05
  • 1
    Nowadays we can run webkit headless with JavaScript engine and all. Check out [phantomjs](http://phantomjs.org/) – holygeek May 26 '12 at 00:48
  • I believe scraping is the way to go (in the absence of direct feeds). I have created a project for parsing HTML tables of financial transactions: https://github.com/bunsn/boiler. To add support for your own bank, you write a basic “statement definition”. It doesn’t handle logging in or crawling multiple statements but may be useful as part of a scraping tool. – Dom Christie Sep 28 '15 at 09:27
  • holygeek, @DomChristie: How do you actually download the *statements* (PDFs) using something like PhantomJS? Doesn't PhantomJS completely fail at *downloading*? – user541686 Feb 14 '17 at 08:53
6

If you're interested in a simple, personal solution, one easy way would have Mint aggregate all your purchases and use an existing scraper to download them. Something like this: https://github.com/mrooney/mintapi.

Slothario
  • 2,830
  • 3
  • 31
  • 47
2

The other main things you may consider is Plaid (more commercial product but looks really good) And https://www.buxfer.com which has a free api. Mint as of late has made it harder for scrapers to scrape it.

JustGage
  • 1,534
  • 17
  • 20
0

Grab your transactions and save them to CSV of XLS to convert to OFX and / QIF using iCreateOFX Basic v4 or try the v5 beta.

Nepaluz
  • 669
  • 1
  • 12
  • 27
-1

I read a comprehensive technical article on this awhile ago, however, I can't find it. From my memory this is what mint does to get data from your bank:

  1. Query API from this company: http://www.finappstore.com/
  2. If bank isn't in above API they fallback to screen scraping
eggie5
  • 1,920
  • 18
  • 25