0

I've read several Stack Overflow threads, I still can't decide what is the best option for my case. And the most secure one.

Here is the story. My webapp is to help users automatically get an overview of some of their data available in some third-party website. I need to store for each user some third-party credentials. Each night or so, my server will connect to the third-party services on the users' behalf and retrieve the required data.

Most of those third-party sites do not implement any API or OAuth mechanism, so I was thinking to do some web scraping.

I've read in many places that storing the credentials in the DB is not a good idea - especially because my app needs access to the password (so it has to be encrypted in such a way I can easily reuse it).

So, I have two options left:

  1. Whenever I access (via webscraping) the third-party service, I store on the server the cookies issued by that service, for future reuse. I encrypt them and keep them encrypted in a DB, and decrypt them only when I need them. The problem is that the cookie can be denied or expired after a while, and so the automatic process wouldn't work any more.
  2. I store the credentials in the environment variables. I will be on Node.js and Heroku. That's an idea I found in another SO thread. But I'm wondering about the security of this idea. Is it really safe? No one can access them but me? And what about if I reach many users. Like 1000 users, with 10 services. That's 10000 credentials to store in the env variables. That doesn't seem like a good idea.

I found two interesting questions on Stack Overflow but they don't fit 100% with my use case.

Community
  • 1
  • 1
nicolasdaudin
  • 302
  • 1
  • 4
  • 15

4 Answers4

1

Storing a lot of data that changes and grows in environment variables will never be practical, no matter if it's secure or not so this is pretty much out of the question, unless if you have a small fixed number of users.

Not storing credentials in the database is a very good advice, but the cookies are credentials and even if you store them encrypted, your app needs to be able to encrypt it to use it. (This is unlike the situation with verifying passwords of your users when you don't need to ever encrypt them, you only need to see if the provided passwords hash to the same values that you have stored).

This is a hard problem because to make it work you need to have some form of credentials (whether those are passwords or cookies) stored and ready to be used unencrypted (even if they are stored encrypted, you need to store the keys to encrypt it as well).

Also, what you are trying to do can be illegal. Mayke sure that you follow the TOC of every service that you're using or otherwise you may face legal trouble.

rsp
  • 107,747
  • 29
  • 201
  • 177
  • I totally agree with rsp on the fact that you have to check the TOC, because I'm pretty sure it's illegal indeed :') – ShellCode Apr 18 '17 at 14:36
  • Thanks guys. Yes this is a hard problem to make it work. I had been thinking about this for months and months and this last few days decided to finally give it a shot. At least think about the details. And now you even tell me that it could be against the TOC - and yes you might be right. *sigh*... ;-) anyway, thanks!! – nicolasdaudin Apr 18 '17 at 14:39
1

Plan for the attacker gaining admin access to the server. Your site will be very attractive to attackers, kind of a one-stop-shop for user credentials so you will need very good security of the login credentials.

There are more than two options for storing the credentials:

  1. Use an HSM for the storage or individual credential encryption keys.

  2. Keep the credentials on another dedicated server with no Internet access, 2-factor authentication and limit admin personal. Rate limit the access to this server and add rate alarms. Access this server on a per user credential basis over a non-Internet connection. The credentials will only be available to the Internet connected server in memory as used, not at-rest in a file.

zaph
  • 111,848
  • 21
  • 189
  • 228
  • Thanks. I didn't know about HSM. Seen the size of my project (and the purpose: it just helps people gather their information in one place, nothing more), it seems overkill but I understand it would actually be a very safe solution. About the other solution, what do you mean "over a non-Internet connection"... ? – nicolasdaudin Apr 18 '17 at 14:42
  • oh and yes, the HSM would not solve the problem of the server itself accessing the credentials, right? – nicolasdaudin Apr 18 '17 at 14:43
  • 1
    An direct ethernet connection between the two servers. The point is not to have a file with all the credentials that only takes one access to obtain. – zaph Apr 18 '17 at 14:45
  • A little more about security overall. All security just raises the work-factor, that is the difficulty. At one end it can be a vulnerability in a server library such as PHP, etc and at an extreme what Apple does with iMessages, admin ceremonies and hashing the HSM access cards in a blender. Attacks can extend to bribes, blackmail and physical harm. – zaph Apr 18 '17 at 14:54
  • Yes, I'm looking for a good compromise between difficulty and security. "Just" a server library might actually be the solution, because my concern is the best strategy to adopt in my source code. I'm not considering any physical solution (extra server, or HSM). – nicolasdaudin Apr 18 '17 at 14:57
  • 1
    The main consideration must be the users. If an attacker can compromise the security they will gain multiple access credentials per user. You will most likely never know that an attack occurred but the users will be comprised across multiple sites. If you are going to put such a system into production you need to have the design and code reviewed by a cryptographic security domain expert, that is what I have done in the past. – zaph Apr 18 '17 at 15:16
  • hi @zaph, just to know, how much did you spend on that expert? and where do you live? – nicolasdaudin Apr 23 '17 at 10:53
  • @nicolasdaudin email me, my profile has a link to my blog that has my email address. – zaph Apr 29 '17 at 20:41
1

I add another answer because maybe this one will do the trick for you. You said the main goal of your website is to have an overview of third party applications. But what if instead of updating this overview every night, you update it when the user logs in ? It changes everything, because you could use the user's password (of your website) as master password to encrypt (using AES) all the others.

If you do that, the communications between your server and the clients have to be encrypted with SSL pinning, because an attacker could perform a MITM, get the master password and all the others stored in the DB... (Even if in practice it's very hard because you need to hack the client AND the server)

ShellCode
  • 1,072
  • 8
  • 17
  • hi @ShellCode ! thanks! Yes, I've read as well somewhere about using the user's password to encrypt the third-party passwords. This is also an option that I have been considering. If it's only when the user logs in, I could even store the credentials on the user device (local cookies) although it's less secure!? What is a MITM? When you say "between your server and the clients" you mean between my server and the users of my app, right? – nicolasdaudin Apr 18 '17 at 14:51
  • Yes you could store credentials in a cookie but I do not recommend it because your DB will store a lot of password and the risk is too big. If you build a such system, I think the safest way is to ask the password to the user every time. A MITM is an attack where the hacker listen the communications between the server and the client (yes I meant the users) and enables him to intercept the password in plaintext, that's why using SSL in a such system is absolutely needed – ShellCode Apr 18 '17 at 15:00
  • @zaph if you have a CA everything's fine I guess ? I mean, X.509 is here for that ? – ShellCode Apr 18 '17 at 15:04
  • @ShellCode That is not a MITM attack. Also clarify what you mean by SSL, do you mean HTTPS? HTTPS (TLS) does not protect against a Man-In-The-Middle (MITM) attack, pinning the certificated does. – zaph Apr 18 '17 at 15:11
  • @zaph man don't be a pain in the a**, I was just simplifying things in order to make him understand... MITM is just creating a bridge between two parties and relaying data (and alter them if needed), but it's mainly used to get plaintext passwords... – ShellCode Apr 18 '17 at 15:16
  • @ShellCode Yes, getting security correct is a PITA. Suggestion: correct the answer/comments. Just using HTTPS will not protect against MITM attacks. – zaph Apr 18 '17 at 15:22
  • @ShellCode A CA does not mitigate MITM attacks, pinning the certificated does. – zaph Apr 18 '17 at 15:24
  • hi both of you! first thanks for your answers and debate! :-) last question what do you mean by "pinning the certificate". The rest is pretty clear. – nicolasdaudin Apr 18 '17 at 21:19
  • Pinning a public key is just the fact to associate a hostname to it, but unfortunately it's not supported by all the browsers, so I don't know if it's acceptable to you. Anyway, I really think the master password sounds like the best solution to your problem, at least the most secure one (and it's easy to do)... – ShellCode Apr 19 '17 at 08:52
  • Pinning a certificate means verifying that the certificate presented by the hiost is not just valid but is the correct certificate for the host you are connecting to. – zaph Apr 19 '17 at 11:58
0

Storing users credentials in a reversible way looks like a terrible idea anyway. But if you really want to store them, I suggest you to use the environment variables solution. But you can improve it. To limit the amount of data you store and don't have 1000000 variables as you said, you can just store an AES encryption key, store all credentials in a DB encrypted with this key, and you just have to get this key (which is in memory) and decrypt the DB. But there is another problem with this solution. As I said, this is stored in RAM memory, so it's not persistent, imagine your server has to reboot for X or Y reason... You will lose the AES key and also the credentials of your users... Moreover, if the attacker performs a memory dump, he will have access to the AES key...

I think the better idea is to store the cookies (in an encrypted way) and when this one expires, you alert the user (by mail, phone, notifications, ...) and ask him to fill his credentials again. But it's not a perfect solution ! Indeed the cookies are a type of credentials and shouldn't be stored either...

ShellCode
  • 1,072
  • 8
  • 17
  • The obvious question is how to protect the encryption key from an attacker, assume the attacker will gain admin access. – zaph Apr 18 '17 at 14:21
  • While there is no perfect solution there are much better solutions, using an HSM it one. – zaph Apr 18 '17 at 14:25
  • 1
    I don't think HSM is a good solution either, because the server needs to access the keys in an automatic way, and if the server can, so can the attacker. – ShellCode Apr 18 '17 at 14:29
  • 1
    The point is not to have a file with all the credentials that only takes one access to obtain. HSMs are how the credit card payment industry protects credentials and how Apple protects iMessage credentials. – zaph Apr 18 '17 at 14:40
  • yes you are right. but theory is one thing, reality is an other and as I said somewhere else, HSM is way too complicated for my use case. Actually, I want to keep it quiet simple. If it's not possible to have something simple and secure as well, maybe I'll just give up! – nicolasdaudin Apr 18 '17 at 21:21