What is the best practice for dealing with passwords in git repositories?

Question

I've got a little Bash script that I use to access twitter and pop up a Growl notification in certain situations. What's the best way to handle storing my password with the script?

I would like to commit this script to the git repo and make it available on GitHub, but I'm wondering what the best way to keep my login/password private while doing this is. Currently, the password is stored in the script itself. I can't remove it right before I push because all the old commits will contain the password. Developing without a password isn't an option. I imagine that I should be storing the password in an external config file, but I thought I'd check to see if there was an established way to handle this before I tried and put something together.

Answer 1

The typical way to do this is to read the password info from a configuration file. If your configuration file is called foobar.config, then you would commit a file called foobar.config.example to the repository, containing sample data. To run your program, you would create a local (not tracked) file called foobar.config with your real password data.

To filter out your existing password from previous commits, see the GitHub help page on Removing sensitive data.

Answer 2

An approach can be to set password (or API key) using an environment variable. So this password is out of revision control.

With Bash, you can set environment variable using

export your_env_variable='your_password'

This approach can be use with continuous integration services like Travis, your code (without password) being stored in a GitHub repository can be executed by Travis (with your password being set using environment variable).

With Bash, you can get value of an environment variable using:

echo "$your_env_variable"

With Python, you can get value of an environment variable using:

import os
print(os.environ['your_env_variable'])

PS: be aware that it's probably a bit risky (but it's a quite common practice) https://www.bleepingcomputer.com/news/security/javascript-packages-caught-stealing-environment-variables/

PS2: this dev.to article titled "How to securely store API keys" may be interesting to read.

Answer 3

What Greg said but I'd add that it's a good idea to check in a file foobar.config-TEMPLATE.

It should contain example names, passwords or other config info. Then it is very obvious what the real foobar.config should contain, without having to look in all the code for which values must be present in foobar.config and what format they should have.

Often config values can be non obvious, like database connection strings and similar things.

Answer 4

Dealing with passwords in repositories would be handled different ways depending on what your exact problem is.

1. Don't do it.

And ways to avoid doing are covered in some replies - .gitignore, config.example, etc

• .gitignore: Store on servers etc but not Git (weak solution)
• Environment variables: https://stackoverflow.com/a/30664318/3070485
• Config.example: https://stackoverflow.com/a/2397905/3070485
• Command line parameter: manually enter password on program startup

or 2. Make repository accessible only to authorized people

I.e. people that are allowed to know the password. chmod and user groups comes to mind; also problems like should Github or AWS employees be allowed to see things if you host your repositories or servers externally?

or 3. Encrypt the sensitive data (purpose of this reply)

If you want to store your config files containing sensitive information (like passwords) in a public location then it needs to be encrypted. The files could be decrypted when recovered from the repository, or even used straight from their encrypted form.

An example javascript solution to using encrypted configuration data is shown below.

const fs = require('fs');
const NodeRSA = require('node-rsa');

let privatekey = new NodeRSA();
privatekey.importKey(fs.readFileSync('private.key', 'utf8'));
const config = privatekey.decrypt(fs.readFileSync('config.RSA', 'utf8'), 'json');

console.log('decrypted: ', config);

So you can recover an encrypted config file writing just a few lines of Javascript.

Note that putting a file config.RSA into a git repository would effectively make it a binary file and so it would lose many of the benefits of something like Git, e.g. ability to cherry pick changes to it.

The solution to that might be to encrypt key value pairs or perhaps just values. You could encrypt all values, for example if you have a separate file for sensitive information, or encrypt just the sensitive values if you have all values in one file. (see below)

My example above is a bit useless to anyone wanting to do a test with it, or as an example to start from as it assumes the existence of some RSA keys and an encrypted config file config.RSA.

So here's some extra lines of code added to create RSA keys and a config file to play with.

const fs = require('fs');
const NodeRSA = require('node-rsa');

/////////////////////////////
// Generate some keys for testing
/////////////////////////////

const examplekey = new NodeRSA({b: 2048});

fs.writeFileSync('private.key', examplekey.exportKey('pkcs8-private'));
fs.writeFileSync('public.key', examplekey.exportKey('pkcs8-public'));

/////////////////////////////
// Do this on the Machine creating the config file
/////////////////////////////

const configToStore = {Goodbye: 'Cruel world'};

let publickey = new NodeRSA();
publickey.importKey(fs.readFileSync('public.key', 'utf8'));

fs.writeFileSync('config.RSA', publickey.encrypt(configToStore, 'base64'), 'utf8');

/////////////////////////////
// Do this on the Machine consuming the config file
/////////////////////////////

let privatekey = new NodeRSA();
privatekey.importKey(fs.readFileSync('private.key', 'utf8'));

const config = privatekey.decrypt(fs.readFileSync('config.RSA', 'utf8'), 'json');
console.log('decrypted: ', config);

Encrypting values only

fs.writeFileSync('config.RSA', JSON.stringify(config,null,2), 'utf8');

You can decrypt a config file with encrypted values using something like this.

const savedconfig = JSON.parse(fs.readFileSync('config.RSA', 'utf8'));
let config = {...savedconfig};
Object.keys(savedconfig).forEach(key => {
    config[key] = privatekey.decrypt(savedconfig[key], 'utf8');
});

With each configuration item on a separate line (e.g. Hello and Goodbye above), Git will recognize better what's going on in a file and will store changes to items of information as differences rather than complete files. Git will also be able to manage merges and cherry picks etc better.

However the more you want to version control changes to sensitive information, the more you are moving towards a SAFE REPOSITORY solution (2) and away from an ENCRYPTED INFO (3) solution.

Answer 5

One can use HashiCorp Vault which secures, stores, and controls access to tokens, passwords, certificates, API keys, etc.

Ansible specifically has a "Vault" feature (unrelated to the HashiCorp product) for encrypting secrets at rest, as well.

Answer 6

Here is a technique I use:

I create a folder in my home folder called: .config

In that folder I place the config files for any number of things that I want to externalize passwords and keys.

I typically use reverse domain name syntax such as:

com.example.databaseconfig

Then in the bash script I do this:

#!/bin/bash
source $HOME/.config/com.example.databaseconfig ||exit 1

The || exit 1 causes the script to exit if it is not able to load the config file.

I used that technique for bash, python, and ant scripts.

I am pretty paranoid and don't think that a .gitignore file is sufficiently robust to prevent an inadvertent check-in. Plus, there is nothing monitoring it, so if a check-in did happen no one would find out to deal with it.

If a particular application requires more than one file I create subfolder rather than a single file.

Answer 7

If you're using ruby on rails, the Figaro gem is very good, easy, and reliable. It has a low headache factor with the production environment too.

Answer 8

Trust but verify.

In .gitignore this would exclude a "secure" directory from the repo:

secure/

But I share @Michael Potter's paranoia. So to verify .gitignore, here's a Python unit test that would raise a klaxon if this "secure" directory ever gets checked in. And to check the check, a legitimate directory is tested too:

def test_github_not_getting_credentials(self):
    safety_url = 'https://github.com/BobStein/fliki/tree/master/static'
    danger_url = 'https://github.com/BobStein/fliki/tree/master/secure'

    self.assertEqual(200, urllib.request.urlopen(safety_url).status)

    with self.assertRaises(urllib.error.HTTPError):
        urllib.request.urlopen(danger_url)

Answer 9

Is there any possibility to tell github to track the file under a different name? Example: Locally, I have a file passwords.config with real passwords, and sample-passwords.config with stubs. However, in public repo, I'd like to have only passwords.config with content from sample-passwords.config and real passwords.config ignored. I know .gitignore, which can hide my passwords.config, but I don't know is there any solution to rename sample-passwords.config while commiting to remote public repo. Of course, I'd like to avoid situation, when my local repo tracks renamed file as if something changed in git status.