Getting BadPaddingException while doing a Sqoop import with encrypted password file

Question

I encrypt file with openssl then put it on HDFS, I used AES/ECB, 128 bits and salt option, and with some research I find out openssl uses PKCS5 padding as default which are all defaults in CryptoFileLoader class. Here is my encryption process:

# echo -n "password" > .pw
# openssl enc -aes-128-ecb -salt -in .pw -out .pw.enc
# hdfs dfs -put .pw.enc /user/user1/

Sqoop version is 1.4.6

Command:

sqoop import \
-Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader \
-Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqoop \
--connect jdbc:oracle:thin:@host/database \
--username user1 \
--password-file /user/user1/.pw.enc \
--table db.table1 \
--hive-import \
--hive-overwrite \
--hive-table hivedb.table1 \
--hive-drop-import-delims

which gives:

17/03/08 15:10:37 WARN tool.BaseSqoopTool: Failed to load password file
java.io.IOException: Can't decrypt the password
        at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
        at org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:81)
        at org.apache.sqoop.util.CredentialsUtil.fetchPassword(CredentialsUtil.java:66)
        at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1042)
        at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:997)
        at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:875)
        at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
        at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
        at javax.crypto.Cipher.doFinal(Cipher.java:2165)
        at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:149)
        ... 12 more
Error while loading password file: Can't decrypt the password

I tried manually giving the other CryptoFileLoader parameters too and also passing local file to the --password-file .

I can decrypt the file back successfully with openssl. I can't decrypt with Java program(?)

I saw there is an issue with padding but I didn't know what it is and how to encrypt the file with a certain padding method or whatever else to do, I'm not experienced with encryption.

There is also org.apache.sqoop.credentials.loader.crypto.iterations parameter in the class which indicates number of PBKDF2 iterations but I don't know if it changes anything.

Thanks for any help.

A bad padding exception is caused by the decrypotion failing. There is an error in data, key, modes and.or iv. But since no example test data is provided it is hard to look into the reason. You need to hex dump the various inputs and add them to the question. — zaph, Mar 08 '17 at 14:17
Do not use ECB mode, it is insecure, see [ECB mode](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_Codebook_.28ECB.29), scroll down to the Penguin. Instead use CBC mode with a random IV, just prefix the encrypted data with the IV for use in decryption, it does not need to be secret. — zaph, Mar 08 '17 at 14:18
Some pointers, use full length keys, for AES that means 128, 192 or 256 bits (16, 24 or 32 bytes). Using keys that are to short will cause an error or undetermined padding that is implementation dependent. — zaph, Mar 08 '17 at 14:22
**Do not encrypt passwords**, when the attacker gets the DB he will also get the encryption key. Iterate over an HMAC with a random salt for about a 100ms duration and save the salt with the hash. Use functions such as password_hash, PBKDF2, Bcrypt and similar functions. The point is to make the attacker spend a lot of time finding passwords by brute force. — zaph, Mar 08 '17 at 14:24
@zaph Thanks for the comments. As I see CBC is more secure but I don't need that much securtiy yet since I only want the database password not to show in Sqoop import configuration or as a plain text in a file. I tried using CBC instead but it wants an IV parameter and I couldn't pass it because it isn't defined how to in the CryptoFileLoader. I guess I will look more into that padding exception and the iteration you mentioned in last comment. — burakongun, Mar 09 '17 at 06:06
If the only passwords are yours that is one thing but if the passwords are from other users it is their security you are making the decision on and the only acceptable solution is to use best practices. — zaph, Mar 09 '17 at 13:25

Simone Cifani · Accepted Answer · 2017-04-06T11:47:20.197

I am not expert with Sqoop and Hadoop but starting from your exception

CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)

I gave a look at the source code of CryptoFileLoader.java

It seems to me that things are a bit different from what you do: the password is stored in an encrypted file using the PBKDF2 algorithm, which is not equivalent to apply AES-128-ECB. From wikipedia:

PBKDF2 applies a pseudorandom function, such as hash-based message authentication code (HMAC), to the input password or passphrase along with a salt value and repeats the process many times to produce a derived key, which can then be used as a cryptographic key in subsequent operations. The added computational work makes password cracking much more difficult, and is known as key stretching.

There is no way to do PBKDF2 from Openssl command line. I made a small test using Java, it could be an alternative

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.NoSuchAlgorithmException;

import javax.crypto.Cipher;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;

public class Test {

    /*Default is AES in electronic code book with padding.*/
    private static String DEFAULT_ALG = "AES/ECB/PKCS5Padding";
    /*Default salt is not much secure, use your own!*/
    private static String DEFAULT_SALT = "SALT";
    /*Iterate 10000 times by default.*/
    private static int DEFAULT_ITERATIONS = 10000;
    /*One of valid key sizes for default algorithm (AES).*/
    private static int DEFAULT_KEY_LEN = 128;

    public static void main(String[] args) throws IOException {

        String inputFileName = "C:\\temp\\in.txt";   /*Enter your input (plain) file path */
        String outputFileName = "C:\\temp\\out.bin"; /*Enter your output (encrypted) file path */
        String passPhrase = "mypassphrase";          /*Enter your passphrase */
        String salt = DEFAULT_SALT;
        String alg = DEFAULT_ALG;
        int iterations = DEFAULT_ITERATIONS;
        int keyLen = DEFAULT_KEY_LEN;

        SecretKeyFactory factory = null;
        try {
            factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
        } catch (NoSuchAlgorithmException e) {
            throw new IOException("Can't load SecretKeyFactory", e);
        }

        SecretKeySpec key = null;
        try {
            String algOnly = alg.split("/")[0];
            key = new SecretKeySpec(
                    factory.generateSecret(
                            new PBEKeySpec(passPhrase.toCharArray(), salt.getBytes(), iterations, keyLen)).getEncoded(),
                    algOnly);
        } catch (Exception e) {
            throw new IOException("Can't generate secret key", e);
        }

        Cipher crypto = null;
        try {
            crypto = Cipher.getInstance(alg);
        } catch (Exception e) {
            throw new IOException("Can't initialize the decryptor", e);
        }

        Path inputFileLocation = Paths.get(inputFileName);
        byte[] decrypted = Files.readAllBytes(inputFileLocation);
        byte[] encrypted;

        try {
            crypto.init(Cipher.ENCRYPT_MODE, key);
            encrypted = crypto.doFinal(decrypted);
        } catch (Exception e) {
            throw new IOException("Can't decrypt the password", e);
        }

        Path outputFileLocation = Paths.get(outputFileName);
        Files.write(outputFileLocation, encrypted);
    }
}

Thanks for the answer. I didn't know I can't do PBKDF2 iterations nor did I need it. I made a Java program based on the code you posted here and the CryptoFileLoader class and I solved my problem. — burakongun, Apr 07 '17 at 07:41

score 1 · Answer 2 · edited May 23 '17 at 11:46

As in Simone's answer - there is a difference in the encryption algorithm between openssl and java implementation. That is why you can decrypt using openssl with no problem (as it is invoking its own (different) algorithm again).

After much digging I found this answer from (Dave Thompson) which states:

Short answer: what openssl enc (without -K for raw) uses is not PBKDF2; it is almost PBKDF1, with iteration count 1.

It seems that there are two ways round this issue, either:

a) Find something in java that can decrypt what openssl is doing - there is a java library 'BouncyCastle' referenced in a post inside this answer (if you are happy using that instead of standard CryptoFile) where they have implemented the exact same algorithm that openssl is using.

or

b) Find some other command line utility to use instead of openssl that implements PBKDF2. A number of implementations in different languages are referenced in the nabble.com posting also mentioned.

(Due credit to Dave for the key observation quoted)

Thanks for the links. They provided me more in depth knowledge about encryption process and stardards. I made a Java program based on my CryptoFile class and Simone's answer, instead of openssl and it is working now. — burakongun, Apr 07 '17 at 07:38

Getting BadPaddingException while doing a Sqoop import with encrypted password file

2 Answers2