0

If I only scale to 1 instance of an Azure Web App my application runs fine.

When I scale to 2 or more instances my application throws exception:

System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed

errors during decryption which appear to be coming from the the second and third instances. When I scale back down to 1 instance the errors go away.

My application draws initialization vectors and ciphertext from a database which means all instances are receiving the same inputs.

I've tried recommendations from related but different questions such as ensuring FlushFinalBlock and explicitly setting padding and block modes mentioned here and here. Yi-Lun Luo experienced an invalid padding error when moving from development to Azure-based production, however I'm not able to find any references to errors being thrown when moving from single to multiple instances.

Has anyone else experienced this problem? Any suggestions of things to try would be greatly appreciated.

Clarification

Anand edited the question by dropping the word "intermittent", which caused an inaccurate description of the problem. When the application is scaled to 2 or more instances, some requests succeed and some throw a cryptographic exception. The application is a rest service. If ten requests are made from a single test page at one time, all will succeed if 1 instance is running. If 2 instances are running roughly half will succeed and half will fail. The trend continues with 3 instances. This behavior makes it seem like the initial instance is running properly and additional instances are not.

Community
  • 1
  • 1
John Clegg
  • 348
  • 3
  • 6
  • Based on a quick search for this error, I'm inclined to say that it is because different machine key on different instances. Can you try by setting machine key in web.config so that all instances use the same machine key? – Gaurav Mantri Jan 04 '16 at 15:09
  • Thanks for the suggestion Gaurav but I can rule out machine key. The data was encrypted by another application on another machine and placed in the database. I have a key that allows it to be decrypted successfully by the first Web App instance, just not by the second or third. – John Clegg Jan 04 '16 at 15:37

1 Answers1

1

I unknowingly used a shared instance of the RijndaelManaged class which lead to a subtle and transient problem.

In the application both encryption key and initialization vector are passed into the decrypt method and then used to set RijndaelManaged object's respective properties (the object is the _algorithm variable in the code below).

        Using _algorithm

        _algorithm.Key = key
        _algorithm.IV = initializationVector

        Dim transform As ICryptoTransform = _algorithm.CreateEncryptor(_algorithm.Key, _algorithm.IV)
        Using memoryStream As New MemoryStream()
            Using cryptoStream As New CryptoStream(memoryStream, transform, CryptoStreamMode.Write)
                Using streamWriter As New StreamWriter(cryptoStream)
                    streamWriter.Write(value)
                End Using
                encrypted = memoryStream.ToArray()
            End Using
        End Using
    End Using

Even though the class was shared (a poor practice) it did not present a problem when only one Web App instance was running. However, as soon as multiple instances existed the value of _algorithm.Key became unstable.

I discovered it by capturing key's value at several points. Capturing key on the first line in the function showed the correct value. Capturing _algorithm.Key immediately after setting would also show the correct value. Capturing it again immediately before Dim transform would show an incorrect value, but only on error, which leads me to believe the shared value was being set elsewhere.

I can't explain why this would happen in multiple Web App instances but not when only a single instance was running. The instances after all should be separate machines with separate memory. However, since making the _algorithm variable non-static the problem has stopped.

John Clegg
  • 348
  • 3
  • 6