Different MD5 outputs in shell and script

Question

This is driving me crazy, im trying to do some MD5 calculation based on the fritzbox SPEC for logging in. Basically you have to convert a challenge and the password into UTF-16LE and then hash it by md5, then concat challenge-md5(uft-16le(challenge-password))

To do so i'm using iconv and md5 from mac OSX in a script

echo -n "challenge-password1234" | iconv -f ISO8859-1 -t UTF-16LE | md5

Which outputs to 2f42ad272c7aec4c94f0d9525080e6de

Doing the exact thing by just pasting it in the shell outputs to 1722e126192656712a1d352e550f1317

The latter one is correct (accepted by fritzbox) the first one is wrong.

Calling the script with bash script.sh results in the proper hash, calling it with sh script.sh results in the wrong hash, which leads to the new question: How come the output is any different between sh and bash?

Answer 1

Different versions of echo behave in very different ways. Some take command options (like -n) that modify their behavior (including -n suppressing the trailing linefeed), and some don't. Some interpret escape sequences in the string itself (including \c at the end of the string suppressing the trailing linefeed)... and some don't. Some do both. It appears the version of echo (/bin/echo) on your system doesn't take options, and therefore treats -n as a string to be printed. If you're using bash, its builtin version overrides /bin/echo, and does interpret flags.

Basically, echo is a mess of inconsistency and portability traps. So don't use it, use printf instead. It's a little more complicated because you have to specify a format string, then the actual stuff you want printed, but it can save a ton of headaches.

$ printf "%s" "challenge-password1234" | iconv -f ISO8859-1 -t UTF-16LE | md5
1722e126192656712a1d352e550f1317

And by the way, here's what the echo command was actually printing:

$ printf "%s\n" "-n challenge-password1234" | iconv -f ISO8859-1 -t UTF-16LE | md5
2f42ad272c7aec4c94f0d9525080e6de

Answer 2

You are specifically asking for the difference between script and command line result. Please note that there can be other cases in which the script result will not be useable for your Fritzbox.

The sample code for the session handling in the AVM Fritzbox documentation is written in C# see

• https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID.pdf

from that code I derived https://github.com/WolfgangFahl/fritz-csharp-api and added some tests:

• https://github.com/WolfgangFahl/fritz-csharp-api/blob/master/fritzsimpletest.cs

which where inspired by the tests in:

• https://github.com/WolfgangFahl/fritzbox-java-api/blob/master/src/test/java/com/github/kaklakariada/fritzbox/Md5ServiceTest.java

Basically there were 5 examples:

1. "" -> "d41d8cd98f00b204e9800998ecf8427e"
2. "secret", "09433e1853385270b51511571e35eeca"
3. "test", "c8059e2ec7419f590e79d7f1b774bfe6"
4. "1234567z-äbc", "9e224a41eeefa284df7bb0f26c2913e2";
5. "!\"§$%&/()=?ßüäöÜÄÖ-.,;:_`´+*#'<>≤|" -> "ad44a7cb10a95cb0c4d7ae90b0ff118a" and yours is now example number 6:

6. "challenge-password1234" -> "1722e126192656712a1d352e550f1317" and these behave the same in the Java and C# implementation. Now trying out these with the bash script below which has

   echo -n "$l_s" | iconv --from-code ISO8859-1 --to-code UTF-16LE | md5sum -b | gawk '{print substr($0,1,32)}'

which is e.g. discussed in https://www.ip-phone-forum.de/threads/fritzbox-challenge-response-in-sh.264639/

as it's getmd5 function gives different results for the Umlaut cases e.g. in my bash on Mac OS Sierra.

There is already some debug output added.

The encoding given for 1234567z-äbc has the byte sequence 2d c3 a4 62 63 while e.g. the java implementation has 2d e4 62 63.

So beware of umlauts in your password - the fritzbox access might fail using this script solution. I am looking for a workaround and will post it here when i find it.

bash script

#!/bin/bash
# WF 2017-10-30
# Fritzbox handling

#
# get the property with the given name
# params
#   1: the property name e.g. fritzbox.url, fritzbox.username, fritzbox.password
#
getprop() {
  local l_prop="$1"
  cat $HOME/.fritzbox/application.properties | grep "$l_prop" | cut -f2 -d=
}

#
# get a value from the fritzbox login_sid.lua
#
getboxval() {
  local l_node="$1"
  local l_response="$2"
  if [ "$l_response" != "" ]
  then
    l_data="&response=$l_response"
  fi
  fxml=/tmp/fxml$$
  curl --insecure -s "${box_url}/login_sid.lua?username=${username}$l_response" > $fxml
  cat $fxml |
  gawk -v node=$l_node 'match($0,"<"node">([0-9a-f]+)</"node">",m) { print m[1] }'
  cat $fxml
  rm $fxml
}

#
# get the md5 for the given string
#
# see https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID.pdf
#
# param
#  1: s - the string
#
# return
#   md5 
#
getmd5() {
  local l_s="$1"
  echo -n "$l_s" | iconv -f ISO8859-1 -t UTF-16LE | od -x
  echo -n "$l_s" | iconv --from-code ISO8859-1 --to-code UTF-16LE | md5sum -b | gawk '{print substr($0,1,32)}'
}

# get global settings from application properties
box_url=$(getprop fritzbox.url)
username=$(getprop fritzbox.username)
password=$(getprop fritzbox.password)

# uncomment to test
getmd5 ""
#   should be d41d8cd98f00b204e9800998ecf8427e
getmd5 secret
#   should be 09433e1853385270b51511571e35eeca
getmd5 test
#   should be c8059e2ec7419f590e79d7f1b774bfe6
getmd5 1234567z-äbc
#   should be 9e224a41eeefa284df7bb0f26c2913e2
getmd5 "!\"§$%&/()=?ßüäöÜÄÖ-.,;:_\`´+*#'<>≤|"
#   should be ad44a7cb10a95cb0c4d7ae90b0ff118a
exit
# Login and get SID
challenge=$(getboxval Challenge "")
echo "challenge=$challenge"
md5=$(getmd5 "${challenge}-${password}")
echo "md5=$md5"
response="${challenge}-${md5}" 
echo "response=$response" 
getboxval SID "$response"