for the libjpeg-turbo port that I did (and am still working on) there the following things to consider:
instruction set (armv6, armv7), SIMD (with or without NEON)
Libjpeg-turbo really shines on armv7 with NEON. NEON is arm's SIMD support.
Tegra, Tegra2 doesn't have NEON but it is armv7 for instance. So libjpeg-turbo can be built right now turning NEON support off but still use some of the armv7 optimizations... it just won't be as fast. It will still be faster than the libjpeg that is currently found in android.
There are armv6 chips out there being put into new phones running Android. Especially for the cheaper markets. Rather then getting into an exhaustive list, what needs to happen in the lib (and this is WIP type stuff for me currently) is getting various armv6 optimizations into libjpeg-turbo so there is an all around good story for libjpeg-turbo on android. Stay tuned.