How to compare the performance of Android Apps written in Java and Xamarin C#? Anyway to check quantitative data (code & results)

Question

I came across Xamarin claims that their Mono implementation on Android and their C# compiled apps are faster than Java code. Did anyone perform actual benchmarks on very similar Java and C# code on different Android platforms to verify such claims, could post the code and results?

Added June 18, 2013

Since there was no answer and could not find such benchmarks done by others, decided to do my own tests. Unfortunately, my question remains "locked" so I cannot post this as the answer, only edit the question. Please vote to re-open this question. For C#, I used Xamarin.Android Ver. 4.7.09001 (beta). The source code, all the data I used for testing and compiled APK packages are on GitHub:

Java: https://github.com/gregko/TtsSetup_Java

C#: https://github.com/gregko/TtsSetup_C_sharp

If someone would like to repeat my tests on other devices or emulators, I'd be interested to learn the results as well.

Results from my testing

I ported my sentence extractor class to C# (from my @Voice Aloud Reader app) and run some tests on 10 HTML files in English, Russian, French, Polish and Czech languages. Each run was performed 5 times on all 10 files, and the total time for 3 different devices and one emulator are posted below. I tested "Release" builds only, without debugging enabled.

HTC Nexus One Android 2.3.7 (API 10) - CyanogenMod ROM

Java: Grand total time (5 runs): 12361 ms, with file reading total: 13304 ms

C#: Grand total time (5 runs): 17504 ms, with file reading total: 17956 ms

Samsung Galaxy S2 SGH-I777 (Android 4.0.4, API 15) - CyanogenMod ROM

Java: Grand total time (5 runs): 8947 ms, with file reading total: 9186 ms

C#: Grand total time (5 runs): 9884 ms, with file reading total: 10247 ms

Samsung GT-N7100 (Android 4.1.1 JellyBean, API 16) - Samsung ROM

Java: Grand total time (5 runs): 9742 ms, with file reading total: 10111 ms

C#: Grand total time (5 runs): 10459 ms, with file reading total: 10696 ms

Emulator - Intel (Android 4.2, API 17)

Java: Grand total time (5 runs): 2699 ms, with file reading total: 3127 ms

C#: Grand total time (5 runs): 2049 ms, with file reading total: 2182 ms

Emulator - Intel (Android 2.3.7, API 10)

Java: Grand total time (5 runs): 2992 ms, with file reading total: 3591 ms

C#: Grand total time (5 runs): 2049 ms, with file reading total: 2257 ms

Emulator - Arm (Android 4.0.4, API 15)

Java: Grand total time (5 runs): 41751 ms, with file reading total: 43866 ms

C#: Grand total time (5 runs): 44136 ms, with file reading total: 45109 ms

Brief discussion

My test code contains mainly text parsing, replacing and Regex searches, perhaps for other code (e.g. more numeric operations) the results would be different. On all devices with ARM processors, Java performed better than Xamarin C# code. The largest difference was under Android 2.3, where C# code run at approx. 70% of Java speed.

On Intel emulator (with Intel HAX technology, emulator runs in fast virt mode), Xamarin C# code runs my sample code much faster than Java - about 1.35 time faster. Maybe Mono virtual machine code and libraries are much better optimized on Intel than on ARM?

Edit July 8, 2013

I just installed Genymotion Android emulator, which runs in Oracle VirtualBox, and again this one uses native Intel processor, not emulating ARM processor. As with Intel HAX emulator, again C# runs here much faster. Here are my results:

Genymotion emulator - Intel (Android 4.1.1, API 16)

Java: Grand total time (5 runs): 2069 ms, with file reading total: 2248 ms

C#: Grand total time (5 runs): 1543 ms, with file reading total: 1642 ms

I then noticed that there was an update to Xamarin.Android beta, version 4.7.11, with release notes mentioning some changes in Mono runtime as well. Decided to quickly test some ARM devices, and big surprise - C# numbers improved:

BN Nook XD+, ARM (Android 4.0)

Java: Grand total time (5 runs): 8103 ms, with file reading total: 8569 ms

C#: Grand total time (5 runs): 7951 ms, with file reading total: 8161 ms

Wow! C# is now better than Java? Decided to repeat the test on my Galaxy Note 2:

Samsung Galaxy Note 2 - ARM (Android 4.1.1)

Java: Grand total time (5 runs): 9675 ms, with file reading total: 10028 ms

C#: Grand total time (5 runs): 9911 ms, with file reading total: 10104 ms

Here C# seems to be only slightly slower, but these numbers gave me a pause: Why the time is longer than on Nook HD+, even though Note 2 has a faster processor? The answer: power saving mode. On Nook, it was disabled, on Note 2 - enabled. Decided to test with power saving mode disabled (as with enabled, it also limits the processor speed):

Samsung Galaxy Note 2 - ARM (Android 4.1.1), power saving disabled

Java: Grand total time (5 runs): 7153 ms, with file reading total: 7459 ms

C#: Grand total time (5 runs): 6906 ms, with file reading total: 7070 ms

Now, surprisingly, C# is slightly faster than Java on ARM processor as well. Big improvement!

Edit July 12, 2013

We all know, that nothing beats native code for speed, and I was not satisfied with the performance of my sentence splitter in Java or C#, particularly that I need to improve it (and thus make it even slower). Decided to re-write it in C++. Here is a small (i.e. a smaller set of files than previous tests, for other reasons) comparison of the speed of native vs. Java on my Galaxy Note 2, with power saving mode disabled:

Java: Grand total time (5 runs): 3292 ms, with file reading total: 3454 ms

Native thumb: Grand total time (5 runs): 537 ms, with file reading total: 657 ms

Native arm: Grand total time (5 runs): 458 ms, with file reading total: 587 ms

Looks like for my particular test, the native code is 6 to 7 times faster than Java. Caveat: could not use std::regex class on Android, so had to write my own specialized routines searching for paragraphs breaks or html tags. My initial tests of the same code on a PC using regex, were about 4 to 5 times faster than Java.

Phew! Waking raw memory with char* or wchar* pointers again, I instantly felt 20 years younger! :)

Edit July 15, 2013

(Please see below, with edits of 7/30/2013, for much better results with Dot42)

With some difficulty, I managed to port my C# tests to Dot42 (version 1.0.1.71 beta), another C# platform for Android. Preliminary results show that Dot42 code is about 3x (3 times) slower than Xamarin C# (v. 4.7.11), on an Intel Android emulator. One problem is that System.Text.RegularExpressions class in Dot42 does not have the Split() function that I used in Xamarin tests, so I used Java.Util.Regex class instead, and Java.Util.Regex.Pattern.Split(), so in this particular place in the code, there is this small difference. Should not be a big problem though. Dot42 compiles to Dalvik (DEX) code, so it cooperates with Java on Android natively, does not need expensive interop from C# to Java like Xamarin.

Just for comparison, I also run the test on ARM devices - here the Dot42 code is "only" 2x slower than Xamarin C#. Here are my results:

HTC Nexus One Android 2.3.7 (ARM)

Java: Grand total time (5 runs): 12187 ms, with file reading total: 13200 ms

Xamarin C#: Grand total time (5 runs): 13935 ms, with file reading total: 14465 ms

Dot42 C#: Grand total time (5 runs): 26000 ms, with file reading total: 27168 ms

Samsung Galaxy Note 2, Android 4.1.1 (ARM)

Java: Grand total time (5 runs): 6895 ms, with file reading total: 7275 ms

Xamarin C#: Grand total time (5 runs): 6466 ms, with file reading total: 6720 ms

Dot42 C#: Grand total time (5 runs): 11185 ms, with file reading total: 11843 ms

Intel emulator, Android 4.2 (x86)

Java: Grand total time (5 runs): 2389 ms, with file reading total: 2770 ms

Xamarin C#: Grand total time (5 runs): 1748 ms, with file reading total: 1933 ms

Dot42 C#: Grand total time (5 runs): 5150 ms, with file reading total: 5459 ms

To me, it was also interesting to note that Xamarin C# is slightly faster than Java on a newer ARM device and slightly slower on the old Nexus One. If anyone would like to run these tests as well, please let me know and I'll update the sources on GitHub. It would be particularly interesting to see results from a real Android device with Intel processor.

Update 7/26/2013

Just a quick update, re-compiled by benchmark apps with the latest Xamarin.Android 4.8, and also with dot42 1.0.1.72 update released today - no significant changes from the results reported before.

Update 7/30/2013 - better results for dot42

Re-tested Dot42 with Robert's (from dot42 makers) port of my Java code to C#. In my C# port done initially for Xamarin, I replaced some native Java classes, like ListArray, with List class native to C#, etc. Robert did not have my Dot42 source code, so he ported it again from Java and used original Java classes in such places, which benefits Dot42, I guess because it runs in Dalvik VM, like Java, and not in Mono, like Xamarin. Now Dot42 results are much better. Here is a log from my testing:

7/30/2013 - Dot42 tests with more Java classes in Dot42 C#

Intel emulator, Android 4.2

Dot42, Greg's Code using StringBuilder.Replace() (as in Xamarin):
Grand total time (5 runs): 3646 ms, with file reading total: 3830 ms

Dot42, Greg's Code using String.Replace() (as in Java and Robert's code):
Grand total time (5 runs): 3027 ms, with file reading total: 3206 ms

Dot42, Robert's Code:
Grand total time (5 runs): 1781 ms, with file reading total: 1999 ms

Xamarin:
Grand total time (5 runs): 1373 ms, with file reading total: 1505 ms

Java:
Grand total time (5 runs): 1841 ms, with file reading total: 2044 ms

ARM, Samsung Galaxy Note 2, power saving off, Android 4.1.1

Dot42, Greg's Code using StringBuilder.Replace() (as in Xamarin):
Grand total time (5 runs): 10875 ms, with file reading total: 11280 ms

Dot42, Greg's Code using String.Replace() (as in Java and Robert's code):
Grand total time (5 runs): 9710 ms, with file reading total: 10097 ms

Dot42, Robert's Code:
Grand total time (5 runs): 6279 ms, with file reading total: 6622 ms

Xamarin:
Grand total time (5 runs): 6201 ms, with file reading total: 6476 ms

Java:
Grand total time (5 runs): 7141 ms, with file reading total: 7479 ms

I still think that Dot42 has a long way to go. Having Java-like classes (e.g. ArrayList) and a good performance with them would make porting code from Java to C# slightly easier. However, this is something I would not be likely to do a lot. I would rather want to use existing C# code (libraries etc.), which will use native C# classes (e.g. List), and that would perform slowly with the current dot42 code, and very well with Xamarin.

Greg

DEBUG mode on Nexus 7 4.2.2 with some optimisations on strings and xamarin alpha 9: Total time: 3907 ms, with file reading total: 4016. What means "5 runs" ? — Softlion, Jul 05 '13 at 19:37
Release mode on same platform: Total time: 2192 ms, with file reading total: 2279 — Softlion, Jul 05 '13 at 19:39
@Softlion, thank you for testing this too! By 5 runs, I mean that I press the "Start Test" button 5 times, then press "back" button on the phone to exit the app, and then it outputs to the debug console the total time of the 5 runs. Please send me the proposed optimizations, best by email or private message, as we can't type much here. But we would have to optimize the Java code in a similar way to have a fair comparison between C# and Java. — gregko, Jul 06 '13 at 21:18
You use too much strings. You should be using only one StringBuilder and string indexes. strings are immutable, and allocates new memory on each concatenation. You alos should not use "Count()" but "Count" instead. This is a "classic" error. — Softlion, Jul 09 '13 at 09:07
@Softlion, I use StringBuilders where I can. In C# I use them also for searching and replacing characters, in Java can't, as it does not have this function. Anyway, for the purpose of this text, the search & replace characters don't matter, as the characters don't occur in my test files - the search is done, but no replace, so no need to change "immutable" strings in Java. I had many similar comments, introduced proposed changes without any significant effect on the result. Please point specific change and I can test again. — gregko, Jul 10 '13 at 11:47
@Softlion, I just reviewed my C# code for .Count vs. .Count() - I use them correctly, use .Count for lists, and .Count() for arrays, like String[] array. It's not a "classic" error, doing otherwise is simply a syntax error that does not let the code compile... — gregko, Jul 10 '13 at 11:58
Count() is a linq method which reads the whole enumerable. For arrays you have to use .Length not Count(). Any other place where you use heavy string concatenations, including in loops, you shouldn't. This imply to rewrite your algorithm as you should use string indexes+lengths and the smallest number of StringBuilders instead of what you do. Also you should use static regex, they are thread safe. — Softlion, Jul 13 '13 at 12:00
@Softlion, when you tell me about methods like this, please specify if you speak of Java or C#. Guess Linq would be C#. Since you can get my source code anytime, please modify it as you propose and post the results. I tried similar modifications in the past and did not notice any improvement in performance. The only place I use .Count() on an array is in my main test loop, to loop through a list of 10 files. Can't imagine how this could change the benchmark result. — gregko, Jul 13 '13 at 18:20
I modified it but I don't understand git enough. I know only basics of java so i won't tell anything about the java code. — Softlion, Jul 17 '13 at 13:58
@Softlion, thank you! If possible, share your C# code modifications by emial to me (gregko AT hyperionics DOT com), I'll run some tests and post them too. In your tests, did you see any difference from your code mods? I'm always eager to learn, thanks again! — gregko, Jul 18 '13 at 11:52
i am curious as to the size of the apk and memory consumption at runtime? — numan salati, Aug 16 '13 at 15:09
@numan - for Xamarin, the APK with both Intel and armeabi-v7a binaries embedded, the package is about 5.6 MB, built in release mode with the latest stable Xamarin. For Dot42 and Java it's below 200 kB. I'm not sure what's the best way to measure memory usage on Android — gregko, Aug 18 '13 at 06:35
@gregko is there no profiler in xamarin? In ADT (eclipse) you can profile the app for memory usage. — numan salati, Aug 18 '13 at 14:49
@numan - Android debug monitor shows very similar heap usage for both C# and Java (about 5 MB for a single run through my test files). I don't see any way to show the total process memory usage there, for sure Xamarin will take more, as it needs first to load the Mono runtime before executing any code, while Java and Dot42 C# code execute directly in Dalvik VM. — gregko, Aug 19 '13 at 08:30
"this question will likely solicit debate, arguments, polling, or extended discussion" <- see above ;) — CodeSmile, Oct 09 '13 at 22:25
@LearnCocos2D - I'm only reporting concrete results and numbers, i.e. facts. Gentlemen don't dispute facts :) — gregko, Oct 11 '13 at 06:05
well, scientists do ;) there's a difference between observed behavior and fact. There's a lot more to it to become fact, and even then applicability to other users/situations remains questionable. This is the crux of benchmarks, they only present facts on the surface - until you find out that vendor x had optimized its driver for a particular benchmark app. On a related note it was once proven that water has memory (ie the homeopathy test), which was disproven after the tester bias was considered and ruled out, then it failed to show any statistical significance. — CodeSmile, Oct 11 '13 at 09:38
plus, with the next +0.1 version bump theseperformance characteristics may change significantly - that's when all your good efforts presented here change from "fact" to "moot". However anyone who comes here may perceive this as fact, and draw the wrong conclusion. Another crux of benchmarks: they are only representative for a given moment in time and versions of the software used. The next day they may no longer reflect reality. You have to keep retesting the results. This is why the results here can be considered subjective and bearing little to no meaning. — CodeSmile, Oct 11 '13 at 09:42
This is a quite interesting comparison. Would you please try to also do the comparison on Android L , since ART has replaced Dalvik, and is supposed to be faster? — android developer, Aug 06 '14 at 12:12
@androiddeveloper, I no longer have Xamarin subscription, so can't do this on Xamarin side. — gregko, Aug 06 '14 at 16:05
I'm voting to close this question as off-topic because the contained information is 3 years stale and this doesn't fit any current criteria of an acceptable question. — spender, Oct 20 '16 at 14:44

score 45 · Answer 1 · edited Mar 27 '23 at 18:49

45

I came across this interesting post:

https://medium.com/@harrycheung/mobile-app-performance-redux-e512be94f976#.kfbauchtz

edited Mar 27 '23 at 18:49

General Grievance

4,555
31
31
45

answered Apr 22 '16 at 09:00

Venkataramana Madugula

13,199
2
18
22

score 35 · Answer 2 · answered Jun 05 '14 at 15:30

We recently investigated using Xamarin for an app. We utilized the C# code we had already written for the Windows RT version of our app. Some specific details had to be rewritten for the Android version.

What we discovered is that I/O in Xamarin C# is approximately 2x slower than Java. Our app is heavily I/O bound. We have not discovered the cause of this yet, but at the moment we are assuming that it is due to marshaling. While we do try to stay inside the Mono VM most of the time, we do not know how Mono actually accesses the disk.

It is also telling that our C# code uses SQLite.NET (https://github.com/praeclarum/sqlite-net). Identical fetches using the SQLite.NET code are also 2x slower than using Android's Java SQLite wrapper. After looking at the source code, it appears to bind directly to the C .dll, so I do not know why it's so much slower. One possibility is that marshaling strings from native to Java may be faster on Android than native to C# is on Xamarin.

This has very likely also to due with "bindings" Xamerin needs to interact with the system. Each and every system call by default goes to a Java class but needs to be delegated to the Mono VM, which takes time. The same also happens in reverse. I have explained this a bit more in my answer: https://stackoverflow.com/a/46973819/1052697 — Rolf ツ, Oct 27 '17 at 11:20

Daniel · Answer 3 · 2016-11-22T09:02:20.213

34

This is another more updated blog post I would like to share with you. He compares Xamarin to native code and Cordova on both IOs and Android.

In a nutshell, Xamarin performs sometimes better than native code. He tested the app size, load times, loading a list from Azure service and prime number computation.

Enjoy!

Edit: I updated the dead link and I noticed that there is a part 2

edited Nov 22 '16 at 09:02

answered Feb 25 '15 at 09:17

Daniel

9,312
3
48
48

Burgito · Answer 4 · 2017-11-17T09:21:44.707

Here are a few informations I found in another test between native, Xamarin and Xamarin.Forms solutions (the tests also include iOS performances) on the two following devices :

Samsung Galaxy A7: Android OS version: 6.0 Central-processing unit: Octa-core 1.9 GHz Cortex-A53 RAM: 3GB Display resolution: 1920×1080

iPhone 6s: iOS version: 10.3.3 Central-processing unit: Dual-core 1.84 GHz Twister RAM: 2 GB Display resolution: 1334×750

Comparison is made on a few common features, each one with its own application :

- Basic “Hello World”
- REST API
- JSON Serialization/Deserialization
- Photo Loading
- SQL Database Insert and Get All

Each test is repeted several times, the graphs show the average results.

Hello World

Rest API

Set of tests aimed at measuring the time it takes for the app to send a request through REST API and receive the response back without further data processing, using OpenWeatherMap API.

JSON Operations Tests made using Newtonsoft Json.net framework to serialize and deserialize JSON objects in all Xamarin apps. Native Android serialization and deserialization tested using two Java libraries: Jackson and GSON.

Two runs are made, one first from scratch and a second one with cached infos and operations

First run :

(Native iOS JSON Operations is killing this test btw, and Xamarin joins it in the second)

Photo Operations

First load on images with three different resolutions :

Resolution – 858×569, Size – 868Kb
Resolution – 2575×1709, Size – 8Mb
Resolution – 4291×2848, Size – 28.9Mb

Something seemed unsure about the Xamarin.Forms results for this test, so it is not included in the graph.

SQLite Operations

Two operations tested :

BulkInsert: Loading rows of data into a database table.
GetAll: Retrieving all data from the database.

With databases having 10,000 records. All operations were processed internally on devices.

Xamarin Native (Xamarin.iOS/Xamarin.Android) show themselves as rather good alternatives to the native code, whereas Xamarin.Forms seems slow in a lot of cases, but it can be a really good solution to develop really simple applications fastly.

Complete test comes from this source :

https://www.altexsoft.com/blog/engineering/performance-comparison-xamarin-forms-xamarin-ios-xamarin-android-vs-android-and-ios-native-applications/

Thank you for giving me the explanations to enhance my answer, hope this helps a little :)

Rolf ツ · Answer 5 · 2017-10-29T07:37:03.487

Performance

Performance is a vague word if you don't define what you mean by performance, if it's plain computation performance Xamarin can be faster than Java depending on the nature of the computation.

Android nativly comes with multipe forms to execute code in:

RenderScript (CPU and GPU)
Java (SDK)
C++ (NDK)
OpenGL (GPU)

It is quite obvious that when executing code the more native the solution the faster it will be. A run-time based language will never beat a language that directly runs on the CPU.

But on the other hand if you want to measure real-life usage performance Java is propbaby going to be faster then Xamarin.

Xamarin and why it can be slower

When comparing Xamarin with plain old Java applications, performance can very well be faster for Xamarin as it can be slower.

In a real world example Xamarin applications are very likely to be slower than Java applications because many Android/Java (system) calls need to be delegated to and from the Xamarin run-time using so called bindings.

There are a few different types of bindings that are important to know:

JNI (Java Native Interface): The binding used in many android applications to interface between Java code (SDK) and native C++ code (NDK).
MCW (Managed Callable Wrappers): A binding that is available in Xamarin to interface from managed C# code to Java code (Android run-time).
ACW (Android Callable Wrappers): A binding that is available in Xamarin to interface from Java code (Android run-time) to managed C# code.

More on MCW and ACW here: https://developer.xamarin.com/guides/cross-platform/application_fundamentals/building_cross_platform_applications/part_1_-_understanding_the_xamarin_mobile_platform/

Bindings are in terms of performance very very costly. Invoking a C++ method from Java adds a huge overhead in calling time, calling a C++ method from within C++ is many many many times faster.

Someone did a performance test to calculate how many Java operations on average a JNI call costs: What is the quantitative overhead of making a JNI call?

But not only JNI calls are costly so are calls to and from MCW and ACW. Real world Xamarin applications make many calls using bindings and because of this real world usage of an Xamarin application can be (and will be in general) slower than a plain old Java application. However depending on how the Xamarin application was designed it is very likely that the user won't even notice the difference.

TLDR/Conclusion: Xamarin needs to using al sorts bindings, which are time costly.

Besides bindings, there are many other factors involved when talking about real-world performance, for example: size of the binary, loading the app in memory, I/O operations and many more. A blog post that investigates some of these things can be found here: https://magenic.com/thinking/mobile-development-platform-performance-part-2-native-cordova-classic-xamarin-xamarin-forms

Denis Gordin · Answer 6 · 2018-03-28T15:36:04.250

4

It's pretty old tests but could be relevant: https://github.com/EgorBo/Xamarin.Android-vs-Java

Arithmetic test

Collections, generics, custom value types

Working with strings

UPD: new data with Google Pixel 2 (thanks yousha-aleayoub)

edited Mar 28 '18 at 15:36

answered Oct 27 '17 at 05:19

Denis Gordin

892
10
22

1

Update 2018 https://camo.githubusercontent.com/76fd25443e2a407665389029ed594643b3f39f5f/68747470733a2f2f686162726173746f726167652e6f72672f776562742f65702f30642f76792f6570306476795f693878375f697a64733135383867796e716468772e706e67 – Yousha Aleayoub Mar 28 '18 at 11:44