16

We've just released an app using the Crittercism framework. After some time, we've had about 125K app loads, and 95 crashes - a rate of less than 0.08%.

One crash happened 19 times, another 10, but the other 41 all occurred 3 or less. If there were any major problems with the app, I would expect to see significantly more failures in particular areas, so I'm happy with the level of figures I'm seeing.

A quick look shows many of them to be low level failures, not obviously caused but programmer error.

Examples

  • The largest group are all to do with CFNetworking on a background thread while static HTML is being being rendered in a web view on the main thread.
  • There are some KVO failures in free_list_checksum_botch

But my question is, in a sufficiently complex OS (iOS in this case), with a sufficiently complex app (which I think it is), should I, as a developer, expect to see this level of "background noise"?

Should I expect to see one app crash per 1-2000 loads, just because the OS isn't perfect? Has anyone else had a similar experience?

(I'm not looking for solutions to the errors themselves.. thanks!)

Ashley Mills
  • 50,474
  • 16
  • 129
  • 160

5 Answers5

6

Crittercism conducted an analysis on app crashes. Their report was based on Android vs iOS crashes.

They concluded that the most popular apps on iOS crash 0.51% of app launches. So @Ashley Mills, if you're getting 0.08%... you're doing well. (i think I have my figures correct, anyway).

Not sure where the original report is, but I read it here:

Forbes app crash rates, conducted by Crittercism

bandejapaisa
  • 26,576
  • 13
  • 94
  • 112
  • So better that average, which is good - but I still don't know what the underlying OS crash rate is. Although, to be honest, I blame the other dev on the project for most of its failings. – Ashley Mills May 31 '12 at 08:58
  • "But my question is, in a sufficiently complex OS (iOS in this case), with a sufficiently complex app (which I think it is), should I, as a developer, expect to see this level of "background noise"? The Crittercism analysis covers tens of thousands of apps and should be a good enough answer for you. Any other single response from a user will just be based on a handful of apps. Now give me the bounty, or the bounty will go on your head. The answer is YES, expect noise. – bandejapaisa May 31 '12 at 09:09
  • How do app launches correlate to user sessions? Does backgrounding of the app and later resuming correspond to the same app launch or separate app launches? – Dalmazio Oct 16 '14 at 16:47
  • I don't know regarding that article, but I would argue that an app launch is launch for first time or launch from the background. So yes, a user session. – bandejapaisa Oct 16 '14 at 18:55
6

Actually another shot in the dark non-technical answer could be. What added value will it bring you (the developer) to keep digging into that particular issue, when you could be spending more time and effort either developing more capability within your tool, or another tool completely.

If your app is simply for fun and learning, then I could see digging into this issue as a fun adventure. From a business perspective, what is your time worth and is figuring out this 0.08% problem going to sell enough (more) copies to make your efforts worth it.

An analogous would be, what requirements are essential, and what requirements are desirements? Just food for thought. I know many of the companies I have worked for don't see the value in stressing about that low of a yielding bug.

trumpetlicks
  • 7,033
  • 2
  • 19
  • 33
  • I agree that it's probably too small to worry about, particularly as the app in question is for consumption of data only, so nothing can really get "lost" in a crash. My question (as yet unanswered I'm afraid) was for my own curiosity. – Ashley Mills May 31 '12 at 08:57
  • 1
    I suspect that (if you look at my comment in an above thread), that 0.08% is almost inevitable. When you look at MS Explorer, or even Safari, it crashes quite regularly. Unfortunately, even if we had perfect code, we could get hit (on iOS devices anyway) with a spurious bit flip in hardware. iOS nor the hardware support ECC, and if you have a bit flip in a code area, your code will act peculiar or even not function, in your data you may venture off (if it used in a branch statement) to an unexpected place. Granted these don't happen often, but when you add everything up....... – trumpetlicks May 31 '12 at 13:17
  • Ah... you've grasped my question exactly. When you launch an app millions of times, occasionally it may crash through something other than programmer error. Now, if only we could see what that rate was... In the meantime, have 50 points :) – Ashley Mills May 31 '12 at 13:24
4

I am an iOS developer, employed professionally. I take it personally when my apps crash, because that is not the user experience I was aiming for. A crash is a bad user experience. One crash, per user, is too many. A crash is a bug.

That being said, I have definitely seen crash logs that appear to be unsolvable because they seem to be indicating a problem way down deep in the SDK. What I have learned, however, is that more than likely there is something about my own code that is ultimately the cause.

There are any number of bizarre crashes that can be caused by timing issues between threads or blocks or just because I did something wrong. Just recently I discovered I was doing something entirely wrong with respect to a complex table I was updating. The crash logs for this problem provided almost no clues except for the general area of code I might look at. As I dug into the code and started experimenting, I realized my mistake, which ultimately was a timing issue caused by what I thought was a clever separation of main-thread vs. non-main-thread activity. I was too clever for my own good, in this case. :-)

So, to sum up:

  • One crash is too many crashes and ultimately a bad user experience.
  • Often, bizarre low-level crashes are the result of your own code's complexity and possibly timing issues there in.

Finally, I offer this question to consider:

  • Are you willing to piss off or dismiss some of your users simply because they fall into the 0.08% of users experiencing crashes?

Food for thought. :-)

Mark Granoff
  • 16,878
  • 2
  • 59
  • 61
  • You're right, of course, 1 crash really is too many. But in this is only a consumption app, so the worst that will happen to a user is they end up back at the springboard (many don't even read this as a crash), and restarting the app puts them back to where they were... no harm done. And I'm fully with you about the cause of many mysterious errors. I'd just like to get an idea of what level (if there is one) of failures are due to inherent OS issues. – Ashley Mills Mar 27 '12 at 15:32
  • 9
    You're almost suggesting that an OS, iOS in this instance, is error free and could not be the cause of issues, and look at the numbers here... he's talking about 0.08% of users. This is a very small amount, and there could be very obscure errors that are never picked up by QA testers or even automated tests - unless they are really stress tested. I'm willing to fix every crash that I possibly can - what I can't afford to do is spend days trying to fix an obscure crash that happens 1 in 50,000 times. – bandejapaisa Mar 27 '12 at 15:35
  • 1
    I know that iOS is not error free. I've seen plenty of inexplicable crashes where my code wasn't even in the offending stack trace. So, no, iOS isn't perfect. But it's close. ;-) Ultimately, you have to balance your effort in solving these kinds of crashes with other data surrounding them (e.g. frequency) and benefit in the end for spending the time to fix them in the first place. – Mark Granoff Mar 27 '12 at 15:39
  • 1
    @bandejapaisa I think it's agreed that though the crashes might be the OS's fault, in *most* cases the root cause is a bug in the app developer's code. Given that, it could be worthwhile business-wise to investigate the bug, since it might point to other dormant bugs which you are not even aware of, and which will be much harder to debug when they wake up a year for now. Again, it's purely a business decision whether or not to investigate such bugs. – Danra May 30 '12 at 16:50
  • 1
    Investigate the bugs. While the overall crash numbers may be low, the crashes could be very repeatable for a few users or on a few devices. Imagine if you downloaded an app and it won't run at all. Those users may never try the app again and just because you never see another crash from them doesn't mean the bug is fixed. It's a good learning experience too. – EricS May 31 '12 at 03:47
  • 1
    I might also add that Hardware is not perfect either. There is a reason that Servers, satellites, and many data and process critical pieces of hardware have things like ECC for memory bit error detection and correction, Flash Memories (many) have BCH codes for multi-bit correction along a multi word field, etc... Testing every possible input to output of a piece of hardware is still theoretically impossible with today' computing power (ironic isn't it). Hardware for the most part is only tested to ~95%. With imperfect hardware and OSs, even perfect app code can appear to be flawed. – trumpetlicks May 31 '12 at 05:54
2

I'm a professional iPhone developer, and what I've seen is the crash frequency isn't what upsets users, it's the funnel for how the crashes happen.

If they are intermittent you are usually okay, the problem starts to occur when one user experiences a specific crash multiple times. This is unacceptable.

Striving to remove every crash is always a good thing, but in many cases it's simply not realistic, you have to decide where your time is best spent. If you have the opportunity to rework a portion of the UX flow or fix an intermittent crash, you need to decide which one benefits more users.

The important thing to remember is, if you choose not to fix a crash that happens 0.08% of the time, you aren't writing off the users experiencing it unless they are experiencing it multiple times.

Andrew Zimmer
  • 3,183
  • 1
  • 19
  • 18
0

Although not a technical answer, on my iPhone I'd personally expect to have an app I use a lot crash at least once or twice over a year. I'd say that level is perfectly acceptable, and since generally I find they crash a lot more on the first time you start it I believe it's something to be expected.

e__
  • 302
  • 3
  • 5
  • 17
  • That's quite a high expectation. As a developer, I strive to remove every single crash. As a user, the occasional crash really doesn't bother me. If it makes the app unusable, or it interrupted my usage (e.g. in the middle of a game) - then I would be concerned. But if i'm using an RSS reader and it crashed.... i just restart it. – bandejapaisa Mar 27 '12 at 15:38