6

Microsoft recently brought Address Sanitizer (ASan) to Microsoft Visual Studio 2019, and I've been experimenting with it. This question is specific to C and C++. My question is this: Is there any reason to have ASan enabled for Release builds, as opposed to having it enabled only for Debug builds? Having ASan turned on is devastating to the performance and memory usage of my program. (CPU performance worse than halved, memory usage tripled.) Therefore my hope is that if I enable ASan just to check for potential issues, and it does not detect any problems in a Debug build, then I can safely assume that there would not be any problems in the Release build that would have otherwise been caught by ASan? Certainly we are not meant to leave ASan enabled in Release/Production builds?

Thanks for any insight.

Ryan Ries
  • 2,381
  • 1
  • 24
  • 33
  • 6
    Address Sanitizer is very useful for finding bugs that are exposed only when optimisation is enabled. Consider creating a 3rd build target for Release with ASan. – Richard Critten Jun 05 '21 at 22:00
  • @RichardCritten Definitely - I long ago split my "Release" configuration into "Develop" and "Publish". – Adrian Mole Jun 05 '21 at 22:02
  • 2
    If you have an integration/system test suite, run that with an optimized Release build and with ASan. – Eljay Jun 05 '21 at 22:03

2 Answers2

4

Therefore my hope is that if I enable ASan just to check for potential issues, and it does not detect any problems in a Debug build, then I can safely assume that there would not be any problems in the Release build that would have otherwise been caught by ASan?

That's not likely (especially for larger complex programs). Consider a function like:

int array[250];

int test1(uint16_t a) {
    return array[ (1016 / (a+1)) & 0xFF];
}

You can test it with lots of values of a (e.g. 0, 2, 3, 4, ...) and it will be fine, and it will pass the address sanitizer's test during debug. Then, after release, it might crash due to a different (untested) value of a (e.g. 1) that triggers the "array index out of range" bug.

To guarantee that something like that can't happen you could test every possible combination of parameters for every function (which would be ludicrously expensive); but (in the presence of global variables/state) that might still not be enough, and (in the presence of multiple threads) it won't find hidden race conditions (e.g. code that almost always works because threadA finishes before threadB, that crashes when subtle differences in timing cause threadB to finish before threadA).

Certainly we are not meant to leave ASan enabled in Release/Production builds?

Correct - it's too expensive to leave enabled.

The general idea is to minimize the risk of bugs (because you can't guarantee that all bugs were found and eliminated). Without much testing you might be able to say that you're "80% sure" there's no bugs in the released version; and with some rigorous testing with Asan (and optimizations) enabled you might be able to say that you're "95% sure" there's no bugs in the released version.

What this really comes down to is economics (money). It's a "cost of software bugs (lost sales, support costs, etc)" vs. "cost of finding bugs" compromise; where something like a computer game might be fine with less testing (because users just assume that games crash occasionally anyway) and something like a life support machine (where bugs cause deaths) you might refuse to use C or C++ in the first place.

The other thing worth considering is that users don't care much why your code failed - if it failed because of a bug then the user will be annoyed; and if it failed because Asan was enabled and detected a bug then the user will be equally annoyed. Leaving Asan enabled in release builds would help the user submit better information in a bug report, but most people don't submit bug reports (they just sit there annoyed; wondering if it was a temporary hardware glitch because they're too cheap to buy ECC RAM, or if it was a bug in the OS or a driver; and then restart the software).

Brendan
  • 35,656
  • 2
  • 39
  • 66
  • 1
    *"Correct - it's too expensive to leave enabled."* - You'll reconsider once a memory safety issue caused your service to leak private information, driving your corporation into bankruptcy, or rendering your Mars Rover useless. – IInspectable Jun 06 '21 at 07:57
  • @IInspectable: In those cases "you might refuse to use C or C++ in the first place". Feel free to benchmark the performance of "C or C++ with Asan running" to something safer like Java (or Ada or Rust or..). – Brendan Jun 06 '21 at 09:20
  • Having no boundary check on stack have no excuse unless you are 100% sure. It can cause very direct security problems and having a branchless boundary check or “unlikely” fail branch would have no noticeable cost there. If you need every bit of speed then you have to check every input there is no excuse if or buts to that. – Abdurrahim Jun 06 '21 at 09:34
  • @IInspectable: Are you suggesting that people should (e.g.) enable Asan when compiling a Linux kernel (or JVM or..)? All those things ship without Asan so... – Brendan Jun 06 '21 at 09:38
  • Yes. That is exactly what I'm suggesting. That won't happen until you give companies and organizations an incentive to deliver software that doesn't stink. Reality now is that every data breach makes the headlines for a couple of weeks, before going back to doing things the way we've always been doing things. Make organizations pay for negligence, the price of the true damage caused, and we'll start seeing release builds with things like ASan in the wild. So long as it's more lucrative to ship shitty software, we keep seeing shitty software everywhere. – IInspectable Jun 06 '21 at 10:06
  • @IInspectable: If you increase "cost of software bugs" you increase "effort to find bugs before release" (and I also think that'd be good); but that will still not cause people to ship with Asan enabled in any released software - it's so expensive (and fails to find far too many problems) that it only makes sense before release (where it's a very valuable tool, almost as good as Valgrind has been for 30+ years). – Brendan Jun 06 '21 at 10:41
  • That's the point where we probably won't agree. You appear to be fine to ship software that is less safe than it could be, with literally zero effort involved, as long as it artificially increases its performance. The performance gain is artificial since it's trivial to optimize code when you lift the requirement of being correct. – IInspectable Jun 06 '21 at 11:28
  • @IInspectable: No; my point is that for released software "70% slower with 240% memory usage to catch almost none of the bugs while lots more bugs are not caught at all" is completely idiotic when there's significantly better (simultaneously more efficient and more able to catch a wider range of bugs) options. It's literally the wrong tool for the job (and the right tool for a completely different job - finding some bugs before release where efficiency doesn't matter). – Brendan Jun 06 '21 at 17:58
  • @IInspectable: In case you're serious about wanting to find out how awful it is, you can find instructions on installing Gentoo with Asan enabled (for all software in the entire OS) here: https://wiki.gentoo.org/wiki/AddressSanitizer – Brendan Jun 06 '21 at 18:32
  • I'm not serious about finding out, how awful it is to bring memory safety to C. I **know** how much of an impact the mitigations have. The issue isn't, that we wouldn't agree on the impact of those mitigations. The issue is, that we do no agree on the **repercussions**. To you it appears that the worst that could happen is a *"crash*". To me the consequences are far more realistic. A memory safety issue is *always* a potential remote code execution, or privilege escalation vulnerability, or something much worse. You seem to ignore any of those very real consequences. – IInspectable Jun 06 '21 at 22:21
  • @IInspectable: Consider all possibilities: assembly or C or C++ or C# or Java or Ada or Haskell or ...; with/without formal verification, with/without strict coding guidelines (e.g. MISRA), with/without peer review, with/without security audits, with none/some/lots of pre-release testing (Asan, unit tests, fuzzing, etc). Determine a scenario specific function to assess possibilities (likely in the form of "`profit = sales(efficiency, safety) - dev_cost`". Finally, plot all of the possibilities on a chart, and fire anyone that thought using Asan after release was sane (after proving it's not). – Brendan Jun 07 '21 at 07:26
1

Hard facts up front: A program's shape - broadly speaking - falls into one of two categories.

  1. Its logical representation expressed as its source code.
  2. Its binary representation, translated by a compiler (and linker) into something a machine can execute.

1. is what the programmer intended the program to do, and 2. is what a program translated that intent into.

Ideally, 1. is perfectly within a programming language's specification, and either representation has identical observable behavior. In reality, 1. is wrong, and 2. does whatever.

The exercise is thus: Evaluate how badly 2. is broken. To do that, you absolutely want a 2. that's as close to what your clients will run as ever possible. Fuzzing a 2. compiled as a Release configuration with ASan enabled gets you there. If you don't feel like reading along, fuzz this target as hard as humanly imaginable, and go your merry way.

That covers the unquestionable facts. What follows is strong opinions on what you (we, us) should really be doing. Stop reading, if you don't like opinions, strong opinions, or ethics.

<Ramblings, not quite done rambling, yet>

IInspectable
  • 46,945
  • 8
  • 85
  • 181
  • You seem to be focusing on pre-release testing (where there's little doubt about the benefits of Asan) The question is about leaving Asan enabled in release/production builds (e.g. after 1 million customers downloaded and installed your software and NOT during pre-release testing). – Brendan Jun 07 '21 at 07:42
  • @bre I'll cover that part below the `
    `, once I get to it. And no, I'm not suggesting to remove ASan prior to shipping.     – IInspectable Jun 07 '21 at 11:21