My question was not phrased completely correctly. I slightly simplified the case. I am using a Spinner, not a Button, to transfer to the next Activity.
Spinner (and Gallery) have a gross bug, not mentioned in the docs - the OnItemSelectedListener event handler is called when a user physically clicks the spinner control, and also when a spinner is first laid out by the framework code. Your spinner handling code must therefore determine if an event was triggered by a user selection or by the spinner being laid out. The easiest way to do this is to make the first item in a Spinner always be "no selection made yet", and to ignore all events on that selection.
See Android Spinner selection and similar posts.
In my case, the orientation change caused the spinner to get laid out again, and I thus got two events from it, the first the layout event, the second from the previously selected entry. And that caused a bogus second activity to be started, and that meant that 3 presses of the back button were needed to "get back" to the first Activity. It was actually going back on the first press, then the spinner fired a layout event and a regular event putting me in the second Activity twice. That wasn't seen on the screen, but was seen using log messages.
When you change orientation, the current Activity is destroyed, and a new Activity created/started.
When you change orientation and press the back key, the previous Activity is popped from the top of the paused stack, destroyed, and a new version of that Activity created/started.
When you change screen orientations, the Activity in the old orientation is never kept. It will be destroyed immediately, or if it is lower down the Paused stack, it will be destroyed when it comes to the top.