Thanks for the replies.
For clarification, by 'upright' I mean pointing at normal (in my equatorial orbit this means north), which, as Reactordrone points out, should maintain that orientation.
I have no joystick, so not that. I'll try turning the trim off later (as Snark suggests) and see if that works. However, quite few weird orbital decays have been appearing in my game since 1.1, so it is probably connected to the bug that swjr-swis mentions.