I figured out that I was hitting Set Orbit maybe before physics had loaded all the way. I was doing this to avoid the craft exploding on the pad. I put a bunch of launch stabilizers on it and waited for it to settle before hitting Set Orbit and it was happy.
After a certain amount of KAS struts being attached the physics does something weird, I get some phantom forces and the Mk2 lander I am using for a root part simply comes unglued from the fuel tank below it. I turned off SAS and everything before going on EVA. Annoying. But this problem with Set Orbit is solved.
Kurld's post in F12 setting orbit cheat bug? was marked as the answer