I have no idea how to measure the error in orientation, but the control algorithm for my previous autopilot mod depended on the angular velocity. The desired angular velocity about whichever axis depended on the error with respect to that axis, and I varied the angular acceleration to maintain the desired angular velocity, so as you're rotating it smoothly just decelerates on the desired direction, and if the ship is rotating like with its orbit then it just maintains the angular velocity. But I don't know how to use quaternions to accomplish that. I just used the down direction and right direction and forward direction to compare to the orientation. And the orbit direction. But I could never figure out how to correct the error like the game itself does. Because it only seems to care about the relative rotation or something, but mine could only depend on the front, down, side directions. And it doesn't seem to care about the roll axis, only yaw and pitch. But that makes it even more confusing for me because I couldn't figure out how to find the desired direction in absolute coordinates, unless it was the orbit direction or something. But the error depends on the angles relative to the forward direction.