If anyone else is having this problem, it is most likely that you are using physical time warp while you are controlling your craft. So the actual problem is that you are adding trim to your craft accidentally. which is left alt + (q, w, e, a, s, or d) (on windows). Try Hitting left alt + x (wich resets trim) and see if that fixes you problem.