I got the same problem during my "atmospheric orbit" thing a few days ago (don't ask). The craft spent 9 hours at least under full physics, without time warp, no drag, and that was enough to significantly move the ap and the pe. However, during half the orbit the ap increased and the pe decreased, but during the other half it was the opposite...
KSP's physics are the best.