I think another cool idea would be implementing the movement mechanics of EVA somehow, allowing the viewer to "float" instead of just move. What i mean by that is maintain inertia, you know, like in EVA.
I'm just now finding this thread via google, but im having the same problem as surge. Im running Linux x64 ksp and my IVA camera rotation is incredibly slow. Not laggy, just slow.