This is a cross platform issue - I'm running the 64 bit Linux executable and everything runs smoothly while sitting at 4.6 - 5.2Gb memory usage, until I start flying low altitude over any celestial body, whereupon the memory usage jumps in 0.5 - 1Gb increments every 30-60 seconds until all 16 gigs of memory is full and the swap file starts filling up in a similar manner (just exited with ksp using over 30Gb of ram and swap!) Like has already been said, turning terrain scatter off increases the time before the spikes begin, but doesn't eliminate the problem. The only solution I've found is to keep an eye on the memory usage, and when it starts to spike, save and restart. Luckily it doesn't crash in Linux, just temporarily freezes, so it's more of a severe annoyance rather than making the game unplayable. Any news on when this issue may be resolved?
