That's actually pretty common for rendering machines and people who host servers. Similar setup on my rendering machine and ksp still slows to a crawl/crashes sometimes. Unity's memory limits are a real bummer.
Is there a resolution limit for personal textures? Started creating a texture for my skyscraper at 1024x1024 px but the landing pad and other areas that the player would be close to look very blurry.