You're talking about nuclear thermal rockets specifically. Actually, there's no point in using any propellant other than LH2 -- that's the only reason they have high Isp. Small molecules move faster than heavy ones -- the speed scales as 1/sqrt(molecule mass) according to equipartition. This translates to exhaust velocity (Isp). NTR's propellant is H2, with a mass of 2 amu: that gives it its edge over chemical LH2/LOX rockets, whose propellant stream is mainly H2O with mass 18 amu. NTR's don't run hotter than chemical rockets, so with a heavier propellant (like CH4), the Isp advantage probably wouldn't be there. Barring exotic ideas like gas-core reactors, they're limited by the temperature of the solid reactor fuel, that's acting as a heat exchanger. It's actually a soluble problem though (LH2 boiloff): you can carry along a refrigeration system to keep the LH2 below boiling. In NASA's bimodal NTR concept, it'd cost about 1 ton of mass and 10 kWe power. If you're building large enough rockets, LH2 is a storable propellant. (There's already precedent for this: Hubble has its own cryogenic cooler. It has this ridiculously tiny gas turbine that keeps liquid neon below 70 K, to keep thermal noise out of the infrared camera).