What you're looking for is a ballistic missile trajectory, since you're looking for the most efficient sub-orbital ellipse that intersects with the ground at an arbitrary ground distance. Someone was nice enough to scan in the full copy of Bate, Muller, and White. Written by three professors of astronautics of the US Air Force Academy, this is literally the textbook on astrodynamics used by almost everyone everywhere, and contains all the math you'd need to suborbit, orbit, or transfer virtually anything anywhere in system, in real life. Equations will be on page 277 (pg 292 of the pdf), Chapter 6, ballistic missile trajectories.