There are a few reasons why typically going into orbit before leaving kerbin is better than not. All of them stem from the oberth effect.
When you are in a circular orbit, your vertical velocity is zero, and if you start burning prograde, it takes some time before you start gaining altitude fast, at least when compared to launching straight up. When launching straight up, your prograde burn happens as you gain quite a lot of altitude. If an orbiting craft and a vertically launched craft do the same prograde burn, the vertical craft will gain less energy. While the vertical craft is moving up, gravity is taking away energy and speed, so as it burns and moves up, it is adding velocity onto a velocity that gravity has taken away from, thus adding less than optimal energy. (KE = 1/2mv^2, so at a higher velocity, a change in velocity adds more energy.) Whereas the orbiting craft loses less energy and velocity due to gravity as it burns since it travels less of a distance upwards. This means that the burn is adding velocity onto a higher velocity/energy, which means that it adds more energy than the same burn that is done on a vertically launched craft. Higher energy means more velocity when the craft finally escapes kerbin.
Another reason orbiting is better is because it allows for ejection burns to be split into multiple orbits, specifically, this allows the split up burns to be completed at the periapsis, where the craft is moving faster, and more energy can added by a certain change in velocity. For a vertical launch, this is not an option, and the longer the burn in a vertical launch, the less efficient it gets, since more burning is happening at a higher altitude. Also, splitting up the orbit into multiple burns means that a small TWR can be used, which allows for a craft to use less engines, leading to a lower dry mass and thus less required fuel.
