Of course, the main problem associated with the direct ascent technique that you have to bring everything you need to get home to the Lunar surface. This means that fuel need for launch and TEI would have to be carried. And that means a heavier and bigger vehicle. And a heavier vehicle means that more descent fuel would have to be carried to bring all of that to the surface. There would have to be a lot of weight compromises.
Or, as Andy44 said, you could go small and use a Gemini capsule. Of course, there are problems with that, too. Even comparing a theoretical Gemini Lunar landing to Apollo 11, Gemini is at a clear disadvantage. A Gemini capsule could not transmit live television pictures for example. Also, there are probably severe weight constaints, too. No experiments could be carried to be left on the Moon, fewer geological and other tools would have to be carried, and the amount of Lunar material that could be returned would be limited by weight and volume. And, obviously, no Lunar Rover. What you get is a classic "flags and footprints" mission.
With the LOR missions used with Apollo, less fuel and consumables would have to be brought to the Lunar surface since a lot of it was left in orbit with the CSM. This ultimately meant more science. And, with the LOR option, NASA found good uses for having a second ship in orbit. One astronaut could take pictures, operate experiments, and (on later flights) use equipment located in SIM bay.
If you look at Apollo 13, a problem in one spacecraft completely crippled it. But they had the LM which was unaffected by the problems in the SM. Having a second spacecraft is a definite advantage in this type of situtation. Something that could not be done with a direct ascent option. Unless every last component was doubly redundant, which, of course, means more weight.
LOR was probably the best option at the time, once rendezvous was proven to be practical.