The two major problems are scale and lighting. You'd only get light from the center of the system, the stars wouldn't emit any light. The problem with scale is that you'd have to shrink stuff together a bit. It works and has been done for some add-on, as the aforementioned star treck.
Depending on how many stars you want there's a third problem, which is performance. But if you keep the amount sensible and don't mind compressed distances and odd lighting, there's no problem whatsoever. You can even give the planets moons, as orbiter supports moons of moons.