Quick Answer
- You can fit D shaped shaft into round hole if diameters are OK.
- D shaft to D hole without a setscrew will be preferable to D shaft to round hole with setscrew if there is no axial movement. Best is D shaft to D hole with setscrew(s). However you can't always find D holes because they're more costly to manufacture.
- Shear forces act on setscrew if holding torque isn't exceeded. You can check holding torque below.
Setscrew holding power
Setscrews' holding power is proportional to its diameter you can check Marks’ Standard Handbook for Mechanical Engineers, Eleventh Edition, Table 8.2.18. gives holding power for Cup-Point Setscrews. Maximum load you can get from a setscrew from that table is 7000lb axial holding power for 1 in setscrew. The torque is
$\frac{F_{a}}{2}D_{s}$
where $F_{a}$ is holding power and $D_{s}$ is shaft diameter in inches. Sorry I don't have metric at hand.
D shaft
Advantage of D shaft is it prevents shaft from spinning inside the gear hub. So it is preffered in gears.
An example
If you choose No0 screw (1.524mm diameter), axial holding power is 50 lb (222N) so your holding torque is 444 Nmm. If you exceed this torque setscrew will fail but because of D shaft it will hold till material failure (on plastic shaft probably).