format_ are modules for working with mediafiles, and codec_* are modules for transcoding human speech.
In general, if we're talking modules, you should enable all codec_ and format_ modules that you have (and then you will enforce the needed codecs via peer configs), because when not used - they basically do nothing. The modules that you shouldn't enable in a bunch are various app_, cel_, cdr_, pbx_ and res_ that you don't use (because these are doing much of unneeded stuff when loaded but not enabled); but you want to have the widest compatibility with incoming/outgoing speech codecs along with on-disk mediafiles formants that you can provide.