I think, it could be a reactor utilizing californium-242 (or, at least, weapon-grade U-235) cooled and moderated by heavy water.
Essentially, it were similar to an atomic bomb, but - of course - it would be optimized for stay around the equilibrial state.
The result were probably a very strong neutron source.
I think, it could be used for various things, mainly in the space applications.
Does any cost/size estimations about this ever created?
The RM-1 Russian submarine reactor had a core of less than one cubic metre. It had about 100kg fuel load, which was 90% enriched (i.e. 90kg) Uranium 235. This was liquid-metal cooled [specifically a "eutectic lead-bismuth alloy (44.5 wt% lead, 55.5 wt% bismuth)" - source as below, p40], so didn't need a moderator.
Submarine 901 had in its right-board reactor just 30.6 kg of Uranium 235; this was at 20% enrichment, so a total fuel load of 153 kg.
These were controllable chain-reaction based reactors.
NKS-138 Russian Nuclear Power Plants for Marine Applications
Ole Reistad, Norwegian Radiation Protection Authority, Norway
Povl L. Ølgaard, Risø National Laboratory, Denmark
Published by Nordic Nuclear Safety Research, April 2006
ISBN: 87-7893-200-9
http://www.nks.org/scripts/getdocument.php?file=111010111120029
Radioisotope thermoelectric generators have been used in space for decades, both on the moon missions and on deep-space probes where solar cells would not be practical.
It largely depends on the critical mass of the fuel - that is, the smallest amount of fuel that is needed for a controlled, sustained reaction to occur. This is often on the order of ten or so kilograms, but can vary with the other aspects of the design (see, for example, this paper on $^{238}\text{Pu}$). The Wikipedia page also has a nice summary from various sources of the size of a sphere containing the critical mass (and the critical mass itself) of quite a few different isotopes. The diameters range from 6.9 centimeters to 30-35 centimeters.
The DOE has a page on the size of fusion reactors; one of the expert responses mentions fission reactors and, while admitting that the critical mass can be very low,
A typical critical mass for a fission reaction would be several kilograms. This would be very portable, if it was not for the shielding needed to keep from frying everybody in the vicinity. Plus, you need cooling and heat exchangers to use the heat that is produced. Both fusion and fission reactors put out heat as their primary method of energy production.
The Depleted Cranium blog notes that the first nuclear reactor, CP-1, was really the smallest ever. It used 40 tons of uranium and 400 tons of graphite to be used as a nuclear moderator. This graphite was made into 45,000 blocks, which created a structure like this drawing:
You need a material like graphite as a moderator, and you need it to be in substantial quantities. CP-1 used the smallest amount possible and produced only a small amount of energy. So you're not going to get under about 450 tons if you want to actually generate a substantial amount of energy.
An almost-critical sphere of fissionable material would do. Add some neutron reflecting material that can be rotated to increase or decrease the neutron reflection. Or you could have two halves of a critical sphere that are moved closer together or further apart to control the reactivity.
A moderator is not necessary if the reactor operates on fast neutrons, thermal (slow) neutron reactors are just easier/cheaper to operate and more proliferation resistant.
If you only operate it at low power levels you would not need a lot of cooling (though the reactor would probably not be very useful).
The largest amount of mass would be the shielding. So as long as you don't care about that, you're fine (and in your question you don't mention it). There has been research into nuclear powered cars, aircraft, and trains, but the size and weight of the shielding was the main technical problem for such applications.
The output of nuclear reactions is heat, if you want to convert it to electricity or another more useful form of energy, you will need some kind of heat engine, which (depending on the temperature) needs to dump 75% of the heat input as low temperature waste heat. So you will need some kind of heat sink for that. But again that is not part of the question.
edit: To be more specific, the output of a nuclear reactor is some heat in the reaction products, and most of the energy in the neutron radiation. Reactors usually also capture those neutrons in order to convert their energy to heat, so that is anoher reason why you might want to use neutron reflectors. The most difficult to shield radiation are the gamma rays, which typically require several meters of concrete or a thick slab of lead to block.
