Reference drivers are produced by the company who makes the chipset on your graphics card and will work on all the cards that use that chipset (e.g. RivaTNT reference drivers should work on all TNT based cards).
The actual manufacturer (e.g. Diamond, STB, etc) can then take the reference drivers, and add their own extra features - for example if your card has a special feature, say TV-Out, then these extra features will need to be added to the manufacturers own drivers.
It depends which graphics card you have as to whether or not the reference drivers will work better than the manufacturers own - certainly with Riva128 based cards, such as the Diamond V330, the reference drivers always worked better than the manufacturers own. On the other hand, with the new RivaTNT cards, the manufacturers own drivers seem to be better - swings and roundabouts!
Davey P