Why are drill chucks used vs collets to hold tools on the mill?

On the mill, we use collets for holding end mills, as they are stiffer and their geometry is better suited for carrying loads from the side. For drilling, we hold tools in a chuck, which can only handle axial loading. For edge finders, you may use either a collet or a chuck, as they do not bear substantial loads.