I've been re-reading a 2005 report by USDA's Kelly Day Rubenstein and colleagues, which explains the problem:
Without continued genetic enhancement using diverse germplasm from both wild and modified sources, the gains in crop yields obtained over the past seven decades are not sustainable, and yields might eventually grow more slowly (or even decline). Agricultural production increasingly relies on “temporal diversity,” changing varieties more frequently to maintain resistance to pests and diseases.See also a recent article at New West -- "Are the Seeds That Spawn America’s Crops Too Homogenized?" -- and a related report from the National Research Council. The new issue of National Geographic magazine has a feature covering this issue.
Three factors contribute to loss of genetic diversity—habitat loss, conversion from landraces (farmer-developed varieties) to scientifically bred varieties, and genetic uniformity in scientifically bred varieties. The loss of wild relatives occurs mainly through habitat conversion for agricultural use. Habitat loss is particularly problematic in developing countries, which often face greater pressures for wild land conversion than do developed countries. Crop genetic diversity also has diminished as landraces are displaced by scientifically developed varieties. Studies show that far less area is planted to landraces worldwide than a century ago. Finally, crop genetic diversity may decline with reductions in total numbers of varieties, concentration of area planted in a few favored varieties, or reductions in the “genetic distance” between these varieties. Thus far, yields for many major crops have been relatively stable as a result, at least in part, of frequent changes in modern varieties and breeders’ continued access to diverse genetic resources.