Heavy metal and pesticide contamination has previously been reported in Chinese Herbal Medicines (CHMs), in some cases at potentially toxic levels. This study was conducted to determine general patterns and toxicological significance of heavy metal and pesticide contamination in a broad sample of raw CHMs. Three-hundred-thirty-four samples representing 126 species of CHMs were collected throughout China and examined for arsenic, cadmium, chromium, lead, and mercury. Of the total, 294 samples representing 112 species were also tested for 162 pesticides. At least 1 metal was detected in all 334 samples (100%) and 115 samples (34%) had detectable levels of all metals. Forty-two different pesticides were detected in 108 samples (36.7%), with 1 to 9 pesticides per sample. Contaminant levels were compared to toxicological reference values in the context of different exposure scenarios. According to a likely scenario of CHM consumption, only 3 samples (1%) with heavy metals and 14 samples (5%) with pesticides were found with concentrations that could contribute to elevated background levels of contaminant exposure. According to the most conservative scenario of CHM consumption, 231 samples (69%) with heavy metals and 81 samples (28%) with pesticides had contaminants that could contribute to elevated levels of exposure. Wild collected plants had higher contaminant levels than cultivated samples. Cadmium, chromium, lead, and chlorpyrifos contamination showed weak correlations with geographic location. Based on our assumptions of the likely mode of consumption of raw CHMs, the vast majority (95%) of the 334 samples in this study contained levels of heavy metals or pesticides that would be of negligible concern. However, given the number of samples with detectable contaminants and the range between the more likely and more conservative scenarios of contaminant exposure, more research and monitoring of heavy metals (especially cadmium and chromium) and pesticide residues (especially chlorpyrifos) in raw CHMs are advised.