Adaptation to Human Populations Is Revealed by Within