Why does eliphalet whittlesey mention that there are more white vagrants in the area than black vagrants?

The reason that the eliphalet whittlesey mentioned that there are more white vagrants in the area that the black vagrants because most leaders during the era is described to be extremely racist that they prefer whites than the blacks and also because of the fact that the blacks are considered as slaves (black codes).