Grey area

Many things have a grey area. A controversial area that one side pushes one way and the other side pushes the other way. Lets look at slavery. It's obvious to our society today that capturing the Africans, bringing them thousands of miles from home and forcing generations of them to go into slave labor is wrong, but slavery isn't all bad to our well thought society. We have people in prison today that do labor for pennies a day, and are they not slaves? They work against their will for nothing. Or virtually nothing. Nevertheless, I'd describe that as slave labor and it's okay. I'm not against that, I'm just saying that somewhere between kidnapping blacks and forcing rapists to do work is ok. Where's the grey area? What's wrong or right? Why?