Buying an organic and non-sweatshop product makes a difference.
People suffer under poor working conditions. People suffer applying toxic pesticide by hand without proper protection. No-one wants this stuff to happen - but it does because of money. Profit margins and awareness are the reasons. Once the big clothing producers know there is an alternative, and know people will buy it, and find out it only costs a little bit more; then they will change too.
So supporting organic and non-sweatshop makes a difference - it gets the ball rolling.