What should be the guiding factor for a designer when using colors in a garment?

That you patters don't clash or your fabrics make sur it somthing original

You could use a colour wheel to make sure the colours suit one another.

You could ask for opinions.

You could make the patterns and their matching colours originally.

What I would do personally would be to find a colour wheel and choose colours for a shirt/skirt/etc depending on what design the garment was and the shape and size. For example, if it were a mini skirt, I would more than likely choose a solid colour. If it were a longer skirt with - say - polka dots, I would choose two colours that compliment one another like blue and orange, or green and red.