In my mind a lot of things can come into play when catching fish and certain lakes/rivers/streams have colors that always produce better than others. That being said I would rather be in the right spot with the wrong bait, than the wrong spot with the right bait. If you aren't putting your bait infront of the fish you won't catch them, but if you put a bait that may not be the exact match infront of enough fish you will get a dumb one from time to time.
My jig box usually only has 4 colors to choose from:
Black/Blue
Dark Green Pumpkin
White
Brown/Chart.
I will mix things up on trailers size/action as I think the trailer itself can be the most important part of a bait as it is what adds the action to a jig or can make a small jig have a big profile, change the rate of fall on the jig, etc...
I know it is a little off the subject but a caouple years back I was fishing a tourny with a friend and we were throwing the exact same jig/trailer combination both using 20lb test. He had put three keepers in the boat, I hadn;t caught anything. As I was going through what we were doing different I realized he was using Yum craw scent and I was using bang pure craw. I asked him if I could use some of his Yum and on my first cast I put a fish in the boat. We ended up killing fish behind other boats that day, but the fish wouldn't touch a jig if it didn't have the yum scent. It was one of the strangest things that I have seen in person on the water as to how picky the fish were being.