That doesn't only happen in surfcasting. I've seen it happen on a boat too, a distance of maybe 2meters from one guy to the next and only one is getting bites. Same bait, rigs, everything.
I reckon it might come down to scent. When you bait up your hooks you must leave a certain amount of your own scent on it. Maybe something you have handled that day that the fish can smell and is putting them off.
Next time it happens borrow a baited up trace from someone and see if you get any bites.