I think the success rate of guessing wildly depends on the server. It only becomes 50/50 success rate when the server decides to abandon his patterns and favorite serves, with the understanding that the opponent is guessing. Until that point, a good "guesser" is in a way better position. Ability to read patterns is definitely a skill.
What people (I think) are frustrated with is the fact that they need to drop the patterns, drop favorite serves. That disturbs their whole game. Another thing that maybe frustrating is the fact that when playing a "guesser" correctly, it becomes some sort of a bizarre lottery game instead of tennis. I can imagine people being not happy with the lack of structure in the match and the chaotic nature. But to each his own.
...at that point, this is no longer "anticipating", rather "guessing". Sure, it may appear to the returner that he is anticipating, but reality is that if the server serves "random" then the returner has nothing to anticipate....he is simply guessing under the false impression that there is some sort of pattern.
Another point from personal perspective: When I "anticipate" I will "cover" a certain side by taking a step right or left, a split second before the server makes contact with the ball. If I anticipated correctly - I will get a good return back - maybe a winner, but if I anticipated wrong - I will still be able to slice it back. Now, "guessing" to me is a totally different concept, it is basically when a returner starts ruining to one or the other direction before the server makes contact. That is a pure guess and will not allow you to cover in case of a mistake. Sure there is an element of "anticipation" in the "guess" because if the server does use patterns - there is something for "guesser" to spot, and therefore he becomes an "anticipator".
As far as correlation to reality - I don't think there is anyone on ATP who is guessing by running to one side or the other, but there are plenty who will cover the BH or FH by slightly moving to that direction before the serve.
So if you are frustrated by a "guesser" guessing your serves correctly - abandon the patterns and randomize your serves. This will get you the desired 50/50 success rate. Increase that success rate slightly by serving down the middle or body serves and you will be golden!

Level 13 Edberg and counting...