I never had a problem listening to very different genres of music (as long as I like them), or felt one meant I should avoid other. Perhaps if I had grown up listening exclusively to one genre things would be different.
I read different authors, too. But I have felt something similar, caused, perhaps, by different circumstances.
I do not have a lot of time to read, so I have to choose what to read carefully, and I definitely cannot read "everything there is". So, when given the choice between a book I really want to read (because I like the author's work, or the series, or whatever) and some other book (different author, different series, or whatever), I know which will win.
Music has the advantage that it takes 4 minutes to listen to a song (or a fragment, if we're talking about a symphony). So I do not have to choose "one or the other". I can have both.
I think the closest thing to that feeling happens to me with Marvel Comics (my favorite) and DC Comics (which I like, but they are not Marvel Comics). I do read DC, and I like it, but the rate of Marvel over DC I read is greater. And there have been times when I may not have read something just because it was DC (I was much younger then, though).