It's a basic truth that the more deeply something speaks to you, the greater chance it has to be completely lost on someone else. More often than not, those things go hand in hand.
The examples of things considered to have "artistic worth" that also speak to the masses are few and far between: The Beatles, Casablanca, Frank Lloyd Wright, Monet. It's just not very often that the Billboard #1 album and the top rated album on Pitchfork.com are one and the same. That site is driven so much on exclusivity and cultural elitism it wouldn't be able to function in a world in which that was possible anyway.
Case in point: as a fan of jazz and blues (which I gather from your interest in Mingus, James and co), what's today considered "indie rock" probably doesn't interest you all that much, yet it means a lot to some people. Thirty years from now, guys like me will wonder why no one remembers Wilco, the Dandy Warhols, Elliott Smith or The Mars Volta... just as you probably wonder how anyone could have gone through life without hearing "Dark Was the Night, Cold Was the Ground."
I think there's a fine line between cultural elitism and seeking out the best and taking necessary steps to appreciate good or great things when you find them. On the surface that can look very similar. Honesty is probably the primary sticking point. Cultural elitists prop up stuff that makes them feel smart and, well, elite. When pressed, they can't properly defend their values or interests. The point of finding the best is pretty simple: you can derive more pleasure from things of high quality.
Seems obvious, but when you're your speaking of subjective interests, people tend to want to say (especially when pressed), 'You can't say that something is good or bad - only that you do or do not like it.' So, say person A's favorite song is 'Smell Like Teen Spirit' (to pick an easy target), and person B's favorite song is 'Coney Island Baby' (to keep this football-ish related) by Lou Reed. Person A can genuinely like the song and connect to it on some emotional level of angst, but beyond that, the song doesn't offer any nuance or opportunity for growth. It's angst in a bucket. 'Coney Island Baby' is crafted in such a way that you can experience it in different lights depending on where you're at in life. There are layers that work together to form a cohesive whole that is genuine, accessible, and difficult. 'Good' and 'bad' doesn't really have a lot of meaning in the world of art (in my opinion). How fully can someone enjoy the given piece of art? That's what matters.
This isn't to say that everyone should enjoy the same things. Each person - whether he/she knows it or not - has his/her own aesthetics and experiences that shape his/her receptiveness to a given piece. I used to be unable to comprehend how anyone could place Lou Reed anywhere near Bob Dylan's level. Now, my feeling is the inverse - except I do understand, having been there. If you're seeking out the best, though, instead of what someone has shoved down your throat, or instead of something that just makes you feel hip, intellectual, etc., you're going to find works of art, aspects of life, etc. that you enjoy much more than you'd be able to enjoy something commonplace, pretentious, etc.
For whatever reason, most people don't want to put effort into enjoying life.