It doesn't matter to me what the images are, but stars are the most common, and I think subconsciously we tend to overrate objects with a star system b/c stars carry a positive connotation.

At least with a Yes/No system, my dislike is correctly weighted.

I have noticed those icons on Mashable's site, but I avoided them.

It doesn't matter to me what the images are, but stars are the most common, and I think subconsciously we tend to overrate objects with a star system b/c stars carry a positive connotation.

At least with a Yes/No system, my dislike is correctly weighted.

I have noticed those icons on Mashable's site, but I avoided them.

I would like to point out that the stars are just an image. Use a different image and you can change the entire experience of rating. I would suggest you take a look at blippr. See it in action here – http://mashable.com/2009/10/06/gmail-accounts-e… You will see small blue icons having the text “=D” alongside site names like Google and Gmail. Hover your mouse on the blue icon and wait a second or so. It is a rating system, but the use of images is quite innovative.

I think the 5 point rating system is just perfect. Many studies on the human mind have shown that our brain is most comfortable in organizing things in 3 or 5 categories. It gets confused with other numbers. In some case 2 categories (simple Yes/No) can be used. But wherever gradings are required and absolute answers do not work – 3 or 5 categories is the best.

I'm not a fan of ranking in a finite list, unless there are concrete data to use, but I think I might be in the minority there. The logic is sound for your plan, call it the eye doctor system; show you artifact A, then ask if B is better/worse, or C, better/worse.

You'll run into issues with coverage though, i.e. if the other options are shown randomly, you may not get enough exposure for all your artifacts to make the comparison valid. For example, A gets ranked 10 times and B 100 times. Not a good data set for comparing.

I also wonder if users would do that without a game; reminds me of that game where you try to match descriptive adjectives of a picture with an anonymous partner. The game is built around creating metadata for photos. Can't recall the name offhand, but that would make rating much less like work.

When you get into complex artifacts like restaurants and apartments, I think you need more granular attributes that roll up into an overall rating. There are too many pieces that make up how you rank restaurants against each other for a single number. But again, you run into user laziness with more input.

I do like the scarce voting model you mention. It creates a better focus for voting, and wrapping the votes around a reputation system (like John mentions) seems like the logical way to get more votes, a la Stackoverflow.

Good stuff. I appreciate the perspective. This is an interesting problem that's fun to discuss, IMO. Stop by anytime.

I'm a developer at a fairly popular ratings website and I've ran across some of problems you mentioned about 5-star ratings (and ratings systems in general). I share your same frustration with Netflix's system as well. There is one idea I've been kicking around in my head, and I'm sure it's not a new one. What are your thoughts on a ranking system in lieu of star ratings? People love to make top 10 and top 5 lists, why not let them do that for the products and services they're interested in. I'm not familiar with your Connect product, so I don't know if it works for you… but hear me out on this, and tell me what you think.

The list of choices will be low (like, say, 3 to 5, and the selected choices will either be randomly chosen by the system or chosen by the user, depending on the nature of the product they are ranking.

Here are a couple of examples of what I mean. Say you have a have a site which you can rate people's ideas (like halfbakery.com; and is that what Connect is?). Raters (rankers) would click through to view the idea, and at the bottom or side of the page, be presented with the option to rank that idea against, say, two other ideas. The number of options has to be kept low in order to encourage higher participation. Then results are aggregated and used to weigh according t rankings to produce the final outcome. For example, user Jack ranks ideas in this order: A-B-C. User Jill ranks the same ideas B-C-A. User Joe ranks those same ideas C-B-A. If you weighed the results as first place: 3 points, 2nd place: 2 points, 3rd place: 1 point, the final results are that B is the best with 7, then B with 6, and A with 5. Very simple concept; it just took me a while to explain.

Similarly on, say, and apartment review website, users can rank their own apartment instead of rating them. But this time, they get to choose what other apartments to rank against since the system does not necessarily know what other apartments they have lived at. (Or if the user has input their rental history, then it CAN display the full, or a limited list to pit the current apartment against).

So I think the advantage of this system over a traditional star rating system is that it eliminates those faults which you outlined in the article. The advantage over a simple vote up/down system is that for a slightly more effort on the user's part, you gain a factor of times more information from the user.

This is what I mean. Say, user Mike has lived at 5 apartments, and he ranks them in order of favorite to least favorite. The system gains 10 data points (i.e. A>B, A>C, A>D, A>E, B>C, etc.). In the star rating system, you only get 5 data points (albeit finer detail per point), but wrought with inaccuracies and bias, as you say. Anecdotely, I find it easier to rank things rather than give them a rating, so I'm more inclined to do a ranking than a rating. Another advantage is you get finer quality data. I.e. Movies A and B may both be 3 stars, but one may actually be a little better than another.

The disadvantages are that on the indivigual (user) level, the final outcome of a ranking are forced to be evenly spaced apart. What I mean is that, there is no way a user can indicate that, for example, restaurant A was just so much better than all the other 4 restaurants on the list. In a ratings system, a user could convey that, but not in the ranking system. What I'm hoping is that with the advantage of more data points, the aggregated data results of the ranking system overcome that disadvantage.

A way to overcome the aforementioned disadvantage would be to do a 'limited point reward' system, like uservoice.com uses. So a user can give 9 votes to idea A, and 1 vote to their second favorite idea B. The only disadvantage I see in this system is that it's slightly more involved for a user to cast their vote than to simply arrange a drag-and-dropable list.

I dunno.. just throwing some thoughts out there.. hope I'm making sense. And sorry for the long comment posting.

Down-voting makes sense sometimes, as you say, not voting and voting against aren't necessarily the same things.

I nearly mentioned reputation, which is my new kick, because this is best way to establish credit in a community. Glad you came to that conclusion without me. It wasn't a test, just good that other smart people think the same way we do. We've code-named our little experiment in reputation 1up. Hoping to get time to build it.

I think star systems have their place, but are mostly relevant for personal ranking systems (e.g. iTunes) or systems where that correlates preferences between users with similar tastes/tendencies (e.g. Netflix). Not sure what to think about downvotes. The problem that occurs to me with assuming that no vote as a proxy for dislike is that no vote could also connote “no interest.” Maybe that's not important; I don't claim to have thought this through too far.

The lack of vote scarcity could be addressed with a reputation system, where users are allocated a certain number of votes based upon both time and level of community participation. One could even work downvotes in there, at a higher level of scarcity, and require a certain rep level before downvoting is even possible. IIRC, that's similar to how Stack Overflow operates. I can't imagine how the karma/rep thing would scale to Digg or Youtube levels, though; it's probably better suited to more bounded communities, like Connect or Mix.