Category Archives: Me

Degree3 is in Beta!

I apologize for the silence lately, but I’ve been cranking pretty hard on my new project, Degree3 Q&A.  It’s a social Q&A system that sites can quickly and easily integrate into their site, helping visitors find answers more easily than using comments or forums.  I’ll be talking more about it later, but for now, if you’re interested in trying it out, drop me a line or apply for our Private Beta at Degree3.com.  In the meantime, try it out on this page, just to the right of this post, and let me know what you think!

MIT Tech Review: Worst review of the Best Q&A Sites

When someone writes for a publication like MIT’s Technology Review, they have an obligation to write articles that are objective and scientifcally sound. To represent a brand like MIT, they have to observe the standards of review journalism such as creating measurable comparison criteria, applying those standards consistently, and giving consistent, even-handed treatment to their subjects. However, Wade Roush of Tech Review last month ignored all of these rules in his article What’s the Best Q&A Site?

Perhaps it was Roush’s objective to take a light-hearted look at the Social Q&A space and therefore was lax in his editorial rigor, but if that’s the case, his review should have been published on a blog somewhere, not on Tech Review, and it should have had the appropriate disclaimers. When MIT Tech Review publishes an articles with hard numbers comparing websites, that review becomes gospel for the hordes of other sites that reference it, so it had better be accurate.

I will show here why that article never should have been published by the MIT Technology Review.

Before we begin, here’s my big, fat disclaimer: I founded Answerbag, one of the subjects of the review, so I’m hardly unbiased. However, to be thorough, I will show here that Roush’s review of all of the QnA sites — not just mine — were cursory, unbalanced, and inaccurate.

I’ll start with Roush’s description of his test:

I also devised a diabolically difficult, two-part test. First, I searched each site’s archive for existing answers to the question “Is there any truth to the five-second rule?” (I meant the rule about not eating food after it’s been on the floor for more than five seconds, not the basketball rule about holding.) Second, I posted the same two original questions at each site: “Why did the Mormons settle in Utah?” and “What is the best way to make a grilled cheese sandwich?” The first question called for factual, historical answers, while the second simply invited people to share their favorite sandwich-making methods and recipes.

The five-second rule in basketball actually has nothing to do with holding, but I’ll let that one go.

I awarded each site up to three points for the richness and originality of its features, and up to three points for the quality of the answers to my three questions, for a total of 12 possible points.

Okay, so here’s the fun part: three points for “richness” and “originality” of its features – these seem like awfully vague terms to use, but it’s only 3 points out of a total 12, so I suppose it’s not that big a deal. He finishes by awarding up to 3 points for answers to each of his three questions, based on the quality of the answer. That seems fair.

Since his review starts with my site, Answerbag, I’ll dig into his review of Answerbag first.

Members get points for asking and answering questions as well as for rating other members’ questions and answers.

Somewhat true. Members can receive points for their questions and answers from other members, but they do not automatically receive points (as they do on some other Q&A systems, a difference that is key to the quality of the answers.) Also, members do not receive points for rating other members’ Q&A. But hey, he says he only had a few days to write this review, so I can’t expect him to get all the details.

He goes on to talk about our Levels system and our Widgets, but never discusses the core differences between Answerbag and other sites, such as the fact that we never close our questions to new answers or ratings on our answers. We also get no credit for pioneering video answers or image answers. We get no credit for allowing our users to sort the questions in a category by popularity so they can easily learn about a topic like “PlayStation 2 troubleshooting” or “foreclosures.” No credit for a unique profile page that lets users see all of the latest activity on the site that involves them (answers to their questions, ratings on their answers, comments on their answers, etc.) No credit for giving more power to users who have earned a lot of points on the site. Then he gives us 1 point out of 3, apparently for not having “rich” and “original” features. Ouch. Well, it’s a subjective measure, so I’ll let it go.

Is there any truth to the five-second rule? All of AnswerBag’s answers about the five-second rule pertained to basketball. Points: 0

Busted. But then, to our answer for why the Mormons settled in Utah, he has this to say:

That’s more or less in line with the best answers to this question at other sites. Points: 1

So…you’re telling me that for an answer that was just as good as the best answers on other sites, we got 1 point out of 3? Is that fair in the review guidelines set forth by MIT’s Tech Review?

To his question on “What’s the best way to make a grilled cheese sandwich?”, he says “I rated the answers to this question purely according to their mouthwateringness.” Very scientific. He received six answers, and gave Answerbag 2 points out of 3 with no explanation whatsoever. I guess they just weren’t mouthwatering enough to get the full 3 points.

Amazon’s Askville

3 points for features – he liked the mechanics of the site. Fair enough, since it was established as a subjective measure.

0/3 points for having no answers to the “five-second rule” question. Bummer.

On the Mormon question, Roush cites a couple of the answers, and then gives 2 points out of 3, with no explanation of what was wrong with the answer, or what a correct answer is.

On the grilled cheese question, he got a couple answers that “provided just the basics” as well as some that were taken from “grilledcheese-contest.com” but complains they were “not very original.” Perhaps if he wanted people to submit their own personal, original grilled cheese recipes, he should have asked for that. Poor Askville only got 1 point out of 3 here thanks to Roush not asking for what he wanted.

Live QnA

For a QnA system that is virtually identical to Yahoo’s, Askville’s, or Yedda’s, MSN gets 1 point from Roush. He complains that as users earn points for their answers on the site, they can raise up in levels but “there are no other rewards.” This was his only criticism, and Yahoo’s point system works virtually the same way. But, MS only scores 1 point from Roush. Maybe he thinks Windows crashes too much.

2/3 points for answers about the “five second rule.” No explanation of what the answers were missing to keep them from getting 3/3.

2/3 points for answers to the Mormon question, again with no explanation of what’s wrong with them.

For the grilled-cheese question, he got three answers, including one he says “sounded delicious,” but I guess it didn’t sound delicious enough. 2/3 points.

Wondir

Roush must have been approaching his deadline. He gave an overview of Wondir’s features, and then gave them 1/3 points with no explanation of what he didn’t like about it at all.

Five second rule: 0/3 points because it was very hard to find answers. Fair enough.

Mormons settling: He got “six answers, 3 of which were useful.” No explanation of what was missing. 2/3 points.

This one is great: for the grilled cheese question, he changed his question (for no apparent reason) to ask how to make a healthy grilled cheese sandwich. He got answers he considered “brief and obvious” like using wheat bread, lowfat cheese, and margarine. I have no clue what he would have considered a good answer. Those answers are the best things I could think of to make a healthy grilled-cheese sandwich; if you’re cooking something with bread, cheese, and butter, what else can you do? Yet, Roush was not satisfied, and this answer got 1/3 points. Brutal.

Yahoo Answers

Roush starts by explaining how Y!Answers has similar features to other sites. He praises their “My QnA” page, but doesn’t acknowledge that almost every other site in the comparison has a similar page. Then he seems to get excited:

One fun twist: users can choose and customize their own cartoon self-portraits, which appear alongside their questions and answers and give the site a surprisingly jaunty feel.

How much is that worth? 3/3 points. THREE POINTS. For an avatar system that anyone who’s used Yahoo (particularly Yahoo Games or Yahoo Instant Messenger) knows has been a feature of Yahoo (not Yahoo Answers) for several years. Somehow their avatars added so much to the experience that he thought Yahoo had the BEST FEATURES out of any of the other QnA sites in the review (tied with Askville.) Perhaps it’s his first time using the web?

Five second rule: This question had been asked 11 times and had 160 answers. It seems that the time it took him to find a good answer under the 11 iterations of the same question was not a hindrance – after reading through the 160 answers (yawn), he apparently liked one that referenced a show from the Discovery Channel as a source. 3/3 points. Alright.

Mormons settling in Utah: Again, Roush inexplicably changed the wording of his question, adding a bit about “why didn’t they go on to California or someplace more desolate.” (Rule #1 of doing experiements: perform the SAME experiment on every subject, Wade.) He got one “offensive” answer, four “cursory” answers, and one he considered good enough for a 2/3, again with no explanation.

Grilled cheese sandwich: This time he just searched for existing answers, giving no word on why he didn’t do this on other services, or why it was good enough to look at existing answers on Yahoo rather than testing how quickly he could get answers to a new question as he had done on other sites. He seemed to be pleased that he had to look at over 100 copies of the same question to get a recipe. Something in there must have sounded yummy to him, but he doesn’t tell us much besides the fact that he found thousands of answers. Yahoo Search returns millions of results to many queries, but that doesn’t make it the best search. Roush gives Y!Answers 3/3 points regardless.

Yedda

His evaluations of Yedda’s features culimates in “In practice, however, I couldn’t see much difference between the answers at Yedda and those at Live QnA, Yahoo Answers, or the other sites.” For being the same as those sites, Yedda only gets a 2/3. No futher explanation offered.

Five second rule: Nothing. 0/3.

Mormons: One answer that didn’t ad dress the question. He’s nice and gives Yedda a 1/3 anyway. How generous.

Grilled-cheese: He got one answer that focused on the cheese used in the sandwich. 1/3 points.

Conclusion

How this article got through MIT Tech Review’s editors is beyond me. Time and time again Roush gives random points to these sites without even saying what he liked or didn’t like about them. He never even established the “correct” answer for the five-second rule or why the Mormons settled in Utah, so we, his readers, have no basis of evaluating the answers he received. At the very least, he could have told us what was wrong with the wrong answers.

Here’s what a good review would look like: come up with a set of real, measurable criteria like: question answering time, accuracy of answers, and intuitiveness. For certain types of questions that are looking for a variety of answers, the number of answers received may be a useful metric (such as grilled-cheese recipes.) It might be nice to see a feature comparison chart of the sites in question, as well as a discussion of why those features are important. You could, perhaps, give one point per useful feature out of a fixed set of features.

Most importantly, apply these measurable standards even-handedly to every subject, and explain why they receive or don’t receive points. These are the basics.

Am I upset just because Yahoo won? No. I’m upset because the review was cursory and didn’t do justice to any of the sites involved. Maybe it was meant to be fun, to be a “jaunty” exploration of Social Q&A, but is Tech Review really the right forum for that? I’m not going to pretend I’m unbiased in the matter, but Roush’s review should not have been represented as a measure of The Best Q&A Site by a well-respected publication.

Answerbag sold to Demand Media. Google buys some online video site

That’s right, last week Demand Media bought my site, Answerbag, from InfoSearch Media, and I will be going along with it. It was just eight months ago that I sold Answerbag to InfoSearch, and now we’ll find a new home again on Santa Monica’s 3rd Street Promenade with Demand Media.

The deal is somewhat bittersweet for me. I am excited to be working with Demand Media, particularly with Rich Rosenblatt and Joe Perez, both of whom I worked with at Intermix Media. They have a large network of sites that will open a lot of opportunities for Answerbag, so that’s a Good Thing.

I am, however, saddened to be leaving InfoSearch Media. The team at ISHM understand the Q&A space as well as just about anyone, with several execs coming from Ask Jeeves including George Lichter (CEO), Claudio Pinkus (Chairman), and David Warthen (CTO, former CTO and co-founder of AskJeeves). Alas, I go with Answerbag, so I’m starting tomorrow at Demand, and I’ll be looking to build Answerbag as much as possible with Demand’s support! They have demonstrated a strong commitment, and indeed they were willing to shell out $3M for it, so I believe Answerbag and I will have a good home.

I am happy for ISHM as well. They made 2x-3x on their investment in Answerbag, and now have a nice little pool of cash to fund their future projects. As George put it, “InfoSearch Media is small company with limited resources — we didn’t want to under nourish either Answerbag or ContentLogic. We saw the opportunity to work with Demand Media as a way to ensure that both groups could have what they needed. It’s kinda like having kids and learning to let them go. A tough decision though, to be sure.”

On to world news – after months of specuation, Google buys YouTube! Google has a lot of smart people, and they must have some secret plan for it, but I just don’t understand the purchase. Youtube has 2 strengths: Brand awareness and traffic. It has no technological or design differentiators over other video sharing sites. What makes YouTube special? Nothing, really. There are dozens of other video sharing sites that do the same thing, but YouTube is just the best recognized one.

So why would Google buy them? They don’t need the traffic, and their brand is already better than YouTube’s. The only thing that makes sense is if they’ve finally figured out a business model for YT, one that has eluded YouTube execs for two years now.

The most obvious route would be to drop video ads before or after the video clips they serve up, and given Google’s recent history of trying to broaden their AdWords/AdSense advertising platform to other mediums, this would be my guess.

The challenge for GooTube will be to integrate ads in a way that doesn’t drive people away from the service. Google revolutionized web advertising once before when they started using text ads that actually performed better than the increasingly annoying graphical ads, such as the infamous “Punch the monkey” or X-15 wireless camera ads. Perhaps this will take the shape of AdSense-style text ads showing at the end of videos. This would reduce the inevitable annoyance from graphical or video ads, and it would allow their existing customer-base to target their ads to YouTube videos with no extra effort.

So, yes, this has great revenue potential for Google, but enough to justify a $1.6 B price tag? And Google is sure to get sued for the copyright infringements on YouTube that went unpunished up until now because YouTube had no cash flow to go after. Only time will tell if this is a good move for Google or a debacle, but it will obvious one way or another in the next year or two.

What’s wrong with the world

I discovered early in my life that I have a gift for criticizing things. No matter what you’re talking about, I can find a downside to it. If I won the Lotto, I’d complain that I have to pay 1/2 of it in taxes, and that Bush will probably spend that money paying Halliburton to build another prison in Iraq, next to the still broken-down water purification plant. When Christmas rolls around, I lament the commercialization of the holiday and how sick and tired I am of hearing the same carols year after year. When I travel, I criticize the ever-changing security procedures at LAX and how poorly documented they are, so you only know what’s going on if you actually ask a security guard.

Does all this mean I’m a pessimist? Perhaps. You can make up your own mind from this blog as I progress.

I’ve been working on this creature we call the Internet for 10 years, and only now am I getting around to starting my own blog. To tell you the truth, I don’t really like blogs. I don’t even read my friends’ blogs, and honestly I’m not expecting my friends to read this blog. There are far too many bloggers who go on and on about their family or what they had for lunch that day or what movies they want to see this weekend. I only care about such things if I’m already a close friend of yours, and if I am a close friend of yours, I’d rather talk to you to catch up with your life than read about it on the web.

Perhaps we can chalk that up to my ADD. We’ll discuss that later on. Personally, I think it’s because you don’t get the “color” by reading something on the web – if I want to learn about your life, I want to hear what you have to say, ask you questions, and have a real conversation. Reading a blog is primarily a one-way conversation, aside from the highly asynchronous “comments” that lend an iota of life to the whole institution.

But here I am anyway. And now that I think about it, why am I doing this? I believe that, despite any pessimistic leaning, I have some observations on the world that I’d like to share. Particularly, observations about Media, Marketing, and Technology, probably the three professional/industrial worlds that I’m closest to in my everyday life. This unholy trinity does have some great things to offer, but they are also abused, and I’d like to take this opportunity to examine how MM&T affect our lives. More often that not, I find that they have a bigger impact than I first recognized.

My next post: Why Wikipedia just doesn’t matter