[No blog post last week--celebrating Memorial Day--but back here today, June 1!]
[This is the fourth post in a 5-part series excerpted from my forthcoming book: Find it Fast: Extracting Expert Infor-mation from Social Networks, Big Data, Tweets, and More, 6th. Ed., (CyberAge Books, Autumn 2015) and the final of three posts on the Wisdom of Crowds ]
In the last blog post, I discussed circumstances when doing research on the Web, where relying on the principle of the wisdom of crowds can work well, and where it may not work so well. Here are two other unusual situations where you might wonder if the principle applies.
The first is whether the wisdom of crowds principle work to help with resolving not a factual matter with a clear correct answer, but a value oriented questions of right and wrong? I asked that question some years ago directly to Cass Sunstein in an interview. Here’s what he replied:
“These are questions like should same sex marriage be permitted. Those questions get into a more philosophical territory, but I will say that we can’t exclude these questions from this process by saying they lack right answers, or else we will fall into the trap of relativism.”
On a personal level, I have to say that I was not completely convinced by Sunstein’s answer, and imagine that there are more ways to look at this intriguing question. However, you are unlikely to have to grapple with this as a researcher. Your task will be assessing and verifying factual matters, not coming to any larger determination on whether or not proscriptive answers or decisions from a larger group represent an objectively better way to do things, make policy, create a higher quality of life etc.
When the Process can Backfire
Based on the Condorcet Jury Theorem, the wisdom of crowds process can also run into problems if group members are mostly poorly informed. This can be of particular concern when a group is asked to make a decision or answer questions that require specialized knowledge and expertise. This can occur not so much for common sense applications (like guessing the number of the jelly beans in a jar), nor for popular culture questions, nor when opinion counts more than factual knowledge (asking a group which news story is most interesting).
But, it can become troublesome for matters that require deeper knowledge or skill based hands on experience. Say a popular political blogger posed a question to his or her readers asking “do you think it will be OK if I plant English lavender in clay soil in my cold climate?” In this case, the aggregated response of a group of that person’s readers is not likely going to be, on the average, more often correct than a single master gardener. And so we could expect that in this case the group would not perform so well.
But, interestingly, if we created a group composed of master gardeners, that group could in fact produce a valid Wisdom of Crowds process in that an aggregated, independently cast and decentralized response of a group of skilled expert gardeners (who together likely have a <50% chance of accuracy) will more often than not be likely to be correct than any individual gardener. (Ideally, that group of gardeners would also encompass some aspects of diversity as well in order to fulfill that requirement).
There is probably also one other exception to relying on the Condorcet Jury Theorem for assigning a high level of credibility to a group—and that is when the right or best answer to a problem is known by only a few—less than a majority, and what we can call ‘fringe thinkers”. So for this reason, it’s best to rely on the wisdom of crowds principle when you looking for answers on more accepted factual matters and not for calling on imaginative or non-mainstream thinking.
[This is the fourth post in a 5-part series excerpted from my forthcoming book: Find it Fast: Extracting Expert Infor-mation from Social Networks, Big Data, Tweets, and More, 6th. Ed., (CyberAge Books, Autumn 2015) and the final of three posts on the Wisdom of Crowds ]
In the last blog post, I discussed circumstances when doing research on the Web, where relying on the principle of the wisdom of crowds can work well, and where it may not work so well. Here are two other unusual situations where you might wonder if the principle applies.
The first is whether the wisdom of crowds principle work to help with resolving not a factual matter with a clear correct answer, but a value oriented questions of right and wrong? I asked that question some years ago directly to Cass Sunstein in an interview. Here’s what he replied:
“These are questions like should same sex marriage be permitted. Those questions get into a more philosophical territory, but I will say that we can’t exclude these questions from this process by saying they lack right answers, or else we will fall into the trap of relativism.”
On a personal level, I have to say that I was not completely convinced by Sunstein’s answer, and imagine that there are more ways to look at this intriguing question. However, you are unlikely to have to grapple with this as a researcher. Your task will be assessing and verifying factual matters, not coming to any larger determination on whether or not proscriptive answers or decisions from a larger group represent an objectively better way to do things, make policy, create a higher quality of life etc.
When the Process can Backfire
Based on the Condorcet Jury Theorem, the wisdom of crowds process can also run into problems if group members are mostly poorly informed. This can be of particular concern when a group is asked to make a decision or answer questions that require specialized knowledge and expertise. This can occur not so much for common sense applications (like guessing the number of the jelly beans in a jar), nor for popular culture questions, nor when opinion counts more than factual knowledge (asking a group which news story is most interesting).
But, it can become troublesome for matters that require deeper knowledge or skill based hands on experience. Say a popular political blogger posed a question to his or her readers asking “do you think it will be OK if I plant English lavender in clay soil in my cold climate?” In this case, the aggregated response of a group of that person’s readers is not likely going to be, on the average, more often correct than a single master gardener. And so we could expect that in this case the group would not perform so well.
But, interestingly, if we created a group composed of master gardeners, that group could in fact produce a valid Wisdom of Crowds process in that an aggregated, independently cast and decentralized response of a group of skilled expert gardeners (who together likely have a <50% chance of accuracy) will more often than not be likely to be correct than any individual gardener. (Ideally, that group of gardeners would also encompass some aspects of diversity as well in order to fulfill that requirement).
There is probably also one other exception to relying on the Condorcet Jury Theorem for assigning a high level of credibility to a group—and that is when the right or best answer to a problem is known by only a few—less than a majority, and what we can call ‘fringe thinkers”. So for this reason, it’s best to rely on the wisdom of crowds principle when you looking for answers on more accepted factual matters and not for calling on imaginative or non-mainstream thinking.
RSS Feed
