Category: Decision Engine

Pres12_WTA
August 16th, 2013 by john.warden@gmail.com

Today I have been thinking about Prediction Markets, and thought I would share some thoughts on how the Decision Engine could be designed to behave very much like a Prediction Market.

If you aren’t familiar with prediction markets, check out this explanation from the world’s leasing prediction market, Intrade.

Verifiable Events

Prediction markets rely on events that will at some point in the future be objectively and unambiguously verifiable (e.g. who won a game, who was elected president). The Decision Engine, on the other hand, is designed for making predictions or decisions on questions that may be matters of opinion or judgement, with no external objective standard for judging correctness. So how could the Decision Engine resemble a prediction market?

The Consensus Index as Market Price

The answer is the consensus index, which is a measure of the consensus on any question. As a discussion proceeds and supporting- and counter-arguments are introduced, discussed, validated or discarded, people’s opinions will change. But (and this is a key aspect of the Decision Engine’s design) it will be calculated not as the percentage of participants that agree/disagree, but the probability that a participant will agree/disagree after reading all the arguments.

The Consensus index will tend to change over time, especially as people introduce convincing arguments. However, as all the arguments are hashed out, I believe a “consensus” index will eventually stabalize.

It is this difference between the initial, pre-argument consensus index and final consensus index (determined by some stopping condition such as time or stability) that makes for the possibility of an interesting prediction market.

Each question could have an underlying virtual security that can be redeemed at a price equal to, say, $1 times the consensus index at some stopping point (e.g. after a certain amount of time, or a certain level of stability). If you believe that by introducing a convincing argument, you can cause the consensus index to fall (and remember, the consensus index is calculated based on the percentage of people who agree/disagree after reading yor argumets), then you have an opportunity to earn points or (virtual) cash by shorting that security!

Example: Common Myths

Common myths make intersting cases for the decision engine, because they should result in a consensus index that changes dramatically after somebody has introduced an argument debunking the myth (which I think should be as easy as linking to Urban Legend or Snopes or Wikipedia).

For example, suppose this question and answer has been posted to the decision engine:

Question: who was the first European to arrive in America?

Top Answer: Christopher Columbus (Initial Consensus Index: 60%)

This is a fantastic opportunity! You know it’s a common myth that Columbus was the first European to discover America (taught in school’s even). All you need to do is short this security, post an argument linking to any authoritative source on Christopher Columbus, and you can bet that the Consensus Index will fall and you’ll make some money!

Moving the Market with Arguments

Not every discussion will be as simple and clear cut as a historical fact. However, even for more complex arguments, facts will matter, as the Decision Engine will ask people for their reasons for supporting a particular positions, and reasons supporting these reasons, and at some point each argument will be supported by a base set of assumptinos that may be more clear-cut. If some position on a complex topic is supported by a reason that is de-bunked, or on a reason that is supported by another reason that is debunked, the engine will adjust the consensus index accordingly. In this way, it may sometimes be possible for one person to topple a whole argument structure that was supported on one weak but commonly accepted assumption, causing the consensus index of a big argument to swing wildly, and make a fortune.

Posted in Decision Engine

globe and wires
June 21st, 2013 by john.warden@gmail.com

In my last post, I introduced the general idea behind the decision engine. This post walks you through an example of what the decision engine might look like from a user’s point of view, and hopefully give you an idea of how it could result in more intelligent group conversations.

You see a sceen that says:

Consider the following statement:

“Romney would make a better President than Obama”

Do you think most people will:

Agree | Disagree

What would you guess? Well, if it’s the months leading up to the 2012 presidential elections, then you know that people are pretty divided on that subject, but you know Obama is ahead in the polls. So you select “Disagree.” You now see this:

CORRECT! Only 47% of people (ironically) agree with this statement

You earned 100 points!

Now, how about you? Do you:

Agree | Disagree

Suppose you agree with the majority that Obama is the better candidate, so you click “Agree”

Thanks for your opinion. You earned an additional 10 points.

Now, earn additional points by responding to one of the arguments below, or posting your own.

Reasons

Wow, it’s easy to earn points in this game. And now I’m curious about what people have to say. So I look under the comments section, and I see this:

Which do you think is the top reason that people DISAGREE:

    • He’s a Republican
    • Mormons are Polygamists

Interesting. I know that Mormons no longer practice Polygamy, and that most (I hope) people know that. I think a lot of people are pretty partisan and choose Obama just because he’s a democrat, so I choose the first.

Sorry! 58% of people that DISAGREE with the original statement chose this as a reason.

Now how about you? Is this one of the reasons you agree?

Yes | No

Convincing People

Darn! Humanity has disappointed me. But I don’t give up without the opportunity to express my opinion. So I click “No.”

Thanks for your input! You earned another 10 points.

Challenge Opportunity

You now have the opportunity to challenge the majority and earn 1000 points. Submit your counter-argument below, and you will gain points for every person that is convinced by your argument.

So I write up my argument:

SUMMARY: That’s a Myth

Okay looks folks, the Mormon Church did away with Polygamy in 1890. Get with the times! And although there are some offshots of the mainstream Mormon church that still practice it, it’s pretty rare. You can find it all in this article.

http://en.wikipedia.org/wiki/Mormonism_and_polygamy

I click submit, and then go on with my day.  The next day, I receive an email with the following email:

SUBJECT: Congratulations! You have convinced the majority. You earn 1000 points.

Only 45% of people are convinced by the argument “Mormon’s are Polygamists“, after reading your argument “It’s a Myth“.

Wow! I didn’t expect that. I changed the percentage of people that were convinced by the silly “Mormon’s are Polygamists” argument from 58% to 45%?

Will People be Rational?

How did that happen?

Well, first of all, a lot of people simply don’t know a lot about Mormons. That’s good for me, because ignorance can be cured with information. For some, just reading my comment or reading the Wikipedia article was an eye opener, and was enough to convince them.

Second, the system was asking new participants on the discussion what they thought after reading your comment. These new participants are less likely to be entrenched in a position, and will not take a position until having read your argument. The fact is, a group of people just asked to answer based on what they know will answer, no average, differently than a group of people that are first asked to reads relevant information.

Third, participants will be forced to consider whether other people will find your counter-argument convincing, before taking a position. This helps replace a knee-jerk reaction with a thought process such as “will I get shot down if I take this position? Is that statement really true? Is that article reliable? Will there be a strong counter-argument?”

Finally, participants have the opportunity to gain lots of points if they correctly guess whether the argument and counter-argument will be accepted or rejected. If they think that the “Mormon’s are Polygamists” argument is ultimately weak, even if it is an argument supporting their position against Romney, they will not only be encouraged to reject the argument but even to pile on their own counter-arguments. If they want to convince people not to support Romney, they will find a stronger argument and support that.

So I hope this example gives you an idea of how the right conversation structure and scoring mechanism can encourage people to act rationally and produce more meaningful, intelligent conversations. I won’t know if this particular process really works until testing with real prototype, but these are the lines along which I’m thinking.

Posted in Decision Engine

delphi
June 20th, 2013 by john.warden@gmail.com

My big project right now is something called a “decision engine”. Put simply, a decision engine is:

“a conversation-based process for group decision making”

At its simplest it is a comments system (like the comments on the bottom of this blog post) that facilitates better online discourse, by adding a layer of structure and process designed to unlock the potential of the group to arrive at positive, useful results — a mechanism for aggregating the collective intelligence of a group. The process ensures that the contributions of each individual are fairly and thoroughly vetted by others in the group, and that relevant information and useful arguments are surfaced and discussed. It requires the final decision to be supported on a solid foundation of reasons, and for those supporting that decision to defend those reasons from counter arguments from other participants in the group — or see the decision reversed.

The process uses principles of game mechanics to create the right motivations, awarding players for introducing and defending reasons for supporting a decision, but also for acknowledging valid counter-arguments, and abandoning an argument if they see they will not be able to defend it.

The decision engine can be used by crowdsourcing projects to harness the wisdom of crowds to create economic value.  It can be used in online Q&A communities to more effectively extract knowledge from a group. It can be used to improve the quality of political discourse, surfacing the strongest arguments on all sides of an issue. It could be used as an alternative process for legal arbitration.

The decision engine is based on a few key assumptions about human behavior: that given the right motivations, most human beings will be willing to give honest, thoughtful opinions and judgements, and most importantly, be willing sometimes to change those opinions in the light of new information. You may not think this sounds like a description of the human beings you are familiar with, especially the sorts of people involved in discussions on the internet. But it may simply be a question of process.

The Process

The process I envision asks people not just for their opinions, but for their reasons — and it requires them to defend these reasons for their opinions to have any weight. Participants can introduce counter-arguments questioning the validity of a reason, and the other side must respond or abandon their position. As the conversation progresses, and new participants enter, weak positions will be abandoned, and the most convincing arguments on both sides will rise to the top.

The final decision will not be based on a simple vote: a participant may support a certain decision, but if they cannot provide a reason, or defend that reason, their support will have no weight in the final results. Instead, the final decision will be based on argument threads, and how many people still support a decision after after reading and responding to the entire argument thread.

A key to making this process work, is that participants may gain points for successfully attacking a weak argument thread even if that argument ultimately supports their own position on a decision. If your friend makes a fallacious argument, you will have nothing to gain nothing by supporting it — instead you can earn easy points by siding against a weak argument that will ultimately fail, and still gain the big points by defending the overall decision with a stronger argument.

People won’t always come to unanimous agreement in the end. But the strongest argument threads on both sides will be surfaced. And only then will the system determine the final decision based on the opinions of participants after taking their positions on the final argument threads.

Gamification of Group Dynamics

Researchers have learned a lot about group dynamics: the simple things that can turn a group of average individuals into a super-intelligent phenomenon or an irrational mob.  We’ve also learned about game design, and the simple things that can convert a task from a chore into an addiction.

The decision engine will award points to participants for sharing not simply what they believe, or for making popular comments, but for estimating what arguments and counter-arguments they believe other participants will find convincing. This will I believe change the dynamic of conversations, making the process into a compelling social game and directing participants mental energy towards truly questioning their opinions and their basis for them.  Participants will learn to anticipate future arguments and counter arguments and make bets on which arguments will ultimately be found to hold water after being hashed out. Participants’ anticipation of other participants’ reasoning will result in the sort of positive group dynamic we know is possible: a feedback loop of self-reinforcing rational behavior.

In future posts, I’ll write more about what this actually looks like, as well as more about the economic potential and philosophy of the decision engine. And in the meantime, I will be working on a prototype.

Posted in Decision Engine