An Artificial Intelligence Agent for Texas Hold'em Poker
Patrick McCurley
University of Newcastle Upon Tyne

Abstract:
Poker is currently the world’s most played card game. Hundreds of thousands of people play
poker every day, and can play in a real life environment or over the internet using a distributed
application running a simulation of the game.
One of the biggest reasons for poker’s recent success is its fundamental dynamics. The ‘hidden’
elements of the game means players must observe their opponent’s characteristics to be able to
arrive at good decisions, given their options. A very good poker player will consistently
dominate a sub-optimal opponent, although stochastic elements apply heavy statistical
variation to the game, allowing weak players to win occasionally.
The game of poker offers a well-defined domain in which to investigate some fundamental
issues in computing science, such as how to handle deliberate misinformation, and how to make
intelligent guesses based on partial knowledge.
This project will aim to investigate what Artificial Intelligence techniques can be applied to the
domain in order to play up to a human standard of decision making. Online poker clients will
provide a reliable test-bed where an agent to be tested continuously and the client interface will
also allow a digitalized game state recording to be read by the agent to aid decisions with no
need for human intervention.

Download: http://pokerai.org/public/aith.pdf

Note: The dissertation is hosted on pokerai.org with the permission of the author.

Poster:

Is this a degree thesys?

This is well written and pretty comprehensive.
I've not found any new and amazing contribution in it, as it mostly apply already read info, but it goes all the way through most important things for botting and it's very clear. I liked the screenscraping part as i have no knowledge in that topic and i found it useful. The neural network performances wasn't amazing but the training process was clearly explained. I also liked the client which shows the game tree . As it has been pointed out the system was a bit slow computationally, which is a pity.

The only big no no is in the testing part, only 2k hands to test the system, i find it to be really too few hands to make conclusions of any sort but if this was a thesis i understand that "i must get my degree" was a stronger statement than "i must extensively test my system" proposition.

Nice publication.

BTW, congrats to everyone that has triggered the above opinion. And to Patrick for the completion of his excellent work.

I still have to read his thesis more seriously, but from what I've seen so far this would be my future insta recommendation for people that just want to start with pokerbotting (next to Johanson's thesis), and before they are into it enough to recommend them the more cutting edge stuff.

btw I front-paged your work on http://pokerai.org

The document is actually a dissertation for my bachelors in computer science (not a masters as it mentions in the front-page). Oh, and the link is also broken.

Many thanks for the positive feedback, it means a lot to have my work acknowledged.

HP

Yes, I just saw that I linked http://localhost/pf3/viewtopic.php?f=64&t=2548. "Works on my machine", as they use to say I'll fix this.

To a person that is unfamiliarized with ANN like myself this is the most comprehensive work I have read so far.
Brilliant job!

Like indiana said, this is a must-read to start poker botting.

Thank You!

Congrats on turning screwing around with online poker bots into a degree!

Hi,

Thank you for this piece of work, it’s a nice summary for anyone starting with bots.

I have a question about your EV calculation on page 12. Wouldn’t make more sense to calculate the EV before going all in?
The whole purpose of EV calculations is to decide whether an action (call/raise/bet) makes sense, hence you should take into account in your EV the “probability to lose at showdown” * “rest of your stack” (negative amount).
What does everyone think?

--sob

sob, the EV calculation on Page 12 is that of an 'all-in' situation because it introduces the reader to EV on the most simple terms possible. In a full blown EV based approach (as demonstrated later in the paper) it is pretty crucial to calculate EV for each possible move, including an all-in.
An example of this can be seen in the GameTree sections where under a non-'all-in' situation (where the agent does not known explicitly the hand of its opponent), each possible hand in the opponents range is iterated, calculated for ev by passing back leaf node EV values, and multiplied by the chance of that node occurring. The move is calculated by aggregating the results.

See Section 4.3.3, Figures 39 - 43.

Thanks for your feedback - if i collect enough I may revise the document at a later date to make some sections clearer.

HP

That makes (some) sense :)

There's a couple typos.

And you could really skip "It can be observed that all probabilities sum to 1."
I assume your readers have higher than 10th grade education ;)

This paper is very well written. It is very informative for a new guy like me . Thanks for making it available.

Very interesting paper indeed! Congratulations on this impressive work.

I like its practical aspect, showing theorical ideas and their direct implementation.

I have two questions:
1. why the bluff parameter totally absent of the hand ranges? Is it for simplification of the opponent model?
2. why was the final simulation only on 2000 hands? Because the bot has been busted, or because all input was manual and it took too much time?

Thanks!

I really enjoyed this thesis and got some new helpful suggestions. I always thought of one big opponent model that was parametrized by poker tracker data. The idea of building clusters and create one model (NN) for each strategy seems a lot easier. But i wonder how many hands one would need in a poker tracker database to define clusters for lets say ten stats (like vpip, pfr, AF, attsb, wtsd, ...). And how many hands are needed to train all permutations of the clusters? Is this realistic or can it be done with fewer samples?

The next thing i'm not sure about is the scenarioParser. Please correct me if i'm wrong. The ScenarioParser collects all important information to train the NN in scenarios. This means one scenario consists of the information that is needed for each input node and the action the player took. During the training phase the input nodes are feeded with information and the output of the NN is compared to the action the player took. Then the weights are adapted using a backpropagation algorithm, for example.
This means that the data of one hand can be devided into maximum three scenarios (one for flop, one for turn and one for the river). Does this sound right?

Again, thank for you thesis!!

Thanks for the feedback, and apologies for not checking this post in a while (quite busy recently).

MurziTheMurr, In response to your questions:

1. why the bluff parameter totally absent of the hand ranges? Is it for simplification of the opponent model?

Sorry, I don't quite understand the question. The bluffing that the agent performs is based upon its $EV predictions in the game tree. If the ANN model picks up that a player is more likely to fold in too many situations, this is exploited when the nodes in the game tree are evaluated against the prediction values. If the bluffing node is the most profitable then the agent will pick that action.

2. why was the final simulation only on 2000 hands? Because the bot has been busted, or because all input was manual and it took too much time?

I was only given a one year space to learn, develop and test the agent - while writing the documentation. I wish I could have tested it further but the agent was only finished at the very end of my year in university. The input was done manually to uphold the ethical guidelines of the project. (see the ethics section in the paper for more details).

Mr.Ruder, the response to your questions:

And how many hands are needed to train all permutations of the clusters? Is this realistic or can it be done with fewer samples?

It is absolutely realistic, and really depends on the quality of your neural network implementation and how many hands you have available. With the HHex on this site containing over a billion hands, you could easily train hundreds of clusters if you needed to. Generally I would use about 1,000,000 scenarios for each player cluster.

This means that the data of one hand can be devided into maximum three scenarios (one for flop, one for turn and one for the river). Does this sound right?

You are absolutely right in your understanding of the theory. However, more than 3 scenarios can arise for each hand.
Say for-instance that there were 4 players who played to the turn. For every action that each player took a scenario can be generated. If there is much 3/4 betting etc on the hand, you can expect quite a few scenarios being extracted.

Thanks for your answers. Ok, you can extract more than three scenarios per hand history. You can extract one scenario for each opponent decision.

I thought a lot about this approach during the last days and i still have one question. If i got you right you used for example hand strength to train your neural net. How did you handle hands where you didn't know the hole cards of your opponents (hands without showdown)?

Ruder,

I only trained on hands that were revealed, or went to showdown - and this is probably one of the most major flaws of the approach. This resulted in a lot of trained predictions being made based on hands that were checked to the end with no betting, so i removed some prior to training to uphold the balance.

A better approach than this would be to hold two neural networks (or more) - one for showdown training, and one that makes prediction without showdown and implement some sort of voting mechanism between them.
Its an interesting problem, and one that applies to pretty much all AI implementations that use hand samples.

Thanks so much for posting this. As a beginner, this has been an immense resource and given me a great base to start from. I guess next I'll be reading Johanson's thesis that indiana mentioned.

In your ethics section you mentioned the "free money" option. Are these the tables that you ended up testing on? Personally, every time I've played at a play money table I've found the players range anywhere from bad to terrible.

Hi, I haven't read all of this yet, but its a great introduction to a lot of botting concepts, and I just wanted to congratulate you on an excellent thesis. I'm sure I'll have more comments when I get around to finishing it.

If only my college would let me do something similar...*sigh*

Nice work.

I have one major question.
How did you define the accuracy measurement of your ANN?
I couldn't find any definition.

This is another thing I can't understand.
How does your network learn that player fold hands?
Usually you can only train the network with the hands which were shown at SD, so the network can never learn which hands opponents are folding.