As I recall, I added some hands that I have personally recorded (> 100,000) so that the agent had a visibility of the actual hole cards in folding scenarios. This helped the agent learn folding probabilities, but is bias as it is only trained against my playing style.
Ideally, training the network on samples containing many different playing styles with visible playing cards and some being folded would create better results. This is another improvement point in the agent.



I cant remember the exact terminology for the accuracy measurement. But as I recall, the ANN accuracy was awarded a correct read if the highest output node matched that of the actual action. Ie - action : fold & output of (0.7, 0.1, 0.2) = correct decision. (Correct decisions / overall decisions) * 100 = ANN accuracy.



For the thesis, I was forced to add this ethical section to ensure that the project was "morale". As you and I both know, a testing environment such as free money is an extremely distorted version of the actual problem, and is of much less use that a real money environment. I will let you decide your own conclusions about this.

How do you separate bet rounds into scenarios using the PT database, given that stats are stored per round per hand? In other words, how does the "ScenarioParser" work?

Have you tried a larger hand sample since? Any plans for the future?

First of all let me say i really enjoyed reading your paper! You introduced and explained most topics pretty well, i would have liked a little more technical details on some things, but im sure it was in line with what your professor required of you. In any case i have a question i hope you can enlighting me a bit about.





This answer interested me a lot, so i checked up on the PT3 database and looked through the tables and available information and it looks a mess! Granted you can extract some kind of hand replay from the player_details/statistics (within certain limits for instance only 3 bets per player per street), however it almost seems like it would be just as easy (or hard depending on how you look at it) to just make a parser of the handhistory text.

How did you go around making the scenario parser, was it a combination of parsing the handhistory text + getting some of the already calculated information or did you actually try to recreate the hand flow from the pokertracker generated values?

Also i assume you meant 4 root scenarios (preflop, flop, turn, river)?

Apologies for the lack of replies to questions, I now work full time and have been rather busy recently.

How do you separate bet rounds into scenarios using the PT database, given that stats are stored per round per hand? In other words, how does the "ScenarioParser" work?

The scenarioparser iterates the database, and extract multiple scenarios from each hand (stored as one row in the PT3 database). I don't currently have the database loaded, so you will have to excuse my inevitable inaccuracy, but I believe that the hand actions are stored something like CRXF meaning Call-Raise-Check-Fold.
From this string logic can be applied to the other fields in the database such as Preflop/Flop/Post-flop calling/raising amounts in order to extract the actions, and to run hand evaluation analysis on each scenario.
If your still stuck with this, give me a PM and ill see if I can dig you out some sample code.

Have you tried a larger hand sample since? Any plans for the future?

Nope, I haven't. I created this bot for the purpose of my bachelor thesis. I entered employment straight after university, so have not had much time for personal development such as poker botting. I can safely say I miss it!

How did you go around making the scenario parser, was it a combination of parsing the handhistory text + getting some of the already calculated information or did you actually try to recreate the hand flow from the pokertracker generated values?

Also i assume you meant 4 root scenarios (preflop, flop, turn, river)?

I hope my previous response has answered this question. I didn;t use any handhistory text, although this would certainly be a worthwhile approach. I actually just extracted all the data from the database, using some extensive LINQ queries. Combining this with handhistory for error checking would be a good idea.

Actually I meant only 3 root scenarios. I did not train the neural networks based on Pre-flop scenarios due to the drastic change in tactics from pre-flop to flop.

How did you determine the bias value when you reweighted hands?

Hi HotPotato,

first, nice work, I agree with sob; it is a good summary for beginners.

Anyway, I have some question , too ^^

1. Did I understand that right: You got one ANN per strategy? And your reason to classify the opponents into different strategy's was to increase the prediction abilities of the ANN's? If so, wouldn't it be enough to just include the statistic's, used for clustering, into the input vector of only one ANN? This should learn the relation ship between different player's strategies by itself. This would lead me to other questions, but I want to clear this up first ^^

2. I was a bit surprised seeing your input vector. You barely encode the action that was before the actual state. (if you do that you may need one ANN per betting round) But I think that would increase your predictability a lot, especially if you try this approach in a full ring game. You said your configuration of input features were "most efficient", did you test other features that contain more information about previous actions?

@ Demonthus
I guess he just normalized the hand-weights

That's correct. Yes your approach would also be viable - but I was also looking into using a neural network voting implementation where several numbers of the same neural networks were queried using slightly different training values, and the prediction values were also extracted from several strategies to get a full picture (like in the example where villian is a semi loose aggro, then both loose aggressive and semi-loose agreesive could be queried. My grand plan would be to have a lot of these pre-trained ANN's available for predictions, as well as a 'real-time trained' network to get the best results. Of course my problem here was that I didn't have enough time, but I suppose to answer your question I chose this layout as a software architecture decision more than anything else.



I think when i mentioned that the input features were "most efficient" I was referring to the different (relatively few) combinations that I tested. I did use the 'last action to bet or raise' input value, but ideally this would be more detailed and more precise. I at no point think that my input values are the most efficient values to use, and I would imagine that other combinations would yield more accurate results with enough persistence and testing.
In the grand scheme of things I was more concerned with getting in the theory and direct implementation to support the paper, rather than striving for optimal configuration. I would thoroughly recommend serious botters to investigate optimal configurations, or indeed other approaches.



Correct!

HP

If you are willing to say ...

Did you implement this into a cash game bot? And if so at what stakes (if any) was it profitable?

Great paper by the way.

I would prefer not to contradict the papers ethical section, and will leave that up to the judgement of the reader. What I will say is that I believe the approach has a good potential to produce a profitable small/micro stake cash game bot, and there would still be a lot left to improve in my implementation.
My results are not conclusive given that my small test hand size is not enough to outweigh the variance associated to the game - but I achieved all the objectives that I set out to accomplish.

Thanks for your feedback.

HP

Very understandable and thanks for the prompt response.

-Hudson

Hi again HP and thanks for the answers in the thread

Could you explain how you "just" normalized this? The bias value is obviously 3.75, but how you got that number I haven’t been able to figure out (if only you had made more than 1 calculation then it would have been easier to deduce).

I have read most of the poker posts and other papers about the weighted tables and hand range prediction. But most are pretty vague about how they technically do it. The most explained is Davidson - Opponent Modeling but there are no bias value there. Also they use noise filtering, did you use that?

Davidson is also talking about the re-weighting system (EHS(1) distribution) and using the 1326 possible pocket hands, from the picture in your thesis it looks like you compacted it into the 169 types instead, is this just an omission for clarity or is this on purpose?
In Davidson’s thesis they start out with all values as 1 and then just decrease from there. It looks like that you have initialized the values to 0.05 for all the pocket hands, but if you do normalize them all from 0-1, then wouldn’t it be 1/169 = 0.00591?

Btw did you use the ppot/npot values calculated from the c# handevaluator lib? The way it is calculated is different from that of the HP algorithm that is used in all of those published papers.

Finally about the bucket system and how to order them into these buckets, where do you know when to cut one bucket from the other? And how do you choose one bucket over the other to represent the hands you think the player has?

I mean if the distribution of EHS values (I just assume we use this for now) is not uniform, as it is now later in the hand, could one bucket hold 2 hands and another 10 (assuming they hold a predetermined range of EHS)? I'm thinking you could use k-means clustering here to properly ensure an evenly distribution of them all in say 10 buckets or is this just making it more difficult than necessary?

h = handweights (vector R^n), 0 <= hi <= 1, i = 1,...,n
p = prediction (scalar R), 0 <= p <= 1

h = h * p
h = h / ||h||

||h|| = sum norm of h, you even don't need to take the absolute value

I assume this is in response to my question about the bias value, however it seems you calculate a new hand weight directly.

In any case, im not sure im following your approach, what norm are you referring to (as there are plenty to choose from)? If we take the example he has shown in the document, where AA has 0.05 weight and prediction says 0.8:

0.04 = 0.05 * 0.8
h = 0.04 / ||h||

where ||h|| = Euclidean norm? or manhatten? or?

Please, an example would be nice


Also i have seen one method where you would simply scale up the numbers by doing the following:



Is that sufficient enough or are there some kind of issue with this?

Hi guys,

I happen to be one of those people that just want to start with pokerbotting and I was wondering where to find Johanson's thesis? Could someone provide me with a link or some good search words so I could find it through Google? Thank you.

PS. As a beginnier, I already find this site very useful...

http://pokerai.org/pf3/viewtopic.php?f=64&t=1192&hilit=johanson
The poker papers subforum is probably the best place to start searching.

Spears: Thank you! That link was really good

This is possible through code by traversing the PT database and creating a game state for every player for every bet round. I have done this by modelling it to Win[Open]holdem symbols. I generate the symbols I need for every player for every action faced. From there you can analyse the data however you wish. This is of course another skill unto itself.