Three flavors of autonomous casino LLM agent validation
If you follow casino news closely, you will probably remember the video poker game that could be hacked with certain button sequences or the UK iGaming operator whose jackpot feature triggered 50x more frequently than expected. We personally know of a casino that accidentally gave $1m a month too much in marketing reinvestment due to a software parameter issue. These are misses in UAT validation.
LLM-driven processes only complicate validation further. LLMs are notorious for hallucinating or producing wildly inconsistent results when asked the same question. Getting this right is the number one roadblock for LLM automation in the casino industry. We are seeing clients have great success amplifying a single user’s productivity with support of foundational models like Claude and Copilot. A single user can create code or do a general analysis far faster today than a couple years ago. But this process is still centered around a user. Scaling to be more reliant on autonomous agents requires assurances that actions are appropriate. Our system – Hex – is based on automating agents and to commercialize, we needed an approach to validation. We hope the major industry testing companies will follow suit.
The validation process we developed is the first of its kind in the casino industry. There are three broad challenges in a casino LLM validation system: are outputs accurate, are they ‘coherent’, and is there MCP/API fidelity. We will take each challenge in turn.
Definition: Before reading on, a note on terminology: we use the term "semantic" often. Semantics refers to meaning rather than wording; two contexts are semantically similar if they describe the same situation in different words. ADT and ATT is normalized spend per day or trip, respectively.
Accuracy
Of the three, accuracy is the easiest to understand. Accuracy validation evaluates if the LLM retrieves / summarizes numbers which are consistently correct. For instance, if a user asks about player reinvestment over the period, do the returned results tie to source systems? With accuracy there is a clear right or wrong answer.
Coherence
Coherence, by contrast, is fuzzier.
Coherence measures how well the LLM produces recommendations that are appropriate given the context. Coherence needs to be consistently appropriate even across semantically similar contexts. For instance, consider the context and recommendation pair below. In this instance, we are using a LLM to recommend a ‘diagnosis’ regarding the state of the player:
Given fine-tuning, access to appropriate examples (embedded in a vector DB), and informed by predictive modelling and history, an LLM can craft a ‘coherent’ output like the below
Diagnosis example: small fine-tuned LLM with access to examples in vector database
| Context | LLM Diagnosis1 |
|---|---|
|
This patron has recently become non-loyal and has likely migrated her entire wallet on most casino visits to another casino. The trips that still occur at the property are probably “change-your-luck” trips, and the majority of her wallet has migrated to a competitor. Expect the patron to stop visiting shortly. |
Poorly supported LLMs will often return ‘incoherent responses’ like the following:
Diagnosis example: leading large parameter frontier model (with no fine-tuning or context)
| Context | LLM Diagnosis2 |
|---|---|
|
This player is an actively declining, at-risk player in the pre-lapse window, likely due to loss fatigue. |
The gap between each model's output and the hand-curated recommendation is what coherence measures. Validation needs a way of quantifying the distance between the approved recommendations (in this case diagnosis) and the LLM generated recommendation. We do that with an LLM-as-judge scoring system.
In order to validate coherence, we go through the following steps:
We first identify a set of potential context situations. We use an analytical approach rooted in conjoint analysis to identify a representative set of context situations.
For each context situation we curate and hand-approve a diagnosis. This takes expert-level experience.
We ask the LLM to create a diagnosis, conclusion, or recommendation. (LLM as a judge if well established evaluation technique).
We use an LLM as a judge to rate the ‘coherence’ between the approved diagnosis and the LLM-created diagnosis. To estimate this score, the LLM judge first extracts the important topics from the diagnosis and then rates semantic similarity between the LLM and approved diagnoses on a limited scale. Depending on the use case, we weight the topics within the score.
Returning to our two examples and assuming equal weighting between the topics and using a 30 point scale:
The fine-tuned small LLM received a score of 22 out of 30, importantly capturing the idea of competitive promiscuity.
The large frontier LLM got an 13 out of 30, focusing on disengagement rather than an understanding of competitive pressures.
This is the logical appropriateness aspect of coherence.
Importantly, the LLM needs to return consistent results given different contexts which are semantically similar. To measure consistency, we vary the contexts and measure the overall consistency. Consider the following variations:
| Context | LLM Diagnosis |
|---|---|
|
This patron has shifted to a non-loyal state and has most likely moved the bulk of her wallet to a competing property, though she has not yet lapsed. Patrons fitting this profile typically carry high churn risk. |
|
The patron is no longer loyal to the property and has probably migrated the majority of her gaming wallet to a competitor; she remains active enough not to be classified as lapsed. Players in this situation are typically at high risk of churning. |
|
This patron has become non-loyal, and the evidence points to most of her wallet migrating to a competitive property while she has not yet fully lapsed. This pattern is characteristic of a high-churn-risk player. |
|
The patron shows the signature of lost loyalty: the majority of her wallet has likely been captured by a competitor, although she has not lapsed yet. Historically, patrons exhibiting this pattern are high churn risks. |
|
This patron has drifted into non-loyalty and has likely redirected most of her spend to a competitor casino, but she is not yet a lapsed player. Patrons with this profile are typically flagged as high churn risk. |
Judged by the LLM, the above batch yields a composite score of 25 out of 30: i.e. the LLM yields consistent results.
Integration or MCP/API fidelity
The final validation step is MCP/API fidelity: confirming that the LLM's intended action is translated into the correct API call. We test this the same way we test coherence, centered around a curated set of contexts, each paired with an approved reference output, but here the reference output is a JSON payload rather than a prose diagnosis. Scoring is also stricter: instead of judging semantic similarity, we check that the generated call matches the reference exactly, including the right endpoint, right parameters, right values. An API call is either correct or it isn't, so this step looks more like accuracy validation than coherence validation.
Simulation
Like the RNG inside a slot machine, LLMs have inherent randomness. To deal with this uncertainty, we simulate this process through thousands of iterations and evaluate the resulting scores saving the responses at each run. We then analyze results of this simulation focusing on the amount of variance with the measures. Our LLM stress-testing validation process is the first such process in the casino industry.
Footnotes:
30B parameter fine-tuned model
Claude Fable 5 model
