This work presents an unsupervised strategy classification pipeline for adaptive adversarial game AI, addressing the challenge of inferring player strategies in real time without labeled behavioral data. Existing approaches optimize for winning, treating difficulty as a scalar quantity that fails to capture the richness of human strategic diversity. To bridge this gap, we adapt the Vector Quantized Variational Autoencoder (VQ-VAE) with a transformer encoder and task-specific decoders to map sequences of game states and actions to discrete latent codes, enabling label-free strategy discovery from raw gameplay. The system is trained and evaluated on a custom turn-based tactical puzzle game played on an 8×8 grid, using a corpus generated by five behaviorally distinct policy variants: balanced, aggressive, defensive, guardian, and efficient.
Figure1. A turn-based tactical puzzle game inspired by Subset Games' Into the Breach.
This project seeks to create a model that both classifies and mitigates player strategies in real-time gameplay. The game used in this study is a turn-based tactical puzzle game inspired by Subset Games' Into the Breach where the player uses a small squad of mechs to manipulate enemy actions on a grid in chess-like scenarios (Figure 1). The player is assigned three different classes of mech on an 8x8 grid that must protect buildings amidst terrain obstacles and targeted attacks by three alien "Vek" over 5 turns. Since the game procedurally generates maps, 40 "winnable" maps were manually selected to guarantee initial states were winnable by the player.
Figure 2. Three-model approach for data gathering, strategy classification, and strategy mitigation.
We compare three decoder configurations: a reconstructive decoder that recovers unit actions and positions, an outcome-predictive decoder that forecasts tactical consequences such as enemy neutralization and building protection, and a label-supervised multi-task decoder that augments reconstruction with an auxiliary policy-variant classification loss.
Figure 3. The three variants of the VQ-VAE (Model 2) decoder configurations.
The key idea of model 2 is that it does not have access to policy labels in the dataset generated by model 1. It has to infer them using a form of Vector Quantized Variational Autoencoder (VQ-VAE) that treats each game state and previous action as a token input to a transformer encoder (Figure 3). The encoder and decoder of model 2 learns to predict 5 features of the previous action that led to the game state via backpropogation, while a codebook of N=5 vectors learns to group the latent space turn vectors into 5 strategy clusters via exponential moving average (EMA).
Both unsupervised models converge on strategy clusters whose policy-variant distributions are near-uniform, indicating that the learned representations reflect emergent behavioral structure rather than the designer's labeling scheme (Figure 4). The reconstructive model discovers five spatially differentiated clusters corresponding to distinct board-region activity patterns, while the outcome-predictive model produces three clusters organized around combat outcome profiles and temporal engagement rhythm. In contrast, the multi-task model with policy-variant classification recovers the imposed policy taxonomy, establishing the decoder objective as the primary driver of codebook organization. These findings validate unsupervised representation learning as a principled foundation for strategy-aware adversarial agent training.
Figure 4. The policy variant distribution per strategy across Models 2.1, 2.2, and 2.3.
Model 3 is a multi-headed PPO adversary that controls the Vek (enemy units) against a frozen player policy (balanced) from Model 1. Instead of learning a single monolithic counter-strategy, it maintains one independent actor-critic policy per VQ-VAE codebook entry, five heads in total, matching the codebook size K=5 (consistent across variants). At each turn, the frozen transformer (Model 2) classifies the player’s most recently completed turn into one of the five codebook strategies, and the adversary routes its experience and action selection to the corresponding head. This allows each head to specialize in countering one particular behavioral cluster of the player.
Figure 4. The integration of model 2 into model 3 for policy selection.
The key idea of model 3 is that during training and inference, the adversary can classify every player turn as using one of the 5 strategies that model 2 was trained to predict (Figure 4). Model 3 can use any architecture that allows for multiple expert heads to exploit the specific detected strategy including Proximal Policy Optimization (PPO), Value-Based Reinforcement Learning System (VBRS), Deep Q-Network (DQN), and hybrid models. These trainable models are then compared against a variety of non-trainable approaches including Monte Carlo Tree Search, AlpaBeta, Minimax, Random, and custom heuristics.
To assess statistical significance, a one-sided sign test was applied to the paired per-map win rate differences between the routed condition and each individual head across all 40 maps. The sign test evaluates whether the routed condition wins more often on a map-by-map basis than the comparison head, under the null hypothesis that neither condition is systematically favored (p = 0.5). The routed condition outperformed head_0 on 16 of 40 maps (p = 0.006), head_1 on 15 of 40 maps (p = 0.010), head_2 on 17 of 40 maps (p = 0.004), head_3 on 16 of 40 maps (p = 0.002), and head_4 on 18 of 40 maps (p = 0.0002). In no case did an individual head outperform the routed condition on more than four maps (Figure 5). All five comparisons are statistically significant at α = 0.01, and four of five are significant at α = 0.005, providing strong evidence that the routed condition’s advantage is not attributable to chance.
Figure 5. Overall adversary win rate by Model 3 policy head
To evaluate whether the advantage of the routed condition extends beyond the training distribution (Model 3 was trained against the balanced opponent policy), Model 3 was also evaluated against four held-out policies, aggressive, defensive, efficient, and guardian, which were not seen during training. Each of the five individual heads (head_0 through head_4), along with the routed condition, was tested against each of these four policies across the full map set. The resulting adversary win rates are reported in Figure 6.
Figure 6. Overall adversary (Model 3) win rate by Model 3 policy head when playing against other player (Model 1) policies
Across all four held-out policies, the routed condition achieved the highest or statistically comparable adversary win rate among the six conditions, indicating that the gains observed against the training policy are not an artifact of overfitting to the balanced opponent. Specifically, the routed condition reached an adversary win rate of 0.18 against aggressive, 0.16 against defensive, 0.13 against efficient, and 0.09 against guardian. In three of the four evaluations (aggressive, efficient, and guardian), the routed condition exceeded every individual head by a meaningful margin; against aggressive, for example, the routed rate of 0.18 was nearly double the best-performing individual head (head_2 and head_3, both at 0.10), and against efficient the routed rate of 0.13 exceeded the best individual head (head_2 at 0.10) by 30%. Against the guardian policy, the most difficult of the four, on which all conditions performed poorly, the routed condition still matched or exceeded every individual head, reaching 0.09 compared to a maximum individual-head rate of 0.08 (head_0).
The one partial exception is the defensive policy, where head_2 achieved a win rate of 0.17, marginally exceeding the routed condition’s 0.16. This single case suggests that for certain opponent types a specialized head may occasionally edge out the router, but the routed condition remained within 0.01 of the best individual head and substantially outperformed the remaining four heads (which ranged from 0.03 to 0.09). Importantly, no individual head dominated across all four held-out policies: head_2 was strongest against defensive and efficient but weak against guardian, while head_0 was strongest against guardian but middling elsewhere. Head_4, the weakest generalizer, failed to exceed a 0.03 win rate against any of the four held-out policies and achieved 0.00 against aggressive.
These results provide evidence that the routed condition’s performance advantage generalizes beyond the training policy. Rather than collapsing to the strategy of any single head, the router appears to select effectively among specialized behaviors in response to opponent type, producing consistently strong, and in most cases best-in-class, performance against policies that were never encountered during training.