PANES: Policy-guided Asymmetric Nash Equilibrium Search

Abstract

Trick-taking card games with mandatory bidding confront reinforcement learning agents with a distinctive two-phase problem: each player must commit to a numeric bid before any cards are played, and whether that bid turns out to be correct hinges on adversarial interactions unfolding over many subsequent tricks. Judgement (also called Oh~Hell) makes this harder still through its Hook rule, which structurally prevents the total bids from summing to the available tricks. This ensures at least one player is guaranteed to fail every round. Here, we introduce a complete framework of reinforcement learning for the Judgement game, consisting of an environment and a specific agent architecture trained together in bidding and trick-play under imperfect information. The environment was built inside the RLCard platform, supporting four-player games with legal-action masking and lightweight state rollback for tree search. On top of this, we developed a PANES agent that pairs Neural Fictitious Self-Play with Information Set Monte Carlo Tree Search. During training, the agent learns through self-play guided by dense trick-level reward shaping; at decision time, it runs multi-step lookahead over determinized game worlds using the learned opponent policies. In a 2,000-game paired-deal arena evaluation, the PANES agent reached 22.7% bid accuracy versus 13.5% for the neural-only baseline---a 68% relative gain, supported by robust statistical significance (p < 0.0001). A closer look at the training dynamics revealed an interesting asymmetry: agents pick up trick-winning skills quickly but struggle much more with deliberate trick avoidance, which we identify as the main learning bottleneck. We also ran an ablation with Prioritized Experience Replay; it cut training loss by roughly a third yet produced no measurable improvement in game-level performance, pointing to the credit assignment horizon as the binding constraint. Taken together, these findings show that merging equilibrium-based self-play with online tree search is a viable path for imperfect-information trick-taking games, while the long-horizon credit assignment problem remains the central open challenge.