Create game explorer with pgn (like lichess explorer)

Question

currently I am looking with python to do a failure "ia" which gives me the opening with the best success rate (using the grand master parts of the lichess database) with a minmax and by taking only the moves with at least 100 position already played so as not to fall on a 100% black success rate with 1 single game. I managed to do it with the lichess api, the problem is that when I request more than 3 depth, lichess blocks me because of spam generated on the api.

I told myself that I was going to do the same thing but instead of using the lichess api, I was going to use a .pgn of grand masters parts and with the library "chess" I could do something similar but currently I am blocked on how to filter this file on by example "e2-e4" and have the success rate of each move and have a recursive function to explore this file.

I can't find a solicitation through the chess-py doc and my googles searches don't come up with anything.

Does someone have an idea ?

api lichess : https://lichess.org/api#tag/Opening-Explorer

png file : https://odysee.com/@Toadofsky:b/Lichess-Elite-Database:b

Answer 1

One approach is by building a database like mongodb.

1. Read each game in the pgn file.
2. Record the epd after every move in the game.
3. Get the result of this game.
4. Record white_win, white_loss, white_draw, black_win, black_loss, black_draw, num_game.
5. Save/update to database the following info.
{
  "epd": <epd> 
  "white_win": ...,
  "white_loss": ..., 
  "...": ...
}

You can now query the database by the epd and get what is there.

You can parse every game in the pgn file using for example the python-chess library.

The advantage of saving it to the database first is that when you query later you will get the result fast.

If you don't like to save it in a database, you can do so but can be expensive if the file is big. You can still use the above algorithm.

Answer 2

Here is an example with the indications of @fougueux (I hadn't thought about creating a database with this file and I missed the "erd" code which is exactly what I need)

Fill the database (dbname = "game", row id, row fen, row result)

import json
import requests
import chess
import chess.pgn
import chess.polyglot

import pymysql


pgn = open('C:/Users/pierr/OneDrive/Bureau/lichess_elite_2020-05.pgn')
result = []

while True:
    game = chess.pgn.read_game(pgn)
    if game is not None:
        board = game.board()
        for move in game.mainline_moves():
            board.push(move)
            result.append([board.epd(), game.headers["Result"]])
    else: 
        break

connection = pymysql.connect(host='localhost',
                             user='root',
                             password='',
                             database='chess-master',
                             cursorclass=pymysql.cursors.DictCursor)

with connection:
    with connection.cursor() as cursor:
        # Create a new record
        for i in result:
            sql = "INSERT INTO `game` (`fen`, `result`) VALUES (%s, %s)"
            cursor.execute(sql, (i[0], i[1]))

And algorithm for search minmax score (score = %winrate white * 1, + %winrate black * 0, + %winrate draw * 0.5), he take a move with more 100 move plays

import json
import requests
import chess
import chess.pgn
import chess.polyglot

board = chess.Board()

def parcourir(board, depth, traitblanc):
    
    scorelist = []
    
    for move in board.legal_moves:
        board.push(move)
                   
        url = 'http://localhost/chess-master/?fen='
        url += board.epd() # fen
        
        r = requests.get(url)
        data  = json.loads(r.text)
        
        somme = int(data[0]['COUNT(result)']) + int(data[1]['COUNT(result)']) + int(data[2]['COUNT(result)'])
       
        if(somme > 100):  
            score = (int(data[0]['COUNT(result)']) * 1) + (int(data[1]['COUNT(result)']) * 0) + (int(data[2]['COUNT(result)']) * 0.5)
            scorelist.append(score)

        board.pop()
        
    if(depth != 0):
        score = []
        for move in board.legal_moves:

            board.push(move)
            score.append(parcourir(board, depth-1, not traitblanc))
            board.pop()
            
    if(traitblanc):
        if(not scorelist):
            return -100
        return max(scorelist)
    else:
        if(not scorelist):
            return 100
        return min(scorelist)
    
print (parcourir(board, 1, True))

being more comfortable with php than python I made the select of the fen via this code:

<?php
    $servername = "localhost";
    $username = "root";
    $password = "";
    $dbname = "chess-master";
    
    $doCount = array("1-0","0-1","1/2-1/2");
    
    // Create connection
    $conn = new mysqli($servername, $username, $password, $dbname);

    $results = array();
    
    foreach($doCount as $c) {
        
        $sql = "SELECT COUNT(result) FROM `game` WHERE fen = '".$_GET['fen']."' and result = '".$c."'";
        $result = $conn->query($sql);
        while($row = $result->fetch_assoc()) {
            $results[] = $row;
            #print_r($row);
        }
    
    }   
    echo json_encode($results)
?>

Thank you for helping :)