Banksia GUI still very buggy

[RubiChess]

Hello guys.

I'm using Arena GUI for a long time but it is has some bugs and is out of development, so I would really like to switch to BanksiaGUI. In fact I already switched to B. on my Linux machine cause the Arena Linux port doesn't work well on my 4k monitor setup.
But BanksiaGUI still is far from stable and I must admit that this didn't change much in the last months or even years.
There are so many situations where B. just doesn't do what it should do (and has done after a clean start) and I need to restart it to fix. Or it just crashes. Fixing two bugs while introducing three new features with four new bugs seems the line of development.

So I would really suggest to start doing some serious testing and fixing all bugs before implementing anything new or different.
If you agree on that, I could help with some exact descriptions/reproductions of the bugs that I noticed.

Regards, Andreas

[pham]

Thanks a lot for the feedback, Matthies!

As we have discussed already, I have been struggling to reproduce those crashes as you have described with RubiChess on Linux. I have run some tests in several Linux systems but still could not reproduce the issue.

If you could, please:

- Try yourself on another Linux system to see if there will be the same crashes
- Find and send me some crash logs (of Linux)

Thanks

[RubiChess]

Hi Pham.

You misunderstood. All the bugs that I have reported in the past are probably solved (as far as I remember) but...
with every new version it seems that new bugs were introduced and some are quite obvious. And this is what some other users are saying, e.g. in some german computerchess forum. So I got the impression that some serious testing is missing in general. And testing means not only isolated testing of a function after a clean start but heavily switching between different functions, tasks, displays and data.

My main workflows in BanksiaGUI are
1. (on-the-fly) analyzing using more than one engine and repetitively feeding the GUI with FEN strings or PGNs from the clipboard
2. playing tournaments with 2-4 engines
In both workflows almost every dock/graph is enabled.
Both workflows are working quite well after a fresh start of BanksiaGUI but after some time and usage (like switching from some FEN/PGN to another, switching engines, ...) I can be sure that something will go wrong. Like missing or wrong outputs in some graph.

Anyway, I will give you exact bug descriptions and a step-by-step instruction how to reproduce once I find them.

Regards, Andreas

[pham]

I have been testing BSG on Ubuntu 18.04 for a while, focusing on tournaments + graphs. So far no problem. Thus it is best if you could give me instructions on how to reproduce the bugs. Otherwise, some screenshots of those bugs should be helpful too. Thanks

[RubiChess]

Okay, after some time not using Banksia I have downloaded 0.56 for Windows and run some tournament.
In this tournament all my versions of RubiChess starting from 0.1 up to current 20221120 were included.
Find below some things that I noticed.

1. Version 0.1 of my engine seems to crash Banksia GUI. I had to remove it from the tournament to continue. This same versions runs fine in Arena. I can send you the binary if you want to test. Even if the engine is faulty in some way (wrong UCI output), the GUI must not crash.

2. Some strange numbers in the Tech table. See screenshot (I hope it will be visible in the post... I will post a link as a fallback). Seems that for some of my older versions the moves/game and time/game is obviously wrong.

https://imgur.com/a/v8cBBEa

3. Looking at Windows taskmanager I can see that Banksia GUI grabs almost 7GB of RAM just after startup even without running any tournament, game or analysis. Why??

Let me know if you need any files or additional informations.

Regards, Andreas

[pham]

Okay, after some time not using Banksia I have downloaded 0.56 for Windows and run some tournament.
In this tournament all my versions of RubiChess starting from 0.1 up to current 20221120 were included.
Find below some things that I noticed.

1. Version 0.1 of my engine seems to crash Banksia GUI. I had to remove it from the tournament to continue. This same versions runs fine in Arena. I can send you the binary if you want to test. Even if the engine is faulty in some way (wrong UCI output), the GUI must not crash.

2. Some strange numbers in the Tech table. See screenshot (I hope it will be visible in the post... I will post a link as a fallback). Seems that for some of my older versions the moves/game and time/game is obviously wrong.

https://imgur.com/a/v8cBBEa

3. Looking at Windows taskmanager I can see that Banksia GUI grabs almost 7GB of RAM just after startup even without running any tournament, game or analysis. Why??

Let me know if you need any files or additional informations.

Regards, Andreas

Thanks for the report!

Yes, could you please send me that binary (0.1, the crash one)? I totally agreed that Chess GUIs should not be crashed with any engine!

I have been checking other issues and will get back soon. Thanks again

[pham]

2. Some strange numbers in the Tech table. See screenshot (I hope it will be visible in the post... I will post a link as a fallback). Seems that for some of my older versions the moves/game and time/game is obviously wrong.

https://imgur.com/a/v8cBBEa

Could you send me the file banksiaplay.json too? I need to check what was wrong with tech stats. Thanks

[pham]

Thank you a lot, Matthies.

I have fixed some bugs based on your information, and data. The new version (today's sub-version D) has run hundreds of games with all your provided engines without any problem with BSG.

Okay, after some time not using Banksia I have downloaded 0.56 for Windows and run some tournament.
In this tournament all my versions of RubiChess starting from 0.1 up to current 20221120 were included.
Find below some things that I noticed.

1. Version 0.1 of my engine seems to crash Banksia GUI. I had to remove it from the tournament to continue. This same versions runs fine in Arena. I can send you the binary if you want to test. Even if the engine is faulty in some way (wrong UCI output), the GUI must not crash.

2. Some strange numbers in the Tech table. See screenshot (I hope it will be visible in the post... I will post a link as a fallback). Seems that for some of my older versions the moves/game and time/game is obviously wrong.

https://imgur.com/a/v8cBBEa

I can't reproduce nor fingered out. Perhaps, that is caused by the (fixed) bug on parsing engines' output. If you see that again, please report.

3. Looking at Windows taskmanager I can see that Banksia GUI grabs almost 7GB of RAM just after startup even without running any tournament, game or analysis. Why??

It is strange. BSG typically takes only a few hundred MB. For example, I run all your engines for almost a day and my Win10 Taskmanager informed it takes under 400 MB.

Perhaps, it may happen with some engines that refused to quit and refused to... die even be killed too (typically those engines have codes of not stopping even if the input returned 0). Those engines still take memory and computer power even their games are completed. BSG creates an instance of an engine for each game and they may take a lot of memory after a long tournament.

Just a curious, Matthies: I used to think RubiChess is named after a gem. But in CPW (Chess Programming Wiki) they illustrated it with a butterfly. Is that (butterfly) correct?

[RubiChess]

Okay, I will retest with version D when I find some time.

About the 7GB RAM: It was definitely no engine process hanging. The RAM was taken immediately after start of Banksia GUI. No engine was running at that time. I even decreased the general Hash setting to 128MB and disabled Syzygy and Banksia still grabbed all the RAM immediately after next start.

About the engine name: It's a totally different story and has to do with a nickname I got at school in the english lessons and with this tv series: https://en.wikipedia.org/wiki/Upstairs, ... TV_series)

Regards, Andreas

[pham]

Could you help me to do a quick test:
- open the configuration folder of BSG (you may go to Settings—>Log & Configuration and click the button to open it)
- close BSG, to make sure it won’t save anything more to that folder
- rename or move that folder to somewhere just temporarily, to make sure BSG can’t read any previous settings and data
- run BSG and check Taskmanager to know how much memory it takes as a new installation
- recover the configuration folder

Thanks a lot for the help

[RubiChess]

I just did that even before I read what you wrote here and indeed the problem with big RAM usage is gone. Or at least far better even after adding some book files in the settings and a 5-men-syzygy folder.
And the crash when playing my version 0.1 seems also fixed.

But... the problem with the wrong stats of my old engines is still there even with 0.56D.
Could it be some regional setting of the OS?
I am using a german Windows and in Germany we use , for a decimal dot for example and have a time format different from US. Maybe Banskia uses some parsing function that uses regional settings of the OS and that doesn't work on an "english" UCI / Banksia json file / whatever.
Otherwise I cannot explain why you can't reproduce the bug playing tourney with my old engines.

Regards, Andreas

[pham]

But... the problem with the wrong stats of my old engines is still there even with 0.56D.

The file you sent me has the wrong numbers already, thus we can't do anything with it. I guess those wrong numbers were caused by parsing engine info when the code has bugs in the previous version. Just ignore. If you see wrong stats again with new tournaments, please inform me.

Could it be some regional setting of the OS?
I am using a german Windows and in Germany we use , for a decimal dot for example and have a time format different from US. Maybe Banskia uses some parsing function that uses regional settings of the OS and that doesn't work on an "english" UCI / Banksia json file / whatever.
Otherwise I cannot explain why you can't reproduce the bug playing tourney with my old engines.

Regards, Andreas

The region of BSG is the one defined by the system. If you need to change it, just set it with the system.

Just note that UCI has no regional relation, all numbers are simple integers, and strings are simple English/Latin only. Thus the communication between chess GUIs and engines should be the same in any region.

[RubiChess]

But... the problem with the wrong stats of my old engines is still there even with 0.56D.

The file you sent me has the wrong numbers already, thus we can't do anything with it. I guess those wrong numbers were caused by parsing engine info when the code has bugs in the previous version. Just ignore. If you see wrong stats again with new tournaments, please inform me.

Come on, I am in IT business, testing and developing software for 25 years and you can trust me that I know how to test and debug an application. We talked about a new/clean configuration folder. Of course I played a NEW tornament with a CLEAN configuration folder and using 56D. This has nothing to do with the file I sent you before. But the new file created with a clean 56D still has "wrong numbers".

Could it be some regional setting of the OS?

Tested it myself. Set my Windows to regional settings of United states (decimal dot etc.) and played a NEW tournament using 56D. And still the old engines show wrong stats.

Meanwhile I did more tests and I am sure that the problem with the wrong stats has to do with the time control.
Playing longer timecontrols like 3 minutes + 1sec. the stats of Rubi-0.5 look quite good.
Playing short tc like 10 seconds + 0.1sec. the stats of Rubi-0.5 get very bad, like 0.1 moves / game.
So please test playing tourneys with my old engines and very short time control then you will see the bug (B.GUI creating the json with the "wrong numbers") yourself.

Andreas

[pham]

I have stored it in my to-do list as a bug since you saw it with the latest release. Thanks.

I am struggling to reproduce it even though I have tested with your engines for a while. Will come back to that issue later.

[RubiChess]

Reading in talkchess that people go back to 0.55 cause "0.56. doesn't work correct(ly)".
In my opinion the only way to improve Banksia GUI (in the sense of making it stable) is to make it open source. I could search for the bug and probably fix it myself instead of arguing with you if there is a bug or not.

Regards, Andreas

[RubiChess]

Next bug: Opening a pgn containing chess960 games.
1. Download and expand FRC games of SF 15.1 played at CCRL: http://ccrl.chessdom.com/ccrl/404FRC/ga ... 00].pgn.7z
2a. Open the pgn in BanksiaGui 0.56 with option "standard chess": Database opens but doesn't show games correctly (e.g. there are games with 0 plies)
2b. Select options "Chess 960" and open the pgn: BanksiaGUI just crashes.

[RubiChess]

Next two problems/bugs/feature requests:
1. The engines log file logengins.txt has logs of many different (replayed) tournaments, so "View engine-log of game x" doesn't load the correct log of game x of the last tournament but it opens the log of the first "game x" it finds in the file. An old one if you don't clean/delete the file manually.
2. Reopening a game doesn't show tb hits in the graph. Probably because banksiaplaying.json doesn't store the tbhits.

Regards, Andreas

[pham]

Next bug: Opening a pgn containing chess960 games.
1. Download and expand FRC games of SF 15.1 played at CCRL: http://ccrl.chessdom.com/ccrl/404FRC/ga ... 00].pgn.7z
2a. Open the pgn in BanksiaGui 0.56 with option "standard chess": Database opens but doesn't show games correctly (e.g. there are games with 0 plies)
2b. Select options "Chess 960" and open the pgn: BanksiaGUI just crashes.

Thank you a lot, especially for the so detailed report (easy to reproduce)! The bug will be fixed for the next release!

[pham]

Next two problems/bugs/feature requests:
1. The engines log file logengins.txt has logs of many different (replayed) tournaments, so "View engine-log of game x" doesn't load the correct log of game x of the last tournament but it opens the log of the first "game x" it finds in the file. An old one if you don't clean/delete the file manually.

Next to the boxes of log paths there are always delete buttons. Users can click on those buttons to quickly delete whole log paths.

I had considered some other solutions, including auto-delete some parts or all log files when BSG is starting. However, there is some risk of losing data when users really need them.

2. Reopening a game doesn't show tb hits in the graph. Probably because banksiaplaying.json doesn't store the tbhits.

Regards, Andreas

tb-hits maybe stored with the game as computing info. If computing info is "Standard", the game could store only "score/depth elapsed nodes" and no place for tb-hits (missed). You may use TCEC standard instead (Settings->Move& Move panel)

[RubiChess]

This time no bug report but a question: What is this "Margin per move" parameter in the time control settings? I could just guess that a time forfeit smaller than "margin per move" doesn't count as a loss?

Regards, Andreas

[pham]

When an engine is time out, BSG will wait for a period of that margin before ruling the game. If the engine could return a result within that period, the game could be continued, otherwise it will be terminated and that engine will be ruled as a loser (forfeited by timeout).

Margins can help to avoid some delays/lagging between network or communicating between engines and chess GUIs or solve the problem with some engines use all allowing time for calculating and no time left for communicating/lagging.

For standard timers, that extra period is still counted (subtracted from time lefts).

[RubiChess]

Two more problems (tested with latest 0.58 but also in older releases)
1. In settings / general the "score in white view" option isn't an option anymore but seems to be part of the radio button group. Means: If you enable "score in white view", neither "pawn unit" nor "centipawn" are selected. And if you select one of them, "score in white view" gets disabled again. This is probably not intended.
2. If you run an analysis of a game (like 10 seconds per move) the eval graph isn't updated. You have to move the banksia window or resize (or close/opn) the score graph dock to get the latest eval dots. Tested under Windows 10.

Regards, Andreas

[pham]

Thanks. All those bugs are fixed for the next release.

[RubiChess]

Long time no bug report... here it comes:
Banksia sends "isready" after setting the options but it doesn't wait for the engines answer "readyok". Instead it directly sets the position and starts the game.
Option setting can take some time (e.g. setting the hash needs clearing the momory which takes some time) and exactly for this the synchronisation via isready/readyok should be used. See https://backscattering.de/chess/uci/#gui-isready

Here is a log of a tourney that uses concurrency 7 so in the very first round 14 engines including setting their options are started in parallel. This results in losing more than 4 seconds on the first move:

Code: Select all

5.15:09:50 RubiChess current Clang> uciok
5.15:09:50 RubiChess current Clang< setoption name Hash value 256
5.15:09:50 RubiChess current Clang< setoption name Move_Overhead value 10
5.15:09:50 RubiChess current Clang< setoption name Ponder value false
5.15:09:50 RubiChess current Clang< setoption name Threads value 1
5.15:09:50 RubiChess current Clang< isready
5.15:09:50 RubiChess current Clang> info string Allocation of memory uses large pages.
5.15:09:50 Komodo 14.1 64-bit< ucinewgame
5.15:09:50 RubiChess current Clang< ucinewgame
5.15:09:50 RubiChess current Clang> info string Allocation of memory: Large pages not available for this size. Disabled for now.
5.15:09:51 RubiChess current Clang< position fen r1bqk1nr/pppp1ppp/2n5/2b1p3/2B1P3/5N2/PPPP1PPP/RNBQK2R w KQkq - 0 2
5.15:09:51 RubiChess current Clang< go wtime 300000 btime 300000 winc 1000 binc 1000
5.15:09:54 RubiChess current Clang> readyok
5.15:09:55 RubiChess current Clang> info depth 1 seldepth 2 multipv 1 time 1 score cp 20 nodes 36 nps 27341 tbhits 0 hashfull 0 pv b1c3