Sunfish severely slows down my PC

[tissatussa]

i used other python chess engines, but esp. SunFish seems to use much CPU time .. my notebook almost freezes now and then while playing in CuteChess (GUI) .. i also tried your packed version and even Pypy but it doesn't help .. did anyone experience this also ? Does an explanation / solution exist ?

[thomasahle]

How do you run it?
Are you saying it loses on time?

[tissatussa]

How do you run it?

just fill the command pypy3 ./sunfish.py in the setup window.

Are you saying it loses on time?

no, it seems when SunFish tries to reach higher depths, it consumes much CPU time .. even my computer clock (with seconds) freeezes and skips .. other processes are almost stuck - for a while .. after Sunfish made its move, CPU is normal again .. i know Python processes can have such behaviour in general .. btw. i use Xubuntu 22.04

[thomasahle]

I mean, it uses 100% of one core while it thinks. All chess engines do that.
Are you perhaps running out of memory?

[tissatussa]

Are you perhaps running out of memory?

i almost never have problems with memory, although normally i have several programs running .. but i checked the memory usage during play, and indeed : my memory (8 Gb total) is almost full .. so, thank you for suggesting, this could certainly be the cause -- eg. i never knew FireFox did take that much memory, having multiple tabs open ..

after closing a few applications, SunFish seems to run fine ..

after closing CuteChess (when the SunFish game ended) my OS made about 5 Gb memory free, that's a lot !? I guess it accumulated during play .. it that normal ?

[thomasahle]

I always assumed sunfish was too slow to accumulate a lot of memory 😅. It should free some every move.
How long games did you play?

[tissatussa]

..It should free some every move..

i seems it doesn't .. i just did another test in CuteChess GUI : SunFish played a 20 minute game (per player), which can be considered "long", and i had very few applications open, using only 1.3 Gb of my max 7.6 .. while the game proceded, SunFish claimed upto ALL memory, leaving only 0.5 Gb at the end of the game .. the other engine is no Python and has a Hash option, which i set to 128 Mb .. letting this engine play with another non-Python one, the memory didn't increase at all until the end of their game .. however, another Python engine called PyGone (v1.5) somewhat has the same behaviour as SunFish : at the end of a (10 minute) game it had claimed about 2 Gb (and PyGone doesn't have a Hash option) .. after starting another game in CuteChess and then closing the tabs of the ended previous matches, memory is made free instantly - as you see in my graph screenshot .. doesn't (your) Python script free up memory after the game is finished ? Or should CuteChess do that ?

thanks again for pointing me to this memory issue concerning Python (chess engines) .. if you want more info, i can supply it.

[thomasahle]

Ok, try to open sunfish.py and after the line self.tp_score.clear() (line 402) insert

if len(self.tp_move) > 10**6:
    self.tp_move.clear()

and see if that helps!
(You could even try it with a lower number than 10^6 too.)

[tissatussa]

that helps ! I did some test games again in CuteChess GUI, 20 minutes per player .. during the first game the memory usage climbed upto only 1.5 Gb .. so, i added a few code lines to log all moments .clear() is done plus the length of self.tp_move .. here's that data :

1366631
1645433
1108689
1061677

the .clear() was done 4 times and the max length once was about 1.6 Gb - am i right?

if you decide to implement this in the main version, does it mean i discovered a bug ?

what is self.tp_move anyway ? Some kind of Hash ? If so, then clearing it could also mean valuable data is lost and thus makes the engine play weaker afterwards ? Btw. you could create a UCI Hash option to set a max, because it's not fair : i gave only 128 Mb Hash to the other engine ..

[thomasahle]

Yes, clearing tp_move often will make sunfish play weaker.
Sunfish is currently quite inefficient with memory because it uses the standard python dictionaries, which store the whole position, rather than just the hash.