@jonas
I've always had cover JPGs in the folders (never auto-downloaded), and this is the only app this has happened on! The folder you've directed me to also doesn't have cached covers (I'd already turned off that tickbox), just "strawberry.db".
I can confirm that Cover Manager shows them separately with the correct covers in the big grid, so it really does seem to be the sidebar with albums as the list that has this issue -- see attached image (RSF are the top set of tracks):
Right Said Fred cover on REM.png