He plays around the same time range every day, early in the day so not many players. Would make more sense just to void the matches completely when a good portion of the match had mostly bots in it no matter how many human players finish the game, the balancer shouldn't update the values if there is less than x humans in the match at then end already, that could also lead to his balancer score not decresing when he's playing against better players until there are enough human players in the match.
The scoreboard on the left isn't related to your score for the balancer, type !stats to see the balancer values.