I personally am torn. What's left on older servers, like 1-3, and probably 4 even, are diehards who love the game, the community, or both. My guild has kept 48-50 players, largely because we are one of the top two guilds. I see growing gaps between the top two guilds and the rest of the server. There are so many contributing factors. (Sorry for getting sidetracked, but I'll come back to the point.)
Players lose interest over time. When new servers come along, sometimes players move on to the new server. Recent discontent has probably pushed away a few as well.
Now for reasons that they stay. The community - do not underestimate this. The recent long overdue updates breathed some life into the game, even though they have not been well received by all. And the game is just hard to put down sometimes, lol.
Since the community is the strongest factor keeping players in, I would hate to rock the boat too much. I wouldn't make guilds smaller. I also wouldn't want to push small guilds too hard to expand beyond their comfortable circle. I think they might benefit to merge and interact with the larger community.
If we can nudge smaller guilds to merge without pushing, I think we can retain more players. So how do we do that? Just brainstorming, I would flatten out the token rewards within the guild, then change the distribution of tokens so that guilds with less than 40 members start at 5th place. I don't want them to get less than they currently do, but I would want the possibility of better rewards to nudge them toward merging.
(This is if I understand your meaning correctly.)