We rank teams even if we don't publish the rankings. It isn't that difficult to figure out which teams are the better teams. Then we arrange the pools accordingly.
Example... 64 team tournament, 16 pools of 4 teams. The top16 teams go to 16 different pools. The next 16 go to 16 different pools, same for the next 16 and the last 16. The goal is to make every pool as equal as possible. Playoffs are done two different ways. some times they need to be predetermined (winner of pool A vs, winner of pool E) to make the schedule work. Other times the teams are seeded after pool play. 16 plays 1, 15 plays 2 and so on.
Obviously as teams do well they move up the ladder or move down in some cases. This is never perfect, but neither is any other way. We have found this is the best way for everyone concerned, especially in highly scouted events. It gives the lesser known teams games against the highest level teams, which gives the lesser known team games that are heavily scouted.
We have been doing it this way for two decades and I can't remember even one complaint. Once again, perfection isn't really possible no matter what you do. Some might say pulling names out of a hat would be the fairest way. We don't want the very best teams in the same pool and the very weakest teams in another pool. of course we have some tournaments that don't have many weak teams. In that case nobody complains because all the pools are difficult.
The whole idea is to be as fair as possible to every team. Try to make the pools as equal as possible. We have a fair share of upsets, but usually at the end the teams playing for the championship are not much of a surprise.