The term Utility player could mean either.
More often than not, In MLB it is a backup player who can play multiple positions. A very valuable asset in a 162 game schedule.
But it can also be a starter like Chone Figgins of the Angels. He starts most games, but his position changes often. He played in 158 games last season, 56 at 3B, 50 in CF, 42 at 2B, 15 in LF. 8 in RF, 7 at DH and 4 at SS.
In youth baseball, players play all over the field, so most players are utility players.