You did not get away with one, but your reasoning is incorrect.
First of all High School rules allows the pitcher to be removed and re-entered as pitcher provided he doesnot leave the game. Thus because of you "local" rule what you did was correct.
The real issue is that once a pitcher a relief pitcher enters the game (this occurs once the pitcher takes his place on the pitcher's plate)he must pitch to a batter.
Winning and losing pitchers: Starting pitcher has pitched the 1st 4 innings. Therefore, regardless of what happened the "Pitcher of Record" was the starting pitcher since he pitched 4 innings.