Some breeds are left virtually untouched until the age of five and for them training has just begun. Many trainers feel there is little to be gained by starting too early and find that horses started later in life catch up with peers whose training began much earlier. Certainly, if the goal is to eliminate any possible strain on a growing body, waiting until the age of five will achieve that goal.
Many performance competitions require a horse to be four or five years old to compete. The American Endurance Riding Conference requires that a horse be at least 5 years old before competing. Canadian Pony Club requires that a pony or horse be at least four years of age for the sake of rider and mount.
Any sport such as jumping, combined driving, reining, dressage or distance riding that puts extra stress on a horse's joints shouldn't be started until the age of four or five (or more, depending on the individual).