Maybe. Certainly he's the greatest swimmer. Even though NBC and other commenters laud Phelps as the greatest Olympian or even the greatest athlete ever, I think there's a great deal of hype in the claim.
Two reasons. First, swimmers have lots of events to compete in (16), thus having lots of opportunities to win a medal. How many different events can boxers compete in? One, boxing. How about shot-putters? One. Field hockey? Pole vaulting? All one event.
Second, the various swimming events are very similar. If you can swin the 200 meter backstroke, obviously you can swim the 100 meter backstroke.
I'd prefer to nominate athletes that compete in multiple sports, like decathletes, or that compete over many years, like Dara Torres, who medaled in 1984, 1988, 1992, 2000, and 2008 as the greatest.