Is archery, or golf not a sport? I'm usually a big detractor of saying that the tolls of eSports can be compared to physical sports, where the toll is equally physical and mental as opposed to eSports which is more mentally draining.
But competitive gamers need coordination and dexterity to make the moves they do. If they were playing a turn-based strategy game like chess, I'd understand, but these players' performances are entirely dependent on their physical dexterity and reaction times.
You can't be a top competitive pro gamer without elite reflexes and hand-eye coordination (heavy on the "hand" part). It's definitely a sport, and some people do end up with injuries (albeit not as serious as Rugby, Boxing, or NFL injuries) like permanently damaged nerves & motor functions in their hands, forearms, and wrists. While the physical exertion is limited, it is still a factor.