Dictionary.com defines sport as "an athletic activity requiring skill or physical prowess". It goes on to specifically mention "racing, baseball, tennis, golf bowling, wrestling, hunting and fishing" as sports. Oxford Dictionary defines sport as "an activity involving physical exertion and skill in which an individual or a team competes against another or others for entertainment".
According to these definitions and my own views, I'd say that a lot of things are sports. Formula 1 is often not considered a sport since the drivers aren't being very physically busy, but real athleticism is displayed in the ability to sit in a tiny bucket at top speed and still operate a vehicle in an efficient way without crashing. Also on E-Sports; it involves exertion of the brain and excessive focus, and there is certainly a need for fast reflexes and dexterity in operating the controller.
But also, if millions of people are entertained, why not call it a sport? What IS e-sports and racing in that case; a very real Netflix series?