[CVPR'21 WAD] Keynote - Andrej Karpathy, Tesla

If you're interested, Tesla's head of AI recently did a public presentation of their new vision based depth system. It's worth a watch if you find this stuff interesting and enjoy learning about the forefront of technology. [VIDEO] A key takeaway is that they've been running this stack in shadow mode (validating output but not controlling the car) on everyone's Tesla for quite some time. Equivalent to 1000 YEARS worth of real world driving. And from this data they've proven it is now superior to radar in all circumstances.

FSD was advertised as paying for something that would come in the future - I bought it under that assumption (and so did everyone I know that has it). That was and remains true. They've continued to make progress towards it: [VIDEO] Autopilot is different and isn't advertised as level 5 - 'autopilot' implies fancy lane keeping not turn by turn point to point autonomy. Elon's FSD timelines were wrong - that isn't fraud. People say everything he's trying to do won't work constantly, just because they're occasionally right doesn't validate all of the times they're wrong (e.g. Model 3 is vaporware will never ship, the company is bankrupt and will close in two weeks, the Model S will never ship, gigafactory is a mistake, roadster means Musk only cares about EVs for the rich etc. etc.) If you can't tell the difference between Elon Musk and the success of Tesla and SpaceX or think it's equivalent to something like Theranos or Nikola I'm not sure what to say about that, there's just not much point in further discussion. Your pejorative use of billionaire also suggests some political element.

They're not just 'tweaking some settings': [VIDEO] What they have today isn't FSD (which they state explicitly), but I'm happy to give them extra money early given the work they're doing on the chance that it works in the future. I think they have the best chance of success of actually shipping something real.

IIRC, in this[1] video Andrej Karpathy mentions radar was mostly redundant, and in addition was occasionally harmful (sending conflicting signals vs. video), so they got rid of it to reduce cost and complexity. [1]: [VIDEO]

No. Newer versions of Tesla's software disable the radar on cars that shipped with it.[1] Vision has gotten so much better (thanks in part to training from radar data) that radar adds noise to their model of the outside world. Andrej Karpathy explained the issues in a talk this summer.[2] 1. [LINK] 2. [VIDEO]

Given that radar is functionally blind[1] to static objects when travelling at high speeds, I'd argue it's not just a good thing, it's a necessary thing. Tesla have demonstrated the superiority of their vision approach and validated it with 1,000 years of real-world driving, running it in shadow mode on customer vehicles.[2] [1] Technically radar isn't blind to static objects, but since it cannot disambiguate them from tunnel entrances and highway overpasses, the data is useless. Therefore it's functionally blind. [2] [VIDEO]

From the keynote of CVPR. CVPR is the premier annual computer vision event comprising the main conference and several co-located workshops and short courses. [LINK] [VIDEO]

> why not use binocular forward facing vision? Because Tesla have demonstrated that it's unnecessary. The depth information they are getting from the forward-facing camera is exceptional. Their vision stack now produces depth information that is dramatically superior to that from a forward-facing radar. [VIDEO] (It's also worth noting that depth information can be validated when the vehicle is in motion, because a camera in motion has the ability to see the scene from multiple angles, just like a binocular configuration. This is how Tesla trains the neural networks to determine depth from the camera data.)

Radar has poor resolution, which means it can't always differentiate between an object near the car's path and an object in the car's path. The radar can give false positives when there's a truck on the side of the road or a metal sign on an overpass. Then you have to use the vision cameras to figure out what's really going on. Radar also loses its "lock" sometimes, which can be an issue with cars braking quickly. The radar will say "no obstacles detected" when the car in front of you is slamming on the brakes. Again, you have to use the vision cameras to figure out what's actually going on. The people at Tesla decided that the radar was creating more noise than it was helping with, so they removed radar last year. Andrej Karpathy explains it in far more detail in this talk: [VIDEO]

> These are the same things that can cause a traffic accident for a human with human eyes, so there's really no logical reason to assume that you can create a more performing and safer computer driver by emulating human vision only. Okay. This is a concrete claim that I think is falsifiable. I'll give not just one, but multiple arguments. Argument 1: Connect Four, Chess, Checkers, Go, Limit Hold'em, No Limit Hold'em, Poker, DOTA, and Starcraft II are all examples of games in which AI outperforms even the best humans despite having equal sensory access. Driving can be reduced to a problem comparable to playing an imperfect information game and we can play imperfect information games at superhuman levels using computers. Therefore, it is reasonable to expect that even human-level sensory access can enable safer than human performance. Argument 2: Currently we have statistics on driver-assistance programs that demonstrate that human + technology assistance is superior to human alone. This holds true even in cases in which those assistance technologies are only using vision. So we have a contradiction. If it wasn't possible to do better than human with only human senses, we wouldn't be able to point to many different situations in which driving with an AI clearly outperforms humans, but we can. For at least those subsets of AI driving, we can say that an AI driver will outperform humans. > I can't understand why they removed radar. Check out [VIDEO] for a very detailed breakdown

Karpathy covered it during a conference keynote here: [VIDEO] It also is pretty common with other vehicles, they just don't get as much press as Tesla: [LINK] [LINK]

Appreciate the downvote/disagreement. But here is Karpathy explaining how vision can be used to measure distance to objects accurately[0] Here is the fact that LiDAR doesn't work in the rain[1]: "... In heavy rain, for example, the light pulses emitted from the lidar system are partially reflected off of rain droplets which adds noise to the data, called 'echoes'." Which logically implies you need to revert to vision, as see [0] also for why Radar is unreliable. [0] [VIDEO] [1] [LINK]

He is talking about the long technical talk at CVPR. [VIDEO] For example look at this section: [VIDEO] Also if you look into it you will find Google's self-driving car project ran into similar issues. They upgraded the radar in response to the issues rather than removing it, but there are actual issues. Vision only was better than radar plus vision in their stack. For real world situations that frequently come up in driving radar wasn't worth it. The theoretical accuracy of the sensor fusion didn't happen in practice because in practice you needed to have vision tell you when sensor fusion wasn't going to be accurate. It isn't naive sensor fusion as in toy math models of sensor fusion. So when there was a disagreement you needed to go with vision. Yet vision was already able to predict both the failure of and the actual result of radar. In theory, you could just have done the more complicated sensor fusion. In practice, the relative advantage of improving other parts of the vision stack far exceeded improving the sensor fusion. Over time one would expect the relative advantage to change as other improvements were made such that this was no longer true. That doesn't mean the time at the decision wasn't correct, it just means people who evaluate it with future changes already in place are suffering from hindsight bias/committing an anachronistic fallacy.

This talk was worth watching (any Karpathy or Carmack talk is imo): [VIDEO] Specifically this is the relevant part: [VIDEO] They removed radar because it was making things worse.