Andrej Karpathy - AI for Full-Self Driving at Tesla

It's not luck. I have a degree in molecular biology, and I took a lot of elective courses in immunology, oncology, virology. I worked in a research lab on HIV vaccine for two years after undergrad before I switched back to software engineering. So I was able to keep a cooler head about the virus than most, which enabled me to buy intelligently. I worked damn hard to earn that degree and in my time in the lab. I was a twitchy premed student that needed every grade to be perfect. NONE of the effort I poured into those six years of my life was luck. It was pure sweat. In the immediate future, I expect that the dip in advertising will cause Q1 earnings from Google and Facebook to be lower than expected. As the Fed continues to inflate stocks, this will cause a lot more buying of TSLA and AAPL. In the 2020s, Tesla will continue to soar because: 1) they have a huge lead in battery tech. they're about to announce that they've achieved the $100/kWh mark, which will let them sell electric cars that go as far as gas ones but don't cost more. 2) batteries are crucial for amazing tech like the emerging eVTOL scene. it's possible Tesla may enter that space and dominate it, because batteries determine everything there, too. in the meantime, I bought some shares of eHang (Chinese company, listed on the Nasdaq under EH) 3) they have a huge lead in autonomous driving. LiDAR is a dead-end, and they're ahead of everyone else there, too: [VIDEO] 4) demand is not a problem at all, especially in C

Here's the most recent talk by Karpathy about this topic: [VIDEO] (there are more if you google for "karpathy talk") Here's the summary via 2 examples. 1. Example of gathering data that needs further labeling To implement neural network (NN) to recognize stop signs they program the cars to recognize things that look like stop signs and deploy that to the fleet of over 600 thousand cars on the road. The cars send those "might or might not be a stop sign" images back to tesla and they get manually labelled and added to a training set. They gather enough images and the "recognize stop signs" feature is done. Apply the same logic to other recognition tasks: recognize cars, pedestrians, animals, speed signs, traffic lights etc. Yes, labeling is expensive but everyone else (including Waymo) has the same cost. But Waymo has even bigger cost of driving around to collect those images. Tesla makes gross profit on every car they sell and they got 600 thousand people driving for them for free. Waymo has 600 cars, each of them reportedly costing over $200k and they have to pay drivers at least minimum wage for each hour of driving. That's why Google is reportedly is spending $1 billion a year on Waymo and why they need $3 billion of additional investment to keep going. 2. Example of gathering data that doesn't need labeling Consider implementing NN to recognize cut in i.e. other cars entering your lane in front of you. They deploy a first version trained on small sample, running in shadow

LIDAR vs camera is a red herring. The fact that Elon and his fan club fixate on this shows you how little they understand about self driving. The fundamental problem is that there is no technology that can provide the level of reasoning that is necessary for self driving. Andrej Karpathy's most recent presentation showed how his team trained a custom detector for stop signs with an "Except right turn" text underneath them [0]. How are they going to scale that to a system that understands any text sign in any human language? The answer is that they're not even trying, which tells you that Tesla is not building a self-driving system. [0] [VIDEO]

If you actually believe that Tesla will fail to deliver full self-driving in the coming years, I have two questions. 1. have you watched this entire technical presentation made by Andrej Karpathy, Senior Director of AI at Tesla? [VIDEO] 2. if you understand what you've seen in that video, why do you think Tesla will fail?

You set goals and work towards them. Becoming an inter-planetary species doesn't happen in a year. I want SpaceX to succeed and they have a track record of execution such that I now believe they really can. I was hopeful before (and if you listen to Musk talk about it he didn't think they'd be able to really pull it off early on either but figured they'd at least make progress towards it even if they failed), but now I think a mars colony is a real possible outcome. It's not a bold take to just state something is impossible until it happens, that's pretty much the default. The bold take is to look at what might be possible and execute goals in pursuit of that. For SpaceX this means reusable rocket technology to bring costs down (massive success here has them ahead of everyone else). Starlink as a revenue source is also a really good approach. For Tesla it's the 'master plan' of roadster -> model s -> model 3, reinvesting in infrastructure and battery technology with vertical integration to build out superchargers and drive costs down. This has been massively successful and their EVs (particularly the model3/y) have no equal at any price point EV or gas. The level 5 autonomy was really a bonus on top of that EV transition that they've added to, and if anyone can pull it off it will be Andrej Karpathy and the fleet of Tesla's they can train with ( [VIDEO] ). Bullshit and really big ideas can sound similar, but that doesn't mean they are - there's a lot of value in being able to

It's not the same: [VIDEO] You can watch that talk and see the approach they're taking. Maybe you're more skeptical than they are about the near term possibility, but you can see that the work and progress is real. The AGI risk is real too. What Milton said doesn't make semantic sense, it's not a question of timelines. [LINK]

Better yet, let's get rid of cars altogether from city streets. This is coming along very nicely... [LINK] Autonomous driving is just around the corner, too, which means that there is no need to waste space on parking structures, either. Here's a Tesla engineer that gave a great technical talk demonstrating their progress as of early this year: [VIDEO] EDIT: yes, there are clear reasons why having an three-dimensional underground structure will absolutely affect the social dynamics above-ground in a city, yet it remains true that it is possible to banish cars from downtown cores in the near future (i.e., say by 2030 or so), given the present shift to electrification and autonomous driving. These issues are worthy of debate, and I'm just trying to shift the conversation to what's current. I'm not trying to rain on anyone's parade! I love the goal of city streets for people, as exemplified by the beautiful laneways in Melbourne (11 minutes): [LINK]

I'm sorry you missed out on one of the best investments of the decade, but there is a lot of room left for this to run. Tesla's coming FSD and robotaxis still haven't been priced in. [VIDEO]

Removing the cameras may be a good idea, but note that their extensive fleet of camera-equipped vehicles is an important part of their potentially VERY lucrative self driving software development. EDIT: I highly recommend watching head of AI at Tesla Andrej Karpathy's talk on their self driving architecture. "The Fleet" as a source of data is a vital functional part of their self driving software development. They may be able to calculate that including cameras and a computer on a car that will ship in high volumes would increase sales of their self driving option across all Tesla vehicles (including that low cost model) enough to justify the inclusion of those parts. [VIDEO]

This is anti-Tesla astroturfing, so I flagged it. If you'd like a high-level technical overview of Tesla's FSD efforts, check out this talk by Andrej Karpathy from February 2020: [VIDEO] There are literally thousands of videos of people filming the FSD beta doing amazing things. They are making great progress very, very quickly: [LINK] As you can see from videos like the one above, real-world data is needed to make FSD practical. There is a very long tail of things to get right, so blocking this progress with regulations would effectively outlaw FSD, or delay its development by a very long time. Given the number of road accidents that human drivers cause (think of just sleep deprivation and drunk driving), delaying FSD through regulation would ultimately cause a great number of uncessary deaths.

Not yet. I have a four wheel drive robot I designed with four 4k cameras feeding in to an Nvidia Jetson Xavier. [1] Just getting it to navigate itself using vision would mean building a complex system with a lot of pieces (beyond the most basic demo anyway). You need separate neural nets doing all kinds of different tasks and you need a massive training system for it all. You can see how much work Tesla has had to do to get a robot to safely drive on public roads. [2] From where I am sitting now, I think we are making good inroads on something like an "Imagenet moment" for robots. (Well, I should note that I am a robotics engineer but I mostly work on driver level software and hardware, not AI. Though I follow the research from the outside.) It seems like a combination of transformers plus scale plus cross domain reasoning like CLIP [3] could begin to build a system that could mimic humans. I guess as good as transformers are we still haven't solved how to get them to learn for themselves, and that's probably a hard requirement for really being useful in the real world. Good work in RL happening there though. Gosh, yeah, this is gonna take decades lol. Maybe we will have a spark that unites all this in one efficient system. Improving transformer efficiency and achieving big jumps in scale are a combo that will probably get interesting stuff solved. All the groundwork is a real slog. [1] [LINK] [2] [VIDEO] [3] [LINK]

You can learn about their systems by watching talks by Andrej Karpathy. As a robotics engineer interested in vision, their architecture is inspiring. This talk [1] is a good overview but each talk he gives is a little different so search for more if you want to know as much as possible. But the big thing is that their autonomy computer can be programmed to look for odd scenarios and send them back home. Tesla uses their fleet of hundreds of thousands of cars to collect edge cases like this, and then they have a kind of compartmentalized neural network system that breaks apart disparate tasks. With their collected examples they can create unit tests to ensure that the moon stops activating the stoplight detector. Once trained, the unit tests presumably help ensure they don't end up with future regressions. So basically every time you see a Tesla do a weird thing, there is a good chance it will stop doing it soon enough. At least if it's hitting hacker news. [1] [VIDEO]

There is a mix of a "neural net planner" and a "explicit planning and control" in traditional code[1]. The explicit planner has the last word, and uses input from both the "vector space" and the neural net planner. Karpathy has commented about the neural nets gradually replacing the traditional code. See his presentation[2] around 18:45. [1] [LINK] [2] [VIDEO]

> All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update. What is this evidence? I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models. [0] [VIDEO]

The actual report is here: [LINK] It failed to detect a stationary mannequin in one specific scenario. Does this generalize? Does this mean there's a major safety problem? These would be assumptions that this test can not prove. A comprehensive safety analysis has to also include these scenarios: [VIDEO]

They are reliable enough to use for FSD: [VIDEO] so i think it's reliable enough for park assist, summon etc. Yeah you're def right about the moisture sealing... it's been hit or miss on production tolerances