Tesla Autonomy Day

Based on what Tesla presented in their autonomy day event ( [VIDEO] ) but it's also what they've been saying publicly for a while now... in a nutshell, they ask the cars to send selected summary data back when interventions happen and in certain other cases.

How can they spend all the money on autonomy [1] while refusing to share any more data [2] beyond the couple of sentences in their "safety reports" [3]? This makes no sense. [1] [VIDEO] [2] [LINK] [3] [LINK]

After the recent "Tesla Autonomy Day" video [VIDEO] (almost 4h long, but worth a watch), I do rather feel like the incumbent services are about to be in a spot of bother. I think there's a bubble in this sector, and it's going to pop.

I was worried about the “diffusion of responsibility” issue, so I rented one on Turo for the day and drove around 200 miles on freeways in Northern California. The user interface is really good, you learn pretty quickly the right amount of force to hold the wheel - then you can let it “autosteer”, but taking control is instantaneous, you just make the movement you want and the car disengages autosteer instantly. I probably took over from the car around 10 times that day, mainly in the left-hand lane with the left edge of the freeway marked with 2-foot-high white concrete barriers, it felt like it wanted to go into them. But that might just have been the car trying to drive “properly” in the center of the lane, while I tend to hug the right side in those situations to have a greater margin of error on the left. It definitely reduces the cognitive load, letting you drive better in my opinion. But there isn’t an idiot-sensor in the car, so like every technology, there will be people who abuse it. The interesting question is whether those same idiots would be safer in a regular car, or if, on balance, the “autopilot+idiot” combination is safer - that seems like an empirical question to me. Even at the current level, it’s a great advance on a regular car (I have no idea how it compares to other expensive cars with various assistive technologies, I usually buy super cheap cars...). I also think that if fully autonomous driving is technically possible, then Tesla will be the one to

> They don't want engineers testing code out on real people's lives. When Tesla pushes and runs code in the shadow mode, it is run in parallel to the "production" version of the software. The actual behavior of a car, even with the test code in, still relies on decisions made by the "production" version of the software[1]. The purpose of such testing is to compare decisions made by the updated code vs those made by the old version, and use this info for fine-tuning the next gen algorithms. [1] [VIDEO]

Actually, they did mention radar during Autonomy Investor Day ( [VIDEO] ). Also, [LINK] says: "A forward-facing radar with enhanced processing provides additional data about the world on a redundant wavelength that is able to see through heavy rain, fog, dust and even the car ahead." So it's rather bizarre that you would speculate about how they're not using radar when they explicitly say they do.

> taking consecutive frames (2D images) we can estimate per pixel depth Yeah, I find it odd that they're bringing up Elon's statement about LiDAR, but then completely ignore that they spoke about creating 3D models based on video. They even showed [0] how good of a 3d model they could create based on dat from their cameras. So they could just as well annotate in 3D. 0: [VIDEO]

Tesla routinely gets data from the fleet and analyzes it. Here is Andre Karpathy speaking on this subject: [VIDEO] 1:49:30. I’m on mobile otherwise I would link to the exact spot. Tesla has tens (hundreds?) of thousands of vehicles on the road right now that are continuously feeding training data back to Tesla. What evidence is there that this is insufficient, or even deficient? From what I can gather, Waymo has limited testing geofenced to Phoenix, Arizona. Tesla has cars all over the world with autopilot, all providing training data. I don’t see a reason for your conviction that Waymo has an advantage here.

It hasn't died, the logic holds, and Tesla is still promising self-driving is actually imminent. In addition to the most data, Tesla also has the best self-driving computing hardware you can buy today. (See autonomy day [VIDEO] ) They also have top software people. I think Tesla may very well release full self-driving in the next couple of years.

Watch the presentation from 6 months ago, where they explain the decision to build their own hardware for inferring : [VIDEO] It's not surprising that they also build the hardware for training. Correct me if I'm wrong, but Google use the same TPUs for training and inference, because the underlying operations are the same : multiply then add numbers. Once Tesla built the hardware for inferring, the design of the hardware for training is probably similar. Unlike Google's TPUs, Tesla have a specific use case for the hardware (computer vision for automotive), and maybe than means they can further optimize the computation pipeline with their own specialized hardware.

Same thing with Tesla cars: they are already remotely asking their AI in their cars to send back recordings that they are interested in by giving example photos. It's used for improving their self-driving and they say it's "anonymized", but it's a little scary thinking about how it could be used in the wrong hands (if there is a hack, or some agency forces them). For example asking for faces or to detect certain behaviors. It's not just stored data, but it's data ready for querying with powerful distributed AI in a self-driving remotely controllable car. [VIDEO]

I heard from a tesla engineer that they do use lidar to train their models, but that they don’t see the value in deploying it to the fleet. It’s worth watching the tesla event from about a year ago if you haven’t seen it for their full position: [VIDEO] The quick notes are that our roads are set up for human vision and a truly capable level 5 system will require human level interpretation of visual data (at which point lidar becomes redundant). He thinks lidar with non human level visual interpretation won’t get us to full autonomy. They see lidar as a short term half measure, a local maximum that’s a distraction from what has to get done anyway.

I don't believe it's a critical component of Autopilot. From the last public video release in 2019 [1], it seems like they use the stereo cameras to derive depth information, and use radar as a kind of ground-truth in informing the model. I would think the source-of-truth for what the Autopilot system thinks is the position of objects is the stereo camera, i.e. the radar is not part of the online flow, but is used offline to correct the model. See specifically at 2:20:22 [2]: > We use that radar to annotate what the vision is seeing, the bounding boxes that come out of the neural networks, so instead of human annotators telling you that this car, in this bounding box, is roughly 25 meters away, you can annotate that data much better using sensors. I take that to mean that the radar is used as part of supervised learning for updating the model offline, whereas just the cameras are used as input to the model. [1] [VIDEO] [2] [VIDEO]

> It's verbatim from Elon Musk on Autonomy Day 2019 No, that was NOT a verbatim quote. Here is the exact quote with me adding emphasis on the critical (but oft omitted) qualifier: "All Teslas being produced right now have this computer. We switched over from the Nvidia solution for S and X about a month ago, and we switched over the Model 3 about ten days ago. All cars being produced have the all the hardware necessary, compute and otherwise, for full self-driving. I'll say that again. All Tesla cars being produced right now have everything necessary for full self-driving. All you need to do is improve the software." [VIDEO] > This is the same event where he promised a million Tesla robotaxis on the road by 2020. That is, in my opinion, not a contextually correct reading of what Elon said. Within the context of the speech it's clear (to me) that for him, "robotaxi" is used as shorthand for "vehicles with the hardware capable of being a robotaxi." Obviously I cannot prove this, and I recognise that responsibility ultimately lies with Elon to be clear. I do agree Elon did say multiple sentences that are absolutely misleading when read in isolation, but his actual words in context (and with some recognition that this is a forward looking statement made with optimism) were far less egregiously wrong than the snipped quote implies on its own. You can watch him say it in context and make up your own mind: [VIDEO]

Thanks for the question and the papers. Like some of those companies, we are believers in the data centric approach to ML. But labeling is not our focus (unlike Snorkel, HumanLoop, Prodigy or Cord.Tech). We focus on model diagnosis and mining the best data to improve them. So there are more similarities with Aquarium or Lightly. There is a great talk from Tesla [1] on what they call the "Data Engine" (that probably inspired some of us :)). One of the things we took from it was that in order to truly close the loop on the ML data flywheel, we needed to turn production into a reliable datasource. It had to become accessible, understandable and minable. To achieve this we took the approach of combining ML observability with Active Learning mining frameworks. Combining both is important in our view because Observability tells how the model behaves in the real world and Active Learning finds the right samples to fix / improve the model on real world data. They go hand in hand. Technically, it means that we integrate with serving and labeling platforms. We ingest data both in streaming and batch. We can mine on production streams including on device (for iot use-cases where accessing data is a challenge). We have an extensive set of metrics to understand model behavior in the wild and solve use cases like data drift (detecting it, triggering mining and sending the data for labeling/retraining). And we are geared toward automation. Regarding which data sampling works well and which

Tesla Autonomy Day 2019-04-22: "By the middle of next year [2020] we'll have over a million Tesla cars on the road with FSD hardware feature complete at a reliably level that we would consider that no one needs to pay attention. Meaning you could go to sleep. From our standpoint, if you fast-forward a year, maybe a year and 3 months, but next year for sure we will have over a million Robotaxis on the road. The fleet wakes up with an over-the-air update. That's all it takes." Source: [VIDEO]

Yes they have. Tesla Autonomy Investor Day 2019 [VIDEO] At 3:13:41. Robotaxis is clearly written on that slide. Elon Musk is talking about robotaxis for minutes as well during the official presentation to investors.

You don't understand. Next year, for sure, Tesla will have over 1 million rob taxis on the road. [1] [1] [VIDEO]

Sources: 1 [LINK] 2 [LINK] 3 [LINK] 4 [LINK] 5 [LINK] 6 [LINK] 7 [LINK] 8 [LINK] 9 [LINK] 10 [LINK] 11 [LINK] 12 [LINK] 13 [LINK] 14 [LINK] 15 [LINK] 16 SXSW 2018 17 [LINK] 18 [LINK] 19 [LINK] 20 [LINK] 21 [VIDEO] 22 [LINK] 23 [LINK] 24 [LINK] 25 [LINK] 26 [LINK] 27 [LINK] 28 [LINK] 29 [LINK] 30 [LINK] 31 [LINK] 32 [LINK] 33 [LINK] 34 [LINK] 35 [LINK] 36 [LINK] 37 [LINK] 38 [LINK] 39 [LINK] 40 [LINK] 41 [LINK]

Oh hey I get to copy paste a list of times elon said FSD would come out next year![1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16] [1] [LINK] [2] [LINK] [3] [LINK] ... [4] [LINK] ... [5] [LINK] [6] [LINK] ... [7] [LINK] [8] [LINK] [9] [LINK] ... [10] [LINK] [11] [VIDEO] [12] [LINK] ... [13] [LINK] [14] [LINK] ... [15] [LINK] ... [16] [LINK] (Or simply, [LINK] )