Mark Rober is perhaps the most successful science & engineering Youtube creator. He has 65 million subscribers--more than Taylor Swift, and many of his videos have over 120 million views, which surpasses the most recent Superbowl. His most recent release compares a Tesla dealing with various road issues, in contrast to a test car with a Luminar LIDAR. He tests how both cars react to a dummy child in the road obscured by fog, water jets, bright lights and even a giant printed poster showing an empty road, as sometimes seen in the Road Runner cartoons.The goal of the test is to compare the performance of the Tesla, which uses only cameras and computer vision, and the LIDAR based system, which uses an advanced scanning laser to create a 3-D view of the road ahead of the car. All working self-driving car systems operating on roads without a human supervisor use LIDAR, radar and cameras and vision, and sometimes more. Tesla, and some startups, are trying to work with just vision.Sadly, Robers large effort here is almost entirely wasted, because while the title of the video says it is a self driving car test, he uses an un-named version of Teslas Autopilot system, which is their older freeway driver-assist tool. Nobody ever considered Autopilot to be a self-driving system. It is more like a fancy cruise control, and it frequently makes mistakes and needs a driver attentive to the road and with hands on the wheel to keep it safe. Tesla moved beyond that system several years ago, and now markets a system called supervised full self driving. In spite of that seemingly explicit name, it also is not a self-driving system, and needs supervision, though Tesla has for many years claimed it would become a self-driving system next year without success. Most recently, it was announced it would operate unsupervised in June in Austin, TX, though few believe this promise will be delivered on in a meaningful way.Nonetheless, the new FSD system (which only works on Teslas latest hardware, found in cars released starting in 2023) is significantly different from the old FSD system, and that is quite different from the old Autopilot as well. The newer Autopilot is derived from the FSD system, but still fairly different. Frankly, few are interested in tests on the old Autopilot, so that makes Robers tests fairly meaningless, even if it is the case that the newest FSD system would have made the same errors--we unfortunately dont know.Luminar has done a similar test at CES with one of their cars and a Tesla with small dummies.AFP via Getty ImagesWhile Rober may not have known about the software versions available for his car, the crew at Luminar, the LIDAR company which assisted with the tests, provided the LIDAR car and system, and has done these tests before, would surely have been aware of them, and should have informed Rober.The tests performed were designed to highlight the difference between computer vision and LIDAR. The LIDAR car performed them all, while the Autopilot system failed in thick fog, very heavy water jets and the Road Runner photographic wall. The Tesla also failed with Autopilot off, leaving just its forward collision avoidance system. The FCA system is designed to delay braking to give the driver a chance to brake first, as such it stopped too late.Whats a good test?Another problem with Robers tests is that the ones where the vision system failed do not model the real world well. While all teams test their vehicles in fog and rain and in the face of bright lights, they generally do not demand success for conditions which do not appear in the real world, and which a human could not handle. They do test these to see how they could improve the systems, but real testing of self-driving cars involves millions of miles and a very broad array of challenges on which the system is hoped to surpass human performance.The road runner wall was a billboard sized canvas printed with a photograph of the empty road beyond the wall. Those are rarely seen in the wild outside of certain regions of New Mexico. Self-driving teams are interested in potential adversarial attacks on cars and how to defend against them, but this is not a likely one; there are simpler and cheaper techniques. Its more in line with the fun that Robers viewers enjoy in his videos than a real test. A visual image is of course something that will foil cameras and not adversely affect radars or LIDARs.All successful self-driving teams, who have managed to deploy a vehicle with no human inside, use a combination of cameras, radars and LIDARs. The vision only approach is used only by Tesla and some smaller startups, and none have made a working self-driving car, or even come close to it, as of yet, though they have hope for the future. Indeed, most experts think that approach may well work in the future, but just when is unknown. For now, they feel that making use of all sensors is important to get the vehicles working, and later work can be done to make things cheaper, including going to just vision if and when that becomes practical.As such, a demonstration of the abilities of LIDAR and the problems vision can have makes sense. Teslas latest FSD probably would have failed some of these tests, but it was not tested so Robers video leaves us without the information we actually might seek. Rober has not, as yet, responded to a request for more information. The problems shown are not the most important and difficult problems for computer vision to solve, however. Its biggest problem is attaining the extremely high reliability level needed to bet your life, not specific hazards like fog or rain or clever coyotes.The Luminar LIDAR performs well. Some LIDARS might have issues with some of the challenges presented, including opaque fog and extremely heavy rain. None would have an issue with the photograph wall. Luminars LIDAR uses long-wave infrared, which allows much more power and range. Waymo has a similar instrument of their own construction on their vehicles. (I worked in a minor way with the team which developed it.)We might hope Rober would redo his test with Teslas latest software and hardware, and also come up with more real-world situations to tell the real story.