2y ago

Rivian says "fat finger" caused software update to brick infotainment systems, physical servicing may be required

Rivian says "fat finger" caused software update to brick infotainment systems, physical servicing may be required::Today’s cars are more like computers on wheels, and even a seemingly routine software update can lead to unexpected consequences. Rivian unfortunately experienced a “fat finger” mishap with their latest software update, bricking infotainment systems [...]

46 comments

I don't have to update my car in order to drive it. Sucks to be them
- I thought the Rivian vehicles were still drivable? It was just the infotainment that’s impacted, no?
  
  Imagine paying $90k for a vehicle and you can't even turn on the radio.
- Congrats on having a nicer experience 2 out of the 365 days in a year. Bravo.
  
  Ok man what if it's an emergency. Here's an example: My wife is in labour, but my car has to update so I can't drive her to the hospital. Now I'll have to pay a more than 2 grand ambulance ride since it was out of network.
I would think they would keep at least 1 of each model/trim of vehicle for testing these things. This leads me to believe one of the following:
Too tight of deadlines
Cheap management won't pay for testing time or units
Culture of pencil whipping
A bad apple didn't do their job, which should be caught by procedures
- I got an offer for software engineering role at Rivian a few years ago. The pay was low.
  
  That's typical of the automotive industry, they don't pay anyone reasonably, they treat the workers like shit, and then they get mad when the workers protest, unionize, or quit. This is how the automotive industry works from the top down, from the highest executive to the lowliest salesman.
  Also from an IT/Development perspective, the money is usually meh, but also the technology is usually expected to be cutting edge, while on the backend they're cutting corners and costs, thus undermining that "cutting edge" factor.
- I think it's more likely just typical dysfunctional software industry workflows. Companies that actually test their software adequately before deployment are the exception, not the norm. That's different from what you said in your second bullet point because it's not even an issue of cheapness, it's an issue of not actually understanding what the best practices need to be.
  
  I mean... those "typical dysfunctions" are what OP described. You're just describing the general state of the industry, not providing further examples of disfunction.
- All of the above.
OTA updates are very much a double-edged sword.
Usually it's good, but sometimes you get crap like this or when Tesla downgraded a bunch of Model S batteries when they saw they weren't holding up as well as expected and wouldn't line up with their battery warranty.
E: and it's insane to me that cars don't have snapshots of previously installed firmware/other software. Or like an A/B partition style thing where you can revert if an update goes bad.
I'm a tech enthusiast, but I increasingly am slowing my roll on all updates but security updates. The era of heavily QAing feature updates is ending, which is leading to some frankly annoying issues across the board, from car software, to phone and computer software. Even GPUs don't seem to be immune from occasional bugs in the latest updates.
Companies and orgs really need to dedicate more time to making sure updates are vetted and clean before releasing them. It might cost you some money up front, but it saves you PR, security issues, and hotfixes in the longer run.
- in the longer run.
  I have yet to work for a company where management cares about this. It's always about what can be sold next week if we rush it enough. Or, more commonly, what was sold weeks ago without any consultation with the technical team.
  
  It's a problem everywhere. Rush to build something, then don't actually finish it, or iron out the bugs before moving on to the next sexy thing. Management wonders why everyone is on the super ancient, rocksolid platform and hasn't really taken the bait on the series of products that have an "agile" development cycle and were dropped after about 5 years.
  Guys....my entire team knows the issue. Support your damn products for years. Commit to improvements for a decade, and help people transition to a new product when the old one has to be sunsetted for technological or knowledge worker retirement reasons.
Weird. My primitive old Samurai started right up this morning, and it's "infotainment" system (bluetooth boombox bungee-corded to the floor behind the seats) still works.
Maybe if I'd paid $90,000 for it I could be privileged enough to have a broken functionality.
- How are you replying from an i386 based pc running windows 3.11?
  
  Slowly
I think while the topic is up it’ll be fun mentioning that the Colorado/Canyon does not have a physical headlight control anymore - in favor of defaulting to Auto and touchscreen controls and the project lead(?) claimed that the system was 100% bug-free.
Also later there was a bug with some OTA update for that model that’ll kill the battery.
Anyways, I bought a 4Runner and immediately called to disable its cellular radio. (dubbed DCM in Toyota-land)
- How do you disable the DCM?
  
  There have been some experimentation but the best (IMO) option is hitting the SOS button to call and navigating the phone menu to get to a representative, not emergency services and having them disconnect your car. You may need your VIN, you’ll need to confirm that you do not want connected services and it may take a day or so to take effect. Now, my SOS button doesn’t have the green light and while the radio - according to the infotainment - is still powered, it is no longer connected to the network.
  Another way if you don’t plan on using the microphone (like for calls) you can pull the DCM fuse but I prefer the above option.
Between the massive privacy issues and cars getting bricked by automatic OTA updates, I currently see zero reason whatsoever to buy a car made after ~2015.
I just recently upgraded to a 2013 Rav4 and it has all the tech I could ever want. I'm gonna keep it as long as I can.
How though
- "According to Rivian the glitch occurred when the wrong build with incorrect security certificates was inadvertently sent out. This error resulted in bricking the infotainment systems of an unknown number of vehicles."
  I always thought that 'fat finger' revered to having fingers that are too big for buttons, but I guess it means sending out wrong software versions.
  
  Fat finger in this context typically means that the person knew what they wanted to do, but typed the wrong thing. Their "fat finger" hit the wrong button. Possible that they wanted certificate 123 in the build, but they accidentally typed 133, or something like that.
  
  I think it's also worth noting that "security certificates" in this context are almost certainly there for the purpose of locking the system away from modification by the vehicle owner, which means they weren't necessary (and indeed, ought to be prohibited by law) to begin with.
  Rivian was hoist on their own consumer-hostile petard.
- They did
  Rm -rf /
  To force remove everything recursively instead of
  Rm -fr /
  To remove the French language pack. Rookie mistake really.
  
  I know you're being funny here, but....
- Dumb-ass managers who thought they could run their embedded software group all Agile and shit, and push stuff out to Production without full testing to meet deadlines, not realizing that they can't just "roll back" this one.
- They had 2 files in one directory. One said “good update” the other said “bad update-don’t use”. When they wanted to push out the update they used a tablet and tapped on “good update” but they accidentally touched the wrong one a bit more with the thumb.
  Something like this. But maybe they don’t want to say wat really happened.

46 comments