Service Degraded
Deploy the Fleet OTA services are currently degraded. This post will be used to track updates.
Update 3 - August 3, 2024 A new version of the Deploy the Fleet Arduino library (1.0.7) has been released which addresses all of the issues identified during this event. We recommend all users update to the latest version.
Update 2 - July 31, 2024 The hosting provider has made a decision not to restore the old certificate. This means, unfortunately, devices running versions of the DTF Arduino library prior to 1.0.6 will need to be manually updated by means other than OTA.
Update 1 - July 29, 2024 The hosting provider is actively working to restore the certificate which was changed on the 24th. This will allow existing devices to receive updates. We strongly encourage users to prepare a firmware update with the latest version (1.0.6) of the DTF Arduino library. We will notify users when this is ready so the update can be pushed. The original certificate has an expiration date of August 23rd, 2024 so firmware updates with the fixed library will need to be delivered before that time.
What Happened?
On July 24, 2024 a 3rd party hosting provider used by Deploy the Fleet altered the SSL certificate to which the DTF firmware library is pinned. This resulted in devices not being able to properly authenticate to Deploy the Fleet servers and, as a result, not receive OTA updates. The Deploy the Fleet web management console is unaffected.
How Do I Fix It?
Devices Under Development
A new version of the DTF_ESP32Update Arduino library(1.0.6) has been published which resolves this issue and uses the new pinned SSL certificate. It also contains some fallback redundancy so a pinned certificate doesn’t create a single point of failure.
Devices In The Field
We are continuing to work with our hosting provider to get the original pinned certificate restored. If this happens, all devices will immediately be able to get OTA updates without any changes.
The existing library is designed to handle failures gracefully so this should not have any impact to the main firmware. Devices simply will not be able to get updates or report check-ins to Deploy the Fleet.
If the certificate is not able to be restored, devices in the field will need to be manually updated via a wired connection. We expect to know if this is necessary within 72 hours.
We also recommend you prepare a firmware update that integrates the latest version of the DTF_ESP32Update Arduino library(1.0.6) to be deployed when devices are able to connect to the service again. The new version of the library contains updates to help prevent issues like this in the future.
Future Prevention
This kind of outage is unacceptable. We want to build the most reliable OTA service available. Here are some steps that are being taken to prevent this type of outage in the future.
- New library which includes fallback logic and less-fragile certificate pinning
- Redundant OTA servers for failover
- SSL certificate guarantees that don’t rely on 3rd party certificate management
- 24/7 active monitoring of certificate changes
Item 1 is complete as of this posting. Items 2-4 are now on the top of the priority list and will be completed as soon as possible with updates posted to this blog.