This Confluence space has been migrated and is no longer current. The content on this space will no longer be accessible after July 1, 2026.
OTIS Backplane Power Cycling Failure Breakdown
Introduction
Based on Power Module data, it appears that we power cycled twice while the backplane was armed. One appears to be most likely during boost and landing.
Relevant Hardware / Software Context
Before we can dig into why I believe the entire backplane power cycled, I should probably give some context on some hardware concepts and how the software works for me to better explain my reasoning.
We get timestamps from the STM32’s real-time clock. A real-time clock keeps track of time even when the CPU (or microcontroller) is not actually running, as long as there is some power source. For example, if you look at your computer’s motherboard, you will notice a coin cell battery. This is the power source that keeps your computer’s RTC running even when your computer is shut down. This allows for your computer to report the correct time when you start it back up and aren’t connected to the Internet). This is the same concept for Backplane modules. So let's say a backplane module’s RTC reports 1 second since an epoch when we first power on the backplane. When we do a core reset while keeping the backplane powered a minute later, we should see the RTC report 60 seconds. However, if we power cycle the backplane by removing the power source and putting it back in, we should expect to see the RTC report ~1 second again on startup.
Time synchronization is a common problem in distributed systems where physical clocks on individual devices can drift. Clock drift can occur for multiple reasons whether its component imperfections, temperature variations, etc. Fortunately, having synchronized time on backplane is not a hard requirement, but it would be ideal to align data produced by different modules together based on their timestamps since the timestamps reported by different modules can by vary a lot. Therefore, I implemented a Network Time Protocol (NTP) server which runs on Radio Module. NTP is how computers synchronize clocks over a network, and its what your computer uses to get the correct time. So even though you have a RTC, you may also have a setting that automatically sets the time for you and that uses NTP to communicate with a server somewhere to get the correct time (with a lot of math involved to account for network propagation delays!). GPS is able to provide the correct time if it gets lock with a satellite which is why the Radio Module is ideal for being a NTP server since it can directly get the correct time based on GPS. Therefore, once GPS lock occurs we set the Radio Module’s RTC. Then each backplane module can send an NTP request from Radio Module for the correct time to synchronize too. Even if we do not get GPS lock, each module’s RTC should be synchronized since it still pulls from the Radio Module’s RTC. Currently, we only do this once at boot, but this will change in the near future when we transition to a callback-based mechanism for receiving network packets. Therefore, we are basically getting the default RTC time of radio module.
To sum up, we log our timestamps at a millisecond scale based on what the STM32 RTC gives us. We synchronize each backplane module’s RTCs based off of Radio Module’s RTC. The Radio Module RTC may, or may not use the correct real-world time based on whether it gets GPS lock. A data point logged afterwards should not have a timestamp that is earlier than a previous data point (value less than the previous). The one of two only possible cases of this occurring is if the 32-bit value rolls over which would have to take around 49 days. The second possible case is if the RTC undergoes power loss and resets itself.
Reasoning
DISCLAIMER: Note that I wiped the filesystem before the backplane got armed on the pad so there should be no previous data logged from testing. Also, you can tell whether its test data or real data since the battery rail voltage would report ~12V from a USB-C input and not the battery we were using.
Below is the data I parsed from the Backplane in the OTIS avionics bay
If you subtract a timestamp from the previous data point’s timestamp you will end up with around ~65000 milliseconds or a little over a minute. This makes sense since we default to logging data at 1 minute intervals while the backplane is powered on. There’s some weird pesky delay in thread scheduling, but ignoring that we can still say that its logging every minute. When it’s in flight, we would log 1.5 times a second though.
Now if you focus on line 42 of the CSV, you will notice that the timestamp reported is 522812868. This is oddly suspicious considering that the previously reported timestamp is 525343616 and 525343616 > 522812868. Also, 522812868 is the same timestamp as line 2, the first every data packet we logged when we first powered on the backplane at the pad!
Doing the math, we get 2530748 milliseconds between line 41 (525343616) and line 2 (522812868). 2530748 milliseconds is 42 minutes and 10 seconds. To confirm the time between boost and around the time we armed, I referenced a few photos and some data. I had a photo of us commanding the the cameras during boost and the radio module reporting the cameras were turned on at 1:33 PM, and other images seem to align with this. I also referenced other data, specifically the LOTUS FeatherWeight’s received GPS data, since the reporting from the avionics bay seemed a little late at around 1:10 PM. The LOTUS FeatherWeight reported being on around 12:55 PM. A few photos also show arming started around 12:55 PM to 1 PM.
I also referenced the timestamps of photos we took at the pad to confirm my rough estimates for timing.
This still leaves around 4 minutes unaccounted for, but this can most likely be attributed to the large inaccuracy of the RTC using the LSI oscillator creating massive clock drift (in testing I noticed that there is a few minutes of clock drift within an hour, which will be addressed in both hardware and software eventually). To further back up my beliefs, I noted the time difference with the second power cycle.
On line 45, the reported timestamp was 522812860 while the previously reported timestamp on line 44 was 522897488. This indicates another power cycle. The difference between line 44 (522897488) and 42 (522812868) is 84620 milliseconds. Since we could’ve power cycled a little after line 44, I am adding a range of an extra 60 seconds. So we had around 1 minute and 24 seconds to 2 minutes and 24 seconds. This was around the time of flight for the rocket based on the timestamps of the photos we took.
Based on how specific the timestamps reported by Power Module are and how they align closely with when the rocket goes through boost and landing, I believe that the entire backplane power cycled during those events. The microcontrollers should not have been under software reset since the RTC timestamps wouldn’t have back pedaled. Also Radio Module only sets its RTC time once at startup and all other modules set their RTC time at startup based on Radio Module’s time. Therefore, even if Radio Module reset its time, no other module would’ve pulled that time, unless they all started up around the same time.