How To Troubleshoot a System

Troubleshooting a multi-part system can be challenging since it’s often unclear which part is contributing to the error, or if there are multiple parts contributing. All you know is the end result isn’t what it should be. I experienced this situation recently, and had the opportunity to use my engineering skills to identify where the error was coming from. I think it serves as a good summary of how to break down a system to identify root causes, so I’m writing this article to share the process I went through in hopes that it will help other engineers out there in their troubleshooting efforts.

The system I troubleshot was purchased from a company called Edelkrone who manufactures cinematic motion products (we’re trying to up our marketing game here at Pipeline). It included the following products (which were effectively subsystems…see Figure 1 below):

• JibOne – a swing arm
• PanPRO – rotates the JibOne
• HeadPLUS – pitch and yaw motions for the camera
• Focus Module – focuses the camera lens

Figure 1. System level broken down by subsystems (individual products)

These products don’t necessarily come as a package, rather they can be mixed as desired. So, it’s not like they all came assembled ready to go from the factory. In addition, to these items, I also already had the camera body and a couple lenses (one Canon, and one Tamron…but the names don’t really matter).

Once I had unboxed and installed them all on my tripod, I started experimenting with the system excited to see the programmable motion come to life in a few test videos. The products work with an app that is simple enough to use. Move the camera to a position, record that position in the app, then move the camera to another position and record that position in the app. Now you can move back and forth between those two positions. Or so it was supposed to be…

Unfortunately, what I found was that positions in the actual trajectory of the system didn’t match those positions in the corresponding program I created (see Figure 2 below). For example, if position A was at x,y,z coordinates 4,4,4, and position B was at x,y,z coordinates 7,7,7, the actual path of the system might end up at A = 4,5,3 and B = 7,6,8…noticeably different than where they should be. I spent hours trying to understand what was going on at a system level…and that was my big mistake. Even after being an engineer for 20 years, it didn’t occur to me to break the system down into its constituent parts for an embarrassingly long time. But I got there eventually.

Figure 2. Video showing discrepancy between positions that were supposed to be the same

Once I realized I was fighting a losing battle troubleshooting at the system level, I started testing each product by itself. Very quickly it became clear where the problems were and were not. The PanPRO by itself showed inconsistent positions while the HeadPLUS was spot on every time. The Focus Module was all over the place when using the Tamron lens (but accurate when using the Canon lens) whereas the JibONE was supremely consistent.

I tracked everything meticulously. I kept a spreadsheet in which I recorded all my results (Figure 3). An organized folder structure for each test (Figure 4). Clearly named subfolders for each trial within each test folder (Figure 5). And clearly labeled individual photos within each trial folder (Figure 6…two poses for each trial, each taken three times so I wasn’t relying on just a single data point). Not only did staying organized help me clearly see where the problems were, but when I’d go back to it a few days later it was easy to pick back up and continue the testing process since results were recorded very clearly.

Figure 3. Master spreadsheet to track results across all tests

Figure 4. Folders to keep each test organized

Figure 5. Subfolders inside the test folders to organize the different trials

Figure 6. Clearly labeled photos to record each trial

Armed with this knowledge I was able to present Edelkrone with actionable information. After a couple of exchanges with support, they determined two things:

1. The PanPRO unit was bad.
2. I was using an unsupported lens (the Tamron…the Canon was supported and did work)

They quickly sent me a new PanPRO unit, and I stopped using the unsupported Tamron lens. Once the new PanPRO unit arrived I installed it (along with only the Canon lens) and voila – everything worked at the system level! Since then we have gone on to use this setup to film multiple projects with wonderful results. We often have customers and industry contacts comment on how well done our photography/videography is (see examples in case studies on our website).

To summarize, here is the process one should consider following when troubleshooting a system that isn’t cooperating:

1. Identify the constituent sub-systems
2. Test each subsystem by itself
3. Record your results in a supremely organized manner
4. Analyze the results to learn if the individual subsystems yield correct results

It’s important to note that this process will work for more than just engineering systems. For example, maybe you’ve just started a diet (which is a system itself) and you’re not getting the results you expected. Break it down into its constituent parts, or subsystems:

• How many total calories am I consuming?
• What is the caloric quality of the foods I’m eating (e.g. pizza vs carrot sticks)
• Am I exercising regularly?
• Am I getting high quality sleep?
• Etc…

Chances are one or more of the subsystems aren’t where they should be. Now that you’ve identified them individually it becomes much easier to troubleshoot each by itself. You get the idea. Good luck troubleshooting your systems, in engineering and in life!