I put my finger on it
Way back in the Dark Ages (the late '70s) I was troubleshooting a military radio that had a phase locked loop (PLL) BFO. I couldn't get it to lock properly. The previous version of the loop worked fine, but the new layout was noisy. I was called in as a consultant because the regulars at the company had worked for six months on the problem and were unable to resolve it. I had solved a PLL problem for a sister company (they had used the wrong order of phase detector), so they thought I'd be able to help them with their problem.
I worked on the problem with the thought that supply-line noise from the surrounding digital circuitry was getting into the PLL and keeping it from properly locking. I worked on it for a month, and the board at the end of that month looked like it had the measles because it had so many capacitors tacked on it. Every day my boss would walk by my test set up and ask, "Any breakthroughs?" That got so annoying that I was given to providing flip answers to the question. Fortunately, the boss indulged me. Then one day I was listening to the raspy signal and happened to put my finger on a random point on the board and voila — the noise went away.
What I found amazed me. The particular section of circuitry that I fingered was a 74F74 divider fed by a VCO. In the earlier version, about 1/2 inch of grounded shield coax had fed the 74F74 and that worked fine. That piece of coax was replaced by a one inch bit of PCB trace in the new layout. Evidently, what was happening (I didn't have the tools to do a measurement and even then, due to probe capacitance, it might not have shown up) was that the 1 inch trace was the conduit for a reflection from the 74F74 back to the VCO and then back to the 74F74, causing occasional double-pulsing of the 74F74 due to the fast rise time of the VCO and the high speed of the 74F74. It was tamed in the original layout by the bit of capacitance that the short length of coax provided. We fixed the problem for the respin of the board by adding a capacitor of a few pF to the output of the VCO to slow the rise time.
And the moral of the story? With today's fast rise and fall times on some parts you have to be very careful about reflections from even short unterminated lines. I learned my lesson a long time ago and was ready for this new age of much faster parts long before the new age was every day.
The other moral of the story is this: I'm sure the regulars had probed this point or accidentally fingered it and did not have the proper mindset to recognize the clue when it arrived. Chance favors the prepared mind. Don't get locked into a mental box. Don't assume you know the nature of any problem. Things can get very weird out there. Second- and third-order effects that are routinely ignored can come back to bite you.
M. Simon's e-mail can be found on the sidebar at Space-Time Productions .