Calculator Accuracy & Usefulness

[Dave Shaffer]

I haven't figured out how to quote part of a previous post, but in reference to

"Crystal accuracy is usually in the range of 5 to 7 sig figs. I have an 8-digit frequency counter with a crystal oven and a trimmer to calibrate it against WWV. The specifications say you can get the error down to 0.1ppm, but by luck I was able to get .01 (although I didn't check to see if that went out the window within the next day or week). Crystals are the most precise components we can get, or at least I can't think of any components more precise off the top of my head."

Check out atomic frequency standards! We use these for our radio astronomical observations. Current Hydrogen maser systems have stability/accuracy at 1e-14, and atomic standards in development are at 1e-17 and hope for 1e-18 !!! There are H masers at radio telescopes all over the world for Very Long Baseline Interferometry (VLBI).

VLBI geodesy approaches 1mm precision on intercontinental baselines (up to 10000 km or so long, or a part in 10 billion (1e10) or thereabouts), so the required calculations are generally done in quadruple precision. We measure 100 GHz frequencies to microhertz and 100 millisecond time delays to picoseconds.

A relatively cheap GPS time and frequency standard will easily give you 1e-9 or 1e-10 precision, so trade in your crystal system!

Cheap rubidium standards (surplus units are ~$100 on Ebay) will give you a few orders of magnitude better, with some care.

The H masers are more like a quarter to half million dollars, and need a nicely controlled thermal chamber (0.001 degree regulation) for best performance. As we discovered once upon a time, they are also magnetometers: the unit in the base of the 140' antenna at NRAO (Green Bank, W.Va.) changed frequency at the 1-e13 level when the metal telescope structure overhead (all 2600 tons!) moved around, changing its self magnetization effects as well as dragging the Earth's magnetic field around.

There are truly some cases where 15 digits or more are useful. (However, we don't use calculators for our analysis!!)