[WP 34s] Pressure Conversion Factors Message #1 Posted by Timothy Roche on 3 Nov 2013, 11:49 p.m.
WP 34s (3.2T 3375)
Pressure conversion factors implemented in the WP 34s calculator seem to be based on the published values found in the NIST Guide to SI Units. (So are all the others, for that matter. I think.) This is a good source for conversion factors (reliable and free), particularly the exact conversion factors highlighted in bold face in the document., There are, however, many listed conversion factors which are approximations in the NIST Guide; results from conversions carried out using these approximated factors must be restricted to the number of given significant figures in the value from the table. As the given approximate conversion factors contain six or seven significant figures (usually) and most people are looking for answers with two, three, or four sigfigs (at least in engineering calculations in my experience), these conversion factors are more than adequate for the job.
EG: The NIST Guide gives the conversion factor from units of mmHg to units of Pa as 1.333 224 E+02. So, given a pressure 760 mmHg the converted result (760 * 133.3224) would be 101 325.024 Pa. Because the factor has seven significant digits the answer should be rounded to no more than seven significant figures: therefore, our rounded answer to seven significant digits is 101 325.0 Pa. You could argue that the answer should not have more than two significant digits based on the given value 760 mmHg, which is true if you did not implicitly mean the number 760 exactly.
Now to my point: Some of the included approximate conversion factors in the NIST Guide have exact conversions which were not included in the publication (NIST is emphasizing the uncertainty of certain measured values like barometric pressures). In particular, the conversions for many of the pressure units: those listed as conventional (mmHg, cmHg, ftHg, mmH2O, cmH2O, inH2O, ftH2O) and those that are related to these conventional pressure units (torr). All these conventional units are based on the pressure unit of atmosphere and the equivalent in these other units. For instance, mmHgtoPa factor is based on the exact definition 1 atm = 101 325 Pa and 1 atm = 760 mmHg (conventional); so the listed value in the NIST Guide 1.333 224 E+02 is the approximation of (101 325/760). I think that the pressure conversion factors implemented in the WP 34s should use the exact definitions of these conventional pressure units. This would eliminate unexpected and slightly wrong conversions displayed on the calculator such as 760 mmHg = 101 325.024 Pa?!
Here are a list of suggested adjusted pressure conversion factors.
760 mmHg = 101 325 Pa, therefore:
1 mmHg = (101 325/760) Pa = 133.322 368 421 0526 Pa
1 torr = 133.322 368 421 0526 Pa
29.921 259 842 519 69 inHg = 101 325 Pa, therefore:
1 inHg = 3 386.388 157 894 736 Pa
1 lbm = 0.453 592 37 kg (exact)
1 lbf = (0.453 592 37 kg)(9.806 65 m s^2) = 4.448 221 615 260 500 N
1 (lbf/in2) = 1 psi = (1 lbf in^2)(4.448 221 615 260 500 N lbf^1)[(39.370 078 740 157 48)^2 in^2 m^2]
1 psi = 6 894.757 293 168 361 Pa
Tim
Edited: 3 Nov 2013, 11:52 p.m.
