Weird result with Matrix determinant

[Dieter]

(03-17-2015 08:16 PM)Tugdual Wrote:  Using Standard number format,
Prime Home says -647,000000002
Prime CAS says -647

Home is pretty close but what kind of algorithm can produce such a funny result to calculate a determinant with only integers?

Well, like any other device with limited working precision (12 resp. 15 digits internally) your calculator will cause a more or less significant roundoff error with every single operation. Evaluating the determinant of a matrix with numerical methods, i.e. not by Cramer's rule or similar exact solutions that are not suitable for larger matrices, usually is done via the well-known Gauss algorithm resp. a LU-decomposition, leaving the product of the diagonal elements as the desired result.

All this involves dozens, if not hundreds of multiplications, divisions and additions. Try a simple example like 2/3 – 2*(1/3) and you'll see how limited the accuracy of even a close-to-perfect calculator can be. A tiny error in the least significant digit is a very good result here. With other matrices the error can grow much, much larger.

Dieter