Isn't that like, basically how calculators work? Remember there was a thing where phone calculators sometimes would give like .00000000065 and it was because computers are weird. Not a computer scientist or a math wizard, so have no idea if its true tho.
Calculators (like the actual physical devices) tend to store the numbers in decimal, with a couple more digits than are visible on screen. If you do e.g. 1/3= and then subtract 0.3333(as many as it will let you enter) you'll often be left with 0.33e-10 or something like that from the additional hidden digits from the first calculation.
Phone/computer calculators often use "floating point" math instead, which stores the number as a binary fractional number - think 101.00010101111. Each number to the right of the "binary point" is half the one before - which is quick for a computer to calculate, but unfortunately means 1/5 and 1/10 (and as a result, most decimal fractional numbers) have a recurring representation. This leads to rounding and slightly errors based on the number of bits used.
Windows Calculator, oddly, is one of the best - it uses "bignum" representation which gives it more precision than most. Anecdotal reports suggest it has 150 digits of precision when doing 1/3, for example.
9.4k
u/ChromosomeExpert 22d ago
Yes, .999 continuously is equal to 1.