In a math library used by BabylonJs, the distance between a plane and a point is calculated like this
The formula is quite different than math books gives, and its also outputting different values. Every book gives a formula where a square root is involved eg:
Is the formula accurate?
Have you checked