if your point to go through does not belong to the defined plane it will be impossible to solve and if it belongs to the plane we do not need the extra condition which is validated already.
So basically you just want to find the point on (pt1,pt2) line which goes through pt4 knowing pt4 needs to belong to the plane (pt1,pt2,pt3)
No, i think you didn’t understand, probably because i was not clear enough.
if you look at the playground, i assume that the pt4 point is ON the (pt1,pt2) line. I sketched a red line between pt1 and pt4 to prove it. So pt4 is already on the plane because the plane is build with the pt1 pt2 and pt3 points.
Now i’d like to know which line is perpendicular to the (pt1, pt2) line, contains the pt4 point AND is on the (pt1,pt2,pt3) plane.
To be perfectly clear, i’ve drawn something on Sketchup.
@sebavan is correct that it is really a maths question you are asking. Once you understand the maths Babylon.js provides the features you need. The following PG gives you the maths you need in Babylon.js form.
If you want the red line to stop at the side pt1_pt3 or pt2_pt3 then you need to find the point where the red line meets one or other of these sides. Search for
how to find where 3d vectors intersect
EDIT PGs #3 and #4 are ones where I misunderstood what you were asking and found the lines (in opposite directions) through pt4 perpendiculat to the plane