I actually have a different approach that I think will simplify things and will be easier to understand. So instead of manually calculating the directions and positions, we can actually use Babylon functions to do the math for us.
Since we always want the ray to come directly from the camera’s look at vector and shoot out into the distance we can actually use a createPickingRay for this. Given a screen position, we can fire the ray in the correct direction all the time without having to worry. Now what screen position should we use? That’s simple! Since the camera is in the middle of the screen we will use the middle of the canvas. I believe the problems in your example arose from picking the sphere as the starting point for your ray. Here is the code for this (line 38-43)
var canvas = document.getElementById("renderCanvas");
var width = canvas.width /2;
var height = canvas.height / 2;
var ray = scene.createPickingRay(width, height, BABYLON.Matrix.Identity(), camera);
var hits = scene.multiPickWithRay(ray);
Now for “change anything it hits” we can do this with the multipick. I think I have some examples above but just in case I include it in this demo as well. Feel free to add more objects to the scene to test it out. In here you can see once the camera is no longer directly looking at the box it won’t continue to go down. There are also some lines you can uncomment (lines 51-53) to visualize the rays but note it will draw a new one every update frame.
Once again, always feel free for follow ups. 
