Doing some console.logs I realized your start position didn’t seem correct. Take a look at line 79. I changed it to be the objects starting position instead of the target.
start = new BABYLON.Vector2(parseFloat(object.left), parseFloat(object.top));
The version you had caused problems because I think you were focusing on the target’s position. In the cases where you had added an item inside of your rectangle the target would be those objects. The positions for those objects are relative to the container they are inside. In this case the position was 0 which caused the jump you saw.