Hard to say without a playground and some experiements. Did you try to increase collisionRetryCount ?
collisionRetryCount