I think this is semi-related to falling through ground. But in your case, it’s the size and normal of the faces colliding.
Try using a Box-shaped physics body since that will totally be sufficient for your use case. Or, if the “hole” is necessary, try using CSG2 to create it.
Edit: another method might be to implement an invisible box at the bottom of each “pole” and create a linear constraint between it and each pearl LOCKED in X and Z. PhysicsBody in ground with 0 mass (or box to pearl constraint with collisions enabled) and each pearl non-zero mass and box physicsBody. Pole is NOT a physics body. With this setup, physics calculations are super simplified (being all boxes and linear constraints).