The collisionFilterGroup and collisionFilterMask might work here.
See example cannon code here cannon.js/collisionFilter.html at master · schteppe/cannon.js · GitHub. Likely you will need to specify these when the impostor is created along with the mass.