I was also looking for the same thing and I found this similar thread: Force chart d3.js inside a triangle.
The most voted answer has a method to detect collisions inside a triangle and also propose a generalized version to make it work with a polygon. You can see it in this demo.
One thing that the answer does not mention and that might be useful to you is to compute the center of your different polygons in order to make the different force layouts centered inside those polygons and I would suggest to use polygonCentroid for that.
var polygon = require('d3-polygon');
var polygon_data = [ [0,0], [10, 0], [10, 10], [0, 10]]; // a small box
var centroid = polygon.polygonCentroid(polygon_data); // [5, 5]
I would definitely love to see this polygon constraint as a feature in d3 but it may be too specific :/
Update:
Just a small correction: the proprosed solution takes account of the polygon's centroid contrary to what I said. My bad.
Update 2:
I've created a block with an implementation for the polygon collision detection: http://bl.ocks.org/pbellon/4b875d2ab7019c0029b636523b34e074.
It uses the collision detection mentionned in the SO's answer I linked & I used it to create a "force" like forceCollide on d3.v4.
It's not perfect but I had a lot of trouble to tweak the way nodes can be repelled from the polygon's border... If anyone has suggestions I would be happy to hear them!