The general picture of this idea is sketched here:
The dust particles in a protoplanetary disk would settle toward the disk midplane due to the vertical component of the stellar gravity. As the sedimentation proceeds, a dust-rich sublayer forms at the midplane. At some point, further sedimentation would be stopped due to sources of turbulence in the disk. Then the question is if this layer is dense enough to trigger GI? It is largely believed that it happes if the dust layer's density rhod exceeds the Roche density, which is about stellar mass over distance cubed, the dust layer becomes unstable and would fragment into km size planetesimals. It is arguable if such high density could be reached , But some recent work, e.g. Aaron and Eugene in 2010, suggest it is achievable if the dust abundance get enriched by only a factor of several.
However, in this work, we question this criterion and actually we find actually this is not the right one to apply. The right density threshold could be much higher than this Roche density.
Next I will use the whiteboard to show you how to derive the new criterion:
1) lets start with one phase medium, e.g. pure gas disk;
The stability is governed by a dimensionless number Q ,which the sound speed times orbital frequency divided by the surface density; this basically tells the pressure and rotation stabilize the disk, while the self-gravity destabilize it. now we can rewrite this, using hg=cg/omega and sigmag=2rhog hg, we can get the critical density is ~ rhog ~ M*/r^3; for pure gas disk, the critical density is indeed the roche density;
2) however,things change when you also well coupled dust; lets also write down the effective Qd for the dust layer; here gas sound speed is replaced by the effective sound speed of the dust+gas mixure, surface density is replaced with the surface density of the dust-rich layer.
For particle size smaller than ~cm at ~ 1AU, the stopping time is much shorter than the dynamical timescale, so we could approximate the mixture as one fluid. In this fluid, only gas provides the pressure, the dust instead adds inertia. So we could find the effective sound speed cd by equate P=rhog cg^2=(rhod+rhog)cd^2. This gives you the sound speed of the mixture is reduced by the dust cd=cg/sqrt(1+rhod/rhog).
Now rewrite Qd with Qg and sigmag ==> finally we could get the critical density rhod ~ Qg (sigg/sigd)^2 *rho_rhoche, which is surprisingly large for typical protplanetary disks. Rhoc is much larger than the Roche density as people usually cited in literature.