Earlier paper of Hill, Goddard, and Visscher and the simpler 2 locus case is discussed here.
Influence of Gene Interaction on Complex Trait Variation with Multilocus ModelsSee also Determination of Nonlinear Genetic Architecture using Compressed Sensing, and related posts on epistasis and additivity.
Asko Mäki-Tanila, William G. Hill
GENETICS September 18, 2014 vol. 198 no. 1 355-367; DOI: 10.1534/genetics.114.165282
Although research effort is being expended into determining the importance of epistasis and epistatic variance for complex traits, there is considerable controversy about their importance. Here we undertake an analysis for quantitative traits utilizing a range of multilocus quantitative genetic models and gene frequency distributions, focusing on the potential magnitude of the epistatic variance. All the epistatic terms involving a particular locus appear in its average effect, with the number of two-locus interaction terms increasing in proportion to the square of the number of loci and that of third order as the cube and so on. Hence multilocus epistasis makes substantial contributions to the additive variance and does not, per se, lead to large increases in the nonadditive part of the genotypic variance. Even though this proportion can be high where epistasis is antagonistic to direct effects, it reduces with multiple loci. As the magnitude of the epistatic variance depends critically on the heterozygosity, for models where frequencies are widely dispersed, such as for selectively neutral mutations, contributions of epistatic variance are always small. Epistasis may be important in understanding the genetic architecture, for example, of function or human disease, but that does not imply that loci exhibiting it will contribute much genetic variance. Overall we conclude that theoretical predictions and experimental observations of low amounts of epistatic variance in outbred populations are concordant. It is not a likely source of missing heritability, for example, or major influence on predictions of rates of evolution.