The anova and mixed commands present their results rather differently. The anova command displays a single test for each factor in the model including factors that have more than one degree of freedom. The mixed command displays estimate for each degree of freedom. Even when you follow the mixed command with test the results often don’t agree with anova except for the highest order interaction. This page will show you how you can get anova type results from mixed by using the margins command.
Example 1
Our first example will use split-plot factorial model with one between subject factor a (2 levels) and one within subject factor b (4 levels). We will begin by loading the data and running the anova model for comparison purposes.
use https://stats.idre.ucla.edu/stat/data/spf24, clear
anova y a / s|a b a#b /
Number of obs = 32 R-squared = 0.9613
Root MSE = .712 Adj R-squared = 0.9333
Source | Partial SS df MS F Prob>F
-----------+----------------------------------------------------
Model | 226.375 13 17.413462 34.35 0.0000
a | 3.125 1 3.125 2.00 0.2070
s|a | 9.375 6 1.5625
-----------+----------------------------------------------------
b | 194.5 3 64.833333 127.89 0.0000
a#b | 19.375 3 6.4583333 12.74 0.0001
Residual | 9.125 18 .50694444
-----------+----------------------------------------------------
Total | 235.5 31 7.5967742
This model has two error terms s and the residual (which is technically the same as the
b#s|a interaction). One advantage to using mixed for these mixed model
ANOVAs is that we do not have to specify which error term is used to test which effect. We do,
however, have to be sure to include all of the necessary random effects in our mixed model.
There is an additional advantage in that you get better estimates using mixed when there
is missing data for some subjects.
We will now run the mixed command followed by tests for each of the effects. The setup
for mixed has the fixed effects to the left of the double pipe (||) and the random
effects to the right. For each test with more than one degree of freedom we will
also show the chi-square value divided by the degrees of freedom so that they are scaled the same
as the F-ratio in the above ANOVA. The reml option must be used to replicate the results of the ANOVA.
mixed y a##b || s: , reml
Performing EM optimization:
Performing gradient-based optimization:
Iteration 0: log restricted-likelihood = -34.824381
Iteration 1: log restricted-likelihood = -34.824379
Computing standard errors:
Mixed-effects REML regression Number of obs = 32
Group variable: s Number of groups = 8
Obs per group:
min = 4
avg = 4.0
max = 4
Wald chi2(7) = 423.89
Log restricted-likelihood = -34.824379 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
y | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
2.a | -2 .6208193 -3.22 0.001 -3.216783 -.7832165
2 | .25 .5034603 0.50 0.619 -.736764 1.236764
3 | 3.25 .5034603 6.46 0.000 2.263236 4.236764
4 | 4.25 .5034603 8.44 0.000 3.263236 5.236764
a#b |
2 2 | 1 .7120004 1.40 0.160 -.3954951 2.395495
2 3 | .5 .7120004 0.70 0.483 -.8954951 1.895495
2 4 | 4 .7120004 5.62 0.000 2.604505 5.395495
_cons | 3.75 .4389855 8.54 0.000 2.889604 4.610396
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
s: Identity |
var(_cons) | .2638887 .2294499 .0480071 1.450562
-----------------------------+------------------------------------------------
var(Residual) | .5069445 .1689815 .2637707 .9743036
------------------------------------------------------------------------------
LR test vs. linear model: chibar2(01) = 3.30 Prob >= chibar2 = 0.0346
estimates store m1 /* save estimate for use later on */
testparm a#b
( 1) [y]2.a#2.b = 0
( 2) [y]2.a#3.b = 0
( 3) [y]2.a#4.b = 0
chi2( 3) = 38.22
Prob > chi2 = 0.0000
display "display as F " r(chi2)/r(df)
display as F 12.739724
testparm i.b
( 1) [y]2.b = 0
( 2) [y]3.b = 0
( 3) [y]4.b = 0
chi2( 3) = 107.88
Prob > chi2 = 0.0000
display "display as F " r(chi2)/r(df)
display as F 35.958898
testparm i.a
( 1) [y]2.a = 0
chi2( 1) = 10.38
Prob > chi2 = 0.0013
As you can see, the only test that has the same value in both the anova and the mixed is the one for the a#b interaction. The reason for this because the factor-variables used in the mixed command use indicator (dummy) coding. The tests of the main effects for a and b are really test of simple
effects for the reference level of the categorical variable.
We can easily get the “correct” tests for the main effects using the margins command.
margins b, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1 | 2.75 .3104096 8.86 0.000 2.141608 3.358392
2 | 3.5 .3104096 11.28 0.000 2.891608 4.108392
3 | 6.25 .3104096 20.13 0.000 5.641608 6.858392
4 | 9 .3104096 28.99 0.000 8.391608 9.608392
------------------------------------------------------------------------------
test (1.b=2.b)(1.b=3.b)(1.b=4.b)
( 1) 1bn.b - 2.b = 0
( 2) 1bn.b - 3.b = 0
( 3) 1bn.b - 4.b = 0
chi2( 3) = 383.67
Prob > chi2 = 0.0000
display "display as F " r(chi2)/r(df)
display as F 127.89039
estimates restore m1
(results m1 are active now)
margins a, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1 | 5.6875 .3124999 18.20 0.000 5.075011 6.299989
2 | 5.0625 .3124999 16.20 0.000 4.450011 5.674989
------------------------------------------------------------------------------
test (1.a=2.a)
( 1) 1bn.a - 2.a = 0
chi2( 1) = 2.00
Prob > chi2 = 0.1573
Note that the tests for a and b main effects now agree with the anova results
in the first part of this example.
For our second example, let’s try a model that is a bit more complex.
Example 2
The second example is a split-plot factorial with one between subject factor a (2 levels)
and two within subject factors b and c (2 levels each). The model has four
separate error terms but
we won’t have to divide any of the chi-square values from the mixed model by their
degrees of freedom because all of the tests have just a single degree of freedom.
use https://stats.idre.ucla.edu/stat/data/spf2-22, clear
anova y a / s|a b a#b / b#s|a c a#c / c#s|a b#c a#b#c /
Number of obs = 32 R-squared = 0.9920
Root MSE = .559017 Adj R-squared = 0.9589
Source | Partial SS df MS F Prob > F
-----------+----------------------------------------------------
Model | 233.625 25 9.345 29.90 0.0002
a | 3.125 1 3.125 2.00 0.2070
s|a | 9.375 6 1.5625
-----------+----------------------------------------------------
b | 162 1 162 199.38 0.0000
a#b | 6.125 1 6.125 7.54 0.0335
b#s|a | 4.875 6 .8125
-----------+----------------------------------------------------
c | 24.5 1 24.5 61.89 0.0002
a#c | 10.125 1 10.125 25.58 0.0023
c#s|a | 2.375 6 .395833333
-----------+----------------------------------------------------
b#c | 8 1 8 25.60 0.0023
a#b#c | 3.125 1 3.125 10.00 0.0195
Residual | 1.875 6 .3125
-----------+----------------------------------------------------
Total | 235.5 31 7.59677419
We begin this next section by using the egen group command to create combinations of variables to act as our random effects. We will need b by s, c by s and b by c by s along with s for our random effects.
We will then run the mixed command followed by test of the a#b#c interaction to show that the results for the highest order interaction is the same as the anova results.
egen bs = group(b s)
egen cs = group(c s)
egen bcs= group(b c s)
mixed y a##b##c || _all: R.bs || _all: R.cs || _all: R.bcs || s:, reml
Performing EM optimization:
Performing gradient-based optimization:
Iteration 0: log restricted-likelihood = -34.084157 (not concave)
Iteration 1: log restricted-likelihood = -34.049357 (not concave)
Iteration 2: log restricted-likelihood = -34.046954
Iteration 3: log restricted-likelihood = -34.046003 (not concave)
Iteration 4: log restricted-likelihood = -34.045909
Iteration 5: log restricted-likelihood = -34.045907
Computing standard errors:
Mixed-effects REML regression Number of obs = 32
-------------------------------------------------------------
| No. of Observations per Group
Group Variable | Groups Minimum Average Maximum
----------------+--------------------------------------------
_all | 1 32 32.0 32
s | 8 4 4.0 4
-------------------------------------------------------------
Wald chi2(7) = 332.00
Log restricted-likelihood = -34.045907 Prob > chi2 = 0.0000
------------------------------------------------------------------------------
y | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
2.a | -2 .6208194 -3.22 0.001 -3.216784 -.7832163
2.b | 3.25 .5303299 6.13 0.000 2.210572 4.289428
a#b |
2 2 | .5 .7499998 0.67 0.505 -.9699725 1.969973
2.c | .25 .4208128 0.59 0.552 -.5747779 1.074778
a#c |
2 2 | 1 .5951191 1.68 0.093 -.1664121 2.166412
b#c |
2 2 | .75 .5590141 1.34 0.180 -.3456476 1.845648
a#b#c |
2 2 2 | 2.5 .7905654 3.16 0.002 .9505203 4.04948
_cons | 3.75 .4389856 8.54 0.000 2.889604 4.610396
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval]
-----------------------------+------------------------------------------------
_all: Identity |
var(R.bs) | .2500028 .2512985 .0348603 1.792908
-----------------------------+------------------------------------------------
_all: Identity |
var(R.cs) | .04167 .1455805 .0000443 39.23096
-----------------------------+------------------------------------------------
_all: Identity |
var(R.bcs) | .2498766 .180416 .0606933 1.028752
-----------------------------+------------------------------------------------
s: Identity |
var(_cons) | .1666639 .2644133 .0074371 3.734895
-----------------------------+------------------------------------------------
var(Residual) | .0626203 . . .
------------------------------------------------------------------------------
LR test vs. linear model: chi2(4) = 4.86 Prob > chi2 = 0.3019
Note: LR test is conservative and provided only for reference.
estimates store m1 /* save estimate for use later on */
testparm a#b#c
( 1) [y]2.a#2.b#2.c = 0
chi2( 1) = 10.00
Prob > chi2 = 0.0016
Yes indeed, the result for the a#b#c interaction is the same in both the mixed and the
anova. Next, we will test each of the main effects and two-way interactions to show that
these results are not equivalent to those in the anova above.
testparm i.a
( 1) [y]2.a = 0
chi2( 1) = 10.38
Prob > chi2 = 0.0013
testparm i.b
( 1) [y]2.b = 0
chi2( 1) = 37.56
Prob > chi2 = 0.0000
testparm a#b
( 1) [y]2.a#2.b = 0
chi2( 1) = 0.44
Prob > chi2 = 0.5050
testparm i.c
( 1) [y]2.c = 0
chi2( 1) = 0.35
Prob > chi2 = 0.5525
testparm a#c
( 1) [y]2.a#2.c = 0
chi2( 1) = 2.82
Prob > chi2 = 0.0929
testparm b#c
( 1) [y]2.b#2.c = 0
chi2( 1) = 1.80
Prob > chi2 = 0.1797
Having verified that the tests for the main effects and two-way interactions differ between mixed and anova, we will finish up by running a series of margin commands testing the main effects and two-way interactions correctly.
margins a, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
c (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1 | 5.6875 .3124997 18.20 0.000 5.075012 6.299988
2 | 5.0625 .3124997 16.20 0.000 4.450012 5.674988
------------------------------------------------------------------------------
test (1.a = 2.a) /* a main effect */
( 1) 1bn.a - 2.a = 0
chi2( 1) = 2.00
Prob > chi2 = 0.1573
estimates restore m1
margins b, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
c (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1 | 3.125 .2724312 11.47 0.000 2.591045 3.658955
2 | 7.625 .2724312 27.99 0.000 7.091045 8.158955
------------------------------------------------------------------------------
test (1.b = 2.b) /* b main effect */
( 1) 1bn.b - 2.b = 0
chi2( 1) = 199.38
Prob > chi2 = 0.0000
estimates restore m1
margins c, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
c (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1 | 4.5 .2473822 18.19 0.000 4.01514 4.98486
2 | 6.25 .2473822 25.26 0.000 5.76514 6.73486
------------------------------------------------------------------------------
test (1.c = 2.c) /* c main effect */
( 1) 1bn.c - 2.c = 0
chi2( 1) = 61.89
Prob > chi2 = 0.0000
estimates restore m1
margins a#b, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
c (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
a#b |
1 1 | 3.875 .3852758 10.06 0.000 3.119873 4.630127
1 2 | 7.5 .3852758 19.47 0.000 6.744873 8.255127
2 1 | 2.375 .3852758 6.16 0.000 1.619873 3.130127
2 2 | 7.75 .3852758 20.12 0.000 6.994873 8.505127
------------------------------------------------------------------------------
test (1.a#1.b - 1.a#2.b - 2.a#1.b + 2.a#2.b = 0) /* a#b interaction */
( 1) 1bn.a#1bn.b - 1bn.a#2.b - 2.a#1bn.b + 2.a#2.b = 0
chi2( 1) = 7.54
Prob > chi2 = 0.0060
estimates restore m1
margins a#c, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
c (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
a#c |
1 1 | 5.375 .3498513 15.36 0.000 4.689304 6.060696
1 2 | 6 .3498513 17.15 0.000 5.314304 6.685696
2 1 | 3.625 .3498513 10.36 0.000 2.939304 4.310696
2 2 | 6.5 .3498513 18.58 0.000 5.814304 7.185696
------------------------------------------------------------------------------
test (1.a#1.c - 1.a#2.c - 2.a#1.c + 2.a#2.c = 0) /* a#c interaction */
( 1) 1bn.a#1bn.c - 1bn.a#2.c - 2.a#1bn.c + 2.a#2.c = 0
chi2( 1) = 25.58
Prob > chi2 = 0.0000
estimates restore m1
margins b#c, post asbalanced
Adjusted predictions Number of obs = 32
Expression : Linear prediction, fixed portion, predict()
at : a (asbalanced)
b (asbalanced)
c (asbalanced)
------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
b#c |
1 1 | 2.75 .3104097 8.86 0.000 2.141608 3.358392
1 2 | 3.5 .3104097 11.28 0.000 2.891608 4.108392
2 1 | 6.25 .3104097 20.13 0.000 5.641608 6.858392
2 2 | 9 .3104097 28.99 0.000 8.391608 9.608392
------------------------------------------------------------------------------
test (1.b#1.c - 1.b#2.c - 2.b#1.c + 2.b#2.c = 0) /* b#c interaction */
( 1) 1bn.b#1bn.c - 1bn.b#2.c - 2.b#1bn.c + 2.b#2.c = 0
chi2( 1) = 25.60
Prob > chi2 = 0.0000
These two examples should give you a good feel for how to use the margins command to get anova type results from mixed.