Simultaneous Inference for General Linear Hypotheses

Simultaneous Inference for General Linear Hypotheses [R logo]

Help pages for package `multcomp' version 0.991-9

adjusted | Methods for General Linear Hypotheses
adjusted_calpha | Methods for General Linear Hypotheses
Chisqtest | Methods for General Linear Hypotheses
cholesterol | Cholesterol Reduction Data Set
coef.glht | Methods for General Linear Hypotheses
confint.glht | Methods for General Linear Hypotheses
contrMat | Contrast Matrices
csimint | Deprecated Functions multcomp Package
csimtest | Deprecated Functions multcomp Package
detergent | Detergent Durability Data Set
Ftest | Methods for General Linear Hypotheses
glht | General Linear Hypotheses
litter | Litter Weights Data Set
mcp | General Linear Hypotheses
modelparm | Generic Accessor Function for Model Parameters
multcomp-deprecated | Deprecated Functions multcomp Package
plot.confint.glht | Methods for General Linear Hypotheses
plot.glht | Methods for General Linear Hypotheses
recovery | Recovery Time Data Set
simint | Deprecated Functions multcomp Package
simtest | Deprecated Functions multcomp Package
summary.glht | Methods for General Linear Hypotheses
univariate | Methods for General Linear Hypotheses
univariate_calpha | Methods for General Linear Hypotheses
vcov.glht | Methods for General Linear Hypotheses
waste | Industrial Waste Data Set


Cholesterol Reduction Data Set

Description

Cholesterol reduction for five treatments.

Usage

data("cholesterol")

Format

This data frame contains the following variables

trt
treatment groups, a factor at levels 1time, 2times, 4times, drugD and drugE.
response
cholesterol reduction.

Details

A clinical study was conducted to assess the effect of three formulations of the same drug on reducing cholesterol. The formulations were 20mg at once (1time), 10mg twice a day (2times), and 5mg four times a day (4times). In addition, two competing drugs were used as control group (drugD and drugE). The purpose of the study was to find which of the formulations, if any, is efficacious and how these formulations compare with the existing drugs.

Source

P. H. Westfall, R. D. Tobias, D. Rom, R. D. Wolfinger, Y. Hochberg (1999). Multiple Comparisons and Multiple Tests Using the SAS System. Cary, NC: SAS Institute Inc., page 153.

Examples

  ### adjusted p-values for all-pairwise comparisons in a one-way layout
  ### set up ANOVA model
  amod <- aov(response ~ trt, data = cholesterol)

  ### set up multiple comparisons object for all-pair comparisons
  cht <- glht(amod, linfct = mcp(trt = "Tukey"))

  ### cf. Westfall et al. (1999, page 171)
  summary(cht, test = univariate())
  summary(cht, test = adjusted("Shaffer"))
  summary(cht, test = adjusted("Westfall"))

  ### use only a subset of all pairwise hypotheses
  K <- contrMat(table(cholesterol$trt), type="Tukey")
  Ksub <- rbind(K[c(1,2,5),],
                "D - test" = c(-1, -1, -1, 3, 0),
                "E - test" = c(-1, -1, -1, 0, 3))

  ### reproduce results in Westfall et al. (1999, page 172)
  amod <- aov(response ~ trt - 1, data = cholesterol)
  summary(glht(amod, linfct = mcp(trt = Ksub[,5:1])),
          test = adjusted("Westfall"))
[Package multcomp version 0.991-9 Index]


Contrast Matrices

Description

Computes contrast matrices for several multiple comparison procedures.

Usage

contrMat(n, type=c("Dunnett", "Tukey", "Sequen", "AVE",
                   "Changepoint", "Williams", "Marcus",
                   "McDermott"), base = 1)

Arguments

n | a (possibly named) vector of sample sizes for each group or a vector of factor levels.
type | type of contrast.
base | an integer specifying which group is considered the baseline group for Dunnett contrasts.

Details

Computes the requested matrix of contrasts for comparisons of mean levels.

Value

The matrix of contrasts with appropriate row names is returned.

References

Frank Bretz, Alan Genz and Ludwig A. Hothorn (2001), On the numerical availability of multiple comparison procedures. Biometrical Journal, 43(5), 645–656.

Examples

 n <- c(10,20,30,40)
 names(n) <- paste("group", 1:4, sep="")
 contrMat(n)    # Dunnett is default
 contrMat(n, base = 2)  # use second level as baseline
 contrMat(n, type = "Tukey")
 contrMat(n, type = "Sequen")
 contrMat(n, type = "AVE")
 contrMat(n, type = "Changepoint")
 contrMat(n, type = "Williams")
 contrMat(n, type = "Marcus")
 contrMat(n, type = "McDermott")


Detergent Durability Data Set

Description

Detergent durability in an incomplete two-way design.

Usage

data("detergent")

Format

This data frame contains the following variables

detergent
detergent, a factor at levels A, B, C, D, and E.
block
block, a factor at levels B_1, ..., B_10.
plates
response variable: number of plates washed before the foam disappears.

Details

Plates were washed with five detergent varieties, in ten blocks. A complete design would have 50 combinations, here only three detergent varieties in each block were applied in a balanced incomplete block design. Note that there are six observations taken at each detergent level.

Source

H. Scheffe (1959). The Analysis of Variance. New York: John Wiley & Sons, page 189.

P. H. Westfall, R. D. Tobias, D. Rom, R. D. Wolfinger, Y. Hochberg (1999). Multiple Comparisons and Multiple Tests Using the SAS System. Cary, NC: SAS Institute Inc., page 189.

Examples

  ### set up two-way ANOVA without interactions
  amod <- aov(plates ~ block + detergent, data = detergent)

  ### set up all-pair comparisons
  dht <- glht(amod, linfct = mcp(detergent = "Tukey"))

  ### see Westfall et al. (1999, p. 190)
  confint(dht)

  ### see Westfall et al. (1999, p. 192)
  summary(dht, test = univariate())
  summary(dht, test = adjusted("Shaffer"))
  summary(dht, test = adjusted("Westfall"))


General Linear Hypotheses

Description

General linear hypotheses and multiple comparisons for parametric models, including generalized linear models, linear mixed effects models, and survival models.

Usage

## S3 method for class 'matrix':
glht(model, linfct,
    alternative = c("two.sided", "less", "greater"),
    rhs = 0, ...)
## S3 method for class 'character':
glht(model, linfct, ...)
## S3 method for class 'expression':
glht(model, linfct, ...)
## S3 method for class 'mcp':
glht(model, linfct, ...)
mcp(...)

Arguments

model | a fitted model, for example an object returned by lm, glm, or aov etc. It is assumed that coef and vcov methods are available for model. For multiple comparisons of means, methods model.matrix, model.frame and terms are expected to be available for model as well.
linfct | a specification of the linear hypotheses to be tested. Linear functions can be specified by either the matrix of coefficients or by symbolic descriptions of one or more linear hypotheses. Multiple comparisons in AN(C)OVA models are specified by objects returned from function mcp.
alternative | a character string specifying the alternative hypothesis, must be one of '"two.sided"' (default), '"greater"' or '"less"'. You can specify just the initial letter.
rhs | an optional numeric vector specifying the right hand side of the hypothesis.
... | additional arguments to function modelparm in all glht methods. For function mcp, multiple comparisons are defined by matrices or symbolic descriptions specifying contrasts of factor levels where the arguments correspond to factor names.

Details

A general linear hypothesis refers to null hypotheses of the form H_0: K β = m for some parametric model model with parameter estimates coef(model).

The null hypothesis is specified by a linear function $K β$, the direction of the alternative and the right hand side $m$. Here, alternative equal to "two.sided" refers to a null hypothesis H_0: K β = m, whereas "less" corresponds to H_0: K β >= m and "greater" refers to H_0: K β <= m. The right hand side vector $m$ can be defined via the rhs argument.

The generic method glht dispatches on its second argument (linfct). There are three ways, and thus methods, to specify linear functions to be tested:

1) The matrix of coefficients $K$ can be specified directly via the linfct argument. In this case, the number of columns of this matrix needs to correspond to the number of parameters estimated by model. It is assumed that appropriate coef and vcov methods are available for model (modelparm deals with some exceptions).

2) A symbolic description, either a character or expression vector passed to glht via its linfct argument, can be used to define the null hypothesis. A symbolic description must be interpretable as a valid R expression consisting of both the left and right hand side of a linear hypothesis. Only the names of coef(beta) must be used as variable names. The alternative is given by the direction under the null hypothesis (= or == refer to "two.sided", <= means "greater" and >= indicates "less"). Numeric vectors of length one are valid values for the right hand side.

3) Multiple comparisons of means are defined by objects of class mcp as returned by the mcp function. For each factor, which is included in model as independent variable, a contrast matrix or a symbolic description of the contrasts can be specified as arguments to mcp. A symbolic description may be a character or expression where the factor levels are only used as variables names. In addition, the type argument to the contrast generating function contrMat may serve as a symbolic description of contrasts as well.

glht extracts the number of degrees of freedom for models of class lm (via modelparm) and the exact multivariate $t$ distribution is evaluated. For all other models, results rely on the normal approximation. Alternatively, the degrees of freedom to be used for the evaluation of multivariate $t$ distributions can be given by the additional df argument to modelparm specified via ....

glht methods return a specification of the null hypothesis H_0: K β = m. The value of the linear function $K β$ can be extracted using the coef method and the corresponding covariance matrix is available from the vcov method. Various simultaneous and univariate tests and confidence intervals are available from summary.glht and confint.glht methods, respectively.

Value

An object of class glht, more specifically a list with elements

model | a fitted model, used in the call to glht
linfct | the matrix of linear functions
rhs | the vector of right hand side values m
coef | the values of the linear functions
vcov | the covariance matrix of the values of the linear functions
df | optionally, the degrees of freedom when the exact $t$ distriubtion is used for inference
alternative | a character string specifying the alternative hypothesis
type | optionally, a character string giving the name of the specific procedure

with print, summary, confint, coef and vcov methods being available. When called with linfct being an mcp object, an additional element focus is available storing the names of the factors under test.

References

Shayle R. Searle (1971), Linear Models. John Wiley & Sons, New York.

Examples

  ### multiple linear model, swiss data
  lmod <- lm(Fertility ~ ., data = swiss)

  ### test of H_0: all regression coefficients are zero
  ### (ignore intercept)

  ### define coefficients of linear function directly
  K <- diag(length(coef(lmod)))[-1,]
  rownames(K) <- names(coef(lmod))[-1]
  K

  ### set up general linear hypothesis
  glht(lmod, linfct = K)

  ### alternatively, use a symbolic description
  ### instead of a matrix
  glht(lmod, linfct = c("Agriculture = 0",
                        "Examination = 0",
                        "Education = 0",
                        "Catholic = 0",
                        "Infant.Mortality = 0"))

  ### multiple comparison procedures
  ### set up a one-way ANOVA
  amod <- aov(breaks ~ tension, data = warpbreaks)

  ### set up all-pair comparisons for factor `tension'
  ### using a symbolic description (`type' argument
  ### to `contrMat()')
  glht(amod, linfct = mcp(tension = "Tukey"))

  ### alternatively, describe differences symbolically
  glht(amod, linfct = mcp(tension = c("M - L = 0",
                                      "H - L = 0",
                                      "H - M = 0")))

  ### alternatively, define contrast matrix directly
  contr <- rbind("M - L" = c(-1, 1, 0),
                 "H - L" = c(-1, 0, 1),
                 "H - M" = c(0, -1, 1))
  glht(amod, linfct = mcp(tension = contr))

  ### alternatively, define linear function for coef(amod)
  ### instead of contrasts for `tension'
  ### (take model contrasts and intercept into account)
  glht(amod, linfct = cbind(0, contr %*% contr.treatment(3)))


Litter Weights Data Set

Description

Dose response of litter weights in rats.

Usage

data("litter")

Format

This data frame contains the following variables

dose
dosages at four levels: 0, 5, 50, 500.
gesttime
gestation time as covariate.
number
number of animals in litter as covariate.
weight
response variable: average post-birth weights in the entire litter.

Details

Pregnant mice were divided into four groups and the compound in four different doses was administered during pregnancy. Their litters were evaluated for birth weights.

Source

P. H. Westfall, R. D. Tobias, D. Rom, R. D. Wolfinger, Y. Hochberg (1999). Multiple Comparisons and Multiple Tests Using the SAS System. Cary, NC: SAS Institute Inc., page 109.

P. H. Westfall (1997). Multiple Testing of General Contrasts Using Logical Constraints and Correlations. Journal of the American Statistical Association, 92(437), 299–306.

Examples

  ### fit ANCOVA model to data
  amod <- aov(weight ~ dose + gesttime + number, data = litter)

  ### define matrix of linear hypotheses for `dose'
  doselev <- as.integer(levels(litter$dose))
  K <- rbind(contrMat(table(litter$dose), "Tukey"),
             otrend = c(-1.5, -0.5, 0.5, 1.5),
             atrend = doselev - mean(doselev),
             ltrend = log(1:4) - mean(log(1:4)))

  ### set up multiple comparison object
  Kht <- glht(amod, linfct = mcp(dose = K), alternative = "less")

  ### cf. Westfall (1997, Table 2)
  summary(Kht, test = univariate())
  summary(Kht, test = adjusted("bonferroni"))
  summary(Kht, test = adjusted("Shaffer"))
  summary(Kht, test = adjusted("Westfall"))
  summary(Kht, test = adjusted("free"))


Methods for General Linear Hypotheses

Description

Simultaneous tests and confidence intervals for general linear hypotheses.

Usage

## S3 method for class 'glht':
summary(object, test = adjusted(), ...)
## S3 method for class 'glht':
confint(object, parm, level = 0.95, calpha = adjusted_calpha(), ...)
## S3 method for class 'glht':
coef(object, rhs = FALSE, ...)
## S3 method for class 'glht':
vcov(object, ...)
## S3 method for class 'confint.glht':
plot(x, xlim, xlab, ...)
## S3 method for class 'glht':
plot(x, ...)
univariate()
adjusted(type = c("free", "Shaffer", "Westfall", p.adjust.methods),
         ...)
Ftest()
Chisqtest()
adjusted_calpha()
univariate_calpha()

Arguments

object | an object of class glht.
test | a function for computing p values.
parm | additional parameters, currently ignored.
level | the confidence level required.
calpha | either a function computing the critical value or the critical value itself.
rhs | logical, indicating whether the linear function K hat{β} or the right hand side m (rhs = TRUE) of the linear hypothesis should be returned.
type | the multiplicity adjustment (adjusted) to be applied. See below and p.adjust.
x | an object of class glht or confint.glht.
xlim | the x limits (x1, x2) of the plot.
xlab | a label for the x axis.
... | additional arguments, such as maxpts, abseps or releps to pmvnorm in adjusted or qmvnorm in confint.

Details

The methods for general linear hypotheses as described by objects returned by glht can be used to actually test the global null hypothesis, each of the partial hypotheses and for simultaneous confidence intervals for the linear function $K β$.

The coef and vcov methods compute the linear function $K hat{β}$ and its covariance, respectively.

The test argument to summary takes a function specifying the type of test to be applied. Classical Chisq (Wald test) or F statistics for testing the global hypothesis $H_0$ are implemented in functions Chisqtest and Ftest. Several approaches to multiplicity adjusted p values for each of the linear hypotheses are implemented in function adjusted. The type argument to adjusted specifies the method to be applied: "free" implements adjusted p values based on the joint normal or $t$ distribution of the linear function, and "Shaffer" and "Westfall" implement logically constraint multiplicity adjustments (Shaffer, 1986; Westfall, 1997). In addition, all adjustment methods implemented in p.adjust are available as well.

Simultaneous confidence intervals for linear functions can be computed using method confint. Univariate confidence intervals can be computed by specifying calpha = univariate_calpha() to confint. The critical value can directly be specified as a scalar to calpha as well. Note that plot(a) for some object a of class glht is equivalent to plot(confint(a)).

All simultaneous inference procedures implemented here control the family-wise error rate (FWER). Multivariate normal and $t$ distributions, the latter one only for models of class lm, are evaluated using the procedures implemented in package mvtnorm.

Value

summary computes (adjusted) p values for general linear hypotheses, confint computes (adjusted) confidence intervals. coef returns estimates of the linear function $K β$ and vcov its covariance.

References

Juliet P. Shaffer (1986), Modified sequentially rejective multiple test procedures. Journal of the American Statistical Association, 81, 826–831.

Peter H. Westfall (1997), Multiple testing of general contrasts using logical constraints and correlations. Journal of the American Statistical Association, 92, 299–306.

Examples

  ### set up a two-way ANOVA with interactions
  amod <- aov(breaks ~ wool * tension, data = warpbreaks)

  ### set up all-pair comparisons for factor `tension'
  wht <- glht(amod, linfct = mcp(tension = "Tukey"))

  ### 95% simultaneous confidence intervals
  plot(print(confint(wht)))

  ### the same (for balanced designs only)
  TukeyHSD(amod, "tension")

  ### corresponding adjusted p values
  summary(wht)

  ### all means for levels of `tension'
  amod <- aov(breaks ~ tension, data = warpbreaks)
  glht(amod, linfct = matrix(c(1, 0, 0,
                               1, 1, 0,
                               1, 0, 1), byrow = TRUE, ncol = 3))

  ### confidence bands for a simple linear model, `cars' data
  plot(cars, xlab = "Speed (mph)", ylab = "Stopping distance (ft)",
       las = 1)

  ### fit linear model and add regression line to plot
  lmod <- lm(dist ~ speed, data = cars)
  abline(lmod)

  ### a grid of speeds
  speeds <- seq(from = min(cars$speed), to = max(cars$speed),
                length = 10)

  ### linear hypotheses: 10 selected points on the regression line != 0
  K <- cbind(1, speeds)

  ### set up linear hypotheses
  cht <- glht(lmod, linfct = K)

  ### confidence intervals, i.e., confidence bands, and add them plot
  cci <- confint(cht)
  lines(speeds, cci$confint[,"lwr"], col = "blue")
  lines(speeds, cci$confint[,"upr"], col = "blue")

  ### simultaneous p values for parameters in a Cox model
  if (require("survival") && require("MASS")) {
      data("leuk", package = "MASS")
      leuk.cox <- coxph(Surv(time) ~ ag + log(wbc), data = leuk)

      ### set up linear hypotheses
      lht <- glht(leuk.cox, linfct = diag(length(coef(leuk.cox))))

      ### adjusted p values
      print(summary(lht))
  }


Generic Accessor Function for Model Parameters

Description

Extract model parameters and their covariance matrix as well as degrees of freedom (if available) from a fitted model.

Usage

modelparm(model, coef., vcov., df, ...)

Arguments

model | a fitted model, for example an object returned by lm, glm, aov, survreg, or lmer etc.
coef. | an accessor function for the model parameters.
vcov. | an accessor function for the covariance matrix of the model parameters.
df | an optional specification of the degrees of freedom to be used in subsequent computations.
... | additional arguments, currently ignored.

Details

One can't expect coef and vcov methods for arbitrary models to return a vector of $p$ fixed effects model parameters (coef) and corresponding $p times p$ covariance matrix (vcov).

The coef. and vcov. arguments can be used to define modified coef or vcov methods for a specific model. Methods for lmer and survreg objects are available (internally).

For objects inheriting from class lm the degrees of freedom are determined from model and the corresponding multivariate $t$ distribution is used by all methods to glht objects. By default, the asymptotic multivariate normal distribution is used in all other cases unless df is specified by the user.

Value

An object of class modelparm with elements

coef | model parameters
vcov | covariance matrix of model parameters
df | degrees of freedom


Deprecated Functions multcomp Package

Description

These functions are provided for compatibility with older versions of multcomp only, and may be defunct in the next release.

Usage

csimint(estpar, df, covm, cmatrix=NULL, ctype="user-defined",
        conf.level=0.95, alternative=c("two.sided","less","greater"),
        asympt=FALSE, eps=0.001, maxpts=1000000)
## Default S3 method:
simint(object,
    type = c("Dunnett", "Tukey", "Sequen", "AVE", "Changepoint",
    "Williams", "Marcus", "McDermott"),
    cmatrix = NULL, conf.level = 0.95,
    alternative = c("two.sided","less","greater"),
    eps = 0.001, maxpts = 1e+06, whichf)
## S3 method for class 'formula':
simint(formula, data=list(), subset, na.action, ...)
## S3 method for class 'lm':
simint(object, psubset = NULL, ...)
csimtest(estpar, df, covm, cmatrix=NULL, ctype="user-defined",
         ttype=c("free","logical"),
         alternative=c("two.sided","less","greater"), asympt=FALSE,
         eps=0.001, maxpts=1000000)
## Default S3 method:
simtest(object,
    type = c("Dunnett", "Tukey", "Sequen", "AVE", "Changepoint",
    "Williams", "Marcus", "McDermott"),
    ttype = c("free", "logical"),
    cmatrix = NULL, conf.level = 0.95,
    alternative = c("two.sided","less","greater"),
    eps = 0.001, maxpts = 1e+06, whichf)
## S3 method for class 'formula':
simtest(formula, data=list(), subset, na.action, ...)
## S3 method for class 'lm':
simtest(object, psubset = NULL, ...)

Arguments

type | the type of contrast to be used.
cmatrix | the contrast matrix itself can be specified. If cmatrix is defined, type is ignored.
conf.level | confidence level.
alternative | the alternative hypothesis must be one of "two.sided" (default), "greater" or "less". You can specify just the initial letter.
asympt | a logical indicating whether the (exact) t-distribution or the normal approximation should be used.
eps | absolute error tolerance as double.
maxpts | maximum number of function values as integer.
psubset | a vector of integers or characters indicating for which subset of coefficients of a (generalized) linear model y simultaneous confidences intervals should be computed.
formula | a symbolic description of the model to be fit.
data | an optional data frame containing the variables in the model. By default the variables are taken from Environment(formula), typically the environment from which simint is called.
subset | an optional vector specifying a subset of observations to be used.
na.action | a function which indicates what should happen when the data contain NA's. Defaults to GetOption("na.action").
whichf | if more than one factor is given in the right hand side of formula, whichf can be used to defined the factor to compute contrasts of.
ttype | Specifies whether the logical constraint method of Westfall (1997) will be used, or whether the unconstrained method will be used.
estpar | estimated parameter vector.
df | degrees of freedom.
covm | estimated covariance matrix of estpar.
ctype | a string describing the kind of contrast matrix used.
object | a fitted model.
... | further arguments to be passed to or from methods.

Details

The generics simint and simtest implement the user interface for multcomp versions up to 0.4-8. The user visible functions simint, csimint, simtest, csimtest have been replaced by one single function glht which offers summary.glht and confint.glht methods for simultaneous tests and confidence intervals, respectively.

Please use glht for your analyses. The old user interface will be removed in future versions of this package.


Recovery Time Data Set

Description

Recovery time after surgery.

Usage

data("recovery")

Format

This data frame contains the following variables

blanket
blanket type, a factor at four levels: b0, b1, b2, and b3.
minutes
response variable: recovery time after a surgical procedure.

Details

A company developed specialized heating blankets designed to help the body heat following a surgical procedure. Four types of blankets were tried on surgical patients with the aim of comparing the recovery time of patients. One of the blanket was a standard blanket that had been in use already in various hospitals.

Source

P. H. Westfall, R. D. Tobias, D. Rom, R. D. Wolfinger, Y. Hochberg (1999). Multiple Comparisons and Multiple Tests Using the SAS System. Cary, NC: SAS Institute Inc., page 66.

Examples

  ### set up one-way ANOVA
  amod <- aov(minutes ~ blanket, data = recovery)

  ### set up multiple comparisons: one-sided Dunnett contrasts
  rht <- glht(amod, linfct = mcp(blanket = "Dunnett"),
              alternative = "less")

  ### cf. Westfall et al. (1999, p. 80)
  confint(rht, level = 0.9)

  ### the same
  rht <- glht(amod, linfct = mcp(blanket = c("b1 - b0 >= 0",
                                             "b2 - b0 >= 0",
                                             "b3 - b0 >= 0")))
  confint(rht, level = 0.9)


Industrial Waste Data Set

Description

Industrial waste output in a manufactoring plant.

Usage

data("waste")

Format

This data frame contains the following variables

temp
temperature, a factor at three levels: low, medium, high.
envir
environment, a factor at five levels: env1 ... env5.
waste
response variable: waste output in a manufacturing plant.

Details

The data are from an experiment designed to study the effect of temperature (temp) and environment (envir) on waste output in a manufactoring plant. Two replicate measurements were taken at each temperature / environment combination.

Source

P. H. Westfall, R. D. Tobias, D. Rom, R. D. Wolfinger, Y. Hochberg (1999). Multiple Comparisons and Multiple Tests Using the SAS System. Cary, NC: SAS Institute Inc., page 177.

Examples

  ### set up two-way ANOVA with interactions
  amod <- aov(waste ~ envir + temp + envir*temp, data=waste)

  ### set up linear hypotheses for all-pairs of both factors
  wht <- glht(amod, linfct = mcp(temp = "Tukey", envir = "Tukey"))

  ### cf. Westfall et al. (1999, page 181)
  summary(wht, test = adjusted("Shaffer"))