^{1}

^{2}

^{3}

The authors have declared that no competing interests exist.

This paper reports the results of a Bayesian analysis on large-scale empirical data to assess the effectiveness of eleven types of COVID-control policies that have been implemented at various levels of intensity in 40 countries and U.S. states since the onset of the pandemic. The analysis estimates the marginal impact of each type and level of policy as implemented in concert with other policies. The purpose is to provide policymakers and the general public with an estimate of the relative effectiveness of various COVID-control strategies. We find that a set of widely implemented core policies reduces the spread of virus but not by enough to contain the pandemic except in a few highly compliant jurisdictions. The core policies include the cancellation of public events, restriction of gatherings to fewer than 100 people, recommendation to stay at home, recommended restrictions on internal movement, implementation of a partial international travel ban, and coordination of information campaigns. For the median jurisdiction, these policies reduce growth rate in new infections from an estimated 270% per week to approximately 49% per week, but this impact is insufficient to prevent eventual transmission throughout the population because containment occurs only when a jurisdiction reduces growth in COVID infection to below zero. Most jurisdictions must also implement additional policies, each of which has the potential to reduce weekly COVID growth rate by 10 percentage points or more. The slate of these additional high-impact policies includes targeted or full workplace closings for all but essential workers, stay-at-home requirements, and targeted school closures.

Many countries, regions, and U.S. states (called ‘jurisdictions’ here) have implemented COVID-control policies such as stay-at-home requirements, workplace closings, school closings, restrictions on gatherings, international travel controls, testing and contact tracing. In this paper, we report on the results of a Bayesian analysis on which policies are most important to infection control. The model generates estimates of the marginal impact of each policy in a jurisdiction after accounting for (i) the overall portfolio of policies adopted by the jurisdiction, (ii) the levels at which the policies are implemented, (iii) the rigorousness of compliance within the jurisdiction, (iv) the jurisdiction’s COVID infections, COVID deaths, and excess deaths, and (v) the performance of the portfolio of policies in other jurisdictions. The purpose is to inform decisions about changing both the level and type of policies.

We find that, across the 40 jurisdictions, the relevance of each of the policies–and the levels at which they must be implemented to drive COVID infection growth to below zero–depends sensitively on compliance in the general population with a core group of policies that are relatively less constraining of social interaction than others. We call these socially tolerable policies the “core” group in our analysis. These include the cancellation of public events, the restriction of gatherings to fewer than 100 people, recommendation to stay at home, recommended restrictions on internal movement, implementation of a partial international travel ban, and coordinated public information campaigns. However, these relatively tolerable core policies are insufficient alone for COVID control in almost all jurisdictions. At least several of the higher-impact, harder-to-tolerate, more restrictive policies must be implemented to drive growth in COVID infections below zero. We refer to these additional policies, each of which must be implemented on top of the core policies to have the estimated effect, as the “additional high-impact” group. We find that, across the 40 jurisdictions, the portfolio of additional high-impact policies includes targeted or full workplace closings for all but essential workers, stay-at-home requirements, and targeted school closings. Each is estimated to reduce the virus growth rate by 10 percentage points or more. The optimal portfolio of additional high-impact policies appropriate for each jurisdiction depends sensitively on compliance with the core policies and on interactions between policies. All data, code, and modeling information used for this project are publicly available through links described below.

Assessing the consequences of changing policies depends on accurate information about infections and deaths [

We incorporate the weekly levels reported in the Oxford COVID-19 Government Response Tracker (OxCGRT) database in eleven categories: School closing, workplace closing, cancel public events, restrictions on gatherings, close public transport, stay-at-home requirements, restrictions on internal movement, international travel controls, public information campaigns, testing, and contact tracing. The level of each policy is assessed daily on a scale that can include up to 4 stages. The lowest level of restriction is level 1 and the greatest level is level 4. The levels are customized for each policy. For example, level 1 of international travel controls is screening arrivals whereas level 4 is closing borders to international travel.

The first set of results is a report on the average impact on COVID infection of each of the eleven policies across all the represented jurisdictions in our model. The results depend on the simultaneous deployment of the various policies across jurisdictions. The second result is a report on the percentage of jurisdictions over time with a given policy in place. The third result is the current status of new infections and growth in infections by jurisdiction. Fourth, we estimate the weekly base growth rate in infections for each jurisdiction under the hypothetical scenario that each adopts four basic policies at level 1, i.e., school closing, workplace closing, cancel public events, and restrictions on gatherings. This allows us to estimate the base rate of compliance with the portfolio of policies by jurisdiction, which in turn enables estimation of the relative impact of each policy for relatively more vs. less compliant jurisdictions. Fifth, we estimate the extent to which a set of widely adopted core policies is effective in locally containing the pandemic, and which additional high-impact policies might potentially be needed to do so. The

Three streams of existing research are most relevant to our study: modeling studies that estimate and predict infection and death rates, medical studies that investigate the important drivers of disease spread, and studies that estimate the effectiveness of specific policies.

Many modeling approaches have been adopted to estimate infection rates, death rates, and disease spread. A well-established class of models relies on differential equations to analyze disease dynamics within susceptible, exposed, infected, and recovered (SEIR) “compartments” of populations [

Despite the limitations, SEIR-type models can serve as a basis for Bayesian inference. For example, Dehning and colleagues combined Bayesian methodology with an SIR model, which is formulated in terms of coupled ordinary differential equations and enables fast evaluation [

A second stream of related research is comprised of medical studies that explore how the virus is transmitted. In contrast to modeling studies that customarily make use of population-level data, these medical studies pinpoint person-to-person transmission mechanisms through clinical verification within identifiable groups of individuals and/or households. An extensive amount of research has been conducted to describe and measure the airborne transmission of COVID [

A number of studies have investigated the impact of health and non-health policies on COVID spread. This stream of research typically combines models with actual infection and death data to gauge policy effectiveness. For instance, Badr and colleagues find that reduced mobility was associated with reduced COVID growth rates in 20 U.S. counties [

Another topic that has received attention is the impact of school closures on COVID spread. Studies have found mixed results. Some have shown that closing schools in U.S. states led to lower case number and infection rates [

While most policy studies focus on a specific set of policies or on a particular country, some studies are broader in scope. For example, Hsiang and colleagues investigate policies (non-pharmaceutical interventions) across six countries using econometric methods and conclude that different types of policies, when combined together, led to significantly reduced growth in infections [

Prior policy papers generally analyze the effectiveness of COVID control within a geographic area rather than assess how individual policies contribute across jurisdictions to the achievement of COVID control. This study is among the first to study how types of policies–implemented with varying intensity levels–perform across a large number of jurisdictions. In the sections below, we describe the model, data, and distinctive features of the approach.

The model is based on an exponential growth curve of infection rates by jurisdiction. This curve arises from a parsimonious SIR model in which the susceptible population (S) is assumed to be substantially greater than the infected (I) or recovered (R) population. The model assumes that, in most jurisdictions, just a minority of the population has been infected, and that herd immunity (i.e., more than 50% of the population has been infected) is remote in all jurisdictions. Under these assumptions, the growth in number of infections ^{gt}. In this equation, the variable

The reasons we model

We assume that the growth rate for each jurisdiction

The first term represents the base growth rate by jurisdiction, while the second term represents the growth rate reduction arising from the implemented policies. The base growth rate is represented by the equation _{it}^{(base)} = _{i}^{(0/1)} + Δ_{it}^{(id)}, which consists of two parts. The term _{i}^{(0)} represents COVID growth prior to the implementation of any policies (e.g., equivalently to _{0} in SIR models). After jurisdictions have implemented basic containment policies, _{i}^{(0)} is replaced by _{i}^{(1)} (see below for detail on policy coding). The second part, Δ_{it}^{(id)}, represents the idiosyncratic growth rate. This term captures any jurisdiction-specific changes over time in COVID growth that cannot be attributed to the policies coded in our database. For this term, we assume a second-order autoregressive process (AR(2)). Mathematical details are provided in the

The term ∑_{p}δ_{ipt}_{p}_{ipt} = 1 if jurisdiction _{ipt} = 0 otherwise. Δ_{p} ≥ 0 is the estimate for the average effect of each policy.

The growth rate is determined by the weekly change in new infections in logarithms: _{it} = log(_{it})–log(_{i,t-1}). Because the number of infections is not directly observable, it is inferred from three different observations by jurisdiction-week: the number of reported COVID cases, the number of reported COVID deaths, and the number of excess deaths. The model considers that reported COVID cases and deaths may be significantly understated, and that the level of underreporting may evolve with changing testing policies. The model also accounts for the lag between infections on the one hand and reported cases and deaths on the other.

We use two sets of input data: policy data to code the variable _{ipt} and epidemiological data to infer the level of new infections _{it}. The policy data are based on the OxCGRT database [

Policies C1-C6; continued in

Policies C7, C8, H1-H3; continued from

ID | Description | Policy level |
---|---|---|

C1 | Record closings of schools and universities | 0—no measures |

C2 | Record closings of workplaces | 0—no measures |

C3 | Record cancelling public events | 0—no measures |

C4 | Record limits on private gatherings | 0—no restrictions |

C5 | Record closing of public transport | 0—no measures |

C6 | Record orders to "shelter-in-place" and otherwise confine to the home | 0—no measures |

C7 | Record restrictions on internal movement between cities/regions | 0—no measures |

C8 | Record restrictions on international travel | 0—no restrictions |

H1 | Record presence of public info campaigns | 0—no COVID-19 public information campaign |

H2 | Record government policy on who has access to testing | 0—no testing policy |

H3 | Record government policy on contact tracing after a positive diagnosis | 0—no contact tracing |

Adapted from:

The model treats the levels cumulatively. For instance, if a jurisdiction is coded as level 2 on policy H2 (virus testing) in a certain week, then we code _{ipt} = 1 for both level 1 and level 2, and _{ipt} = 0 for level 3. In other words, the effect of H2 policies on the overall growth rate _{it} is modelled as Δ_{p = H2-1} + Δ_{p = H2-2,} with H2-

Because significant policy efforts in essentially all jurisdictions began with C1 (school closing), C2 (Workplace closing), C3 (Cancel public events), and C4 (restrictions on gatherings), we use the implementation of any of these policies at level 1 or above as the criterion for shifting the time-constant part of the base growth rate from _{i}^{(0)} (free spread of the virus corresponding to _{0}) to _{i}^{(1)} (spread of the virus with only basic policies in place). Because C1-C4 are implemented almost simultaneously at an initiating level 1 or greater, they are highly collinear in the estimation of Eq (_{p}. For the same reason we also exclude policy levels that are implemented in fewer than 5% of all policy-weeks. (We also used several different sets of base policies for _{i}^{(1)} and of excluded variables. The marginal effects for these (unreported) analyses were not significantly different from those reported here.) This results in 23 distinct policy levels that we include in our analysis.

The epidemiological data arise from two sources. COVID case and death data are taken from the Johns Hopkins Coronavirus Resource Center to calculate weekly counts of new cases and deaths by jurisdictions. Excess death data are drawn from The Economist, which reports weekly and monthly total deaths above what was projected based on pre-COVID trends for a limited set of countries and U.S. states. The excess deaths over expected deaths serve as a proxy for the underreporting of COVID-19 deaths.

In our analysis, we include all countries and U.S. states with at least 5 million residents for which The Economist reports excess deaths weekly. Forty jurisdictions meet these criteria. The

Both policy and epidemiological data are used to estimate the effect of various policies on COVID control. The primary variable of interest is Δ_{p}, which represents the marginal effect of each implemented policy, _{it}^{(base)}, the base growth rate in infections independently of the implemented policies coded in Δ_{p}. Finally, we use the model to make inferences by jurisdiction of weekly new infections _{it} and the infection growth rate _{it}.

The Bayesian model generates point estimates (posterior median) and 95% intervals (posterior quantiles) for each of the variables of interest. Eq (_{it} = log(_{it})–log(_{i,t-1}), is not directly observable because we do not know the actual level of new infections _{it}. Therefore, this variable must be estimated probabilistically based on observations of reported COVID cases, reported COVID deaths, and excess deaths. Bayesian modeling is uniquely suited to accomplish this. For the reported case count, _{it}^{(case)}, we adopt a negative binomial model with rate parameter

The rate parameter _{ipt}^{(case)} that is reported in jurisdiction _{s} that is reported with a lag of _{it}. Similar negative binomial models are assumed for reported COVID deaths and excess mortality.

One main advantage of the Bayesian approach is that we can use the observed data to make stochastic inferences (with 95% intervals) about the three parameters

A second advantage of the Bayesian model is that it is robust to outliers [_{outlier}, and that the observation is drawn from the distribution represented by Eq (_{outlier} = 10^{−3} for reported COVID cases and deaths (no assumption is necessary for excess deaths because these do not exhibit outliers). In the ^{−2} and 10^{−4}.

Third and finally, Bayesian analysis accommodates further constraints and/or stochastic assumptions. For instance, we impose a constraint that policy effects reduce rather than increase infection by assuming Δ_{p} ≥ 0. Other constraints include that the fraction _{it}^{(id)} (see Equation (S3) of the

For the population of countries and U.S. states included in our analysis, the marginal impact Δ_{p} of each policy

Bars = median estimate by policy level; Lines = 95% intervals for maximum policy level.

Several important results are evident in

_{it}. For Italy and Switzerland, new infections are more than 200 per 10,000 people weekly, which is close to the highest numbers of infections in the early stages of the pandemic in Spain, New York, and New Jersey. Growth rates of new infections are highest in Sweden, Colorado, and Pennsylvania. In these jurisdictions, new infections are growing at a weekly rate around or above _{it} = 0.4, which corresponds to an increase of 49% new cases every week (e^{0.4}–1 = 0.49). In other jurisdictions, the growth rate is closer to or below zero (although presently the rate is estimated to be below zero with at least 95% certainty only in Belgium, Chile, Denmark, and France). These differences arise from variation in the policies in place as represented in Figs

Dots = median estimates; Lines = 95% intervals.

_{it}^{(base)} by jurisdiction in the hypothetical situation in which only level 1 policies were implemented on school closing, workplace closing, canceling public events, and restrictions on gatherings (see Eq (_{i}^{(1)}. The model does not, however, offer reasons for this outcome. For example, Sweden’s relatively low population density, weather characteristics, and demographics likely play a role in the result. In recent months, Sweden’s base rate of growth in infection is estimated to have increased, while other jurisdictions such as Denmark and South Africa have estimated _{it}^{(base)} that have declined to levels below that of Sweden. By contrast, Brazil, Chile, Mexico, and Spain have had at least some periods in which implementation of only base policies would likely have not had as strong an impact on COVID control.

A lower number indicates a higher effectiveness to contain the virus. The total growth rate _{it} of the virus is the number in this chart minus the sum of the effects of implemented policies in

^{(base)} for each category of jurisdiction: the most compliant 10%; the median by compliance; and the least compliant 10%. For jurisdictions in each of these categories, the figure shows how much reduction in the growth rate

Core policies include the cumulative effects of C3–2, C4–3, C6–1, C7–1, C8–3, H1–2, H2–2, and H3–2. Dots = median estimates; Lines = 95% intervals.

The figure shows that the set of core policies lead to COVID control only for jurisdictions with strong policy compliance (i.e., the lowest ^{(base)}). In contrast, for the median jurisdiction, the core policies are insufficient for driving COVID growth below zero despite having a strong effect, i.e., they reduce growth from an estimated 1.3 cases per week corresponding to an increase in virus spread by e^{1.3}–1 = 270% to an estimated 0.4 cases per week corresponding to a growth of e^{0.4}–1 = 49%. Hence, for 90% of jurisdictions, the core policies are insufficient to achieve COVID control. Almost all jurisdictions must implement more stringent policies to achieve a further reduction of the weekly growth rate _{p} above 0.1) are:

Targeted workplace closings (C2–2; Δ_{p} = 0.20)

Full workplace closings (C2–3; Δ_{p} = 0.14, on top of C2–2)

Stay-at-home requirement with exceptions (C6–2; Δ_{p} = 0.14)

Targeted school closings (C1–2; Δ_{p} = 0.14)

The next two most effective policies that must be implemented in addition to the core policies are (with median estimated effectiveness Δ_{p} between 0.05 and 0.1):

Full school closings (C1–3; Δ_{p} = 0.07, on top of C1–2)

Restrictions on gatherings below 10 (C4–4; Δ_{p} = 0.07)

Each of these policies has significant societal costs. Unfortunately, however, unless governments can compel greater compliance with core policies, jurisdictions must implement several of these additional policies to ensure that virus spread does not grow exponentially.

As with any study, these results must be interpreted within the limitations of the methods used. One limitation is that we need to make several assumptions as described in the Methods section about disease epidemiology and the impact of government policy. In the

The analysis generates several important insights. First, important differences arise across jurisdictions in infection levels, death rates, policy implementation, and compliance with policies. These differences have major implications for COVID control. The portfolio of the eleven types of policies–implemented with different levels of intensity–necessary to drive COVID growth rate below zero depends on the jurisdiction’s COVID burden and compliance, which reflects behavioral and demographic characteristics of the jurisdictions. Second, we estimate that a core set of socially tolerable policies lead to COVID control only in those jurisdictions that have unusually high levels of compliance. The socially tolerable core policies alone are meaningful and significant, but insufficient by themselves for preventing escalating growth in infections in 90% of the jurisdictions analyzed. For these jurisdictions, one or more from a set of additional high-impact but difficult-to-tolerate policies must be implemented to achieve COVID control. Third, the impact of testing and contact tracing has been lower than the impact of other policies. Fourth, for the jurisdictions covered in this analysis, the policies with the greatest marginal impact for achieving COVID control mainly involve restrictions on adults through workplace closings and stay-at-home requirements, although targeted school closings are also in the group of additional high-impact policies. Altogether, the analysis indicates that, in all but a few highly compliant jurisdictions, relatively significant social costs must be incurred to reduce COVID growth below zero. The model points to significant opportunities for cultivating deeper understanding of the drivers of compliance and of variation in the impact of policies, which are potentially rewarding topics for future research.

Dots = reported; X = outlier; Solid lines = model fit; Dashed lines = 95% intervals.

(DOCX)

Dots = median estimates; Lines = 95% intervals; Red = Base estimate outside of 95% posterior interval of changed specification.

(DOCX)

(DOCX)

Dots = reported; Solid lines = model prediction; Dashed lines = 95% intervals.

(DOCX)

(DOCX)

(DOCX)

(DOCX)