%!TEX TS-program = xelatex
%!TEX encoding = UTF-8 Unicode

%
% Policy analysis is often presented as either ‘art’ or ‘science’. With reference to the course materials in weeks 1-6 (inclusive) and additional research, including practical examples, please choose one side of the debate and make your case.
%
% Created by Milan Ilnyckyj on 2014-02-07
%  Copyright (c) 2014 Milan Ilnyckyj. All rights reserved.
%

\documentclass[12pt]{article}

% This is the preamble

% Letter sized paper with 1 inch margins
\usepackage[paperwidth=8.5in,paperheight=11in,left=1in,top=1in,right=1in]{geometry}

% Use system fonts
\usepackage{fontspec}
\defaultfontfeatures{Mapping=tex-text}
\setromanfont{Perpetua}

% Set double spacing
\usepackage{setspace}
\doublespacing

% Allow URLs
\usepackage{url}
\usepackage[colorlinks=true,allcolors=navy]{hyperref}

% Indent first graphs
\usepackage{indentfirst}

% Letter sized paper with 1 inch margins
\usepackage[paperwidth=8.5in,paperheight=11in,left=1in,top=1in,right=1in]{geometry}

% Define colours for use in this document
\usepackage{color}
\definecolor{dkred}{rgb}{0.9,0.05,0.05}
\definecolor{navy}{rgb}{0.1,0.1,0.4}
\definecolor{gray75}{gray}{0.75}
\definecolor{gray20}{gray}{0.2}

% Code for fancy section headings
\usepackage{titlesec, blindtext, color}
\newcommand{\hsp}{\hspace{20pt}}
\titleformat{\section}[hang]{\normalsize\bfseries}{\thesection\hsp\textcolor{gray75}{|}\hsp}{0pt}{\normalsize\bfseries}

\usepackage[version=3]{mhchem} % allows \ce{*formula*} for chemical notation. CO2 should be written \ce{CO2}

% Single space in quotes
\makeatletter
\g@addto@macro\quote\singlespace
\makeatother

% Bibliography settings
\usepackage[style=authortitle-icomp,natbib=true,sortcites=true,block=space]{biblatex}
\bibliography{PolAnal-EssayOne}

\title{The semi-empirical `science' of policy analysis}
\author{Milan Ilnyckyj}
\date{2014-02-07}

% The preamble is ending now



\begin{document}



\begin{titlepage}

\maketitle

\thispagestyle{empty}

\end{titlepage}



\clearpage

% Policy analysis is often presented as either `art' or `science'. With reference to the course materials in weeks 1-6 (inclusive) and additional research, including practical examples, please choose one side of the debate and make your case.
% Approximately 10 double-spaced pages (excluding references and foot/end notes), 12-point font, one-inch margins.
% Cite practical examples as well as sources; show that you have done outside thinking; consider cases from the readings and your own expertise



\doublespacing
\setcounter{page}{1}



%%% INTRODUCTION %%%



Assessing whether policy analysis is better described as an `art' or a `science' leads us inevitably into the realm of semantic disagreement.\footcite[Furthermore, there are many additional models of the policy-making process, including policy-making as ``a political game'', as ``discourse'', as ``an institutional process'' and others. See: ][p. 12]{Enserink2013}
Neither `art' nor `science' has a universally-agreed definition.
Indeed, both might be fairly categorized as ``essentially contested concepts'', as defined by Walter Bryce Gallie.\footcite[][]{Gallie1956a} \footcite[][]{Gallie1956b} \footcite[][]{Gallie1957} \footcite[][p. 157--191]{Gallie1964}
These are concepts for which ``there is no one clearly definable general use... which can be set up as the correct or standard use'' and where no general definition ever emerges because groups with differing interpretations ``continue... to defend [their] cases with what [they] claim... to be convincing arguments, evidence and other forms of justification''.\footcite[][p. 168]{Gallie1956a}
At the same time as there is no consensus on the meaning of either term, there are very different interpretations of what policy analysis comprises, with different scholars emphasizing different aspects of the work.
In particular, there are those who emphasize the internal politics of bureaucracy and the interpersonal diplomacy necessary to function effectively within the civil service or other organizations engaged in policy analysis, and then there are those who emphasize the efforts of policy analysts to develop and justify empirical hypotheses.\footcite[For a well-argued example of the first kind, see: ][]{Prince2007}
Viewed in the first way, policy analysts must be ``astute, shrewd, and subtle''.\footcite[][]{Prince2007}
The first interpretation, which focuses on the behaviour of individual analysts within bureaucracies, may fit more naturally with the perspective of policy analysis as `art'.
The second interpretation, which focuses on the effort to produce robust empirical claims, falls more naturally under the heading of `science'.



Of the two terms, calling policy analysis a `science' is arguably far more specific and therefore potentially subject to more meaningful evaluation.
`Art' offers such an enormous latitude of interpretation --- from the roof of the Sistine Chapel to Sun Tzu's `art' of war --- that it does little to aid our understanding of what policy analysis involves.
The `science' categorization, by contrast, is most easily challenged by pointing out the inescapable presence of value judgments and normative issues in policy-making and policy analysis.
While the effort to produce and evaluate empirical claims is at the heart of what a non-partisan and merit-based civil service is meant to do, it clearly cannot do so in isolation from normative considerations.
These considerations arise not only when trying to decide what course or action would be wisest, most just, or most equitable --- but also during the theoretically more objective process of identifying potential policies and attempting to project their consequences.
It seems most justifiable, therefore, to characterize policy analysis as a `science' profoundly intertwined with non-scientific normative considerations.



In addition to offering somewhat greater precision than the `art' categorization, the `science' perspective carries less danger of leading to a disempowering form of relativism.
If policy analysis is a deeply personal and impressionistic undertaking, we have few mechanisms for distinguishing between examples of it being done well or badly.
This sits at odds with the apparent possibility of comparative public policy analysis that reaches fairly robust conclusions about how effectively different public policy options serve particular objectives.
Furthermore, the `art' characterization risks obscuring some of the normative questions involved and ethical decision-making that necessarily accompanies even seemingly technical choices.
For those reasons, it is most credible to think of policy analysis as a `science' in symbiosis with ethics, rather than as an `art'.\footcite[For the purposes of this analysis, only policy-making in a contemporary context is being considered. It is possible that the relative plausibility of `science' and `art' frames for understanding public policy has varied across time as the discipline of public policy has evolved. For more on the subject, see: ][p. 8--27]{Radin2013} \footcite[See also: ][p. 79--95]{deLeon1994} \footcite[][p. 411--425]{Lynn1999}



%%% DEFINITIONS %%%
\section{Definitions}



To categorize public policy analysis as either an `art' or `science' is to apply one inappropriate label or another. 
Forced to do so, it can best be described as a deeply imperfect `science' which can never achieve the level of normative detachment aspired to in the `pure' or `natural' sciences.
The \emph{Oxford English Dictionary} contains a short descriptive passage distinguishing between `science' and `art':
\begin{quote}
The distinction as commonly apprehended is that a science is concerned with theoretic truth, and an art with methods for effecting certain results. Sometimes, however, the term science is extended to denote a department of practical work which depends on the knowledge and conscious application of principles; an art, on the other hand, being understood to require merely knowledge of traditional rules and skill acquired by habit.\footcite[][]{OEDScience}
\end{quote}
While accurately reflective of the usage of the two terms, this distinction leaves things rather muddled.
Certainly, policy analysis is concerned with producing ``certain results'', but it is also clearly ``practical work'' that depends on intentionally-acquired knowledge, rather than an automatic product of habit.
Helpfully, the \emph{OED} also provides a more precise definition of `science' that may be of greater use here:
\begin{quote}
In a more restricted sense: A branch of study which is concerned either with a connected body of demonstrated truths or with observed facts systematically classified and more or less colligated by being brought under general laws, and which includes trustworthy methods for the discovery of new truth within its own domain.
\end{quote}
The basic positivist impulse in the social sciences is to apply the methods of the natural sciences to social and political circumstances, attempting to evaluate hypotheses on the basis of their conformity or divergence from observed empirical reality.
A scientific approach to policy analysis is one that seeks to generate and maintain such a connected body of truths, evaluated on the basis of empirical observation and sometimes experiment, and which seeks to use data and the generation and evaluation of hypotheses to make novel, generalizable, and falsifiable claims about matters related to public policy.
While such a perspective does not account for every important feature of policy analysis, it does reflect the persistent and general impulse to understand the connections between causes and effects on the basis of data from the real world.



None of this is to deny that it is possible to convincingly frame policy analysis as an `art'.
Indeed, the extremely wide latitude of interpretation associated with the word `art' encompasses several distinct interpretations of policy analysis.
Furthermore, since `art' is an introspective activity and one constant undertaking within the artistic community has been interrogating and challenging the boundaries of `art', there is still more semantic terrain to work with.\footnote{Consider how much effort artists have devoted to convincing critics that soup cans, urinals, and slashed blank canvases are `art', and those are just some of the most prominent successful examples.}
In particular, there is value in interpreting policy analysis as a kind of performance or ongoing act of diplomacy, as Michael Prince does.\footcite[][]{Prince2007}
If we take `policy analysis' to mean everything policy analysts do in the course of their work --- from engaging in intraoffice politics to framing arguments in terms likely to be well-received by their superiors to jockeying for promotion --- this perspective of policy analysis as performance art may have greater validity.\footcite[For more commentary on these dimensions of the lives of policy analysts, see: ][p. 628--34]{LidmanSommers2005} \footcite[See also: ][p. 414--5]{Lynn1999} \footcite[][p. 613--33]{HowlettWellstead2011}
It is possible, however, to confine our analysis to the task of identifying policy options and evaluating their probable consequences.\footcite[As Bryan Jones identifies: ``Decision makers did not simply need to choose among alternatives; they had to generate the alternatives in the first place. Problems were not givens; they had to be defined. Solutions did not automatically follow problems; sometimes actors had set solutions ready to apply to problems that could occur.''][p. 302--3]{Jones1999}
While the actual enterprise of doing these things is inevitably coloured by personal and small-p `political' matters, policy analysis can also be interpreted as an idealized method of reaching the most defensible possible empirical conclusions and then using them as inputs for a normative process of policy-making.
In elaborating that idealized `scientific' view, I will first consider the empirical aspects of policy analysis, before moving on to normative questions.



%%% EMPIRICAL ASPIRATIONS %%%
\section{The empirical aspiration in policy analysis}



As Gallie argues, ``the words `science' and `scientific' do not stand for a single definable concept''.\footcite[][p. 120]{Gallie1957}
That being acknowledged, a commitment to the evaluation of hypothesis with reference to externally observed empirical phenomena is probably the closest thing to a working definition of `science'.\footnote{As `Zombie Feynman' asserts in an XKCD comic: ``Ideas are tested by experiment. That is the core of science''. \url{https://xkcd.com/397/}. Alternatively, as expressed by Leonardo da Vinci: ``First I shall do some experiments before I proceed farther, because my intention is to cite experience first and then, with reasoning, show why such experience is bound to operate in such a way... And this is the true rule by which those who speculate about the effects of nature must proceed''. \url{https://www.youtube.com/watch?v=kMf8hFBJylA}}
Virtually all areas of public policy-making involve judgments on empirical questions: the size of fish stocks must be estimated, the relative effectiveness of medical treatments must be assessed, the ability of various policy measures to alleviate poverty must be evaluated, and the ability of different air defence systems to intercept incoming aircraft and missiles must be determined.
Answering these questions involves methods closely akin to those employed routinely in the natural sciences.
They may rely on the collection of large datasets, possibly using sampling methodology derived from the natural sciences, followed by the quantitative analysis of the data using tools that would be familiar to a scientific researcher.
As former Clerk of the Privy Council Mel Cappe argues:
\begin{quote}
Ultimately, the application of science, evidence, data, analysis, modeling and econometrics will improve our understanding of our identity, our problems and our solutions.\footcite[][p. 6]{Cappe2011}
\end{quote}
Expertise is a major justification for the existence and influence of the community of policy analysts within and outside of government, and the expertise that is relevant largely consists of the rigorous use of such methods.\footcite[][p. 14]{Enserink2013} \footcite[As Michael Howlett and Adam Wellstead identify, the specific toolset employed varies between analysts and further empirical work on the precise approach used by different policy analysts may enrich our general understanding of the policy-making process.][p. 624--6]{HowlettWellstead2011} \footcite[On the relationship between expertise and influence, see also: ][p. 255--273]{Page2010} \footcite[Other literature considers the role of civil servants in terms of competing values of ``detachment'' and ``responsiveness''. See: ][p. 1250--71]{Montpetit2011}



In many cases, experiments are the gold standard of empirical science.
This is because they permit researchers to keep every variable aside from the independent variable which is being tested constant, and thus be able to make strong causal claims on the basis of the observed results.
If you have a control group and an experimental group, and each experiences identical treatment aside from one variable being tested, researchers are left in a fairly good position to ascribe causal importance from the variable being manipulated to systematic variations in outcomes between the two treated groups.
While certainly desirable, it is worth noting that meeting this standard is not necessary in order for one's work to be seen as well within the generally-accepted boundaries of `science'.
Astronomers, for instance, are rarely able to evaluate their hypotheses through conventional experiments.
They can hardly create control and treatment groups for stars or galaxies --- to say nothing of big bangs --- in which they can alter a single variable and observe the effects.
Nonetheless, their hypotheses can be evaluated to some degree or another on the basis of experiments, as well as falsified by new empirical observations that contradict existing theories.
For instance, spectroscopy is a key technique for evaluating the chemical composition of objects that are too distant to be physically sampled.
The methods of spectroscopy, however, can be evaluated and refined on the basis of experiments in which known combinations of substances are examined.
Furthermore, irresolvable contradictions between our theories about how the chemical composition of objects affects the light passing through or radiating from them and our observations of the physical universe can draw our attention to problems with our observations, hypotheses, or both.
Within the field of policy analysis, there may be similar examples of cases where empirical evaluation can normally only be done observationally and not by experiment, but in which hypotheses may sometimes be substantiated or refuted on the basis of experiment or empirical observation.
We cannot, for instance, randomly assign different developing countries to different treatment and control groups to evaluate the effects of policy choices on rates of economic development.
We can, however, take advantage of `natural experiments', in which the unmodified unfolding of real-world conditions approximates this ideal, as well as use historical data to attempt to assess the accuracy and generalizability of causal claims.



Beyond empiricism, Gallie highlights ``inherent progressiveness'' as the hallmark of a science.\footcite[][p. 136]{Gallie1957}
Under this view, sciences evolve as new contributions build upon what was established in the past.
Today's researchers ``stand on the shoulders of giants'', to use Isaac Newton's famous metaphor.
By this criterion, policy analysis is arguably in more trouble as a `science'.
It's not clear that a linear or quasi-linear process of intellectual development takes place within policy analysis.\footcite[See: ][p. 81]{deLeon1994}
Rather, there may be different threads of interpretation that only engage with one another to a very limited degree --- consider, for instance, rational choice analyses in comparison with Marxist analyses and radically post-modern feminist analyses.\footcite[See: ][p. 13--37]{ShepsleBonchek1997} \footcite[][p. 45--61]{Simon1995} \footcite[][p. 297--321]{Jones1999} \footcite[][p. 29--39]{Pal2011}
In part, this is because of how the ideological preconceptions of policy analysts affect what they perceive, as well as what they advocate doing about it.
Empirical interpretations can shift in light of changing ideological commitments.
For instance, Sylvia Bashevkin argues that the neoliberal Thatcher / Mulroney / Reagan period saw welfare recipients re-categorized as morally underserving, and that this understanding carried over to their centre-left successors in Blair, Chretien, and Clinton.\footcite[][]{Bashevkin2002}
If it is strongly the case that the ideological preconceptions of policy analysts affect their interpretation of apparently empirical data, the possibility that public policy thinking can accumulate and progress in the way Gallie points to becomes more implausible.
The difficulty or impossibility of separating values from facts in the evaluation of policy options requires us to restrict and specify what we mean by policy-making as a `science'.



%%% NORMATIVE %%%
\section{The relationship between empirical and normative questions}



Policy analysis may have the appearance of a science because it involves empirical claims and methods for evaluating them, but policy success or failure is ultimately a normative matter.\footcite[For a good discussion of this basic point in the context of environmental policy, see: ][]{Schmid2002}
Whether they are acknowledged overtly or not, policy decisions inevitably have normative consequences.
This is the major reason why policy analysis cannot simply be characterized as a `science', regardless of how rigorous the tools used to make empirical judgments become.
Over and above the normative consequences of making policy choices, there are arguably normative assumptions that underlie the process of policy analysis itself.
Policy analysts sometimes aspire to be impartial arbiters of human relations, distinguishing between policy options that are objectively superior or inferior on the basis of tools that do not inherently privilege one philosophical perspective or one set of interests over another.
While elements of this aspiration are laudable, it is doubtful that it serves as a credible description of how policy analysis is actually practiced.



Consider cost-benefit analysis.
Even if the actual tabulation of costs and benefits can be achieved in a truly objective way, it is necessary to make a decision at the outset about whose interests matter: those of the citizens of one state only?
People around the world?
What about members of future generations?
How many such generations?
What about non-human nature --- should we accord any importance to protecting coral reefs or redwoods or tigers, beyond the utility that human beings derive from their existence?
Furthermore, should we respect the existing distribution of resources, or should we be willing to substantially redistribute them in order to achieve utility gains?
As Paul Thompson identifies in the context of environmental policy-making:
\begin{quote}
Efficiency criteria are determined relative to an existing system of rights and privileges. They provide no basis for evaluating proposals to change that system.\footcite[][p. 109]{Thompson1995}
\end{quote}
These major normative questions are a necessary part of the cost-benefit analysis process, yet the fact that they are even being asked and answered is frequently obscured.
We can therefore identify at least three sites where values crop up when analysts are trying to reach empirical judgments about the world: in their assumptions about what outcomes matter, in their evaluation of the relative desirability of different outcomes, and --- most problematically --- in their interpretation of empirical data itself.
While the first and last intersection can conceivably be teased apart from empirical analysis through careful consideration of normative assumptions and preferences, the intermingling of analysis with normative preferences and assumptions produces an inescapable problem for policy analysis as an empirically objective enterprise.\footcite[See: ][p. 631]{LidmanSommers2005}



A similar point is made by Laurence Lynn:
\begin{quote}
Policy analysis must create problems that decision-makers are able to handle with the variables under their control and in the time available... by specifying a desired relationship between
manipulable means and obtainable objectives... Unlike social
science, policy analysis must be prescriptive; arguments about
correct policy, which deal with the future, cannot help but be willful and therefore political.\footcite[][p. 417]{Lynn1999}
\end{quote}
Without question, normative issues bear upon policy analysis, both in terms of how they affect the interpretation of those trying to make empirical judgments and in terms of how policy choices inevitably have normative consequences.
In the face of this, one may be tempted to substitute the looser term of `art' to describe the practice.
One danger in so doing is distancing oneself from the very aspiration of objectivity.
While it may be laudable to honestly acknowledge the complex intersubjective characteristics of policy analysis, going further to say that there are no empirical means through which competing claims can be evaluated risks pushing us into extreme relativism.
Rather than be empowering, such a shift may serve to obscure the ideological and normative agendas that exist adjacent to and often penetrate public policy analysis.
If policy analysis is an `art' and radically open to interpretation, we have few means to make authoritative assertions of any kind about policy processes or outcomes.
This risks being needlessly disempowering, including for groups that suffer significantly under the \emph{status quo} and who might otherwise be able to use convincing empirical claims as grounds for encouraging policy reform.



Additional normative considerations arise in terms of policy analysis procedure itself.
Is there a normative value in having processes of policy evaluation be democratic, and is that value instrumental (in terms of better odds that good policy options will be identified, or strong empirical analysis will be carried out) or inherent (based on the moral importance of citizens playing a role in the processes through which public policies are selected)?\footcite[See: ][p. 77--95]{deLeon1994} \footcite[][p. 419--21]{Lynn1999}
It is plausible that in a democracy citizen participation in the deliberations of government has inherent value.
If so, this is a further factor distinguishing policy analysis from sciences where no such value exists.
As identified by Peter deLeon, it also creates new dangers, including an increased risk of government paralysis.\footcite[See: ][p. 90]{deLeon1994}
Other scholars have interpreted the shift toward more horizontal and democratic modes of decision-making as substantially altering the role of policy analysts themselves, replacing the traditional role of ``speaking truth to power'' with  ``sharing truth with many actors of influence''.\footcite[][p. 85]{CraftHowlett2012}



\section{A practical example}



Consider the case of alternative public policy approaches to the problem of opiate addiction, and specifically to heroin.
In evaluating options, there is empirical data to draw upon.
We can examine the policies that have been implemented in various jurisdictions at various times, ranging from `tough on crime' policies that focus on harsh criminalization and punishment to `harm reduction' policies that treat addiction as a health care problem rather than one of public security.
While those with strong views on what sort of policy is appropriate are likely to interpret empirical findings that are supportive of their existing position more favourably than empirical findings that challenge it, it does seem possible that with a good methodology and access to a sufficiently large body of data we can meaningfully distinguish between the outcomes of different approaches using metrics like:
\begin{itemize}
 \item the rate of addiction in the population; 
 \item the severity of policy side effects such as crime driven by the need to secure funds to purchase illicit drugs, the role of organized crime in providing them, the consequences of large-scale imprisonment on communities, deaths and injuries from adulterated drugs or those of unpredictable potency, and rates of infectious diseases associated with drug use; 
 \item and the differential impact of various public policy options on at-risk groups within society, different ethnic groups, or groups that have historically suffered from discrimination.
\end{itemize}
Even with a robust set of empirical findings on all of these dimensions, however, we are left with many normative questions that bear upon the matter of what sort of public policy is most desirable.\footcite[Edward Page comments on the temptation to ``inflate the natural science boundary'' to expand the range of issues that can be addressed through purely technocratic means.][p. 263]{Page2010}
Do we accept or reject the idea that people in a free society should have the liberty to make choices about what substances they put into their own bodies, even when the substances are harmful?\footcite[In philosophically complex ways, these moral questions relate to other empirical questions about the nature of human reasoning and will and what it means for people to be rational, as well as what human freedom means, and how that relates to the legitimate operation of government. For some discussion related to this, see: ][p. 297--321]{Jones1999}
Is drug use itself a moral failing which the government can justly punish?
If so, how do we decide which substances are vices that should be prohibited (heroin, under Canada's present policies), which should be tolerated but discouraged (alcohol and tobacco), and which should be permitted ubiquitously (caffeine)?
What level of public coercion is justified by the objective of reducing the incidence of addiction and the harm associated with it?
What kind of claims about individual rights, or about societal utility, ought we to bear in mind when deciding how to proceed?
Finally, in terms of public participation, what sort of process for choosing between drug policies appropriately involves the citizenry?



Drug policy, like all areas of public policy, requires a fusion of the quasi-scientific consideration of empirical facts and outcomes with a process of normative argumentation, prioritization, and decision-making.
Calling public policy analysis a `science' without taking into consideration the normative dimension obscures many of the most important dimensions of what is being done.
Empirical evaluation of the relative effectiveness of alternative air defence systems doesn't tell us what our defence policy should be; increasingly sophisticated global climate models do not tell us how much temperature increase we should impose on future generations; nor does the best empirical data on the experiences and prospects of those with terminal illnesses tell us what our health policies should be.
These questions cannot be answered with `science' alone, no matter how robust or credible.
At the same time, adopting the extremely broad term `art' to describe public policy analysis risks distracting us from the twofold task of generating and testing empirical hypotheses and advancing, comparing, and adjudicating between moral arguments.



\section{Conclusions}



Laurence Lynn argues that: ``Policy analysis has never been monolithic, never beyond the sobering influences of political and social life, and never about making policy according to strict positivist canon'' and that it is ``clearly associated with the world of political action, both normative and prescriptive''.\footcite[][p. 417]{Lynn1999}
Indeed, the intertwined character of empirical and normative work within policy analysis is the major reason why it can never be classified as a `science' in the comparatively unproblematic way that physics, chemistry, or genetics can be.
Yet, without skirting the issue of the inherently normative dimensions of policy analysis and policy-making, we can identify the importance of empirical evaluation within each as grounds for considering them semi-scientific in terms of both aims and methods.
By considering policy analysis as a kind of quasi-science, we can recognize that empirical claims have enormous importance in the generation and evaluation of effective public policies.
While actually going from a strong empirical and normative understanding of a problem to the implementation of sound public policies in response certainly requires `soft skills' of a sort that can be characterized as `arts', that process might be more appropriately described as `politics' --- even if it is happening within a non-partisan bureaucracy --- than as part of the quasi-scientific policy analysis process itself.



\clearpage

\onehalfspacing
\printbibliography
	
\end{document}