Weick, 8 Wiener, 9 Winograd, 10 and Woods 11

Vicente, 7 Weick, 8 Wiener, 9 Winograd, 10 and Woods 11

in shaping our vision of sensemaking and the impli- cations for collaborations between smart people and smart machines to solve “wicked problems.”

Wicked Problems

Nousala, Dubberly, and Pangaro wondered at our use of the term wicked problems in our conceptual model of a sociotechnical system (Figure 1 in the target article). We chose the label to represent the kinds of problems that sociotechnical systems must grapple with as a way to emphasize that these are ‘open systems,’ and that the problems they must solve are ‘complex.’ The systems are open to the extent that there are external disturbances that cannot be fully specified or anticipated, and the problems are com- plex in the sense that there are typically large num- bers of incommensurate variables—in some cases, an unspecifiable number of variables—that could poten- tially be relevant.

Additionally, the problems are such that there is typically no unambiguously correct solution or right answer. Thus, these problems are only solvable in the sense of satisficing—in other words, the solutions can only be evaluated pragmatically—are the conse- quences of decisions/actions deemed satisfactory with respect to the mission of the organization. The key is that there is no analytical basis for identifying any particular solution as optimal or absolutely right.

Note that satisficing is generally discussed relative to human limitations, or bounded rationality. The implication is that the choice of a less than optimal solution is due to humans’ internal constraints. How- ever, we are using the term satisficing here to reflect

a property of the problems. In this case, satisficing is not a compromise relative to a potentially optimal solution, but rather it indicates that it is impossible to specify an optimal solution. A solution either leads to satisfying or unsatisfying consequences, but there is no absolute basis for identifying a particular solution or set of solutions as absolutely better than another set. Further, in some cases there will be no a priori normative basis to predict whether a solution will

be satisfying or even to judge post hoc whether the choices made were good ones. Healthcare provides

a good example—in cases of diseases such as cancer, it will often be difficult to unambiguously judge

whether a patient recovered or died because of or in spite of the treatments. Were negative side effects from treatments avoidable or not? At what point do the negative side effects outweigh any benefits of extending life? Was the ultimate outcome worth the suffering?

Derek Miller’s commentary provides a clarifica- tion and amplification of our use of the term wicked problems. The problems of sociotechnical systems often involve conflicts of values. These values are often incommensurate in the sense that there is no common metric for integrating them into a unified cost-function or single figure of merit to unambig- uously determine the quality of a solution. Miller suggests some examples, including how much liberty we should sacrifice in pursuit of order or security. The critical thing here is that the right balance across competing values cannot be absolutely prescribed in- dependently from the context—including economic, social, political, and cultural considerations—and these contexts are changing, or non-stationary. A bal- ance that is stable in one context or at one time may not be stable in another context.

We would further claim that wicked problems are far more common than is generally thought. Picking a graduate school, buying a house, choosing

a mate, diagnosing and treating a cancer patient, and selecting a career path would all fit our concept of what a wicked problem is, due to the trade-off of incommensurate values they require, their non-sta- tionary dynamics, and the size and dimensionality of their possibility sets. This should be even more obvious in the case of larger sociotechnical organiza- tions that are dealing with problems with more global political, social, and cultural implications—managing

a national healthcare system, for example.

There may be better terms for characterizing the complex challenges of sociotechnical systems. For ex- ample, Simon and Newell use the term “ill-structured problems”:

“Problems are ill-structured when they are not well-structured. In some cases, for example, the essential variables are not numerical at all, but symbolic or verbal…. Second, there are many important situations in everyday life where the objective function, the goal, is vague and non- quantitative…. Third, there are many practical problems—it would be accurate to say ‘most practical problems’—for which computational

algorithms simply are not available.” 12

The essential motivation behind the use of wicked problems as a term is to counter a tendency for The essential motivation behind the use of wicked problems as a term is to counter a tendency for

human fulfillment is a chimera,” 13 we would argue

that the hope that sociotechnical problems are analyt- ically tractable or solvable in any absolute sense is a chimera. The best we can hope for is a dynamic sta- bility where the inevitable mistakes that the system makes are at least survivable—satisfactory—and perhaps that the system learns from each mistake, so that fitness generally improves with experience. This

perception-action loop is closed through a complex or wicked problem

leads to a second concern of Dubberly and Pangaro.

domain. Image © 2016 by John M. Flach and Kyle J. Behymer.

particularly reflected in the clash of values described

Control versus Stability

so well by Miller in his commentary. In the dynamic Dubberly and Pangaro also raised a concern about the

of sociotechnical systems, the stable solutions are not fact that there is no explicit goal in our model ( Figure

specified a priori as goals and/or cost-functions. Rather,

1 in the target paper) noting that a goal is “a key com- the stable solutions typically emerge as the result of

contests between competing values. This is more sim- of any explicit goal is intentional and in fact, we

ponent of any control system model.” 14 The omission

ilar to a prey-predator system than a servomechanism. think it is misleading to call this a “control system

That is, in the same way that stable population levels model.” Although this is a circular system, where

can emerge from competition in natural ecologies, the consequences of actions feed back as informa-

stable solutions can emerge in sociotechnical systems tion that can shape future actions, it is definitely

as a result of competition among diverse values. Lind- not a simple servomechanism. There is no explicit

blom has described this dynamic of sociotechnical goal, and thus there is no clear basis for computing

systems as “muddling through” or “incrementalism.” 16 an “error signal.” As noted in the previous section,

Solutions typically emerge as a result of incremental rather than a well-specified goal, performance in

adjustments in organizations that are highly resistant sociotechnical systems is often influenced by a host

to change. This resistance reflects a friction that may of diverse values that may or may not be shared by 17 often be essential for stability in a complex ecology.

the distributed agents involved in observing the In a self-organizing system, the stable points are state of the world, and acting upon that world.

not imposed from without—an explicit a priori goal— Unfortunately, our choice to use the labels “con-

rather, the stable points emerge from the internal dy- troller” and “observer” in the original figure contrib-

namics of the coupling. In the prey-predator example, utes to confusion about the kind of system we are

the critical question from a control theoretic perspec- describing. Thus, we have revised the figure to use the

tive is to understand the conditions that lead to stable alternative labels of “perception” and “action” ( Figure

populations, rather than the collapse of the ecosystem R1 ). We have also changed the labels on the compo-

resulting in extinction. In sociotechnical systems, nents to eliminate the label “control.” The original

stability will be a function of the capabilities of the choice of terms was intended to emphasize paral-

distributed organization relative to the demands of lels to concepts used in control theory to describe

the wicked problems. This leads naturally to another systems. However, we see control theory as a field

of Dubberly and Pangaro’s concerns. concerned primarily with understanding stability in closed-loop systems. Thus, our view of control theory may be somewhat inconsistent with the way many

Requisite Variety

social scientists understand it to be—a servomecha- As Dubberly and Pangaro note, there are two primary nism. In our view, control theory considers trade-offs

sources of variety in our model of a sociotechnical across many potential solutions to achieving stability,

system. One source is the variety associated with the where the simple servomechanism is but one of the

wicked problem. This determines the constraints asso- simpler, more primitive examples. 15 ciated with the possible outcomes that are, in essence,

So what kind of system do we envision in Figure the problem space—like the beach in Simon’s classic R1 ? We see this as a self-organizing system. This is

analogy. 18 The other source is the variety associated

From Autonomous to Sociotechnical Systems

A B C analogy. The key implication of Ashby’s Law of Requi-

site Variety 19 is that complete control—the ability of the ant to go anywhere on the beach—requires that the organization has at least as much variety as the problem space. In other words, if the organization is

more constrained than the problem space, there will Organiza on

Wicked

Problem

be some states that will not be reachable. To frame this in terms of the functional components of the

variety associated with a wicked problem (solid circles) and the variety

organization, lower variety in the organization of associated with an organization of agents (dashed circles). Image © 2016

by John M. Flach and Kyle J. Behymer.

agents implies that either there are important dis- criminations with respect to the problem state that

illustrates a bounded rationality in which the variety the perception component is incapable of making, or

of the problem exceeds the variety of the organiza- the problem requires some actions or moves that the

tion. In this case, full or optimal control is not pos- action component cannot make. In terms of Simon’s

sible. However, it still may be possible for the system analogy, for example, constraints on the locomotion

to achieve a stable equilibrium or to satisfice. In other capabilities of the ant might make it impossible for it

words, the system may not be able to go everywhere, to reach some areas of the beach.

but it may be able to avoid catastrophe most of the In the classical use of the term satisficing, there

time—for example, through the use of smart heu- is at least an implication that the requisite variety of

ristics. The second relation (B) illustrates a situation the unaided human is often less than the problem—

where the organization has more variety than the there are internal constraints, biases or limitations—

wicked problem. In this case, full control is possible. that require the human to settle for a less than

On the positive side, the excess variety may allow in- optimal solution. For example, Flach has suggested

creased flexibility and redundancy, increasing system that typically no single agent in an organization will

stability. On the negative side, the excess variety may satisfy the requisite variety demands for many soci-

be only so much “noise,” possibly making it difficult

for the system to discover or settle on even a satis- that it may be possible for a collection of diverse

otechnical problems. 20 However, Flach also suggests

fying solution.

agents—including human and autonomous systems— The third relation in Figure R2 (C) illustrates to satisfy the demands of Ashby’s law. However, there

what is probably the most common situation— a is another dimension of variety that connects to the

partial overlap between the variety associated with previous ideas of wicked problems and satisficing that

the problem and the variety associated with the Dubberly and Pangaro raise. This is the potential for

organization. In this context, it is easier to frame the there to be variety within the organization above and

discussion in terms of constraints, or factors that beyond that required to meet the demands of requi-

limit variety. Thus, the constraints associated with the site variety.

organization are not fully aligned with the constraints The difficulty of wicked problems may lie in part

of the wicked problem. This implies that the ratio- with the fact that there is greater variety in the orga-

nality of the organization will be bounded relative to nization than in the problem. One obvious source of

the problem and that there may be internal noise that this variety, suggested in Miller’s commentary, would

increases the difficulty of identifying satisfying strat-

be differences in the values among the various agents egies for achieving stability. The critical point is that in an organization. For example, even if a solution is

in shaping an organization to deal with wicked prob- satisfactory because the organization remains viable,

lems, one must often consider both 1) how to scale-up there may often not be consensus across the agents

the limited variety of individual agents to satisfy the about whether any particular solution is satisfying. In

requisite demands of the wicked problem, and 2) how solving sociotechnical problems, no matter the quality

to filter out the noise, or extract the signal from the of a solution, there will almost always be people

noise that reflects a diversity of opinions within the within the organization who are unsatisfied and who

organization. This leads naturally to the last of Dub- will second-guess decisions and actions taken.

berly and Pangaro’s concerns.

Figure R2 illustrates some of the potential re- lations between variety associated with a wicked problem (solid circle) and variety associated with the

The Quality of Communications

organization (dashed circle). The first relation (A) Dubberly and Pangaro note that our model “does

she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016 she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016

and ‘effective.’” 21 Then they pose the million-dollar tations are typically constructed using spatial analogs question: “How do we achieve it?” In the target paper,

or metaphors. The incorporation of the mouse and we finessed this by providing examples of systems

the desktop metaphor that transformed the com- where we feel that the communications were effective

puter from a scientific instrument to an all-purpose (Chef Watson) and ineffective (Clippy). We did this

personal information system is a classical example of partly because we don’t think there is a single, simple

how design can improve the quality of communica- prescription for quality communications. The design

tions between humans and computers. Shneiderman’s challenge will always be to align the constraints

concept for direct manipulation interfaces is another (variety) internal to the organization with the con-

example of how representations can facilitate coor- straints (variety) of the wicked problems to be solved.

dination between humans and computers to manage As suggested in the previous section, this will involve 24 complex problems.

aligning the internal constraints with the problem Another potentially powerful way to tune the in- constraints, which may require 1) diversifying the or-

ternal constraints of an organization to better match ganization to scale up the internal variety to meet the

the deep structure of the problem domain is through demands of Ashby’s Law, and 2) tuning the internal

narratives or stories that reflect the insights of ex- constraints to pull out the signal from the noise.

perts in dealing with critical events. Klein, Snowden, One reason that it is difficult to specify a priori

and others have found that narratives provide a useful rules or standards for quality communications is 25 vehicle for both mining expertise and for training.

that communication and language both appear to We hypothesize that the reason that analogs, met-

be dynamic, self-organizing processes. For example, aphors, and narratives facilitate communications Rączaszek-Leonardi writes:

is that they translate constraints of the problem “Language is thus claimed to be embodied, sit-

into terms that are more familiar with the agents, uated and distributed, and thus linguistic struc-

making the signals—the problem constraints— tures, instead of being independent ‘vessels of

more salient relative to the background noise. meaning’, are seen as immersed in a variety of

Another important consideration in the design dynamical events that give rise to them and are,

of organizations to facilitate communications is in turn, controlled by them … communication,

the formalization of lines of communication and instead of being meaning transfer between in-

authority. For example, a major source of discus- dividuals is rather seen as creating something-

sion in both economic and military systems is in-common, or communion, that is, formation of

the relative benefits and costs of centralization, temporal, adaptive wholes, in which individuals

or hierarchy, relative to more distributed organi-

22 coordinate in functional synergies.” 26 zations, or networks. The key again seems to

be discovering the appropriate balance between With respect to the role of effective communications

internal constraints and the demands of particular and collaboration, there is a kind of chicken and

problem domains. For example, we can image egg problem that is always associated with circular

situations where conversations should be inclu- dynamics. Each is both simultaneously being shaped

sive and collaborative, taking full advantage of by and shaping the other. Effective communication

the diversity among a network of collaborating is simultaneously a pre-requisite for, and a product

agents, as did the puzzle-solving exercises used in of, effective coordination. Thus, as designers, it may 27 the MIT study on group intelligence. However,

not be possible to predetermine the quality of com- we can also image situations where the addition munications. However, there may be ways that de-

of more hierarchical constraints will reduce the signers can influence or constrain communications in

internal variety—potential noise—in ways that will potentially productive ways. One way that designers

facilitate achieving stability. For example, Sage and can do this is to align the internal constraints with

Cuppan describe a federalist style of organization the problem constraints through the construction of

that can be very effective in helping various agen- representations. This is the central focus of Ecolog-

cies—like police departments, fire departments, ical Interface Design—construct graphical interfaces

and hospitals—to coordinate activities in response that represent the deep structure of problems in a 28 to regional disasters/emergencies. Thus, our

way that is compatible with users’ capabilities—for guess is that various organizational forms—hier- example, by designing interfaces that represent the

archies, heterarchies, networks, federalism—will

From Autonomous to Sociotechnical Systems

8 Karl E. Weick, Sensemaking in Organizations (Thousand Oaks, CA:

to meeting the requisite variety demands for Sage, 1995). achieving stability in different contexts. There is no 9 Norbert Wiener, Cybernetics: Or Control and Communication in the

Animal and the Machine (Cambridge, MA: MIT Press, 1948).

single one-size-fits-all solution.

10 Terry Winograd and Fernando Flores, Understanding Computers and Cognition: A New Foundation for Design (Boston, MA: Addi - son-Wesley, 1986).

Conclusion

11 David D. Woods and Erik Hollnagel, Joint Cognitive Systems: Patterns

Again, we thank the editors of She Ji and the authors

in Cognitive Systems Engineering (Boca Raton, FL: 2006).

of the commentaries for enriching the discussion of

12 Herbert A. Simon and Allen Newell, “Heuristic Problem Solving: The Next Advance in Operations Research,” Operations Research

design thinking relative to improving the quality of

6, no. 1 (1958): 5.

collaborations. As a result of the feedback from the

13 Derek B. Miller, “Value-Pluralism and the Collaboration Impera-

commentaries, we would like to reframe the discus-

tive in Sociotechnical Systems,” commentary to Kyle Behymer

sion from “Designing for Effective Collaboration” to

and John Flach, “From Autonomous Systems to Sociotechnical Systems: Designing Effective Collaborations,” She Ji: The Journal of

“Enabling Effective Collaboration.” This is to empha-

Design, Economics, and Innovation 2, no. 2 (2016): 115.

size the self-organization dynamic of sociotechnical

14 Hugh Dubberly and Paul Pangaro, “Distinguishing Between

systems. Thus, we are skeptical that it is possible for

Control and Collaboration—and Communication and Conver-

designers to fully determine the quality of commu-

sation,” commentary to Kyle Behymer and John Flach, “From

nications or collaborations through Autonomous Systems to Sociotechnical Systems: Designing a priori design

Effective Collaborations,” She Ji: The Journal of Design, Economics

decisions. Rather we think that the quality of both

and Innovation 2, no. 2 (2016): 117.

will be an emergent property of the system dynamic.

15 For example, see John M. Flach and Fred A. Voorhorst, What

However, we think that focusing design thinking on

Matters?: Putting Common Sense to Work (Dayton: Wright State

the sociotechnical dynamic—communication and

University Library, 2016).

collaboration—rather than exclusively on the techno-

16 Lindblom, “Science of ‘Muddling Through’,” 88; Lindblom, “Still Muddling,” 517.

logical artifacts—autonomous systems—is critical to

17 Nordal Åkerman, ed., The Necessity of Friction: Nineteen Essays on a

enabling quality collaborations to emerge. As Dub-

Vital Force, reprint (Boulder: Westview, 1998).

berly and Pangaro note, this is not a new or novel in-

18 Simon,

sight. Clearly, this was a critical insight that led to the

19 Ashby W. Ross, An Introduction to Cybernetics (London: Chapman &

development of personal computers and later devel-

Hall, 1956).

opments such as smart phones. Yet, despite this his-

20 John M. Flach, “Supporting Self-Designing Organizations,” com-

tory, this is a lesson that we have to keep reminding

mentary to Donald A. Norman and Pieter Jan Stappers, “DesignX:

ourselves of in order not to become seduced by the

The Design of Complex Sociotechnical Systems,” She Ji: The Journal of Design, Economics and Innovation 2, no. 1 (2016): 95–99.

growing powers of information technologies and the

21 Dubberly and Pangaro, “Distinguishing Between Control and

illusion that with enough computational power it will

Collaboration,” 117.

be possible to completely tame the wickedness that is

22 Joanna Raczaszek-Leonardi, “Multiple Systems and Multiple Time

inherent in a complex world.

Scales of Language Dynamics: Coping with Complexity,” Cybernet- ics & Human Knowing 21, no. 1-2 (2014): 37–38.

1 Sidney Dekker, Drift into Failure: From Hunting Broken Components 23 Kevin B. Bennett and John M. Flach, Display and Interface Design: to Understanding Complex System (Farnham, UK: Ashgate, 2011).

Subtle Science, Exact Art (Boca Raton: CRC Press, 2011). 2 Charles E. Lindblom, “The Science of ‘Muddling Through’,” Public

24 Ben Shneiderman, Designing the User Interface (Reading, MA: Administration Review 19, no. 2 (1959): 79–88; Charles E. Lindblom,

Addison Wesley, 1992).

“Still Muddling, Not Yet Through,” Public Administration Review 39, 25 Gary A. Klein, “Using Knowledge Engineering to Preserve Corpo- no. 6 (1979): 517–26.

rate Memory,” in The Psychology of Expertise: Cognitive Research 3 Donald A. Norman and Stephen W. Draper, eds., User Centered

and Empirical AI, ed. Robert R. Hoffman (New York: Springer-Verlag, System Design: New Perspectives on Human-Computer Interaction

1992), 170–87; David J. Snowden, “Narrative Patterns, The Perils (Hillsdale: Lawrence Erlbaum Associates, 1986).

and Possibilities of Using Story in Organizations,” Knowledge 4 James G. March, Decisions and Organizations (Oxford: Basil Black-

Management 4, no. 10 (2001): 1–14.

well, 1989). 26 John Arquilla and David Ronfeldt, In Athena’s Camp: Preparing for 5 Jens Rasmussen, Information Processing and Human-Machine

Interaction: An Approach to Cognitive Engineering (New York: Elsevier 27 Anita Williams Woolley et al., “Evidence for a Collective Intelli- Science Ltd., 1986).

gence Factor in the Performance of Human Groups,” Science 330, 6 Herbert A. Simon,

no. 6004 (2010): 686–88.

MA: MIT Press, 1996). 28 Andrew P. Sage and Christopher D. Cuppan, “On the Systems Engi- 7 Kim J. Vicente, Cognitive Work Analysis: Toward Safe, Productive,

neering and Management of Systems of Systems and Federations and Healthy Computer-Based Work (Mahwah: Lawrence Erlbaum

of Systems,” Information, Knowledge Systems Management 2, no. 4 Associates, 1999).

she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016

Jordan Beck , School of Informatics and Computing, Indiana University, USA Erik Stolterman , School of Informatics and Computing, Indiana University, USA

Examining Practical, Everyday Theory Use in Design Research

Abstract This paper discusses how theories (as objects) are used in articles

Keywords

published in Design Studies. While theory and theory construction have

Design research

been given time and attention in the literature, less is known about how

Design theory Research methodology

researchers put theories to work in their written texts—about “practical, everyday” theory use. In the present paper, we examine 32 articles and syn- thesize six models of “theory use” based on our examination.

Received October 4, 2015 Accepted January 14, 2016

Emails

Jordan Beck (corresponding author) joebeck@indiana.edu

Erik Stolterman estolter@indiana.edu

Copyright © 2016, Tongji University and Tongji University Press. Publishing services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ). The peer review process is the responsibility of Tongji University and Tongji University Press.

http://www.journals.elsevier.com/she-ji-the-journal-of-design-economics-and-innovation http://dx.doi.org/10.1016/j.sheji.2016.01.010

Examining Practical, Everyday Theory Use in Design Research

1 Ken Friedman, “Theory

Theory is integral to any academic discipline. It takes researchers beyond obser-

Construction in Design Re- search: Criteria: Approaches,

vation and interpretation into the realm of sharable knowledge. Theory provides

and Methods,” Design Studies

us with the means to structure knowledge, to evaluate and assess it, to construct

24, no. 6 (2003): 507–22, DOI:

it, and to share it. In the everyday practice of research, theory can be seen as a

http://doi.org/10.1016/S0142- 694X(03)00039-5 ; Terence

concrete tool used for practical purposes. For instance, theory can be used as an

Love, “Philosophy of Design: A

explanatory tool, or as a predictive one. But these are not the only ways in which

Meta-Theoretical Structure for

theory functions as a tool. The use of theory is multifaceted and complex. 1 How-

Design Theory,” Design Studies 21, no. 3 (2000): 293–313, DOI:

ever, in this article our aim is not to demarcate or define theory. Instead, our aim is

http://doi.org/10.1016/S0142-

to investigate how researchers present their use of theory in written texts.

694X(99)00012-5 ; Karl E. Weick,

The reason for our investigation originates in a parallel study 2 which engages

“Theory Construction as Disci- plined Imagination,” Academy

with the question of what constitutes a design theory. We have examined publica-

of Management Review 14, no. 4

tions from many different domains of design research, and we have found that the

(1989): 516–31, DOI: http://doi.

way theory is mentioned, written about, and used can be difficult to understand. This

org/10.5465/AMR.1989.4308376 ; Karl E. Weick, “What Theory Is

difficulty pointed us towards the important two-part question of (1) how researchers

Not, Theorizing Is,” Administra-

in the design field use theory, and (2) how that use manifests in their writings.

tive Science Quarterly 40, no.

Our interest is primarily in the everyday practice of theory use. By “everyday

3 (1995): 385, DOI: http://doi. org/10.2307/2393789 .

practice,” we mean to distinguish between what we might call “revolutionary contributions” to the field and contributions that might be considered normal or

2 Jordan Beck and Erik Stolter-

routine. In order to accomplish this, we have examined a selection of articles from

Theories of Design That Do Not

a single calendar year in one of the most prestigious journals in the design research

Scientize Design?” in Proceed-

community—Design Studies.

ings, from the 11th European Academy of Design Conference:

We propose six models that capture the different ways researchers use theory

The Value of Design Research

in their publications. We suggest that these models support a deeper understanding

(Boulogne: Paris Descartes Uni-

of the structure of publications in Design Studies. In addition, and perhaps more im-

versity, 2015), accessed August

portantly, these models facilitate interesting and useful questions about the state of

3, 2016, http://www.academia. edu/11241766/Can_there_be_sci-

theory use in design research in general, such as: why is theory used more in some

HQWLÀFBWKHRULHVBRIBGHVLJQBWKDWB

ways than others? How do authors employ theory in different ways in the same

do_not_scientize_design .

texts? To what extent does current theory use in design research tell us something

3 Friedman, “Theory Construc-

about design theory, as opposed to other kinds of theory? What is the current state of

tion,” 513.

theory use in design research?

4 John Ziman, Real Science:

We intend this paper to make two primary contributions to the field: First it

What It Is, and What It Means

describes the current state of theory use in design research; and second, it explores

(Cambridge, UK: Cambridge University Press, 2002), 128.

the implications of this existing state when it comes to design research in general.

A deeper understanding of how theory is used in the design research community

5 Simon Blackburn, The Oxford Dictionary of Philosophy, 2nd

can potentially better position its constituents to be more intentional in their

ed. (Oxford: Oxford University

theory use.

Press, 2005), 363. 6 John S. Gero, “Design

Prototypes: A Knowledge

Theory as an Object

Representation Schema for Design,” AI Magazine 11, no. 4

Theory can be understood in many different ways. For instance, it has been de-

(1990): 26; John S. Gero and Udo

scribed as a model, 3 likened to a map, 4 and defined as a way of looking at some

Kannengiesser, “The Situated

phenomena “with explanatory or predictive implications.” 5 A simple distinction

Function-Behaviour-Structure Framework,” Design Studies

that we have found useful for our purposes is that theory can be understood either

25, no. 4 (2004): 373–91,

as an object (i.e., as a kind of knowledge entity) or as a process (i.e., as theorizing).

DOI: http://doi.org/10.1016/j.

When a researcher develops an explanation of how or why some phenom-

destud.2003.10.010 ; John S. Gero and Udo Kannengiesser, “The

enon occurs, they are engaging in theorizing—in a process. The explanation itself

Function-Behaviour-Structure

becomes a theoretical object. For instance, the FBS framework, 6 CK theory, 7 dis-

Ontology of Design,” in An An-

tributed cognition, 8 and strong concepts, 9 could all be interpreted as theoretical

thology of Theories and Models of Design, ed. Amaresh Chakrabarti

objects. A parallel distinction can be made between designing and a design: when a

and Lucienne T. M. Blessing

designer designs, they are engaged in design as a process, while the outcome of that

(London: Springer-Verlag, 2004),

process becomes the design(ed) object.

263–83, http://link.springer.com/ chapter/10.1007/978-1-4471-

Both aspects are of course of great importance. So when we consider our re-

6338-1_13 .

search question, “How is theory used in written texts in design research,” we have

she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016 she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016

7 Armand Hatchuel and Benoît

studying one would be quite different than the other. For our purposes, we made Weil, “A New Approach of

Innovative Design: An Introduc-

the decision to frame theory as an object.

tion to CK Theory,” in DS 31:

Framing theories as objects can serve many different ends. For instance, it

Proceedings of ICED 03, The 14th

would be possible to study theories (as objects) in design to explore their constitu- International Conference on En-

gineering Design, Stockholm, ed.

tion and structure, or to study their internal consistency, or their scope or depth.

A Folkeson, K Gralen, M Norell,

It would also be possible to compare and contrast theories with the purpose of

and U Sellgren (2003), 109–10;

gaining insight into the field where they are used. These ways of investigating

Armand Hatchuel and Benoît Weil, “C-K Design Theory:

theories can lead to detailed analyses of their intrinsic status. However, such ap-

An Advanced Formulation,”

proaches would require the theories under investigation to be opened up, and their

Research in Engineering Design

inner workings exposed. We have taken a simpler approach: We are only interested

19, no. 4 (2009): 181–92, DOI: http://doi.org/10.1007/s00163-

in how theories are used as knowledge objects, and we are not interested (in this

008-0043-4 .

context) in the internal constitution of the object itself.

8 James Hollan, Edwin Hutchins,

Distinguishing a theoretical knowledge object in a publication is an important

and David Kirsh, “Distributed

issue to address. For our purposes, we err on the side of inclusivity, since previous

Cognition: Toward a New

work has shown that theories can be referred to in research publications as every-

Foundation for Human-Comput- er Interaction Research,” ACM

thing—from “theory” or “model” to “perspective,” “rationale,” “idea,” “process,”

Transactions on Computer-Hu-

and “phenomenon.” 10 We are interested in identifying what could reasonably be

man Interaction (TOCHI) 7, no.

construed as theoretical knowledge objects based on our own understanding of

2 (2000): 174–96, DOI: http://doi. org/10.1145/353485.353487 .

theory and our understanding of how other researchers in the design field under-

stand and talk about theory. 11 As a consequence, the way we frame theory could

9 Kristina Höök and Jonas Löwgren, “Strong Concepts:

be seen in alignment with a broad, inclusive definition of theory “as [an] abstracted

Intermediate-Level Knowledge

knowledge [object] that tells us something about fundamental entities at the core

in Interaction Design Research,”

of a discipline.” 12

ACM Transactions on Comput- er-Human Interaction (TOCHI)

The way we frame theories suggests that the use of a theory can at least be un-

19, no. 3 (2012): 23; Jonas

derstood in three ways: as a tool, as a reference, and as a knowledge contribution.

Löwgren, “Annotated Portfolios

In the following section, we will briefly discuss each of these aspects of theory as an

and Other Forms of Intermedi- ate-Level Knowledge,” interac-

object. There are of course other ways to understand theories as objects, but these

tions 20, no. 1 (2013): 30–34.

are the ones we have chosen to focus on in our analysis.

10 Jo E. Hannay, Dag I. K. Sjøberg, and Tore Dybå, “A

Theory as a Tool

Systematic Review of Theory

Framing theory as a tool implies that a user uses a theory for a particular purpose.

Use in Software Engineering

13 Experiments,” Theory has been described and defined by many researchers in terms of its utility. IEEE Transactions

on Software Engineering 33, no.

It has been framed as a tool for explaining, describing, or predicting phenomena.

2 (2007): 87–107, DOI: http://doi.

It has been described in design research as a tool for “binding together” our knowl-

org/10.1109/TSE.2007.12 .

edge of design practice and as a tool for “providing an understanding” of design

11 Steven R. Haynes and John

writ large. 14 Tools have functions that are in most cases related to a particular need

M. Carroll, “The Range and

or purpose. For instance, when a designer needs to communicate an idea quickly Role of Theory in Information

Systems Design Research: From

and cheaply to a stakeholder, she uses a sketch. Similarly, when researchers need

Concepts to Construction,” in

to explain how or why some phenomenon occurs, they use theory—since theory

2010 International Conference

provides a means of structuring knowledge in precise ways to share with other on Information Systems (ICIS

2010) Proceedings, Paper 136,

researchers. 15 There are of course situations when particular tools are not well

accessed August 3, 2016, http://

suited for the task at hand. Sketching is not necessarily an ideal tool for “selling”

aisel.aisnet.org/cgi/viewcontent.

a design to a client. Similarly, theory is not necessarily an ideal tool for explaining cgi?article=1139&context=i-

cis2010_submissions .

reality in the same way that an analogy or metaphor might be. Finally, the tool metaphor lends itself to the idea of appropriation. Tools that were designed for one 12 Erik Stolterman and Mikeal

Wiberg, “Concept-Driven

purpose often end up serving others. Pencil and paper, as tools for sketching, have

Interaction Design Research,”

structural properties conducive to low-fidelity 3D prototyping. Theory, a tool for

Human–Computer Interaction 25,

explaining, predicting, or describing, has structural properties conducive to other no. 2 (2010): 95–118. purposes as well.

13 Yvonne Rogers, “HCI Theory: Classical, Modern,

and Contemporary,” Theory as a Reference Synthesis

Lectures on Human-Centered

Common definitions of reference include: the action of mentioning or alluding

Informatics 5, no. 2 (2012): 1–129,

Examining Practical, Everyday Theory Use in Design Research

DOI: http://doi.org/10.2200/

to something or the use of information to ascertain something. When theory is

S00418ED1V01Y201205HCI014 ; Ben Shneiderman, Catherine

used as a reference, it most often appears in the introductory and/or background

Plaisant, Maxine S. Cohen, and

sections of a given written text. With this type of theory use, researchers often

Steven M. Jacobs, Designing

reference frameworks and models instead of referencing theory per se. Other inves-

the User Interface: Strategies for Effective Human-Computer

tigations of theory use in written texts have found that authors refer to theories as

Interaction, 5th ed. (Boston:

everything, from “theory,” “model,” “perspective,” “rationale,” “idea,” “process,”

Prentice Hall, 2009).

to “phenomenon.” 16 In many cases, the reference remains vague. For instance, the

14 Amaresh Chakrabarti

reference might amount to little more than authors’ names, or a paper title and

and Lucienne T.M. Blessing,

the publication year. The names and the publication year “stand in” for an explicit

“Theories and Models of Design: A Summary of Findings,” in

citation of a particular theory or theories. Theory as a reference can perform sev-

An Anthology of Theories and

eral functions that may appear in concert or individually in a given research paper.

Models of Design, ed. Amaresh

For instance, a theory can be used to establish the basis for a research project, to

Chakrabarti and Lucienne T.M. Blessing (London: Springer-Ver-

situate a text within a lineage, to establish the knowledge base of the writer, or to

lag, 2014), 15, accessed August

show connection to a community or school of thought. We will discuss some of

3, 2016, http://link.springer.

these in greater detail when we present our models of theory use.

com/chapter/10.1007/978- 1-4471-6338-1_1 ; Friedman, “Theory Construction,” 511;

Theory as a Knowledge Contribution

“Understanding Designing:

When theory is used as a tool or as a reference, it usually means that the theory in

What Are Designers Doing When They Design (John Gero),”

question already exists. Characterizing theory as a knowledge contribution suggests

YouTube video, 1:01:30, from the

that it had not existed prior to the researcher’s articulation of it. The researchers

ASP Course “Design Methods”

brought it into being by formulating their results such that they would be under-

coordinated by Gaetano Cascini,

stood as a theory, for instance, as a set of constructs and their definitions, as well as a set of propositions about how the constructs relate to one another. by “Alta Scuola Politecnica,” 17 This

October 8, 2013, https://www.

formulation of a theory becomes a knowledge object. How this object is to be

youtube.com/watch?v=TRiET-

“used” by others is partly a consequence of the authors’ claims about it, and partly

j2ujvQ .

a consequence of the readers’ interpretation. For example, the author might claim

15 Hannay, Sjøberg, and Dybå,

that the knowledge contribution relates to existing theory, or the research problem

“A Systematic Review of Theory Use”; Ziman, Real Science.

at hand, or to something else entirely. But it would also be possible for a reader to see applications beyond the authors’ claims. For example, the mathematicians who

16 Hannay, Sjøberg, and Dybå, “A Systematic Review of Theory

developed tensor calculus did so “without any premonition of its later use in phys-

Use,” 87–107.

ics,” 18 in particular in the General Theory of Relativity. In our analysis, we have

17 Karl E. Weick, “Theory

primarily focused on what the resulting theory is claimed to contribute and not on

Construction as Disciplined

reader interpretation.

Imagination.”

With the notions of theory as a tool, a reference, and a knowledge contribution in

18 Ziman, Real Science, 139.

mind, we turn our attention to describing our approach, including the conceptual-

19 George Psathas, Conver-

ization, design, and implementation of the current research project.

sation Analysis: The Study of Talk-in-Interaction, Qualitative Research Methods Series, vol.

Approach

35 (Thousand Oaks, CA: Sage

Publications, 1995); Harvey

Prior to assembling a corpus of texts, we established that we would engage in a

round of “unmotivated looking” in 19 Structures of Social Action: in order to see how the texts could be said to use

Sacks, “Notes on Methodology,”

Studies in Conversation Analysis,

theory without establishing or imposing strict definitions of “theory” or of “use.”

ed. J. Maxwell Atkinson and

One of the challenges we faced at the outset of our work was that of defining

John Heritage (New York: Press

“theory use.” How might we capture what it means to “use” theory? We found that

Syndicate of the University of Cambridge, 1984), 21–27.

framing theory as an object pointed us in the direction of an answer. When theory

[Editor’s note] This reference

is seen as an object—or a thing—then it becomes possible to see a theory as a tool.

note is added in the light of

When theory is seen as a tool then we can define “use” in terms of how it functions

Danah Henriksen’s comments.

in a text. What purpose does theory fulfill in a given written text?

20 Barney G. Glaser and Anselm

These initial questions and decisions were not guided by existing “theoretical

L. Strauss, The Discovery of frameworks,” but rather by a “general perspective… [and a] general problem area.” Grounded Theory: Strategies for 20 Qualitative Research (Piscataway,

These initial decisions could therefore be seen to be in alignment with the initial

NJ: AldineTransaction, 2009), 45.

decisions of a grounded theory approach.

There are many different publication venues one might look at in order to

she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016 she ji The Journal of Design, Economics, and Innovation Volume 2, Number 2, Summer 2016

21 Friedman, “Theory Con-

year’s worth of articles (n=32) published in Design Studies, which is one of the top struction,” 507–22; Terence Love,

“Philosophy of Design,” 293–313;

journals in the field. It is one of the oldest design research journals, and it aims to

Terence Love, “Constructing

provide an interdisciplinary forum for inquiry into design activities. It has been

a Coherent Cross-Disciplinary

a key source of material for scholars seeking answers to questions about how to Body of Theory about Designing

21 define, construct, and evaluate theory with a particular focus on design theory. and Designs: Some Philosoph- We ical Issues,” Design Studies

do not position our analysis as a response to or critique of this existing content.

23, no. 3 (2002): 345–61, DOI:

Rather, we see our work as complementary: a contribution to our collective knowl-

http://doi.org/10.1016/S0142- 694X(01)00043-6 .

edge of theory in design research. In order to determine which function theory performs in the works included

22 Wayne C. Booth, Gregory G. Colomb, and Joseph M. Williams,

in our corpus, we adopted an emergent coding approach. We separately reviewed

The Craft of Research, 2nd ed.

the corpus, and recorded preliminary observations about how theory functioned in

(Chicago: University of Chicago

each text. Then we reconvened to analyze and compare our observations.

Press, 2003); Kate L. Turabian, A Manual for Writers of Research

This stage yielded four groupings, which we named originating, positioning,

Papers, Theses, and Dissertations:

shaping, and shaped. Originating papers were those whose research questions orig-

Chicago Style for Students and

inated from theory, or whose question was about a specific theory or theories. Po-

Researchers, 8th ed. (Chicago: University of Chicago Press,

sitioning papers were those that positioned their work in relation to existing work

that may or may not have included theory per se. Shaping papers were those that used theory in a manner similar to positioning papers, with the difference being that theory actually shaped—or re-shaped—the original research question. Finally, shaped papers were papers that used theory as a filter through with to pass prelimi- nary research findings. We saw findings as shaped by theory.

We adopted these four groupings as a coding scheme, and independently applied them to the texts in the corpus. However, we found that this preliminary coding scheme did not yield an acceptable level of reliability. So we continued discussing the similarities and differences in our application, and we iterated upon the coding scheme accordingly.

In parallel with this iteration, we developed models to represent the relation- ships and interactions between theory and the other core elements in our selected papers. In attempting to establish what could constitute the core elements of our selected papers, we made the decision to model as few elements as possible: fewer elements reduce the complexity of the models but also lead to coarser descriptions. And while we relied on grounded theory techniques to generate kinds of theory use, we developed our list of core elements of a research publication based on two

widely used reference texts on research and academic writing. 22 We ended up with

the following core elements: question, examination, findings, and theory. Questions identify the needs or interests that the researcher deems worthy of understanding, explaining, predicting, or describing. For example, our framing question identifies an interest in understanding how theory is used in design research. Examination captures the approach taken to answer the question—that is, it includes all forms of analytical or empirical work done by the researcher to investigate the question at hand. Findings refer to the outcome of this examination. Theory is the fourth core element in each of our models.

It is obvious that by using so few elements to capture complex objects such as research papers, there is a concomitant loss of precision. However, it is not our intent to precisely capture on a detailed level the intrinsic structure of each article. Instead, our purpose is to find a level of analysis that makes it possible to extract patterns, similarities and differences across articles in a manageable way. We be- lieve that our level of analysis has made this possible.

After we developed the original set of four models, we critiqued and iterated on them by independently applying them to the papers in our corpus. Our cri- tique focused primarily on the match between the models and the papers. Did the models accurately capture the essential structure of the core elements in each

Examining Practical, Everyday Theory Use in Design Research

a methodological and/or analytical tool. We developed additional models to account for these types of theory use. In the interest of completeness, we also developed a model to account for papers that do not show any use of theory. However, this type of paper did not appear in our corpus, which is itself interesting.

Our critique and iteration finally led to six models:

0) No theory,

1) Theory as the object of study,

2) Theory as a contextualizing tool,

3) Theory as a shaping tool,

4) Theory as a methodological tool, and

5) Theory as an analytical tool. In the following section, we will explain each of these models and point to relevant

examples from our corpus. For each model, we include textual excerpts from our corpus in order to illustrate the type of content that pointed towards a particular model. We want to enable readers to evaluate the texts on similar terms as we did.