Never mind the quality, feel the width…

“integrity” flickr photo by drumminhands https://flickr.com/photos/drumminhands/7114464945 shared under a Creative Commons (BY) license

In preparing for a forthcoming supervisory meeting, I’ve been asked to share what I felt were the standout insights from my empirical observations, but for each one explain how I know, how I convinced myself of that, and how I can convince others. I guess what I’m being asked here is to justify my claims to knowledge; how do I assert that my interpretations are plausible? Lincoln and Guba (1985:290) phrase it as follows:

“How can an inquirer persuade his or her audiences (including self) that the findings of an inquiry are worth  paying attention to, worth taking account of?”

For them it is about trustworthiness and the arguments which can be mounted to make the case, however, assessing the quality of research findings is far from straightforward and is contested in a number of ways. Traditionally, research quality has been judged on the criteria of validity, reliability, generalisability, and objectivity. Validity, simply put, is the extent to which an account adequately represents the phenomenon it purports to. Reliability is related to the replicability of the data generation and analysis; if different people conducted the same study, or the same person on different occasions, would the outcomes be the same? Generalisability refers to the extent what has been learned can be extended to wider populations and objectivity, to how the biases and interests of researcher and researched have been reduced or accounted for.

These principles were inherited from the methods of natural science and are generally to be found in positivist-leaning, often quantitative studies. Although they continue to be applied within qualitative research (Morse et al, 2002), a debate amongst qualitative researchers developed asking how and even whether rigour can be established in this way. Participants in this debate fall into three camps (Denzin, 2009): foundational, quasi-foundational and non-foundational. Foundationalists contend ‘that research is research, quantitative of qualitative. All research should conform to a set of shared criteria’ (e.g. validity, credibility etc). Quasi-foundationalists seek a set of criteria unique to qualitative research, whereas non-foundationalists propose that criteria are not appropriate since they view ‘inquiry within a moral frame, implementing an ethic rooted in the concepts of care, love, and kindness.’

Given the uneasy truce emerging from the paradigm wars of the late 20th century (Bryman, 2008), a common set of criteria with which to judge all research endeavours might be seen as one route through which to reinforce parity between quantitative and qualitative research. Some, like Hammersley (2007) seek to restore the perceived devaluing of qualitative research by the drive towards evidence-based practice. This he argues might be achieved by a move towards at least some common criteria by which rigour might be established.

A helpful summary of some of the quality principles which have been proposed across the research can be found in this table from Johnson and Rasulova (2017):

 

Authors Principles of quality and rigour
Agar (1986) Credibility, accuracy of representation, and authority of the writer
Guba (1981) Truth value, applicability, consistency, and neutrality
Kirk and Miller (1986) Consistency of results, stability over time, similarity within a given time period
Brink (1991) Stability, consistency, equivalence
Lincoln and Guba (1985) Credibility, confirmability, dependability, transferability, authenticity

 

They point out the degree of overlap of criteria, but note that it is Lincoln and Guba’s (1985) principles which seems to have garnered most traction. Helpfully, Johnson and Rasulova also summarise these principles:

 

Qualitative research Questions that underpin the principles of qualitative research (Pretty, 1994; p. 42) Quantitative research concepts Techniques to establish the principles include:
Credibility How can we be confident about the ‘truth’ of the findings? Internal validity Prolonged engagement, persistent observation, triangulation, peer debriefing, negative case analysis, member check.
Confirmability How can we be certain that the findings have been determined by the subjects and contexts of the inquiry, rather than the biases, motivations and perspectives of the investigator? Objectivity
Dependability Would the findings be repeated if the inquiry were replicated with the same (or similar) subjects in the same or similar context? Reliability An audit trail subject to external audit
Transferability Can we apply these findings to other context or with other groups of people? Generalisation Thick description
Authenticity Have people been changed by the process? To what extent did the investigation prompt action? Fairness, together with ontological, educative, catalytic and tactical authentication.

 

(The additional criterion of authenticity was added by Lincoln and Guba in 1986 to respond to criticisms. I have added the fifth column to provide a little more detail )

This closer look at these criteria (and other similar lists) make it apparent how they attracted criticisms levelled at their positivist origins and the way they appear to be seeking absolute trustworthiness. Coming firmly from the non- (or anti-) foundational camp, Smith (1984) goes so far as to reject the possibility of criteria completely, stating that

“The assumptions of naturalistic inquiry—that reality is mind-dependent and that there are multiple realities—are incompatible with the idea of an independently existing reality that can be known through a neutral set of  procedures.”

This shift towards ontology makes plain the argument which is being made relating to truth or reality. The criteria offered thus far seem to have been formulated in an attempt to uncover the nature of an single, external, independent reality. Furthermore that the researcher and researched, knower and known are independent of one another, that biases can be eliminated (or accounted for) and that objectivity should be a target. At least for the first four of Lincoln and Guba’s criteria.

Arising from her proclivities for interpretive and poststructural research, and seeking to distance herself from the charge of pandering to post-positivist derived criteria, Tracy (2010) developed her ‘Eight ”Big-Tent” Criteria.’ She contends that high quality qualitative methodological research can be judged by the presence of (a) a worthy topic, (b) rich rigor, (c) sincerity, (d) credibility, (e) resonance, (f) a significant contribution, (g) ethics, and (h) meaningful coherence. These are presented as ‘universal’ criteria, that is, in order to be considered quality research, each of these criteria must be satisfied.

The various approaches presented so far fall into what Burke (2016) and Sparkes and Smith (2006) have termed the ‘criteriological approach.’ This propounds the belief that “criteria for judging qualitative research need to be, and can be, predetermined, permanent and applied to any form of inquiry regardless of its intents and purposes” (Smith and McGannon, 2017). They see the universal application of criteria as problematic, in how it requires any and all research to be judged in ‘preordained and set ways’ (which of course was precisely the reason proposed as one means to address the criticisms of the positivists). They offer as an alternative the ‘relativist’ approach in which criteria are instead a socially constructed list of characterising traits, and are applied in a manner that is contextually situated and flexible (Sparkes and Smith, 2009). These lists are not fixed in advance, but are open-ended, responding to what unfolds by adding or subtracting characteristics as required. This feels somewhat woolly, leaving itself open to the charge from realists that ‘anything goes.’ However, proponents of a relativist approach counter on ontological grounds, claiming that there is no social reality independent of our interests to act as a reference point against which to judge claims to knowledge. The only option when judging the quality of research is to appeal to ‘time and place contingent lists of characteristics to sort out the good from the not so good’ (Smith and Hodkinson, 2009).

In a rather more radical take, Richardson and St. Pierre (2005) use ‘triangulation’ as a point of departure to move towards the notion of ‘crystallization.’ This conception moves beyond the two-dimensional, triple perspective of the triangle, and embraces the way crystals grow and change shape to produce a wealth of symmetries, facets, shapes and potential angles of approach. As such, they embrace a much broader range of methods, approaches to research and the subsequent analyses. Crystallized projects should be evaluated on the basis of their ‘substantive contribution, aesthetic merit, reflexivity, and impact,’ although Richardson and St. Pierre note these are only four of the criteria they use. So despite the unfortunate pseudoscientific associations with crystals, even here we have a list of criteria, although one acknowledged as having emerged from postmodern thinking.

My research project is neither built on a realist ontology, nor a positivist epistemology, nor does it employ quantitative methods. It will come as no surprise therefore that the traditional criteria of validity, reliability, generalisability and objectivity hardly seem appropriate here. However, nor do those which are somewhat inspired by them, like confirmability, dependability and transferability. These all assume a reality ‘out there’ which can somehow be known ‘in here. With an actor-network theory sensibility however, reality can be out there, independent of the knower, but only if it made that way.

“Realities are made. They are effects of the apparatuses of inscription. At the same time, since there are such apparatuses already in place, we also live in and experience a real world filled with real and more or less stable objects.” (Law, 2004)

What emerges from my research will be made through the practices and methods I have employed. The apparatuses by which validity, reliability and so forth are inscribed elsewhere will indeed be appropriate for some studies, however, not so for my research I’d argue. As a consequence, I think this is better served by a relativist, rather than a criteriological approach. I also feel that using some of the original terminology (trustworthiness, rigour etc) carries baggage, so I propose to use terms which are less used, and for me, describe better the characteristics on which this project should be judged. It is my hope that the reader, for it is they who will be the arbiter of the quality of this research, feel that it has been conducted with integrity, fidelity and honesty. To that end, in a forthcoming post, I’ll offer characteristics assembled from different studies (Elliott et al, 1999; Lincoln and Guba, 1985; Lincoln, 1995; Richardson, 2000; Spencer et al, 2012; Tracy, 2010;) which are appropriate to the circumstances within which this research was conducted.

References

Burke, S. (2016). Rethinking validity and trustworthiness in qualitative inquiry. In B. M. Smith, & A. C. Sparkes (Eds.), Routledge handbook of qualitative research in sport and exercise (pp. 330-339). London: London : Routledge
Bryman, A. (2008). The end of the paradigm wars? In P. Alasuutari, L. Bickman & J. Brannen (Eds.), The SAGE handbook of social research methods (pp. 13-25). London: SAGE Publications. doi:10.4135/9781446212165
Hammersley, M. (1990). Reading ethnographic research : A critical guide. Longman.
Hammersley, M. (1992). What’s wrong with ethnography? : Methodological explorations. Routledge.
Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research & Method in Education, 30(3), 287-305.
Johnson, S., & Rasulova, S. (2017). Qualitative research and the evaluation of development impact: Incorporating authenticity into the assessment of rigour. Journal of Development Effectiveness, 9(2), 263-276.
Lieblich, A., Tuval-Mashiach, R., & Zilber, T. (1998). Narrative research: Reading, analysis and interpretation (Applied social research methods ; 47). Thousand Oaks, Calif. ; London: Sage Publications.
Lincoln, Y. (1995). Emerging Criteria for Quality in Qualitative and Interpretive Research. Qualitative Inquiry, 1(3), 275-289.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry Sage.
Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New directions for evaluation, 1986(30), 73-84.
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International journal of qualitative methods, 1(2), 13-22.
Nolan, M., Hanson, E., Magnusson, L., & Andersson, B. A. (2003). Gauging quality in constructivist research: The Äldre Väst Sjuhärad model revisited. Quality in Ageing and Older Adults, 4(2), 22-27.
Richardson, L. (2000). Evaluating ethnography. Qualitative inquiry, 6(2), 253-255.
Richardson, L., & St Pierre, E. A. (2005). Writing: A method of inquiry. In N. K. Denzin, & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed. ed., pp. 959-978). Thousand Oaks, Calif. ; London: Thousand Oaks, Calif. ; London : Sage Publications.
Smith, John K. (1984). The Problem of Criteria for Judging Interpretive Inquiry. Educational Evaluation and Policy Analysis, 6(4), 379-91.
Smith, John K., & Hodkinson, Phil. (2009). Challenging Neorealism: A Response to Hammersley. Qualitative Inquiry, 15(1), 30-39.
Smith, B., & McGannon, K. (2017). Developing rigor in qualitative research: Problems and opportunities within sport and exercise psychology. International Review of Sport and Exercise Psychology, 1-21.
Sparkes, & Smith. (2009). Judging the quality of qualitative inquiry: Criteriology and relativism in action. Psychology of Sport & Exercise, 10(5), 491-497.
Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2012). Quality in qualitative evaluation: A framework for assessing research evidence. (). London: Cabinet Office.
Tracy, S. (2010). Qualitative Quality: Eight “Big-Tent” Criteria for Excellent Qualitative Research. Qualitative Inquiry, 16(10), 837-851

Advertisements

What do teachers do on Twitter? Emerging findings.

“Sharing” flickr photo by bengrey https://flickr.com/photos/ben_grey/4582294721 shared under a Creative Commons (BY-SA) license

When teachers are active on Twitter, they participate in a wide range of practices. From the outset it is important to state how their activity is underpinned by the principle of reciprocity. When one teacher claims that Twitter helps them keep abreast of current educational research, it follows that at some point, someone else must have contributed that information. When teachers describe how they contribute ideas, share experiences and signpost resources, that might not immediately appear to be enhancing their learning. However, when taken in the round, at some point the same teachers will have benefited from others having done the same. Professional practice within this communal space is based on exchange, both giving and receiving.Read More »

Connecting

One of the most frequently mentioned activities when teachers discuss their use of Twitter is ‘connecting,’ and the purpose of connecting is in order to learn. Principally, they’re seeking connections with other people, but also in order to link between different spaces, for example between Twitter and blog posts. Some of the most important connections they’re seeking to make are with other teachers, especially those who share similar interests, passions, and are of a ‘like-mind.’

Whilst connecting with specific individuals can lead to strong and trusting relationships forming and reforming, there’s a recognition that by doing so, you’re also connecting with the whole network of other connections that person benefits from; it’s a cumulative or even exponential process. Knowing the importance of making and maintaining connections, some act as brokers in bringing together people and things they think would interest or assist one another.

…or seeing someone tweeting about some subject and you think, well you need to speak to this person; those sorts of connections I think are incredibly important and that’s not really to do with the technology, it’s just about being a nice person …
(J Dale)

Read More »

Groups, communities, collectives or …?

Recently Stephen Downes released an updated version of a graphic outlining how groups and networks differ in what he proposes as ‘The Semantic Condition,’ a network design principle. He helpfully explains how the diagram was produced in this video:

In addition to the criteria he uses to distinguish between the two ways in which people might organise or aggregate, what attracted my attention was the appearance of Twitter within the arguments. One of the aspects coming through in the data from my research, is how educators on Twitter describe the ways in which they come together. The most common term people seem to use is community, but group, tribe and network also appear. Although these terms are conceptualised differently, I suspect in most cases, a particular term is used simply because it happens to be the favoured choice, rather than having an awareness that there is a distinction between it and the others. If I was to explore this more carefully, I might be able to tease apart the ways in which people see these different terms, but suspect that what for one person is a community, could just as easily be what a tribe is for another. It was from wondering how these different groupings are distinguished from one another in the literature, that I was attracted to Stephen’s graphic.Read More »

Schrödinger’s … medical condition?

“The Body Multiple” flickr photo by IaninSheffield https://flickr.com/photos/ianinsheffield/38894472832 shared under a Creative Commons (BY-NC-SA) license

One of the touchstone references regularly encountered when reading sociomaterial accounts is The Body Multiple by Annemarie Mol. Since we didn’t have it in our Uni library and because it seemed so important, I took the rather rash step (for a Yorkshireman) of buying a copy. Although it took me a while to fully get to grips with the concepts therein, I now know why it is such a classic. I really should have reviewed it on the blog, however my small but select bunch of educational readers might not have found much of interest in an ethnographic text of medical practice … or perhaps they would. Put far too simply, it’s an ethnographic telling of how the arterial disease, atherosclerosis, is enacted in a hospital. Enacted, yes, because this is tale of ontology and how reality comes to be.

The messages in The Body Multiple resurfaced for me this week as I cycled into Uni., pondering the events of a few months ago, back in August (2017). I’d been invited by my local health centre for a Health Check. No biggie; just one of those things to which people of ‘a certain age’ here in the UK can benefit from. If I had my cynical head on, then I might say my clinic had invited me in order to meet its targets, or to reduce the likelihood of me becoming a future financial burden on an ever more stressed national health service. Since I’m feeling much more generous, I’d say it’s to help spot early symptoms of diseases which might make my later life less pleasant, or even foreshorten it. The check requires you to provide blood and urine samples, then re attend a week later for a nurse to go through the results with you and suggest lifestyle changes if necessary. I wasn’t worried; my last check six or so years ago indicated nothing more than cholesterol levels slightly above those recommended by the medical establishment. In the intervening period, I made adjustments to my diet and was confident I’d now be within the margin. What I hadn’t expected was the phone call from the clinic the day after the blood tests, asking if I could make an appointment to see a doctor, and could I come in the following day.Read More »

Visualising assemblage

Having decided to attempt to describe certain phenomena on Twitter as learning assemblage, I now find myself in somewhat of a quandary. Earlier yesterday, whilst teaching a group of undergrad BEd with Science QTS students about circular motion, we were discussing the importance of sketching free-body diagrams to aid understanding and problem solving. So perhaps it’s the scientist in me that generates the proclivity to want to summarise situations by using visualisations of one sort or another. A quick scan through the back catalogue of this blog will reveal many examples, however I now find myself struggling and somewhat dissatisfied.

I’ve recently been drafting vignettes in which I describe groups and activity on Twitter as assemblage, but I feel the need to produce a visualisation which captures a sense of what that is. The problem of course is that I’m trying to render assemblage, a dynamic process, as a static representation. But why should that be a problem? That’s precisely what I’ve been doing when producing physics free-body diagrams isn’t it? Representing a dynamic situation through a static diagram?

“Marble loop” flickr photo by IaninSheffield https://flickr.com/photos/ianinsheffield/24447353008 shared under a Creative Commons (BY-NC-SA) license

Read More »

“Can Twitter learn?”

“Can Twitter learn? Can a tweet learn?”

“Twitter Expert” by mkhmarketing https://flickr.com/photos/mkhmarketing/8540535352 is licensed under CC BY

Two questions one of my supervisors posed in the feedback on a recent draft thesis section I’d submitted. Despite knowing this was playfully provocative, I’m only too well aware that I need to be able to answer questions like this, whatever their intention. In response, I first need to clarify what learning is within the context of my study. Although there’s an imperative to lay that out within my thesis, I haven’t yet done so because I’ve been wrestling with how learning is conceived through a sociomaterial perspective. What better time to grasp that nettle?

What constitutes learning is often unproblematically taken for granted amongst most educators, however, during twenty years of teaching I can’t recall ever discussing it explicitly as an isolated concept. Read More »

It might be a little way off yet, but …

“thesis” flickr photo by Liz Henry https://flickr.com/photos/lizhenry/147745444 shared under a Creative Commons (BY-ND) license

The traditional way that most theses are presented is in the form of an 80 000(ish) word report. University regulations usually specify that this should be bound in hard copy format, and ready to be posted onto the shelves in the Library stacks. Recently, in the spirit of openly sharing knowledge, it is becoming common for Universities to also require a digital copy of the thesis for posting to the institutional digital repository. For me then, this will be through the Sheffield Hallam University Research Archive, SHURA. We are also now required, where permissible, to post the data that our research generates. This aligns with my own feelings about research being as open as is ethically permissible, so I have no problem with any of this.

As I have begun drafting my thesis, the necessity for it to conform to the constraint of A4-shaped dead trees is causing me a few headaches. I don’t have a problem with writing text … actually I do, but that’s another story! Read More »

Wither wends my tweet?

I was attempting to write a vignette yesterday about how the tweet which prompted this post actually got in front of the eyes of some people who might be so inclined to respond. If we’re asking a question or seeking advice, rather than sharing a resource or thought, then the audience becomes even more significant than it normally would. Without an audience, like the falling tree in the forest needing someone to hear it, the tweet and the query it carries may as well not be there.Read More »