2 Reflections on starting my DPhil
I use qualitative methods throughout my thesis. Reflexivity is important in qualitative research as “A researcher’s background and position will affect what they choose to investigate, the angle of investigation, the methods judged most adequate for this purpose, the findings considered most appropriate, and the framing and communication of conclusions” [1]. More succinctly, “Despite the sterility of the instruments, we never come innocent to a research task” [2]. In this chapter, I examine my own lack of innocence.
Reflexivity requires self reflection to acknowledge one’s own beliefs, bias, and judgement systems before, during, and after the research process (Shaw 2010). Researchers engage with self reflection to different degrees, and Woolgar [3] describes a continuum from positivists’ benign (often retrospective) introspection to constructivists’ radical constitutive reflexivity. My reflexivity falls in the middle and shifted over time. Reflexivity was a new concept to me when I started my DPhil and in my early chapters (3–4) I was mainly concerned with representing participants’ views accurately, thereby placing me at the positivist’s end of Woolgar’s continuum. In this chapter, I reflect on my experiences with reporting guidelines before starting my DPhil, and the experiences of my supervisors. This account documents my my initial stance and helps explain where this work grew from. Wilkinson [4] describes this as personal reflexivity – how a researchers’ identity and experience may influence what they study.
Wilkinson [4] differentiates personal reflexivity from functional reflexivity (how researchers undertake research and how research shapes them in return) and disciplinary reflexivity (how prevailing paradigms, attitudes and power structures shape research). As my DPhil progressed my thoughts strayed further into functional reflexivity. I became increasingly aware of my inescapable role in knowledge generation in later chapters, especially in the workshops I led with reporting guideline experts (chapter 7), and when collecting feedback on design decisions (chapters 10). I kept a research diary throughout my DPhil and the remaining chapters end with shorter reflexivity sections where I consider how my beliefs may have shaped my research and how doing the research had, in turn, shaped my beliefs. This chapter, however, focuses solely on personal reflexivity.
This thesis marks the ten-year anniversary of my attempts to improve research reporting. My early interest came from reading about psychology’s reproducibility crisis [5], the cancer reproducibility project [6], Ben Goldacre’s book Bad Pharma [7], and Ioannidis’ essay “Why Most Published Research Findings Are False” [8]. I was working in a psychology laboratory at the time, having recently finished a MSc in Neuroscience. The compelling Lancet series on research waste [9] drilled home a magnitude of inefficiencies in medical research. I was most stuck by the article on incomplete or unusable research reports [10], where a key takeaway was that “although reporting guidelines are important, major improvements need active enforcement” by editors or reviewers. I took Paul Glasziou’s words as a call to action, and I set about creating tools to help editors enforce good reporting.
I began by writing software to check whether a manuscript cited a reporting guideline. Next I wrote code to check the reporting of statistical analyses. These were both pet projects. My first “proper” software was a manuscript checker that evaluates whether a manuscript adheres to journal guidelines [11]. I began working with academic journals, and came to appreciate that reporting guidelines are one of many priorities for editors, and often fall behind more pressing issues like ethics, consent, image duplication, and paper mills.
I later worked with the UK EQUATOR Centre to create GoodReports.org [12], a website where authors can find and complete reporting checklists. EQUATOR and I collaborated with BMJ Open in 2018 to see whether authors improved their manuscripts after completing a checklist as part of manuscript submission [13]. The results were disappointing. Few authors made any changes.
I was awfully demoralized. It was tempting to blame the failure on lazy authors paying lip service to the checklists because they know that equally-lazy editors will not check. Indeed, this is a refrain I have heard bandied around by frustrated guideline developers. However, a tenet of software development is that when it comes to how a user experiences a product (or service), the user is never wrong. A software developer that blames poor design on “lazy users” will quickly find themselves out of a job. If users do not use your product, or do not use it how you would expect them to, then your product may lack usability (users cannot understand or use it), product market fit (users do not need or want it), or awareness (users do not know about it). It is the creator’s responsibility to make their product known, foolproof, and useful.
Understanding users’ experiences and needs is therefore central to making a successful product. It is also very difficult. Like many developers, I am happier behind a screen and talking to users never came naturally to me. Instead, I often latched on to an idea before understanding the people I was trying to help, or the problem I was trying to solve. This was a good strategy for building things nobody wanted.
I almost fell into the same trap when starting my DPhil. In my interview I adamantly pitched a tool to create personalised checklists by combining reporting guidelines with journal, funder, and institutional requirements. I envisioned something akin to WebCONSORT on steroids. The old me would have happily spent months falling down that rabbit hole and building something (probably) useless; the custom checklists would have been incredibly long and probably confusing. The new me, the me writing this paragraph, is glad I paused and spent a year trying to better understand authors’ experiences and needs.
On starting my DPhil I had never used a reporting guideline myself, despite recommending others to do so. Yet I held them in high regard, as did my three initial supervisors. Jen, Michael, and Gary are affiliated with the EQUATOR Network. Gary is Director of the UK EQUATOR Centre and an author of many reporting guidelines and a working-group member for many others. Michael studies reporting completeness, the robustness of reporting guideline development methods, and the consolidation of different reporting guidelines for the reporting of studies of nutritional interventions. Jen is involved in many studies investigating reporting and reporting guidelines, and runs training on writing and reporting guidelines.
Charlotte joined my supervision team in my second year, once we realised my thesis was fixed on a qualitative course. Qualitative behaviour change research was new territory for Gary, Jen, Michael, and I, and we needed an expert. Charlotte came to the rescue. She has a lot of experience in qualitative research and behaviour change theory, both from her own research and from leading the Oxford course on qualitative methods. However, she had never studied reporting guidelines before, nor any other meta-research phenomenon. Charlotte has published a suggested amendment to a reporting guideline she uses in her own work [14]. Whereas Jen, Michael, and Gary can be described as reporting guideline advocates, Charlotte was a little more cool-headed. Although she considered reporting guidelines useful for writing up quantitative work, she found guidelines for qualitative research frustrating because they were developed from a positivist perspective, and so did not fit all qualitative research.
In summary, I came to this DPhil with an existing passion for improving research, a deep-but-naïve respect for reporting guidelines and EQUATOR, and with determination to make something helpful after rebounding from the disappointing BMJ Open study. Sitting atop some rusty, decade-old research experience, I had a software developer’s vocabulary and mindset. Core to this mindset was a belief that if people do not use what you have made in the way you want them to, it is not their responsibility to change their own behaviour. It is your responsibility to change what you have made. The challenge is figuring out what changes you need to make.
Having set the scene, I’ll begin my thesis where I began my research: by trying to understand why authors do not adhere to reporting guidelines. In the previous chapter I described the evidence that reporting guidelines have had little effect on reporting quality. Next I wanted to know why. Chapters 3 - 5 describe my mixed methods approach to finding possible answers to this question, beginning with a qualitative evidence synthesis.